Hello
MonetDB Nov2008 release, built with all standard options on RedHat EL 5
(2.6.18-92.el5 #1 SMP), on HP Proliant DL380 8-way, 16GB RAM / 16GB swap, with
dbfarm on a RAID0 internal 7-disk array.
I'm doing bulk load tests (using mclient to run COPY INTO table FROM file),
repeatedly loading 500,000 row (68M) CSV files into a fact table, while
periodically running a suite of test queries in a separate process to test
query times as the fact table grows.
The problem is that as the fact table grows to around 80Million records, the
database starts to consume all available system memory and swap, the system
thrashes and comes to its knees. I also get database corruption causing
foreign key violations. (More details below)
Has anyone else seen this? Any ideas if it's a bug, memory leak? Or could this
be caused by operator error? Suggestions?
Cheers Bob
I am experiencing similar problems. The testcase is different, but the symptoms are the same. In my case, the symptoms go away when i drop all constraints, suggesting that the problem is related to that.
Is it possible for you to run your test without any primary/foreign key constraints on the database?
--
====================================================================
CWI, Kamer C0.03 Centrum voor Wiskunde en Informatica
Kruislaan 413 Email: arjen.de.rijke@cwi.nl
1098 SJ Amsterdam tel: +31-(0)20-5924305
Nederland +31-(0)6-51899284
fax: +31-(0)20-5924312
=====================
http://www.cwi.nl/~rijke/ ====================
> PS More detailed account of the problem below:
>
> Everything starts off fine, files load in 8-10 seconds, queries quite fast.
>
> As the table starts to fill, the queries (not unexpectedly) start to slow. But
> (unexpectedly) the memory consumption of the mserver5 process steadily
> increases.
>
> At some point, around 80Million records, the system performance rapidly starts
> to degrade. Query times dramatically increase, load time also dramatically
> increases, and the entire system thrashes and becomes unusable.
>
> Running 'top' shows that mserver5 has consumed vurtually all physical memory,
> and most of the swap too.
>
> top - 16:33:34 up 28 days, 23:52, 8 users, load average: 1.68, 1.51, 1.96
> Tasks: 224 total, 1 running, 223 sleeping, 0 stopped, 0 zombie Cpu(s): 0.1%us,
> 1.5%sy, 0.0%ni, 85.9%id, 12.4%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 16439196k total,
> 16298404k used, 140792k free, 2300k buffers Swap: 18876364k total, 13315960k
> used, 5560404k free, 9824352k cached PID USER PR NI VIRT RES SHR S %CPU %MEM
> TIME+ COMMAND 32380 monetdb 18 0 54.6g 14g 9.4g S 12 94.9 107:19.46 mserver5
> 29217 bostr 15 0 290m 5552 2264 S 1 0.0 12:01.37 gnome-terminal
>
> The first time this happened, I stopped the load scripts, restarted the
> database, and that helped things for a little while, but it wasn't long before
> the memory was all consumed again, and performance went to the dogs. Then
> loads started failing with foreign-key violation errors.. Inspection showed a
> corrupted dimension table that somehow ended up with a bunch of bad values in
> it..
>
> So I blew away, and recreated the database.
>
> This time I tried adding the line 'gdk_nr_threads=1' to the monetdb5.conf file
> - having seen a defect report related to 'COPY INTO
>
> ' being buggy and this being the solution. But the same thing happened again..
> it ran fine again until I had about 100Million rows, then memory problems/
> thrashing. Restarted the server again, and shortly after the data loads failed
> again with foreign-key violations and a corrupted dimension table..
> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
> View this message in context: mserver memory grows, system thrashes and hangs
> Sent from the monetdb-users mailing list archive at Nabble.com.
>
> ------------------------------------------------------------------------------
> SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada.
> The future of the web can't happen without you. Join us at MIX09 to help
> pave the way to the Next Web now. Learn more and register at
>
http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/...
> MonetDB-users mailing list
> MonetDB-users@lists.sourceforge.net
>
https://lists.sourceforge.net/lists/listinfo/monetdb-users