Hi Stephan, I will reply below :
________________________________
De : Stefan Manegold Stefan.Manegold@cwi.nl À : sylver_b sylver_b@yahoo.com Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 10h28mn 41s Objet : Re: [Monetdb-developers] Re : mclient running out of memory, crashing mserver5
Yes
How big is/was your mserver5 when running or just before crashing?
missed to check that, but this is what i have at the moment :
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 31679 0.0 3.2 1950816 131056 pts/3 Sl+ 10:05 0:00 | _ mserver5 --dbinit=include sql;
Your machine and OS seem to be 64-bit, right?
32-bit
What about your MonetDB installation? Could you please provide us with the output of `mserver5 --version` ?
# mserver5 --version
MonetDB server v5.8.0 (32-bit), based on kernel v1.26.0 (32-bit oids) Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008- MonetDB B.V., all rights reserved Visit http://monetdb.cwi.nl/for further information Configured for prefix: /root/MonetDB Libraries: openssl: OpenSSL 0.9.8b 04 May 2006 (compiled with OpenSSL 0.9.8b 04 May 2006) Compiled by: root@ulys Compilation: gcc -O2 -std=c99 Linking : /usr/bin/ld
How much free space is on your disk partition where your dbfarm is located?
# df -h
Filesystem Size Used Avail Use% Mounted on /dev/sda1 212G 180G 22G 90% / varrun 2.0G 56K 2.0G 1% /var/run varlock 2.0G 0 2.0G 0% /var/lock procbususb 10M 52K 10M 1% /proc/bus/usb udev 10M 52K 10M 1% /dev devshm 2.0G 0 2.0G 0% /dev/shm
Is there anything to do to prevent this ? should the DB swap at all ?
MonetDB does in-memory processing, i.e., all columns that are active at a time need to be in the processes address space, either being loaded or being memory mapped; so yes, with huge amount of data MonetDB will also use virtual memory, either as swap or as memory-mapped files.
Stefan
De : Stefan de Konink stefan@konink.de À : sylver_b sylver_b@yahoo.com Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 1h09mn 13s Objet : Re: [Monetdb-developers] mclient running out of memory, crashing mserver5
sylver_b wrote:
I don't think i'm running out of memory. I had to stop and restart mserver5, flush the table, then re-run the command - all the records were inserted. Also I noticed this type of messages on the mserver5 console when piping records through the mclient (over 1million records) . It seems that the mclient is quiet instable (cf my post about "COPY, terminating connection" few months ago). So, is there a way to increase the memory limit or make sure mserver5 won't crash when running bash inserts via a cron job ?
The only thing I was able to do against that; more swap for the job. And of course the standard stuff as loading CSV files, preferably no indices on the table you are loading huge amounts of data to.
Stefan
This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers