On Wed, Jan 21, 2009 at 02:07:04AM -0800, sylver_b wrote:
Hi
Interesting..
Even worst, this morning i found out that all the inserts of last night failed !! :(
#BBPTRIM_ENTER: memsize=1144385432,vmsize=1599613808 #BBPTRIM: memtarget=819717481 vmtarget=0 #BBPTRIM_EXIT: memsize=366265384,vmsize=1599605544 #BBPTRIM_ENTER: memsize=1173710640,vmsize=1628939016 #BBPTRIM: memtarget=849042689 vmtarget=0 #BBPTRIM_EXIT: memsize=366265384,vmsize=1628939016 #BBPTRIM_ENTER: memsize=1143635024,vmsize=1629665320 #BBPTRIM: memtarget=0 vmtarget=19052584 #BBPTRIM_EXIT: memsize=366265384,vmsize=1598862320 #BBPTRIM_ENTER: memsize=1143633968,vmsize=1645786120 #BBPTRIM: memtarget=0 vmtarget=35173384 #BBPTRIM_EXIT: memsize=366265384,vmsize=1645786120 #BBPTRIM_ENTER: memsize=1143633992,vmsize=1676588064 #BBPTRIM: memtarget=0 vmtarget=65975328 #BBPTRIM_EXIT: memsize=366265384,vmsize=1598862320 #BBPTRIM_ENTER: memsize=1143633944,vmsize=1613149168 #BBPTRIM: memtarget=0 vmtarget=2536432 #BBPTRIM_EXIT: memsize=366265384,vmsize=1582347224 #BBPTRIM_ENTER: memsize=1143633920,vmsize=1620095960 #BBPTRIM: memtarget=0 vmtarget=9483224 #BBPTRIM_EXIT: memsize=366265384,vmsize=1620095960 #GDKmmap(148111360) fail => BBPtrim(enter) usage[mem=1143633920,vm=1496757208] #BBPTRIM_ENTER: memsize=1143633920,vmsize=1496757208 #BBPTRIM: memtarget=0 vmtarget=1073741824 #BBPTRIM_EXIT: memsize=366265384,vmsize=1496757208 #GDKmmap(148111360) fail => BBPtrim(ready) usage[mem=1143633920,vm=1496757208] !ERROR: GDKload: cannot mmap(): name=32/3267, ext=theap !OS: Cannot allocate memory !ERROR: GDKload: failed name=32/3267, ext=theap
top - 10:00:34 up 249 days, 20:31, 4 users, load average: 0.28, 0.12, 0.04 Tasks: 86 total, 1 running, 85 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 4025600k total, 3633228k used, 392372k free, 107544k buffers Swap: 9502408k total, 92k used, 9502316k free, 1520104k cached
Is that after mserver5 had crashed?
How big is/was your mserver5 when running or just before crashing?
Your machine and OS seem to be 64-bit, right?
What about your MonetDB installation? Could you please provide us with the output of `mserver5 --version` ?
How much free space is on your disk partition where your dbfarm is located?
Is there anything to do to prevent this ? should the DB swap at all ?
MonetDB does in-memory processing, i.e., all columns that are active at a time need to be in the processes address space, either being loaded or being memory mapped; so yes, with huge amount of data MonetDB will also use virtual memory, either as swap or as memory-mapped files.
Stefan
De : Stefan de Konink stefan@konink.de À : sylver_b sylver_b@yahoo.com Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 1h09mn 13s Objet : Re: [Monetdb-developers] mclient running out of memory, crashing mserver5
sylver_b wrote:
I don't think i'm running out of memory. I had to stop and restart mserver5, flush the table, then re-run the command - all the records were inserted. Also I noticed this type of messages on the mserver5 console when piping records through the mclient (over 1million records) . It seems that the mclient is quiet instable (cf my post about "COPY, terminating connection" few months ago). So, is there a way to increase the memory limit or make sure mserver5 won't crash when running bash inserts via a cron job ?
The only thing I was able to do against that; more swap for the job. And of course the standard stuff as loading CSV files, preferably no indices on the table you are loading huge amounts of data to.
Stefan
This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers