[Monetdb-developers] mclient running out of memory, crashing mserver5
Hi,
I've finally found a better way to bulk load records via perl & odbc - a lot more stable.
However, I've noticed many times mclient crashing mserver5 when loading records via the interactive interface. For instance, I tried to load about 130 inserts from a sql file using this command : mclient -lsql -d mydb < public_sysuser.sql
After few seconds I started to see :
#BBPTRIM: memtarget=17978753 vmtarget=0 #BBPTRIM_EXIT: memsize=24100864,vmsize=273741960 [ 1 ] [ 1 ] [ 1 ] [ 1 ] [ 1 ] [ 1 ] [ 1 ] [ 1 ] [ 1 ] #BBPTRIM_ENTER: memsize=275281456,vmsize=275166768 #BBPTRIM: memtarget=19273201 vmtarget=0 #BBPTRIM_EXIT: memsize=24100864,vmsize=275158816 [ 1 ] [ 1 ] [ 1 ] [ 1 ] [ 1 ] [ 1 ] [ 1 ] #BBPTRIM_ENTER: memsize=276161752,vmsize=276047064 #BBPTRIM: memtarget=20153497 vmtarget=0 #BBPTRIM_EXIT: memsize=24100864,vmsize=276056368 [ 1 ] #BBPTRIM_ENTER: memsize=276322608,vmsize=276207920 #BBPTRIM: memtarget=20314353 vmtarget=0 #BBPTRIM_EXIT: memsize=24100864,vmsize=276198064
Top is showing :
top - 23:26:14 up 249 days, 9:57, 5 users, load average: 0.00, 0.20, 0.45 Tasks: 96 total, 1 running, 95 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 4025600k total, 2433324k used, 1592276k free, 191788k buffers Swap: 9502408k total, 92k used, 9502316k free, 1993392k cached
I don't think i'm running out of memory. I had to stop and restart mserver5, flush the table, then re-run the command - all the records were inserted. Also I noticed this type of messages on the mserver5 console when piping records through the mclient (over 1million records) . It seems that the mclient is quiet instable (cf my post about "COPY, terminating connection" few months ago).
So, is there a way to increase the memory limit or make sure mserver5 won't crash when running bash inserts via a cron job ?
Please advise, Thank you SB
sylver_b wrote:
I don't think i'm running out of memory. I had to stop and restart mserver5, flush the table, then re-run the command - all the records were inserted. Also I noticed this type of messages on the mserver5 console when piping records through the mclient (over 1million records) . It seems that the mclient is quiet instable (cf my post about "COPY, terminating connection" few months ago).
So, is there a way to increase the memory limit or make sure mserver5 won't crash when running bash inserts via a cron job ?
The only thing I was able to do against that; more swap for the job. And of course the standard stuff as loading CSV files, preferably no indices on the table you are loading huge amounts of data to.
Stefan
Hi
Interesting..
Even worst, this morning i found out that all the inserts of last night failed !! :(
#BBPTRIM_ENTER: memsize=1144385432,vmsize=1599613808 #BBPTRIM: memtarget=819717481 vmtarget=0 #BBPTRIM_EXIT: memsize=366265384,vmsize=1599605544 #BBPTRIM_ENTER: memsize=1173710640,vmsize=1628939016 #BBPTRIM: memtarget=849042689 vmtarget=0 #BBPTRIM_EXIT: memsize=366265384,vmsize=1628939016 #BBPTRIM_ENTER: memsize=1143635024,vmsize=1629665320 #BBPTRIM: memtarget=0 vmtarget=19052584 #BBPTRIM_EXIT: memsize=366265384,vmsize=1598862320 #BBPTRIM_ENTER: memsize=1143633968,vmsize=1645786120 #BBPTRIM: memtarget=0 vmtarget=35173384 #BBPTRIM_EXIT: memsize=366265384,vmsize=1645786120 #BBPTRIM_ENTER: memsize=1143633992,vmsize=1676588064 #BBPTRIM: memtarget=0 vmtarget=65975328 #BBPTRIM_EXIT: memsize=366265384,vmsize=1598862320 #BBPTRIM_ENTER: memsize=1143633944,vmsize=1613149168 #BBPTRIM: memtarget=0 vmtarget=2536432 #BBPTRIM_EXIT: memsize=366265384,vmsize=1582347224 #BBPTRIM_ENTER: memsize=1143633920,vmsize=1620095960 #BBPTRIM: memtarget=0 vmtarget=9483224 #BBPTRIM_EXIT: memsize=366265384,vmsize=1620095960 #GDKmmap(148111360) fail => BBPtrim(enter) usage[mem=1143633920,vm=1496757208] #BBPTRIM_ENTER: memsize=1143633920,vmsize=1496757208 #BBPTRIM: memtarget=0 vmtarget=1073741824 #BBPTRIM_EXIT: memsize=366265384,vmsize=1496757208 #GDKmmap(148111360) fail => BBPtrim(ready) usage[mem=1143633920,vm=1496757208] !ERROR: GDKload: cannot mmap(): name=32/3267, ext=theap !OS: Cannot allocate memory !ERROR: GDKload: failed name=32/3267, ext=theap
top - 10:00:34 up 249 days, 20:31, 4 users, load average: 0.28, 0.12, 0.04 Tasks: 86 total, 1 running, 85 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 4025600k total, 3633228k used, 392372k free, 107544k buffers Swap: 9502408k total, 92k used, 9502316k free, 1520104k cached
Is there anything to do to prevent this ? should the DB swap at all ?
________________________________ De : Stefan de Konink stefan@konink.de À : sylver_b sylver_b@yahoo.com Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 1h09mn 13s Objet : Re: [Monetdb-developers] mclient running out of memory, crashing mserver5
sylver_b wrote:
I don't think i'm running out of memory. I had to stop and restart mserver5, flush the table, then re-run the command - all the records were inserted. Also I noticed this type of messages on the mserver5 console when piping records through the mclient (over 1million records) . It seems that the mclient is quiet instable (cf my post about "COPY, terminating connection" few months ago). So, is there a way to increase the memory limit or make sure mserver5 won't crash when running bash inserts via a cron job ?
The only thing I was able to do against that; more swap for the job. And of course the standard stuff as loading CSV files, preferably no indices on the table you are loading huge amounts of data to.
Stefan
On Wed, Jan 21, 2009 at 02:07:04AM -0800, sylver_b wrote:
Hi
Interesting..
Even worst, this morning i found out that all the inserts of last night failed !! :(
#BBPTRIM_ENTER: memsize=1144385432,vmsize=1599613808 #BBPTRIM: memtarget=819717481 vmtarget=0 #BBPTRIM_EXIT: memsize=366265384,vmsize=1599605544 #BBPTRIM_ENTER: memsize=1173710640,vmsize=1628939016 #BBPTRIM: memtarget=849042689 vmtarget=0 #BBPTRIM_EXIT: memsize=366265384,vmsize=1628939016 #BBPTRIM_ENTER: memsize=1143635024,vmsize=1629665320 #BBPTRIM: memtarget=0 vmtarget=19052584 #BBPTRIM_EXIT: memsize=366265384,vmsize=1598862320 #BBPTRIM_ENTER: memsize=1143633968,vmsize=1645786120 #BBPTRIM: memtarget=0 vmtarget=35173384 #BBPTRIM_EXIT: memsize=366265384,vmsize=1645786120 #BBPTRIM_ENTER: memsize=1143633992,vmsize=1676588064 #BBPTRIM: memtarget=0 vmtarget=65975328 #BBPTRIM_EXIT: memsize=366265384,vmsize=1598862320 #BBPTRIM_ENTER: memsize=1143633944,vmsize=1613149168 #BBPTRIM: memtarget=0 vmtarget=2536432 #BBPTRIM_EXIT: memsize=366265384,vmsize=1582347224 #BBPTRIM_ENTER: memsize=1143633920,vmsize=1620095960 #BBPTRIM: memtarget=0 vmtarget=9483224 #BBPTRIM_EXIT: memsize=366265384,vmsize=1620095960 #GDKmmap(148111360) fail => BBPtrim(enter) usage[mem=1143633920,vm=1496757208] #BBPTRIM_ENTER: memsize=1143633920,vmsize=1496757208 #BBPTRIM: memtarget=0 vmtarget=1073741824 #BBPTRIM_EXIT: memsize=366265384,vmsize=1496757208 #GDKmmap(148111360) fail => BBPtrim(ready) usage[mem=1143633920,vm=1496757208] !ERROR: GDKload: cannot mmap(): name=32/3267, ext=theap !OS: Cannot allocate memory !ERROR: GDKload: failed name=32/3267, ext=theap
top - 10:00:34 up 249 days, 20:31, 4 users, load average: 0.28, 0.12, 0.04 Tasks: 86 total, 1 running, 85 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 4025600k total, 3633228k used, 392372k free, 107544k buffers Swap: 9502408k total, 92k used, 9502316k free, 1520104k cached
Is that after mserver5 had crashed?
How big is/was your mserver5 when running or just before crashing?
Your machine and OS seem to be 64-bit, right?
What about your MonetDB installation? Could you please provide us with the output of `mserver5 --version` ?
How much free space is on your disk partition where your dbfarm is located?
Is there anything to do to prevent this ? should the DB swap at all ?
MonetDB does in-memory processing, i.e., all columns that are active at a time need to be in the processes address space, either being loaded or being memory mapped; so yes, with huge amount of data MonetDB will also use virtual memory, either as swap or as memory-mapped files.
Stefan
De : Stefan de Konink stefan@konink.de À : sylver_b sylver_b@yahoo.com Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 1h09mn 13s Objet : Re: [Monetdb-developers] mclient running out of memory, crashing mserver5
sylver_b wrote:
I don't think i'm running out of memory. I had to stop and restart mserver5, flush the table, then re-run the command - all the records were inserted. Also I noticed this type of messages on the mserver5 console when piping records through the mclient (over 1million records) . It seems that the mclient is quiet instable (cf my post about "COPY, terminating connection" few months ago). So, is there a way to increase the memory limit or make sure mserver5 won't crash when running bash inserts via a cron job ?
The only thing I was able to do against that; more swap for the job. And of course the standard stuff as loading CSV files, preferably no indices on the table you are loading huge amounts of data to.
Stefan
This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
Hi Stephan, I will reply below :
________________________________
De : Stefan Manegold Stefan.Manegold@cwi.nl À : sylver_b sylver_b@yahoo.com Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 10h28mn 41s Objet : Re: [Monetdb-developers] Re : mclient running out of memory, crashing mserver5
Yes
How big is/was your mserver5 when running or just before crashing?
missed to check that, but this is what i have at the moment :
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 31679 0.0 3.2 1950816 131056 pts/3 Sl+ 10:05 0:00 | _ mserver5 --dbinit=include sql;
Your machine and OS seem to be 64-bit, right?
32-bit
What about your MonetDB installation? Could you please provide us with the output of `mserver5 --version` ?
# mserver5 --version
MonetDB server v5.8.0 (32-bit), based on kernel v1.26.0 (32-bit oids) Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008- MonetDB B.V., all rights reserved Visit http://monetdb.cwi.nl/for further information Configured for prefix: /root/MonetDB Libraries: openssl: OpenSSL 0.9.8b 04 May 2006 (compiled with OpenSSL 0.9.8b 04 May 2006) Compiled by: root@ulys Compilation: gcc -O2 -std=c99 Linking : /usr/bin/ld
How much free space is on your disk partition where your dbfarm is located?
# df -h
Filesystem Size Used Avail Use% Mounted on /dev/sda1 212G 180G 22G 90% / varrun 2.0G 56K 2.0G 1% /var/run varlock 2.0G 0 2.0G 0% /var/lock procbususb 10M 52K 10M 1% /proc/bus/usb udev 10M 52K 10M 1% /dev devshm 2.0G 0 2.0G 0% /dev/shm
Is there anything to do to prevent this ? should the DB swap at all ?
MonetDB does in-memory processing, i.e., all columns that are active at a time need to be in the processes address space, either being loaded or being memory mapped; so yes, with huge amount of data MonetDB will also use virtual memory, either as swap or as memory-mapped files.
Stefan
De : Stefan de Konink stefan@konink.de À : sylver_b sylver_b@yahoo.com Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 1h09mn 13s Objet : Re: [Monetdb-developers] mclient running out of memory, crashing mserver5
sylver_b wrote:
I don't think i'm running out of memory. I had to stop and restart mserver5, flush the table, then re-run the command - all the records were inserted. Also I noticed this type of messages on the mserver5 console when piping records through the mclient (over 1million records) . It seems that the mclient is quiet instable (cf my post about "COPY, terminating connection" few months ago). So, is there a way to increase the memory limit or make sure mserver5 won't crash when running bash inserts via a cron job ?
The only thing I was able to do against that; more swap for the job. And of course the standard stuff as loading CSV files, preferably no indices on the table you are loading huge amounts of data to.
Stefan
This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
On Wed, Jan 21, 2009 at 03:12:20AM -0800, sylver_b wrote:
Hi Stephan, I will reply below :
De : Stefan Manegold Stefan.Manegold@cwi.nl À : sylver_b sylver_b@yahoo.com Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 10h28mn 41s Objet : Re: [Monetdb-developers] Re : mclient running out of memory, crashing mserver5
Yes
How big is/was your mserver5 when running or just before crashing?
missed to check that, but this is what i have at the moment :
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 31679 0.0 3.2 1950816 131056 pts/3 Sl+ 10:05 0:00 | _ mserver5 --dbinit=include sql;
This while being idle or while loading data?
In any case, with almost 2 GB vitual size, your mserver5 has obviously reached the addressable limit on a 32-bit system (well, theoretically, 32-bit systems can address 4 GB, but in practice processes are often only allowed to address 2 GB).
Your machine and OS seem to be 64-bit, right?
32-bit
What about your MonetDB installation? Could you please provide us with the output of `mserver5 --version` ?
# mserver5 --version
MonetDB server v5.8.0 (32-bit), based on kernel v1.26.0 (32-bit oids) Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008- MonetDB B.V., all rights reserved Visit http://monetdb.cwi.nl/for further information Configured for prefix: /root/MonetDB Libraries: openssl: OpenSSL 0.9.8b 04 May 2006 (compiled with OpenSSL 0.9.8b 04 May 2006) Compiled by: root@ulys Compilation: gcc -O2 -std=c99 Linking : /usr/bin/ld
Ok, on a 32-bit system, 32-bit MonetDB can only handle data sizes up to 2 GB at a time. Th efollwoing two limitations apply: each column of your tables (called BAT in MonetDB lingo) cannot exceed 2 GB AND the total size of all columns (BATs) active at a time while evaluating queries cannot exceed 2 GB, i.e., the addressable data size on 32-bit systems. When bulkloading data, all columns (BATs) of a table are active concurrently, hence, the total table size cannot exceed 2 GB.
The GDKmmap failure in you previous posting suggetst that MonetDB cannot allocate more memory (or to be more precise: address space), most probably as the available 2 GB address space is already exhausted.
As Stefan (de Konink) suggested, you can test whether loading your data succeeds without constraints (primary keys, foreign keys, etc.) defined on your table(s). If not, your data size might just be too large for a 32-bit MonetDB, and you'd need to resort to a 64-bit machine + 64-bit MonetDB or consider fragmenting your data.
Hope this helps ...
Stefan
How much free space is on your disk partition where your dbfarm is located?
# df -h
Filesystem Size Used Avail Use% Mounted on /dev/sda1 212G 180G 22G 90% / varrun 2.0G 56K 2.0G 1% /var/run varlock 2.0G 0 2.0G 0% /var/lock procbususb 10M 52K 10M 1% /proc/bus/usb udev 10M 52K 10M 1% /dev devshm 2.0G 0 2.0G 0% /dev/shm
Is there anything to do to prevent this ? should the DB swap at all ?
MonetDB does in-memory processing, i.e., all columns that are active at a time need to be in the processes address space, either being loaded or being memory mapped; so yes, with huge amount of data MonetDB will also use virtual memory, either as swap or as memory-mapped files.
Stefan
De : Stefan de Konink stefan@konink.de À : sylver_b sylver_b@yahoo.com Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 1h09mn 13s Objet : Re: [Monetdb-developers] mclient running out of memory, crashing mserver5
sylver_b wrote:
I don't think i'm running out of memory. I had to stop and restart mserver5, flush the table, then re-run the command - all the records were inserted. Also I noticed this type of messages on the mserver5 console when piping records through the mclient (over 1million records) . It seems that the mclient is quiet instable (cf my post about "COPY, terminating connection" few months ago). So, is there a way to increase the memory limit or make sure mserver5 won't crash when running bash inserts via a cron job ?
The only thing I was able to do against that; more swap for the job. And of course the standard stuff as loading CSV files, preferably no indices on the table you are loading huge amounts of data to.
Stefan
This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/%C2%A0 | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
On Wed, Jan 21, 2009 at 02:07:04AM -0800, sylver_b wrote:
Hi
Interesting..
Even worst, this morning i found out that all the inserts of last night failed !! :(
#BBPTRIM_ENTER: memsize=1144385432,vmsize=1599613808 #BBPTRIM: memtarget=819717481 vmtarget=0 #BBPTRIM_EXIT: memsize=366265384,vmsize=1599605544 #BBPTRIM_ENTER: memsize=1173710640,vmsize=1628939016 #BBPTRIM: memtarget=849042689 vmtarget=0 #BBPTRIM_EXIT: memsize=366265384,vmsize=1628939016 #BBPTRIM_ENTER: memsize=1143635024,vmsize=1629665320 #BBPTRIM: memtarget=0 vmtarget=19052584 #BBPTRIM_EXIT: memsize=366265384,vmsize=1598862320 #BBPTRIM_ENTER: memsize=1143633968,vmsize=1645786120 #BBPTRIM: memtarget=0 vmtarget=35173384 #BBPTRIM_EXIT: memsize=366265384,vmsize=1645786120 #BBPTRIM_ENTER: memsize=1143633992,vmsize=1676588064 #BBPTRIM: memtarget=0 vmtarget=65975328 #BBPTRIM_EXIT: memsize=366265384,vmsize=1598862320 #BBPTRIM_ENTER: memsize=1143633944,vmsize=1613149168 #BBPTRIM: memtarget=0 vmtarget=2536432 #BBPTRIM_EXIT: memsize=366265384,vmsize=1582347224 #BBPTRIM_ENTER: memsize=1143633920,vmsize=1620095960 #BBPTRIM: memtarget=0 vmtarget=9483224 #BBPTRIM_EXIT: memsize=366265384,vmsize=1620095960 #GDKmmap(148111360) fail => BBPtrim(enter) usage[mem=1143633920,vm=1496757208] #BBPTRIM_ENTER: memsize=1143633920,vmsize=1496757208 #BBPTRIM: memtarget=0 vmtarget=1073741824 #BBPTRIM_EXIT: memsize=366265384,vmsize=1496757208 #GDKmmap(148111360) fail => BBPtrim(ready) usage[mem=1143633920,vm=1496757208] !ERROR: GDKload: cannot mmap(): name=32/3267, ext=theap !OS: Cannot allocate memory !ERROR: GDKload: failed name=32/3267, ext=theap
top - 10:00:34 up 249 days, 20:31, 4 users, load average: 0.28, 0.12, 0.04 Tasks: 86 total, 1 running, 85 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 4025600k total, 3633228k used, 392372k free, 107544k buffers Swap: 9502408k total, 92k used, 9502316k free, 1520104k cached
Is that after mserver5 had crashed?
Hi Stefan,
Very interesting. Thanks for your detailed reply.
I will go for the 64 bit architecture and re-run the trial.
Thanks again SB
________________________________ De : Stefan Manegold Stefan.Manegold@cwi.nl À : sylver_b sylver_b@yahoo.com Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 11h48mn 42s Objet : Re : [Monetdb-developers] Re : mclient running out of memory, crashing mserver5
On Wed, Jan 21, 2009 at 03:12:20AM -0800, sylver_b wrote:
Hi Stephan, I will reply below :
De : Stefan Manegold Stefan.Manegold@cwi.nl À : sylver_b sylver_b@yahoo.com Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 10h28mn 41s Objet : Re: [Monetdb-developers] Re : mclient running out of memory, crashing mserver5
Yes
How big is/was your mserver5 when running or just before crashing?
missed to check that, but this is what i have at the moment :
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 31679 0.0 3.2 1950816 131056 pts/3 Sl+ 10:05 0:00 | _ mserver5 --dbinit=include sql;
This while being idle or while loading data?
In any case, with almost 2 GB vitual size, your mserver5 has obviously reached the addressable limit on a 32-bit system (well, theoretically, 32-bit systems can address 4 GB, but in practice processes are often only allowed to address 2 GB).
Your machine and OS seem to be 64-bit, right?
32-bit
What about your MonetDB installation? Could you please provide us with the output of `mserver5 --version` ?
# mserver5 --version
MonetDB server v5.8.0 (32-bit), based on kernel v1.26.0 (32-bit oids) Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008- MonetDB B.V., all rights reserved Visit http://monetdb.cwi.nl/for further information Configured for prefix: /root/MonetDB Libraries: openssl: OpenSSL 0.9.8b 04 May 2006 (compiled with OpenSSL 0.9.8b 04 May 2006) Compiled by: root@ulys Compilation: gcc -O2 -std=c99 Linking : /usr/bin/ld
Ok, on a 32-bit system, 32-bit MonetDB can only handle data sizes up to 2 GB at a time. Th efollwoing two limitations apply: each column of your tables (called BAT in MonetDB lingo) cannot exceed 2 GB AND the total size of all columns (BATs) active at a time while evaluating queries cannot exceed 2 GB, i.e., the addressable data size on 32-bit systems. When bulkloading data, all columns (BATs) of a table are active concurrently, hence, the total table size cannot exceed 2 GB.
The GDKmmap failure in you previous posting suggetst that MonetDB cannot allocate more memory (or to be more precise: address space), most probably as the available 2 GB address space is already exhausted.
As Stefan (de Konink) suggested, you can test whether loading your data succeeds without constraints (primary keys, foreign keys, etc.) defined on your table(s). If not, your data size might just be too large for a 32-bit MonetDB, and you'd need to resort to a 64-bit machine + 64-bit MonetDB or consider fragmenting your data.
Hope this helps ...
Stefan
How much free space is on your disk partition where your dbfarm is located?
# df -h
Filesystem Size Used Avail Use% Mounted on /dev/sda1 212G 180G 22G 90% / varrun 2.0G 56K 2.0G 1% /var/run varlock 2.0G 0 2.0G 0% /var/lock procbususb 10M 52K 10M 1% /proc/bus/usb udev 10M 52K 10M 1% /dev devshm 2.0G 0 2.0G 0% /dev/shm
Is there anything to do to prevent this ? should the DB swap at all ?
MonetDB does in-memory processing, i.e., all columns that are active at a time need to be in the processes address space, either being loaded or being memory mapped; so yes, with huge amount of data MonetDB will also use virtual memory, either as swap or as memory-mapped files.
Stefan
De : Stefan de Konink stefan@konink.de À : sylver_b sylver_b@yahoo.com Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 1h09mn 13s Objet : Re: [Monetdb-developers] mclient running out of memory, crashing mserver5
sylver_b wrote:
I don't think i'm running out of memory. I had to stop and restart mserver5, flush the table, then re-run the command - all the records were inserted. Also I noticed this type of messages on the mserver5 console when piping records through the mclient (over 1million records) . It seems that the mclient is quiet instable (cf my post about "COPY, terminating connection" few months ago). So, is there a way to increase the memory limit or make sure mserver5 won't crash when running bash inserts via a cron job ?
The only thing I was able to do against that; more swap for the job. And of course the standard stuff as loading CSV files, preferably no indices on the table you are loading huge amounts of data to.
Stefan
This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/%C2%A0 | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
On Wed, Jan 21, 2009 at 02:07:04AM -0800, sylver_b wrote:
Hi
Interesting..
Even worst, this morning i found out that all the inserts of last night failed !! :(
#BBPTRIM_ENTER: memsize=1144385432,vmsize=1599613808 #BBPTRIM: memtarget=819717481 vmtarget=0 #BBPTRIM_EXIT: memsize=366265384,vmsize=1599605544 #BBPTRIM_ENTER: memsize=1173710640,vmsize=1628939016 #BBPTRIM: memtarget=849042689 vmtarget=0 #BBPTRIM_EXIT: memsize=366265384,vmsize=1628939016 #BBPTRIM_ENTER: memsize=1143635024,vmsize=1629665320 #BBPTRIM: memtarget=0 vmtarget=19052584 #BBPTRIM_EXIT: memsize=366265384,vmsize=1598862320 #BBPTRIM_ENTER: memsize=1143633968,vmsize=1645786120 #BBPTRIM: memtarget=0 vmtarget=35173384 #BBPTRIM_EXIT: memsize=366265384,vmsize=1645786120 #BBPTRIM_ENTER: memsize=1143633992,vmsize=1676588064 #BBPTRIM: memtarget=0 vmtarget=65975328 #BBPTRIM_EXIT: memsize=366265384,vmsize=1598862320 #BBPTRIM_ENTER: memsize=1143633944,vmsize=1613149168 #BBPTRIM: memtarget=0 vmtarget=2536432 #BBPTRIM_EXIT: memsize=366265384,vmsize=1582347224 #BBPTRIM_ENTER: memsize=1143633920,vmsize=1620095960 #BBPTRIM: memtarget=0 vmtarget=9483224 #BBPTRIM_EXIT: memsize=366265384,vmsize=1620095960 #GDKmmap(148111360) fail => BBPtrim(enter) usage[mem=1143633920,vm=1496757208] #BBPTRIM_ENTER: memsize=1143633920,vmsize=1496757208 #BBPTRIM: memtarget=0 vmtarget=1073741824 #BBPTRIM_EXIT: memsize=366265384,vmsize=1496757208 #GDKmmap(148111360) fail => BBPtrim(ready) usage[mem=1143633920,vm=1496757208] !ERROR: GDKload: cannot mmap(): name=32/3267, ext=theap !OS: Cannot allocate memory !ERROR: GDKload: failed name=32/3267, ext=theap
top - 10:00:34 up 249 days, 20:31, 4 users, load average: 0.28, 0.12, 0.04 Tasks: 86 total, 1 running, 85 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 4025600k total, 3633228k used, 392372k free, 107544k buffers Swap: 9502408k total, 92k used, 9502316k free, 1520104k cached
Is that after mserver5 had crashed?
By the way ... what will happen if i only have 4GB of memory on a 64-bit system and table size over 10GB (for example) ?
________________________________ De : Stefan Manegold Stefan.Manegold@cwi.nl À : sylver_b sylver_b@yahoo.com Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 11h48mn 42s Objet : Re : [Monetdb-developers] Re : mclient running out of memory, crashing mserver5
On Wed, Jan 21, 2009 at 03:12:20AM -0800, sylver_b wrote:
Hi Stephan, I will reply below :
De : Stefan Manegold Stefan.Manegold@cwi.nl À : sylver_b sylver_b@yahoo.com Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 10h28mn 41s Objet : Re: [Monetdb-developers] Re : mclient running out of memory, crashing mserver5
Yes
How big is/was your mserver5 when running or just before crashing?
missed to check that, but this is what i have at the moment :
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 31679 0.0 3.2 1950816 131056 pts/3 Sl+ 10:05 0:00 | _ mserver5 --dbinit=include sql;
This while being idle or while loading data?
In any case, with almost 2 GB vitual size, your mserver5 has obviously reached the addressable limit on a 32-bit system (well, theoretically, 32-bit systems can address 4 GB, but in practice processes are often only allowed to address 2 GB).
Your machine and OS seem to be 64-bit, right?
32-bit
What about your MonetDB installation? Could you please provide us with the output of `mserver5 --version` ?
# mserver5 --version
MonetDB server v5.8.0 (32-bit), based on kernel v1.26.0 (32-bit oids) Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008- MonetDB B.V., all rights reserved Visit http://monetdb.cwi.nl/for further information Configured for prefix: /root/MonetDB Libraries: openssl: OpenSSL 0.9.8b 04 May 2006 (compiled with OpenSSL 0.9.8b 04 May 2006) Compiled by: root@ulys Compilation: gcc -O2 -std=c99 Linking : /usr/bin/ld
Ok, on a 32-bit system, 32-bit MonetDB can only handle data sizes up to 2 GB at a time. Th efollwoing two limitations apply: each column of your tables (called BAT in MonetDB lingo) cannot exceed 2 GB AND the total size of all columns (BATs) active at a time while evaluating queries cannot exceed 2 GB, i.e., the addressable data size on 32-bit systems. When bulkloading data, all columns (BATs) of a table are active concurrently, hence, the total table size cannot exceed 2 GB.
The GDKmmap failure in you previous posting suggetst that MonetDB cannot allocate more memory (or to be more precise: address space), most probably as the available 2 GB address space is already exhausted.
As Stefan (de Konink) suggested, you can test whether loading your data succeeds without constraints (primary keys, foreign keys, etc.) defined on your table(s). If not, your data size might just be too large for a 32-bit MonetDB, and you'd need to resort to a 64-bit machine + 64-bit MonetDB or consider fragmenting your data.
Hope this helps ...
Stefan
How much free space is on your disk partition where your dbfarm is located?
# df -h
Filesystem Size Used Avail Use% Mounted on /dev/sda1 212G 180G 22G 90% / varrun 2.0G 56K 2.0G 1% /var/run varlock 2.0G 0 2.0G 0% /var/lock procbususb 10M 52K 10M 1% /proc/bus/usb udev 10M 52K 10M 1% /dev devshm 2.0G 0 2.0G 0% /dev/shm
Is there anything to do to prevent this ? should the DB swap at all ?
MonetDB does in-memory processing, i.e., all columns that are active at a time need to be in the processes address space, either being loaded or being memory mapped; so yes, with huge amount of data MonetDB will also use virtual memory, either as swap or as memory-mapped files.
Stefan
De : Stefan de Konink stefan@konink.de À : sylver_b sylver_b@yahoo.com Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 1h09mn 13s Objet : Re: [Monetdb-developers] mclient running out of memory, crashing mserver5
sylver_b wrote:
I don't think i'm running out of memory. I had to stop and restart mserver5, flush the table, then re-run the command - all the records were inserted. Also I noticed this type of messages on the mserver5 console when piping records through the mclient (over 1million records) . It seems that the mclient is quiet instable (cf my post about "COPY, terminating connection" few months ago). So, is there a way to increase the memory limit or make sure mserver5 won't crash when running bash inserts via a cron job ?
The only thing I was able to do against that; more swap for the job. And of course the standard stuff as loading CSV files, preferably no indices on the table you are loading huge amounts of data to.
Stefan
This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/%C2%A0 | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
On Wed, Jan 21, 2009 at 02:07:04AM -0800, sylver_b wrote:
Hi
Interesting..
Even worst, this morning i found out that all the inserts of last night failed !! :(
#BBPTRIM_ENTER: memsize=1144385432,vmsize=1599613808 #BBPTRIM: memtarget=819717481 vmtarget=0 #BBPTRIM_EXIT: memsize=366265384,vmsize=1599605544 #BBPTRIM_ENTER: memsize=1173710640,vmsize=1628939016 #BBPTRIM: memtarget=849042689 vmtarget=0 #BBPTRIM_EXIT: memsize=366265384,vmsize=1628939016 #BBPTRIM_ENTER: memsize=1143635024,vmsize=1629665320 #BBPTRIM: memtarget=0 vmtarget=19052584 #BBPTRIM_EXIT: memsize=366265384,vmsize=1598862320 #BBPTRIM_ENTER: memsize=1143633968,vmsize=1645786120 #BBPTRIM: memtarget=0 vmtarget=35173384 #BBPTRIM_EXIT: memsize=366265384,vmsize=1645786120 #BBPTRIM_ENTER: memsize=1143633992,vmsize=1676588064 #BBPTRIM: memtarget=0 vmtarget=65975328 #BBPTRIM_EXIT: memsize=366265384,vmsize=1598862320 #BBPTRIM_ENTER: memsize=1143633944,vmsize=1613149168 #BBPTRIM: memtarget=0 vmtarget=2536432 #BBPTRIM_EXIT: memsize=366265384,vmsize=1582347224 #BBPTRIM_ENTER: memsize=1143633920,vmsize=1620095960 #BBPTRIM: memtarget=0 vmtarget=9483224 #BBPTRIM_EXIT: memsize=366265384,vmsize=1620095960 #GDKmmap(148111360) fail => BBPtrim(enter) usage[mem=1143633920,vm=1496757208] #BBPTRIM_ENTER: memsize=1143633920,vmsize=1496757208 #BBPTRIM: memtarget=0 vmtarget=1073741824 #BBPTRIM_EXIT: memsize=366265384,vmsize=1496757208 #GDKmmap(148111360) fail => BBPtrim(ready) usage[mem=1143633920,vm=1496757208] !ERROR: GDKload: cannot mmap(): name=32/3267, ext=theap !OS: Cannot allocate memory !ERROR: GDKload: failed name=32/3267, ext=theap
top - 10:00:34 up 249 days, 20:31, 4 users, load average: 0.28, 0.12, 0.04 Tasks: 86 total, 1 running, 85 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 4025600k total, 3633228k used, 392372k free, 107544k buffers Swap: 9502408k total, 92k used, 9502316k free, 1520104k cached
Is that after mserver5 had crashed?
On Wed, Jan 21, 2009 at 04:16:00AM -0800, sylver_b wrote:
By the way ... what will happen if i only have 4GB of memory on a 64-bit system and table size over 10GB (for example) ?
MonetDB will use virtual memory, mostly memory mapped files, partly swap --- until the available disk capacity or address space is exceeded --- whatever limit comes first (holds for both 32- & 64-bit systems).
Stefan
De : Stefan Manegold Stefan.Manegold@cwi.nl À : sylver_b sylver_b@yahoo.com Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 11h48mn 42s Objet : Re : [Monetdb-developers] Re : mclient running out of memory, crashing mserver5
On Wed, Jan 21, 2009 at 03:12:20AM -0800, sylver_b wrote:
Hi Stephan, I will reply below :
De : Stefan Manegold Stefan.Manegold@cwi.nl À : sylver_b sylver_b@yahoo.com Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 10h28mn 41s Objet : Re: [Monetdb-developers] Re : mclient running out of memory, crashing mserver5
Yes
How big is/was your mserver5 when running or just before crashing?
missed to check that, but this is what i have at the moment :
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 31679 0.0 3.2 1950816 131056 pts/3 Sl+ 10:05 0:00 | _ mserver5 --dbinit=include sql;
This while being idle or while loading data?
In any case, with almost 2 GB vitual size, your mserver5 has obviously reached the addressable limit on a 32-bit system (well, theoretically, 32-bit systems can address 4 GB, but in practice processes are often only allowed to address 2 GB).
Your machine and OS seem to be 64-bit, right?
32-bit
What about your MonetDB installation? Could you please provide us with the output of `mserver5 --version` ?
# mserver5 --version
MonetDB server v5.8.0 (32-bit), based on kernel v1.26.0 (32-bit oids) Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008- MonetDB B.V., all rights reserved Visit http://monetdb.cwi.nl/for further information Configured for prefix: /root/MonetDB Libraries: openssl: OpenSSL 0.9.8b 04 May 2006 (compiled with OpenSSL 0.9.8b 04 May 2006) Compiled by: root@ulys Compilation: gcc -O2 -std=c99 Linking : /usr/bin/ld
Ok, on a 32-bit system, 32-bit MonetDB can only handle data sizes up to 2 GB at a time. Th efollwoing two limitations apply: each column of your tables (called BAT in MonetDB lingo) cannot exceed 2 GB AND the total size of all columns (BATs) active at a time while evaluating queries cannot exceed 2 GB, i.e., the addressable data size on 32-bit systems. When bulkloading data, all columns (BATs) of a table are active concurrently, hence, the total table size cannot exceed 2 GB.
The GDKmmap failure in you previous posting suggetst that MonetDB cannot allocate more memory (or to be more precise: address space), most probably as the available 2 GB address space is already exhausted.
As Stefan (de Konink) suggested, you can test whether loading your data succeeds without constraints (primary keys, foreign keys, etc.) defined on your table(s). If not, your data size might just be too large for a 32-bit MonetDB, and you'd need to resort to a 64-bit machine + 64-bit MonetDB or consider fragmenting your data.
Hope this helps ...
Stefan
How much free space is on your disk partition where your dbfarm is located?
# df -h
Filesystem Size Used Avail Use% Mounted on /dev/sda1 212G 180G 22G 90% / varrun 2.0G 56K 2.0G 1% /var/run varlock 2.0G 0 2.0G 0% /var/lock procbususb 10M 52K 10M 1% /proc/bus/usb udev 10M 52K 10M 1% /dev devshm 2.0G 0 2.0G 0% /dev/shm
Is there anything to do to prevent this ? should the DB swap at all ?
MonetDB does in-memory processing, i.e., all columns that are active at a time need to be in the processes address space, either being loaded or being memory mapped; so yes, with huge amount of data MonetDB will also use virtual memory, either as swap or as memory-mapped files.
Stefan
De : Stefan de Konink stefan@konink.de À : sylver_b sylver_b@yahoo.com Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 1h09mn 13s Objet : Re: [Monetdb-developers] mclient running out of memory, crashing mserver5
sylver_b wrote:
I don't think i'm running out of memory. I had to stop and restart mserver5, flush the table, then re-run the command - all the records were inserted. Also I noticed this type of messages on the mserver5 console when piping records through the mclient (over 1million records) . It seems that the mclient is quiet instable (cf my post about "COPY, terminating connection" few months ago). So, is there a way to increase the memory limit or make sure mserver5 won't crash when running bash inserts via a cron job ?
The only thing I was able to do against that; more swap for the job. And of course the standard stuff as loading CSV files, preferably no indices on the table you are loading huge amounts of data to.
Stefan
This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/%C2%A0 | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
On Wed, Jan 21, 2009 at 02:07:04AM -0800, sylver_b wrote:
Hi
Interesting..
Even worst, this morning i found out that all the inserts of last night failed !! :(
#BBPTRIM_ENTER: memsize=1144385432,vmsize=1599613808 #BBPTRIM: memtarget=819717481 vmtarget=0 #BBPTRIM_EXIT: memsize=366265384,vmsize=1599605544 #BBPTRIM_ENTER: memsize=1173710640,vmsize=1628939016 #BBPTRIM: memtarget=849042689 vmtarget=0 #BBPTRIM_EXIT: memsize=366265384,vmsize=1628939016 #BBPTRIM_ENTER: memsize=1143635024,vmsize=1629665320 #BBPTRIM: memtarget=0 vmtarget=19052584 #BBPTRIM_EXIT: memsize=366265384,vmsize=1598862320 #BBPTRIM_ENTER: memsize=1143633968,vmsize=1645786120 #BBPTRIM: memtarget=0 vmtarget=35173384 #BBPTRIM_EXIT: memsize=366265384,vmsize=1645786120 #BBPTRIM_ENTER: memsize=1143633992,vmsize=1676588064 #BBPTRIM: memtarget=0 vmtarget=65975328 #BBPTRIM_EXIT: memsize=366265384,vmsize=1598862320 #BBPTRIM_ENTER: memsize=1143633944,vmsize=1613149168 #BBPTRIM: memtarget=0 vmtarget=2536432 #BBPTRIM_EXIT: memsize=366265384,vmsize=1582347224 #BBPTRIM_ENTER: memsize=1143633920,vmsize=1620095960 #BBPTRIM: memtarget=0 vmtarget=9483224 #BBPTRIM_EXIT: memsize=366265384,vmsize=1620095960 #GDKmmap(148111360) fail => BBPtrim(enter) usage[mem=1143633920,vm=1496757208] #BBPTRIM_ENTER: memsize=1143633920,vmsize=1496757208 #BBPTRIM: memtarget=0 vmtarget=1073741824 #BBPTRIM_EXIT: memsize=366265384,vmsize=1496757208 #GDKmmap(148111360) fail => BBPtrim(ready) usage[mem=1143633920,vm=1496757208] !ERROR: GDKload: cannot mmap(): name=32/3267, ext=theap !OS: Cannot allocate memory !ERROR: GDKload: failed name=32/3267, ext=theap
top - 10:00:34 up 249 days, 20:31, 4 users, load average: 0.28, 0.12, 0.04 Tasks: 86 total, 1 running, 85 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 4025600k total, 3633228k used, 392372k free, 107544k buffers Swap: 9502408k total, 92k used, 9502316k free, 1520104k cached
Is that after mserver5 had crashed?
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/%C2%A0 | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
Thanks..
________________________________ De : Stefan Manegold Stefan.Manegold@cwi.nl À : sylver_b sylver_b@yahoo.com Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 12h40mn 10s Objet : Re: [Monetdb-developers] Re : mclient running out of memory, crashing mserver5
On Wed, Jan 21, 2009 at 04:16:00AM -0800, sylver_b wrote:
By the way ... what will happen if i only have 4GB of memory on a 64-bit system and table size over 10GB (for example) ?
MonetDB will use virtual memory, mostly memory mapped files, partly swap --- until the available disk capacity or address space is exceeded --- whatever limit comes first (holds for both 32- & 64-bit systems).
Stefan
De : Stefan Manegold Stefan.Manegold@cwi.nl À : sylver_b sylver_b@yahoo.com Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 11h48mn 42s Objet : Re : [Monetdb-developers] Re : mclient running out of memory, crashing mserver5
On Wed, Jan 21, 2009 at 03:12:20AM -0800, sylver_b wrote:
Hi Stephan, I will reply below :
De : Stefan Manegold Stefan.Manegold@cwi.nl À : sylver_b sylver_b@yahoo.com Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 10h28mn 41s Objet : Re: [Monetdb-developers] Re : mclient running out of memory, crashing mserver5
Yes
How big is/was your mserver5 when running or just before crashing?
missed to check that, but this is what i have at the moment :
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 31679 0.0 3.2 1950816 131056 pts/3 Sl+ 10:05 0:00 | _ mserver5 --dbinit=include sql;
This while being idle or while loading data?
In any case, with almost 2 GB vitual size, your mserver5 has obviously reached the addressable limit on a 32-bit system (well, theoretically, 32-bit systems can address 4 GB, but in practice processes are often only allowed to address 2 GB).
Your machine and OS seem to be 64-bit, right?
32-bit
What about your MonetDB installation? Could you please provide us with the output of `mserver5 --version` ?
# mserver5 --version
MonetDB server v5.8.0 (32-bit), based on kernel v1.26.0 (32-bit oids) Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008- MonetDB B.V., all rights reserved Visit http://monetdb.cwi.nl/for further information Configured for prefix: /root/MonetDB Libraries: openssl: OpenSSL 0.9.8b 04 May 2006 (compiled with OpenSSL 0.9.8b 04 May 2006) Compiled by: root@ulys Compilation: gcc -O2 -std=c99 Linking : /usr/bin/ld
Ok, on a 32-bit system, 32-bit MonetDB can only handle data sizes up to 2 GB at a time. Th efollwoing two limitations apply: each column of your tables (called BAT in MonetDB lingo) cannot exceed 2 GB AND the total size of all columns (BATs) active at a time while evaluating queries cannot exceed 2 GB, i.e., the addressable data size on 32-bit systems. When bulkloading data, all columns (BATs) of a table are active concurrently, hence, the total table size cannot exceed 2 GB.
The GDKmmap failure in you previous posting suggetst that MonetDB cannot allocate more memory (or to be more precise: address space), most probably as the available 2 GB address space is already exhausted.
As Stefan (de Konink) suggested, you can test whether loading your data succeeds without constraints (primary keys, foreign keys, etc.) defined on your table(s). If not, your data size might just be too large for a 32-bit MonetDB, and you'd need to resort to a 64-bit machine + 64-bit MonetDB or consider fragmenting your data.
Hope this helps ...
Stefan
How much free space is on your disk partition where your dbfarm is located?
# df -h
Filesystem Size Used Avail Use% Mounted on /dev/sda1 212G 180G 22G 90% / varrun 2.0G 56K 2.0G 1% /var/run varlock 2.0G 0 2.0G 0% /var/lock procbususb 10M 52K 10M 1% /proc/bus/usb udev 10M 52K 10M 1% /dev devshm 2.0G 0 2.0G 0% /dev/shm
Is there anything to do to prevent this ? should the DB swap at all ?
MonetDB does in-memory processing, i.e., all columns that are active at a time need to be in the processes address space, either being loaded or being memory mapped; so yes, with huge amount of data MonetDB will also use virtual memory, either as swap or as memory-mapped files.
Stefan
De : Stefan de Konink stefan@konink.de À : sylver_b sylver_b@yahoo.com Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 1h09mn 13s Objet : Re: [Monetdb-developers] mclient running out of memory, crashing mserver5
sylver_b wrote:
I don't think i'm running out of memory. I had to stop and restart mserver5, flush the table, then re-run the command - all the records were inserted. Also I noticed this type of messages on the mserver5 console when piping records through the mclient (over 1million records) . It seems that the mclient is quiet instable (cf my post about "COPY, terminating connection" few months ago). So, is there a way to increase the memory limit or make sure mserver5 won't crash when running bash inserts via a cron job ?
The only thing I was able to do against that; more swap for the job. And of course the standard stuff as loading CSV files, preferably no indices on the table you are loading huge amounts of data to.
Stefan
This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/%C2%A0 | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
On Wed, Jan 21, 2009 at 02:07:04AM -0800, sylver_b wrote:
Hi
Interesting..
Even worst, this morning i found out that all the inserts of last night failed !! :(
#BBPTRIM_ENTER: memsize=1144385432,vmsize=1599613808 #BBPTRIM: memtarget=819717481 vmtarget=0 #BBPTRIM_EXIT: memsize=366265384,vmsize=1599605544 #BBPTRIM_ENTER: memsize=1173710640,vmsize=1628939016 #BBPTRIM: memtarget=849042689 vmtarget=0 #BBPTRIM_EXIT: memsize=366265384,vmsize=1628939016 #BBPTRIM_ENTER: memsize=1143635024,vmsize=1629665320 #BBPTRIM: memtarget=0 vmtarget=19052584 #BBPTRIM_EXIT: memsize=366265384,vmsize=1598862320 #BBPTRIM_ENTER: memsize=1143633968,vmsize=1645786120 #BBPTRIM: memtarget=0 vmtarget=35173384 #BBPTRIM_EXIT: memsize=366265384,vmsize=1645786120 #BBPTRIM_ENTER: memsize=1143633992,vmsize=1676588064 #BBPTRIM: memtarget=0 vmtarget=65975328 #BBPTRIM_EXIT: memsize=366265384,vmsize=1598862320 #BBPTRIM_ENTER: memsize=1143633944,vmsize=1613149168 #BBPTRIM: memtarget=0 vmtarget=2536432 #BBPTRIM_EXIT: memsize=366265384,vmsize=1582347224 #BBPTRIM_ENTER: memsize=1143633920,vmsize=1620095960 #BBPTRIM: memtarget=0 vmtarget=9483224 #BBPTRIM_EXIT: memsize=366265384,vmsize=1620095960 #GDKmmap(148111360) fail => BBPtrim(enter) usage[mem=1143633920,vm=1496757208] #BBPTRIM_ENTER: memsize=1143633920,vmsize=1496757208 #BBPTRIM: memtarget=0 vmtarget=1073741824 #BBPTRIM_EXIT: memsize=366265384,vmsize=1496757208 #GDKmmap(148111360) fail => BBPtrim(ready) usage[mem=1143633920,vm=1496757208] !ERROR: GDKload: cannot mmap(): name=32/3267, ext=theap !OS: Cannot allocate memory !ERROR: GDKload: failed name=32/3267, ext=theap
top - 10:00:34 up 249 days, 20:31, 4 users, load average: 0.28, 0.12, 0.04 Tasks: 86 total, 1 running, 85 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 4025600k total, 3633228k used, 392372k free, 107544k buffers Swap: 9502408k total, 92k used, 9502316k free, 1520104k cached
Is that after mserver5 had crashed?
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/%C2%A0 | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
participants (3)
-
Stefan de Konink -
Stefan Manegold -
sylver_b