[Monetdb-developers] Re : Re : Re : mclient running out of memory, crashing mserver5

sylver_b sylver_b at yahoo.com
Wed Jan 21 13:12:59 CET 2009


Hi Stefan,

Very interesting. Thanks for your detailed reply.

I will go for the 64 bit architecture and re-run the trial.

Thanks again
SB




________________________________
De : Stefan Manegold <Stefan.Manegold at cwi.nl>
À : sylver_b <sylver_b at yahoo.com>
Cc : monetdb-developers at lists.sourceforge.net
Envoyé le : Mercredi, 21 Janvier 2009, 11h48mn 42s
Objet : Re : [Monetdb-developers] Re : mclient running out of memory, crashing mserver5

On Wed, Jan 21, 2009 at 03:12:20AM -0800, sylver_b wrote:
> Hi Stephan,
>  
> I will reply below :
> 
> 
> 
> ________________________________
> 
> De : Stefan Manegold <Stefan.Manegold at cwi.nl>
> À : sylver_b <sylver_b at yahoo.com>
> Cc : monetdb-developers at lists.sourceforge.net
> Envoyé le : Mercredi, 21 Janvier 2009, 10h28mn 41s
> Objet : Re: [Monetdb-developers] Re : mclient running out of memory, crashing mserver5
> 
> 
> >> Yes
> 
> How big is/was your mserver5 when running or just before crashing?
> 
> >> missed to check that, but this is what i have at the moment :
>  
> USER       PID     %CPU %MEM    VSZ        RSS     TTY      STAT START   TIME COMMAND
> root          31679  0.0       3.2          1950816 131056 pts/3     Sl+    10:05      0:00  |   \_ mserver5 --dbinit=include sql;

This while being idle or while loading data?

In any case, with almost 2 GB vitual size, your mserver5 has obviously
reached the addressable limit on a 32-bit system (well, theoretically,
32-bit systems can address 4 GB, but in practice processes are often only
allowed to address 2 GB).

> Your machine and OS seem to be 64-bit, right?
> >> 32-bit

> What about your MonetDB installation?
> Could you please provide us with the output of
>     `mserver5 --version`
> ?
> >> # mserver5 --version
> MonetDB server v5.8.0 (32-bit), based on kernel v1.26.0 (32-bit oids)
> Copyright (c) 1993-July 2008 CWI
> Copyright (c) August 2008- MonetDB B.V., all rights reserved
> Visit http://monetdb.cwi.nl/for further information
> Configured for prefix: /root/MonetDB
> Libraries:
>   openssl: OpenSSL 0.9.8b 04 May 2006 (compiled with OpenSSL 0.9.8b 04 May 2006)
> Compiled by: root at ulys
> Compilation: gcc -O2 -std=c99
> Linking    : /usr/bin/ld

Ok, on a 32-bit system, 32-bit MonetDB can only handle data sizes up to 2 GB
at a time. Th efollwoing two limitations apply: each column of your tables
(called BAT in MonetDB lingo) cannot exceed 2 GB AND the total size of all
columns (BATs) active at a time while evaluating queries cannot exceed 2 GB,
i.e., the addressable data size on 32-bit systems. When bulkloading data,
all columns (BATs) of a table are active concurrently, hence, the total
table size cannot exceed 2 GB.

The GDKmmap failure in you previous posting suggetst that MonetDB cannot
allocate more memory (or to be more precise: address space), most probably
as the available 2 GB address space is already exhausted.

As Stefan (de Konink) suggested, you can test whether loading your data
succeeds without constraints (primary keys, foreign keys, etc.) defined on
your table(s). If not, your data size might just be too large for a 32-bit
MonetDB, and you'd need to resort to a 64-bit machine + 64-bit MonetDB or
consider fragmenting your data.

Hope this helps ...

Stefan

> How much free space is on your disk partition where your dbfarm is located?
> >> # df -h
> Filesystem            Size Used Avail Use% Mounted on
> /dev/sda1             212G  180G   22G  90% /
> varrun                2.0G   56K  2.0G   1% /var/run
> varlock               2.0G     0  2.0G   0% /var/lock
> procbususb             10M   52K   10M   1% /proc/bus/usb
> udev                   10M   52K   10M   1% /dev
> devshm                2.0G     0  2.0G   0% /dev/shm
> 
> 
> > Is there anything to do to prevent this ? should the DB swap at all ?
> 
> MonetDB does in-memory processing, i.e., all columns that are active at a time
> need to be in the processes address space, either being loaded or being
> memory mapped; so yes, with huge amount of data MonetDB will also use
> virtual memory, either as swap or as memory-mapped files.
> 
> Stefan
> 
> > 
> > 
> > 
> > 
> > 
> > ________________________________
> > De : Stefan de Konink <stefan at konink.de>
> > À : sylver_b <sylver_b at yahoo.com>
> > Cc : monetdb-developers at lists.sourceforge.net
> > Envoyé le : Mercredi, 21 Janvier 2009, 1h09mn 13s
> > Objet : Re: [Monetdb-developers] mclient running out of memory, crashing mserver5
> > 
> > sylver_b wrote:
> > > I don't think i'm running out of memory. I had to stop and restart mserver5, flush the table, then re-run the command - all the records were inserted. Also I noticed this type of messages on the mserver5 console when piping records through the mclient (over 1million records) . It seems that the mclient is quiet instable (cf my post about "COPY, terminating connection" few months ago).
> > >  So, is there a way to increase the memory limit  or make sure mserver5 won't crash when running bash inserts via a cron job ?
> > 
> > The only thing I was able to do against that; more swap for the job. And of course the standard stuff as loading CSV files, preferably no indices on the table you are loading huge amounts of data to.
> > 
> > 
> > Stefan
> > 
> > 
> > 
> >      
> > ------------------------------------------------------------------------------
> > This SF.net email is sponsored by:
> > SourcForge Community
> > SourceForge wants to tell your story.
> > http://p.sf.net/sfu/sf-spreadtheword
> > _______________________________________________
> > Monetdb-developers mailing list
> > Monetdb-developers at lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/monetdb-developers
> 
> 
> -- 
> | Dr. Stefan Manegold | mailto:Stefan.Manegold at cwi.nl |
> | CWI,  P.O.Box 94079 | http://www.cwi.nl/~manegold/  |
> | 1090 GB Amsterdam  | Tel.: +31 (20) 592-4212      |
> | The Netherlands    | Fax : +31 (20) 592-4312      |
> 
> On Wed, Jan 21, 2009 at 02:07:04AM -0800, sylver_b wrote:
> > Hi 
> > 
> > Interesting..
> > 
> > Even worst, this morning i found out that all the inserts of last night failed !! :(
> > 
> > 
> > #BBPTRIM_ENTER: memsize=1144385432,vmsize=1599613808
> > #BBPTRIM: memtarget=819717481 vmtarget=0
> > #BBPTRIM_EXIT: memsize=366265384,vmsize=1599605544
> > #BBPTRIM_ENTER: memsize=1173710640,vmsize=1628939016
> > #BBPTRIM: memtarget=849042689 vmtarget=0
> > #BBPTRIM_EXIT: memsize=366265384,vmsize=1628939016
> > #BBPTRIM_ENTER: memsize=1143635024,vmsize=1629665320
> > #BBPTRIM: memtarget=0 vmtarget=19052584
> > #BBPTRIM_EXIT: memsize=366265384,vmsize=1598862320
> > #BBPTRIM_ENTER: memsize=1143633968,vmsize=1645786120
> > #BBPTRIM: memtarget=0 vmtarget=35173384
> > #BBPTRIM_EXIT: memsize=366265384,vmsize=1645786120
> > #BBPTRIM_ENTER: memsize=1143633992,vmsize=1676588064
> > #BBPTRIM: memtarget=0 vmtarget=65975328
> > #BBPTRIM_EXIT: memsize=366265384,vmsize=1598862320
> > #BBPTRIM_ENTER: memsize=1143633944,vmsize=1613149168
> > #BBPTRIM: memtarget=0 vmtarget=2536432
> > #BBPTRIM_EXIT: memsize=366265384,vmsize=1582347224
> > #BBPTRIM_ENTER: memsize=1143633920,vmsize=1620095960
> > #BBPTRIM: memtarget=0 vmtarget=9483224
> > #BBPTRIM_EXIT: memsize=366265384,vmsize=1620095960
> > #GDKmmap(148111360) fail => BBPtrim(enter) usage[mem=1143633920,vm=1496757208]
> > #BBPTRIM_ENTER: memsize=1143633920,vmsize=1496757208
> > #BBPTRIM: memtarget=0 vmtarget=1073741824
> > #BBPTRIM_EXIT: memsize=366265384,vmsize=1496757208
> > #GDKmmap(148111360) fail => BBPtrim(ready) usage[mem=1143633920,vm=1496757208]
> > !ERROR: GDKload: cannot mmap(): name=32/3267, ext=theap
> > !OS: Cannot allocate memory
> > !ERROR: GDKload: failed name=32/3267, ext=theap
> > 
> > top - 10:00:34 up 249 days, 20:31,  4 users,  load average: 0.28, 0.12, 0.04
> > Tasks:  86 total,   1 running,  85 sleeping,   0 stopped,   0 zombie
> > Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
> > Mem:   4025600k total,  3633228k used,   392372k free,   107544k buffers
> > Swap:  9502408k total,       92k used,  9502316k free,  1520104k cached
> 
> Is that after mserver5 had crashed?
> 
> 
>      
-- 
| Dr. Stefan Manegold | mailto:Stefan.Manegold at cwi.nl |
| CWI,  P.O.Box 94079 | http://www.cwi.nl/~manegold/  |
| 1090 GB Amsterdam  | Tel.: +31 (20) 592-4212      |
| The Netherlands    | Fax : +31 (20) 592-4312      |



      
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.monetdb.org/pipermail/developers-list/attachments/20090121/08de2c60/attachment.html>


More information about the developers-list mailing list