[Monetdb-developers] Could not create hash table for key test

Stefan de Konink stefan at konink.de
Sun Nov 23 13:40:11 CET 2008


Hi Peter,


Peter Boncz wrote:
> Like Martin indicated, MonetDB runs out of memory here, when trying to obtain
> a contiguous area of 14GB.

But is a contiguous area of 14GB required for this? That is much memory!

> The error messages show that MonetDB is pulling all the stops to try to get
> this memory from the system. That is, a so-called "trim" buffer manager thread
> tries to unload all unpinned other tables. To no avail here. You see the
> actvity of the trim thread only when a GDKmalloc has failed already, and the
> system tries again after unloading what it can.

I'm now trying to import the relative 'smaller' integer only tables 
without references; but it ends up also in:

#BBPTRIM_ENTER: memsize=180224,vmsize=4443275264
#BBPTRIM: memtarget=0 vmtarget=0
#BBPTRIM_EXIT: memsize=14925824,vmsize=4443275264
#BBPTRIM_ENTER: memsize=180224,vmsize=4443275264
#BBPTRIM: memtarget=0 vmtarget=0
#BBPTRIM_EXIT: memsize=14925824,vmsize=4443275264
#GDKmalloc(1374238248) fail => BBPtrim(enter) 
usage[mem=180224,vm=4443275264]
#BBPTRIM_ENTER: memsize=180224,vmsize=4443275264
#BBPTRIM: memtarget=4611686018427387904 vmtarget=0
#BBPTRIM_EXIT: memsize=14925824,vmsize=4443275264
#GDKmalloc(1374238248) fail => BBPtrim(ready) 
usage[mem=180224,vm=4443275264]
!ERROR: GDKmallocmax: failed for 1374238248 bytes
!ERROR: GDKload: failed name=31/3141, ext=tail

> The fact that though it tries to GDKmalloc you see the GDKload error message
> indicates that it actually tries to get the 14GB by creating a file and memory
> mapping on it.
> 
> The cause for failure can thus be:
> (1) your file system is full (test just after the error message appears,
> because a MonetDB restarts first clears out all temp files it created on the
> last run)

MonetDB runs on 95GB of free data. Resulting in 16GB in MonetDB5, having 
inserted 19GB worth of textual content.

> (2) you try to process such large loads on a 32-bts machine (naughty naughty)

I'm not naughty, only have sick mind to try to run this amount of data 
on only 2GB of RAM.

Linux srv1 2.6.27-gentoo-r4 #2 SMP Sat Nov 22 18:34:42 CET 2008 x86_64 
Intel(R) Pentium(R) Dual CPU E2180 @ 2.00GHz GenuineIntel GNU/Linux


Stefan




More information about the developers-list mailing list