[Monetdb-developers] Could not create hash table for key test

Martin Kersten Martin.Kersten at cwi.nl
Sun Nov 23 08:32:00 CET 2008


Dear Stefan,
Let's explain as far as possible.

Stefan de Konink wrote:
>   #GDKmalloc(14777222432) fail => BBPtrim(enter) 
> usage[mem=14840896,vm=27527806976]
> #BBPTRIM_ENTER: memsize=14840896,vmsize=27527806976
> #BBPTRIM: memtarget=4611686018427387904 vmtarget=0
> #BBPTRIM_EXIT: memsize=19120128,vmsize=27527806976
>   
The system is trying to malloc space for (in this case) a hash table,
but that instruction fails. Then it attempts to free up memory by
swapping out tables.
Given the fact that after this sweep it still fails, indicates that
your  swap memory has become fragmented, such that there is no
consecutive area of 14 GB left.

For large table loads (relative to the swap size) it is advisable to 
load in multiple steps.
First load the data into the tables without (foreign) key checks.
After this step ALTER the tables to respect the (foreign) keys.

Better solutions are being worked on.
> #GDKmalloc(14777222432) fail => BBPtrim(ready) 
> usage[mem=11538120,vm=27527806976]
> #BATpropcheck: BAT tmp_1260(-688): could not allocate hash table for key 
> test
> #GDKmalloc(14777222432) fail => BBPtrim(enter) 
> usage[mem=11537664,vm=27527806976]
> #BBPTRIM_ENTER: memsize=11537664,vmsize=27527806976
> #BBPTRIM: memtarget=4611686018427387904 vmtarget=0
> #BBPTRIM_EXIT: memsize=19120128,vmsize=27527806976
> #GDKmalloc(14777222432) fail => BBPtrim(ready) 
> usage[mem=10416208,vm=27527806976]
> #BATpropcheck: BAT tmp_1135(-605): could not allocate hash table for key 
> test
> MAPI  = monetdb at localhost:50000
> QUERY = COPY 773410980 RECORDS INTO node_tags from 
> '/mnt/data2/csv/node_tags.csv' USING DELIMITERS ',', '\n', '''';
> ERROR = !MALException: GDKmallocmax: failed for 1892320544 bytes
>          !ERROR: GDKload: failed name=12/1252, ext=thash
>          !ERROR: GDKmallocmax: failed for 1892320544 bytes
>          !ERROR: GDKload: failed name=12/1253, ext=thash
>
> What kind of voodoo is this?
>
> I'm trying to import a 42GB csv file using copy into, on a native 
> system, 2GB of memory, 4GB of swap. Latest CVS version, this time 
> compiled for CoreDuo, without debuging, with optimise.
>
>
> Stefan
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Monetdb-developers mailing list
> Monetdb-developers at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/monetdb-developers
>   





More information about the developers-list mailing list