[Monetdb-developers] Dealing with large tables

Agustin Schapira schapira at cs.umass.edu
Thu Jun 14 17:59:24 CEST 2007


Dear Monet developers,

Once again, here I am, writing for help ;-)

We are working with a 'large' table, with 250 million rows, 11  
columns. This takes up a lot of memory space, as you can imagine: a  
single column of 250 million 'ints' requires almost a gigabyte, and  
there's several of those; in fact, I think that the entire table  
takes up a bit over 6GB in memory (some columns are strings, and can  
be compressed). The problem is that we have a machine with 8GB, and  
every time we try to do something with that table, Monet (4.16.2)  
crashes. For example, I've been trying to delete rows from the table  
where two columns have the same value:

var var_0 := [=](col1,col2).select(true).mirror();
col1.delete(var_0);
commit();
col2.delete(var_0);
commit();
col3.delete(var_0);
commit();
...

After deleting from a few of those columns, Monet dies:

!ERROR: BATSIGcrash: Mserver internal error (Segmentation fault),  
please restart.
!ERROR: (One potential cause could be that your disk might be full...)


Do you know if there a fix for this? In principle, Monet should be  
able to unload the BATs that it has already deleted from, to release  
memory. Also, would doing a semijoin() instead of a delete() help? (I  
though that a semijoin would need to create a new BAT, thereby using  
even more memory!). And, in general: is Monet 5 capable of accessing  
BATs that don't entirely fit in memory? Would the new version fix  
this problem? Maybe it's time for us to adapt it...

Thanks again for all your help, and regards from Amherst,

-- Agustin




More information about the developers-list mailing list