[Monetdb-developers] Dealing with large tables
Martin.Kersten at cwi.nl
Thu Jun 14 18:29:05 CEST 2007
Agustin Schapira wrote:
> Dear Monet developers,
> Once again, here I am, writing for help ;-)
> We are working with a 'large' table, with 250 million rows, 11
> columns. This takes up a lot of memory space, as you can imagine: a
> single column of 250 million 'ints' requires almost a gigabyte, and
I would say 1GB is a minimum. What if you are on a 64bit machine?
And also, if this table requires a hash index, then add another 1GB/table.
> there's several of those; in fact, I think that the entire table
> takes up a bit over 6GB in memory (some columns are strings, and can
> be compressed). The problem is that we have a machine with 8GB, and
> every time we try to do something with that table, Monet (4.16.2)
> crashes. For example, I've been trying to delete rows from the table
> where two columns have the same value:
> var var_0 := [=](col1,col2).select(true).mirror();
Hard to deduce what's happening.
I would first try to make sure if the commit
is not sitting in the way.
Do you need to commit here ?
> After deleting from a few of those columns, Monet dies:
> !ERROR: BATSIGcrash: Mserver internal error (Segmentation fault),
> please restart.
> !ERROR: (One potential cause could be that your disk might be full...)
> Do you know if there a fix for this? In principle, Monet should be
> able to unload the BATs that it has already deleted from, to release
> memory. Also, would doing a semijoin() instead of a delete() help? (I
> though that a semijoin would need to create a new BAT, thereby using
> even more memory!). And, in general: is Monet 5 capable of accessing
> BATs that don't entirely fit in memory? Would the new version fix
> this problem? Maybe it's time for us to adapt it...
> Thanks again for all your help, and regards from Amherst,
> -- Agustin
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> Monetdb-developers mailing list
> Monetdb-developers at lists.sourceforge.net
More information about the developers-list