[Monetdb-developers] Dealing with large tables

Peter Boncz P.Boncz at cwi.nl
Thu Jun 14 19:34:15 CEST 2007


Hi Augustin,

The crash is a bug. You have not told us about your platform (Darwin, I
presume) and whether you have compiled with 64-bits enabled. Considering the
data is 'large', it may be an out-of-memory problem but then in any case it is
a bug, because lack of resources should lead to the query failing rather than
crashing.

So, first provide us the missing platform info (is it 64 bits????), and
second, recompile with debugging enabled (configure with: --enable-debug=yes
--enable-optimize=no). Since this problem seems highly reproducable, then run
again following my instructions:

(0) start Mserver in gdb (gdb --args Mserver ...args''')
(1) modify the MIL script to set the debugmask 19
(2) also insert some extra info() calls
(3) when it crashes, type in the gdb console: bt
(4) send us all the output


Your modified MIL script should resemble:

debugmask(19);
var var_0 := [=](col1,col2).select(true).mirror();
var_0.info().print();
col1.info().print();
col1.delete(var_0);
commit();
col2.info().print();
col2.delete(var_0);
commit();
col3.info().print();
col3.delete(var_0);
commit();


thanks in advance,

Peter


------------------------------

Message: 6
Date: Thu, 14 Jun 2007 11:59:24 -0400
From: Agustin Schapira <schapira at cs.umass.edu>
Subject: [Monetdb-developers] Dealing with large tables
To: monetdb-developers at lists.sourceforge.net
Message-ID: <9FAD9406-07DC-454A-9AB0-A9C9DFD05153 at cs.umass.edu>
Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed

Dear Monet developers,

Once again, here I am, writing for help ;-)

We are working with a 'large' table, with 250 million rows, 11
columns. This takes up a lot of memory space, as you can imagine: a
single column of 250 million 'ints' requires almost a gigabyte, and
there's several of those; in fact, I think that the entire table
takes up a bit over 6GB in memory (some columns are strings, and can
be compressed). The problem is that we have a machine with 8GB, and
every time we try to do something with that table, Monet (4.16.2)
crashes. For example, I've been trying to delete rows from the table
where two columns have the same value:

var var_0 := [=](col1,col2).select(true).mirror();
col1.delete(var_0);
commit();
col2.delete(var_0);
commit();
col3.delete(var_0);
commit();
...

After deleting from a few of those columns, Monet dies:

!ERROR: BATSIGcrash: Mserver internal error (Segmentation fault),
please restart.
!ERROR: (One potential cause could be that your disk might be full...)


Do you know if there a fix for this? In principle, Monet should be
able to unload the BATs that it has already deleted from, to release
memory. Also, would doing a semijoin() instead of a delete() help? (I
though that a semijoin would need to create a new BAT, thereby using
even more memory!). And, in general: is Monet 5 capable of accessing
BATs that don't entirely fit in memory? Would the new version fix
this problem? Maybe it's time for us to adapt it...

Thanks again for all your help, and regards from Amherst,

-- Agustin



------------------------------

Message: 7
Date: Thu, 14 Jun 2007 18:29:05 +0200
From: Martin Kersten <Martin.Kersten at cwi.nl>
Subject: Re: [Monetdb-developers] Dealing with large tables
To: Agustin Schapira <schapira at cs.umass.edu>
Cc: monetdb-developers at lists.sourceforge.net
Message-ID: <46716CD1.8080104 at cwi.nl>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

Agustin Schapira wrote:
> Dear Monet developers,
>
> Once again, here I am, writing for help ;-)
>
> We are working with a 'large' table, with 250 million rows, 11
> columns. This takes up a lot of memory space, as you can imagine: a
> single column of 250 million 'ints' requires almost a gigabyte, and
I would say 1GB is a minimum. What if you are on a 64bit machine?
And also, if this table requires a hash index, then add another 1GB/table.

> there's several of those; in fact, I think that the entire table
> takes up a bit over 6GB in memory (some columns are strings, and can
> be compressed). The problem is that we have a machine with 8GB, and
> every time we try to do something with that table, Monet (4.16.2)
> crashes. For example, I've been trying to delete rows from the table
> where two columns have the same value:
>
> var var_0 := [=](col1,col2).select(true).mirror();

Hard to deduce what's happening.
> col1.delete(var_0);
> commit();
> col2.delete(var_0);
> commit();
> col3.delete(var_0);
> commit();
I would first try to make sure if the commit
is not sitting in the way.
Do you need to commit here ?
> ...
>
> After deleting from a few of those columns, Monet dies:
>
> !ERROR: BATSIGcrash: Mserver internal error (Segmentation fault),
> please restart.
> !ERROR: (One potential cause could be that your disk might be full...)
>
>
> Do you know if there a fix for this? In principle, Monet should be
> able to unload the BATs that it has already deleted from, to release
> memory. Also, would doing a semijoin() instead of a delete() help? (I
> though that a semijoin would need to create a new BAT, thereby using
> even more memory!). And, in general: is Monet 5 capable of accessing
> BATs that don't entirely fit in memory? Would the new version fix
> this problem? Maybe it's time for us to adapt it...
>
> Thanks again for all your help, and regards from Amherst,
>
> -- Agustin
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> _______________________________________________
> Monetdb-developers mailing list
> Monetdb-developers at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/monetdb-developers




------------------------------

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

------------------------------

_______________________________________________
Monetdb-developers mailing list
Monetdb-developers at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/monetdb-developers


End of Monetdb-developers Digest, Vol 13, Issue 6
*************************************************




More information about the developers-list mailing list