Resource limits in MonetDB
I am experimenting with using MonetDB for a project where the most natural implementation would involve ~100K tables (with the huge majority being near empty, a few incredibly massive) with approx. 10 columns each. MonetDB choked on this, with the following error:
!FATAL: BBPextend: trying to extend BAT pool beyond the limit (16384000)
I am assuming that I hit this limit because of the extreme amount of tables. I am wondering if there are any other hard limits to be aware of in MonetDB that you could hit before hitting system limits.
Examining the source code (gdk_bbp.c and gdk.h), one finds a compile-time limit on the number of BATs (binary association tables), i.e. the basic internal unit of operation within MonetDB.
I've not tried its practical limits myself and haven't done the analysis for your case below (partly because exact numbers are not provided) but you may be able to extend this limit by rebuilding the dbms from source with a larger value for N_BBPINIT, for instance.
Regards, Masood
P.S. Here's a relevant fragment from gdk.h
#define N_BBPINIT 1000 #if SIZEOF_VOID_P == 4 #define BBPINITLOG 11 #else #define BBPINITLOG 14 #endif #define BBPINIT (1 << BBPINITLOG) /* absolute maximum number of BATs is N_BBPINIT * BBPINIT */
On Fri, Jan 18, 2013 at 8:19 AM, Charles Milutinovic milutino@gmail.comwrote:
I am experimenting with using MonetDB for a project where the most natural implementation would involve ~100K tables (with the huge majority being near empty, a few incredibly massive) with approx. 10 columns each. MonetDB choked on this, with the following error:
!FATAL: BBPextend: trying to extend BAT pool beyond the limit (16384000)
I am assuming that I hit this limit because of the extreme amount of tables. I am wondering if there are any other hard limits to be aware of in MonetDB that you could hit before hitting system limits. _______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
Dear Masood,
The BAT pool has an arbitrary limit set. Be aware that each BAT descriptor takes about 0.5K and the structure lives in your malloc arena. Repeatedly extending it may lead to memory fragmentation.
Overall, I would re-consider the reason for having >100K tables of which most are empty. Aside from BAT storage overhead, the database catalog table structures are even worse and syntax/semantic checks could become a severe performance issue. We haven't tested that extreme case (aside from a single table with 4 M attributes).
regards, Martin
On 1/21/13 8:08 AM, Masood Mortazavi wrote:
Examining the source code (gdk_bbp.c and gdk.h), one finds a compile-time limit on the number of BATs (binary association tables), i.e. the basic internal unit of operation within MonetDB.
I've not tried its practical limits myself and haven't done the analysis for your case below (partly because exact numbers are not provided) but you may be able to extend this limit by rebuilding the dbms from source with a larger value for N_BBPINIT, for instance.
Regards, Masood
P.S. Here's a relevant fragment from gdk.h
#define N_BBPINIT 1000 #if SIZEOF_VOID_P == 4 #define BBPINITLOG 11 #else #define BBPINITLOG 14 #endif #define BBPINIT (1 << BBPINITLOG) /* absolute maximum number of BATs is N_BBPINIT * BBPINIT */
On Fri, Jan 18, 2013 at 8:19 AM, Charles Milutinovic <milutino@gmail.com mailto:milutino@gmail.com> wrote:
I am experimenting with using MonetDB for a project where the most natural implementation would involve ~100K tables (with the huge majority being near empty, a few incredibly massive) with approx. 10 columns each. MonetDB choked on this, with the following error: >!FATAL: BBPextend: trying to extend BAT pool beyond the limit (16384000) I am assuming that I hit this limit because of the extreme amount of tables. I am wondering if there are any other hard limits to be aware of in MonetDB that you could hit before hitting system limits. _______________________________________________ users-list mailing list users-list@monetdb.org <mailto:users-list@monetdb.org> http://mail.monetdb.org/mailman/listinfo/users-list
users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
participants (3)
-
Charles Milutinovic -
Martin Kersten -
Masood Mortazavi