[Monetdb-developers] [MonetDB-users] SQL server startup time problem (undo/redo problem?)

Martin Kersten Martin.Kersten at cwi.nl
Mon Jun 18 23:50:47 CEST 2007


Dear James,

Can you specify the system you are working on, e.g. hardware/os.

You are really beating the system. I don't know a system that
can handle a TPC-H SF100 out of the box. This size often
require a multistage process and careful setting of the system
parameters.

But, let's see what you have done and learn from your experience.
James Laken wrote:
> Dear MonetDB Developers and Users,
> 
> My TPC-H SF100 test is still producing interesting problems. After 
> several hours of work I have managed to import near 420 million record 
> to the lineitem table (7x 60 million record slice). Accidentally I 
Ok, you perform a sliced base load. i.e. 7 x a SF-10
> killed the import process, and stopped the server process. I have tried 
Did you stop and restart the server between the loads? If not, then
from a recovery point of view all 420M are stored in a single log file
and become the target of a single reload. It behaves as if you loaded
the 7x60M as a single batch.

Killing a database process is of course hard. In that case, the
recovery process has to reload the data and enters a really expensive
part of TPC-H: ensure correctness of the integrity relationships.

Protection against this is hard, because it requires that integrity
rules enforcement should either be disabled (the method persued in MySQL).

> to restart the server process but after three hours of intensive 
> processing the sql module still not started. Please note that the 
> initialization process allocated nearly all memory and swap.
This is what we expect. Your tables require a lot of space, because
MonetDB does not automatically partitioning it. (That's scheduled
for an upcoming release ;-))
> 
> I have attached a gdb to the server process and the execution stacks 
> looks like this:
> 
> Program received signal SIGINT, Interrupt.
> 0x00002b240cb1c2a4 in file_read () from /usr/lib/libstream.so.0
> (gdb) where
> #0  0x00002b240cb1c2a4 in file_read () from /usr/lib/libstream.so.0
> #1  0x00002b240cb1b744 in stream_readLngArray () from 
> /usr/lib/libstream.so.0
> #2  0x00002b240c67a27f in lngRead () from /usr/lib/libbat.so.0
> #3  0x00002b240c788e90 in logger_readlog () from /usr/lib/libbat.so.0
> #4  0x00002b240c789b0c in logger_create () from /usr/lib/libbat.so.0
> #5  0x00002aaaaab42658 in store_init () from 
> /usr/lib/MonetDB5/lib/lib_sql.so
> #6  0x00002aaaaab179b3 in mvc_init () from /usr/lib/MonetDB5/lib/lib_sql.so
> #7  0x00002aaaaaacf9f4 in SQLinit () from /usr/lib/MonetDB5/lib/lib_sql.so
> #8  0x00002b240bf89e3a in initScenario () from /usr/lib/libmal.so.0
> #9  0x00002aaaaaacf968 in SQLsession () from 
> /usr/lib/MonetDB5/lib/lib_sql.so
> #10 0x00002b240bf6536e in runMALsequence () from /usr/lib/libmal.so.0
> #11 0x00002b240bf6697b in runMAL () from /usr/lib/libmal.so.0
> #12 0x00002b240bf5f3f3 in MALengine () from /usr/lib/libmal.so.0
> #13 0x00002b240bf5e3ab in callString () from /usr/lib/libmal.so.0
> #14 0x0000000000402a65 in main ()
> (gdb) c
> Continuing.
> 
> Program received signal SIGINT, Interrupt.
> 0x00002b240d6c31a0 in malloc () from /lib/libc.so.6
> (gdb) where
> #0  0x00002b240d6c31a0 in malloc () from /lib/libc.so.6
> #1  0x00002b240c675c04 in GDKmallocmax () from /usr/lib/libbat.so.0
> #2  0x00002b240c675da9 in GDKmalloc () from /usr/lib/libbat.so.0
> #3  0x00002b240c6787ee in strRead () from /usr/lib/libbat.so.0
> #4  0x00002b240c788e90 in logger_readlog () from /usr/lib/libbat.so.0
> #5  0x00002b240c789b0c in logger_create () from /usr/lib/libbat.so.0
> #6  0x00002aaaaab42658 in store_init () from 
> /usr/lib/MonetDB5/lib/lib_sql.so
> #7  0x00002aaaaab179b3 in mvc_init () from /usr/lib/MonetDB5/lib/lib_sql.so
> #8  0x00002aaaaaacf9f4 in SQLinit () from /usr/lib/MonetDB5/lib/lib_sql.so
> #9  0x00002b240bf89e3a in initScenario () from /usr/lib/libmal.so.0
> #10 0x00002aaaaaacf968 in SQLsession () from 
> /usr/lib/MonetDB5/lib/lib_sql.so
> #11 0x00002b240bf6536e in runMALsequence () from /usr/lib/libmal.so.0
> #12 0x00002b240bf6697b in runMAL () from /usr/lib/libmal.so.0
> #13 0x00002b240bf5f3f3 in MALengine () from /usr/lib/libmal.so.0
> #14 0x00002b240bf5e3ab in callString () from /usr/lib/libmal.so.0
> #15 0x0000000000402a65 in main ()
> 
> Any idea?
> 
> Regards,
> J.
> 
> 
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> _______________________________________________
> MonetDB-users mailing list
> MonetDB-users at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/monetdb-users





More information about the developers-list mailing list