Hi Michael,

On 19-02-2008 20:22:12 +0100, Michael Schmidt wrote:

Hi,

I'm trying to run a benchmark using the latest February stable (on an Ubuntu 7.04 - Feisty Fawn), and want to cleanup the database entirely between two single runs. Basically, my benchmark script is as follows:

//////////////////// START CODE // kill all processes and remove databases) // (I know this is not the intended way, but using // > monetdb stop test // > monetdb destroy -f test // while not restarting merovingian basically results in // in the same error)

...

killall -9 merovingian killall -9 mserver5 killall -9 mclient rm -Rf /usr/local/src/MonetDB/var/MonetDB5/dbfarm/*

I fixed a couple of bugs in merovingian/monetdb tonight, and added a feature in the current branch for monetdb stop to have a -w flag that waits for the action to complete.

I don't understand yet why you need the kill -KILL on merovingian (it whould kill all mservers in a much nicer way for you than with -KILL when you just normally kill merovingian), but I from what I understand merovingian is just not stopping the databases?

If so, could you please send me the merovingian.log file?

// start the merovingian server

...

merovingian sleep 2

// wait until ports 50000 and 50001 are free, // as they might still be blocked (I inserted this piece // of code because I thought busy port allocation might // be the cause of failure)

...

[code works fine (ommited)]

// create and start the database

...

monetdb create test; sleep 2; monetdb start test; sleep 2;

with the new code in the HEAD you could do monetdb start -w test instead avoid the sleep, and also cover cases where 2 secs is not enough ;)

I just realise that I forgot to make monetdb have an appropriate exit code when it detects the started server crashes, but that shouldn't spoil the fun for now.

On 20-02-2008 02:26:30 +0100, Michael Schmidt wrote:

Hi Fabian,

thanks for your response. I tried to install the lates branch, but get a compile error. The latest nightly build works fine, but does not yet contain the changes, so I will probably wait until tomorrow evening.

Unfortunately I had to do a lot of cross work, so all fixes are in stable only, whereas the feature you want are in the current branch, which doesn't include the fixes (yet).

In meantime, I changed the script as follows:

=============== begin code // create database (I will replace the ugly sleeps as soon as your fixes are available to me)

...

monetdb create SchemaDBLP1000 sleep 2

The sleep here is already not necessary, as the tool already only returns once the necessary directories are created.

...

monetdb start SchemaDBLP1000 sleep 2

// load document

...

mclient -lsql -dSchemaDBLP1000 < document.sql

// stop the database again

...

monetdb stop SchemaDBLP1000 sleep 2 monetdb destroy -f SchemaDBLP1000 sleep 2

Same for destroy, if it returns before the database is really destroyed, then that is a bug.

=============== end code

The merovingian process is now started initially, from outside the script. Starting and stopping seems to work fine. First of all, do you agree that this is the intended way (except for the sleeps, which may fail)?

Yes, your script looks fine. You could add some monetdb create bla || die "Failed to create bla" kind of structures to catch problems, but if everything succeeds then you script should work.

However, also in the scenario above exactly every second run fails. The log for four consecutive runs is provided at the end of the mail.

Ok.

...

If so, could you please send me the merovingian.log file?

Here it is, for four consecutive runs of the script above. In run 2 and 4 (which fail), there is a "MSG merovingian[5790]: database 'SchemaDBLP1000' (5832) was killed by signal 11", which seems to reflect the problem. Unfortunately, I do not know what this message means?

signal 11 is SIGSEGV, so the database just segfaulted (crashed).

Your log shows an interesting pattern. It just looks like the database on the even startups sees some left over garbage from the previous run (e.g. an logfile). However, that should not be possible, as destroy is supposed to throw that away as well.

I'm affraid I can't answer this right away, we will have to reproduce this to see what's going on exactly.

=============== begin log ////////////////// INITIAL STARTUP TME 2008-02-20 03:12:30 MSG merovingian[5790]: Merovingian 0.6 starting ... MSG merovingian[5790]: monitoring dbfarm /usr/local/src/MonetDB/var/MonetDB5/dbfarm MSG merovingian[5790]: listening for connections on gemma:50000 ////////////////// RUN 1 MSG merovingian[5790]: starting database SchemaDBLP1000 due to control signal MSG SchemaDBLP1000[5801]: executing '/usr/local/src/MonetDB/bin/mserver5' for database 'SchemaDBLP1000' MSG merovingian[5790]: starting database 'SchemaDBLP1000', up min/avg/max: 0/0/0, crash average: 0.00 0.00 0.00 (0-0=0) MSG SchemaDBLP1000[5801]: !WARNING: BBPdir: initializing BBP. MSG SchemaDBLP1000[5801]: # MonetDB server v5.4.1, based on kernel v1.22.1 MSG SchemaDBLP1000[5801]: # Serving database 'SchemaDBLP1000' MSG SchemaDBLP1000[5801]: # Compiled for i686-pc-linux-gnu/32bit with 32bit OIDs dynamically linked MSG SchemaDBLP1000[5801]: # Copyright (c) 1993-2008 CWI, all rights reserved MSG SchemaDBLP1000[5801]: # Visit http://monetdb.cwi.nl/ for further information MSG SchemaDBLP1000[5801]: # Listening for connection requests on mapi:monetdb://gemma:50001/ MSG merovingian[5790]: database 'SchemaDBLP1000' already running since 2008-02-20 03:12:41, up min/avg/max: 0/0/0, crash average: 0.00 0.00 0.00 (1-0=0) MSG merovingian[5790]: redirecting client 127.0.0.1:24780 for database 'SchemaDBLP1000' to mapi:monetdb://gemma:50001/ MSG merovingian[5790]: stopping database SchemaDBLP1000 due to control signal MSG SchemaDBLP1000[5801]: # MonetDB/SQL module v2.22.1 loaded MSG merovingian[5790]: database 'SchemaDBLP1000' (5801) has exited with exit status 0 ////////////////// RUN 2 MSG merovingian[5790]: starting database SchemaDBLP1000 due to control signal MSG SchemaDBLP1000[5832]: executing '/usr/local/src/MonetDB/bin/mserver5' for database 'SchemaDBLP1000' MSG merovingian[5790]: starting database 'SchemaDBLP1000', up min/avg/max: 0/0/0, crash average: 0.00 0.00 0.00 (0-0=0) MSG SchemaDBLP1000[5832]: !WARNING: BBPdir: initializing BBP. MSG SchemaDBLP1000[5832]: # MonetDB server v5.4.1, based on kernel v1.22.1 MSG SchemaDBLP1000[5832]: # Serving database 'SchemaDBLP1000' MSG SchemaDBLP1000[5832]: # Compiled for i686-pc-linux-gnu/32bit with 32bit OIDs dynamically linked MSG SchemaDBLP1000[5832]: # Copyright (c) 1993-2008 CWI, all rights reserved MSG SchemaDBLP1000[5832]: # Visit http://monetdb.cwi.nl/ for further information MSG SchemaDBLP1000[5832]: # Listening for connection requests on mapi:monetdb://gemma:50001/ MSG merovingian[5790]: database 'SchemaDBLP1000' already running since 2008-02-20 03:12:51, up min/avg/max: 0/0/0, crash average: 0.00 0.00 0.00 (1-0=0) MSG merovingian[5790]: redirecting client 127.0.0.1:24782 for database 'SchemaDBLP1000' to mapi:monetdb://gemma:50001/ MSG merovingian[5790]: database 'SchemaDBLP1000' (5832) was killed by signal 11 ////////////////// RUN 3 MSG merovingian[5790]: starting database SchemaDBLP1000 due to control signal MSG SchemaDBLP1000[5868]: executing '/usr/local/src/MonetDB/bin/mserver5' for database 'SchemaDBLP1000' MSG merovingian[5790]: starting database 'SchemaDBLP1000', up min/avg/max: 0/0/0, crash average: 0.00 0.00 0.00 (0-0=0) MSG SchemaDBLP1000[5868]: !WARNING: BBPdir: initializing BBP. MSG SchemaDBLP1000[5868]: # MonetDB server v5.4.1, based on kernel v1.22.1 MSG SchemaDBLP1000[5868]: # Serving database 'SchemaDBLP1000' MSG SchemaDBLP1000[5868]: # Compiled for i686-pc-linux-gnu/32bit with 32bit OIDs dynamically linked MSG SchemaDBLP1000[5868]: # Copyright (c) 1993-2008 CWI, all rights reserved MSG SchemaDBLP1000[5868]: # Visit http://monetdb.cwi.nl/ for further information MSG SchemaDBLP1000[5868]: # Listening for connection requests on mapi:monetdb://gemma:50001/ MSG merovingian[5790]: database 'SchemaDBLP1000' already running since 2008-02-20 03:12:59, up min/avg/max: 0/0/0, crash average: 0.00 0.00 0.00 (1-0=0) MSG merovingian[5790]: redirecting client 127.0.0.1:24784 for database 'SchemaDBLP1000' to mapi:monetdb://gemma:50001/ MSG merovingian[5790]: stopping database SchemaDBLP1000 due to control signal MSG SchemaDBLP1000[5868]: # MonetDB/SQL module v2.22.1 loaded MSG merovingian[5790]: database 'SchemaDBLP1000' (5868) has exited with exit status 0 ////////////////// RUN 4 MSG merovingian[5790]: starting database SchemaDBLP1000 due to control signal MSG SchemaDBLP1000[5911]: executing '/usr/local/src/MonetDB/bin/mserver5' for database 'SchemaDBLP1000' MSG merovingian[5790]: starting database 'SchemaDBLP1000', up min/avg/max: 0/0/0, crash average: 0.00 0.00 0.00 (0-0=0) MSG SchemaDBLP1000[5911]: !WARNING: BBPdir: initializing BBP. MSG SchemaDBLP1000[5911]: # MonetDB server v5.4.1, based on kernel v1.22.1 MSG SchemaDBLP1000[5911]: # Serving database 'SchemaDBLP1000' MSG SchemaDBLP1000[5911]: # Compiled for i686-pc-linux-gnu/32bit with 32bit OIDs dynamically linked MSG SchemaDBLP1000[5911]: # Copyright (c) 1993-2008 CWI, all rights reserved MSG SchemaDBLP1000[5911]: # Visit http://monetdb.cwi.nl/ for further information MSG SchemaDBLP1000[5911]: # Listening for connection requests on mapi:monetdb://gemma:50001/ MSG merovingian[5790]: database 'SchemaDBLP1000' already running since 2008-02-20 03:13:09, up min/avg/max: 0/0/0, crash average: 0.00 0.00 0.00 (1-0=0) MSG merovingian[5790]: redirecting client 127.0.0.1:24786 for database 'SchemaDBLP1000' to mapi:monetdb://gemma:50001/ MSG merovingian[5790]: database 'SchemaDBLP1000' (5911) was killed by signal 11

On 20-02-2008 12:54:25 +0100, Michael Schmidt wrote:

Hi Fabian,

...

...

thanks for your response. I tried to install the lates branch, but get a compile error. The latest nightly build works fine, but does not yet contain the changes, so I will probably wait until tomorrow evening.

Unfortunately I had to do a lot of cross work, so all fixes are in stable only, whereas the feature you want are in the current branch, which doesn't include the fixes (yet).

OK, am I right that all changes will be included in the coming nightly build. If so, I will wait until tomorrow and try again with '-w' flag and your new fixes.

As far as I can see, the fixes from stable have not yet been propagated to the current (HEAD) branch yet. In that respect I am sorry to tell you that you still have to wait for it.

...

Yes, your script looks fine. You could add some monetdb create bla || die "Failed to create bla" kind of structures to catch problems, but if everything succeeds then you script should work.

Yes, but the problem is not database creation. Rather than that, mserver5 crashes /not before/ loading data into the database. Before connecting to mserver5 (i.e. by loading the sql file), everything looks fine, i.e. mserver5 appears in the process list (and I guess monetdb status indicates that the server is fine).

...

Your log shows an interesting pattern. It just looks like the database on the even startups sees some left over garbage from the previous run (e.g. an logfile). However, that should not be possible, as destroy is supposed to throw that away as well.

I totally agree. This is why I initially thought it might be a port allocation problem. I tried waiting for ports to be freed again, and also tried timeouts of about 10mins between two consecutive runs, but still the same behavior. There must be some kind of garbage that causes mserver5 to crash as soon as a connection/any other action is triggered.

...

I'm affraid I can't answer this right away, we will have to reproduce this to see what's going on exactly.

It seems that this error is independent from the data files being loaded into the DB, but if you need some of the data files I am testing with let me know!

Niels thinks that the crashes are due to the sql_logs not being thrown away. Could you check if the var/MonetDB5/sql_logs/DBNAME is removed by monetdb destroy -f as well as var/MonetDB5/dbfarm/DBNAME?

On 21-02-2008 14:55:10 +0100, Michael Schmidt wrote:

Hi Fabian,

...

As far as I can see, the fixes from stable have not yet been propagated to the current (HEAD) branch yet. In that respect I am sorry to tell you that you still have to wait for it.

you're right, I've installed the latest nightly build and the -w option is not yet supported. But don't worry, I'll try again in the coming days.

The fixes have just been merged into the curent branch, so either use CVS, or check tomorrow's nightlies for the "current" branch.

...

Niels thinks that the crashes are due to the sql_logs not being thrown away. Could you check if the var/MonetDB5/sql_logs/DBNAME is removed by monetdb destroy -f as well as var/MonetDB5/dbfarm/DBNAME?

Great, this solves my problem. I simply extended my script by

...

rm -Rf .../sql_logs/DBNAME

just after destroying the database, and everything works fine now. You were right that this log file is not removed by the destroy command (FYI: the dbfarm subfolder is removed correctly).

Please let me know when you have fixed this bug, then I will remove the rm -Rf again, and can of course test for you whether everything still is fine.

Could you file a bug for this issue with monetdb?

Dear Fabian,

I think I found a possible cause for this issue. Tomorrow's nightlies should include some code that may reveal a problem encountered by the monetdb utility here.

ok, thank you again.

...

Could you file a bug for this issue with monetdb?

I submitted the bug, you will find a short description of the problem in the MonetDB bug tracker system. Please note that I did not know how to categorize the bug, so I left these fields open.

Kind regards, Michael