[MonetDB-users] failed to fork mserver: database 'dw' has crashed after starting, manual intervention needed, check monetdbd's logfile for details
Hi folks,
How can I recover from a crashed database?
I got this error when I restart a running database 'dw', Do you have any ideas on this?
2012-02-15 13:59:28 MSG control[27007]: (local): served status list
2012-02-15 13:59:28 MSG merovingian[27007]: startup of database under maintenance 'dw' forced
2012-02-15 13:59:28 MSG merovingian[27007]: database 'dw' has crashed after start on 2012-02-15 13:58:47, attempting restart, up min/avg/max: 1d/1d/1d, crash average: 1.00 1.00 0.97 (45-1=44)
2012-02-15 13:59:28 MSG dw[27831]: arguments: /usr/lib/monetdb/bin/mserver5 --set gdk_dbfarm=/data/warehouse --dbname=dw --set merovingian_uri=mapi:monetdb://monet00:50000/dw --set mapi_open=false --set mapi_port=0 --set mapi_usock=/data/warehouse/dw/.mapi.sock --set monet_vault_key=/data/warehouse/dw/.vaultkey --set gdk_nr_threads=24 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes
2012-02-15 13:59:28 MSG dw[27831]: # MonetDB 5 server v11.7.5 "Dec2011"
2012-02-15 13:59:28 MSG dw[27831]: # Serving database 'dw', using 24 threads
2012-02-15 13:59:28 MSG dw[27831]: # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked
2012-02-15 13:59:28 MSG dw[27831]: # Found 23.533 GiB available main-memory.
2012-02-15 13:59:28 MSG dw[27831]: # Copyright (c) 1993-July 2008 CWI.
2012-02-15 13:59:28 MSG dw[27831]: # Copyright (c) August 2008-2012 MonetDB B.V., all rights reserved
2012-02-15 13:59:28 MSG dw[27831]: # Visit http://www.monetdb.org/ for further information
2012-02-15 13:59:28 MSG dw[27831]: # Listening for UNIX domain connection requests on mapi:monetdb:///data/warehouse/dw/.mapi.sock
2012-02-15 13:59:28 MSG dw[27831]: # MonetDB/SQL module loaded
2012-02-15 13:59:28 MSG merovingian[27007]: database 'dw' (27831) was killed by signal SIGSEGV
2012-02-15 13:59:28 ERR control[27007]: (local): failed to fork mserver: database 'dw' has crashed after starting, manual intervention needed, check monetdbd's logfile for details
The following is my platform information:
MonetDB 5 server v11.7.5 "Dec2011" (64-bit, 64-bit oids)
Copyright (c) 1993-July 2008 CWI
Copyright (c) August 2008-2012 MonetDB B.V., all rights reserved
Visit http://www.monetdb.org/ for further information
Found 23.5GiB available memory, 24 available cpu cores
Libraries:
libpcre: 6.6 06-Feb-2006 (compiled with 6.6)
openssl: OpenSSL 0.9.8e-rhel5 01 Jul 2008 (compiled with OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008)
libxml2: 2.6.26 (compiled with 2.6.26)
Compiled by: root@monet00 (x86_64-unknown-linux-gnu)
Compilation: gcc -g -O2
Linking : /usr/bin/ld -m elf_x86_64
Thanks in advance,
Tony
Hi Tony,
On 15-02-2012 14:12:15 +0800, Tony wrote:
How can I recover from a crashed database?
I got this error when I restart a running database ‘dw’, Do you have any ideas on this?
I selected the two most important lines from your log:
2012-02-15 13:59:28 MSG merovingian[27007]: database 'dw' (27831) was killed by signal SIGSEGV 2012-02-15 13:59:28 ERR control[27007]: (local): failed to fork mserver: database 'dw' has crashed after starting, manual intervention needed, check monetdbd's logfile for details
This is quite unconvenient. Your database won't start any more, most probably because the data is in a state which makes the server crash.
In order to help us to deal with this issue, the following questions: How large is your dbfarm (could we e.g. download it). Could you compile monetdb with --enable-debug and run mserver5 separately in gdb?
We need a stack trace at least. If you could tell us what's in the database (how many tables, how many columns, how many rows, constraints, keys) and what you did as last thing before it went wrong, that might give us clues as well. Last but not least, if you can reproduce this in a way, we'd like to know how.
participants (2)
-
Fabian Groffen -
Tony