MonetDB error while restarting the database

Ashish Kumar Singh ashishkumar.singh at altair.com
Thu Oct 30 15:54:22 CET 2014


Thanks All!

we made sure now that database connection pool has released all
connections before we shutdown monetdb.

But still infrequently we are seeing this error popping up and we have
double checked that client has returned all the connections.

Before stopping the database a data load process happened from the client
to monetdb, can it be contributed to that??



Regards,
Ashish

-----Original Message-----
From: Sjoerd Mullender <sjoerd at monetdb.org>
Reply-To: "developers-list at monetdb.org" <developers-list at monetdb.org>
Date: Monday, 27 October 2014 11:07 pm
To: "developers-list at monetdb.org" <developers-list at monetdb.org>
Subject: Re: MonetDB error while restarting the database

>
>
>On 27/10/14 17:42, Dimitar Nedev wrote:
>> Hi Ashish,
>> 
>> What happened there is monetdbd send a TERM signal to the mserver
>>process (the actual MonetDB database process), which killed all running
>>threads. The messages "#GDKexit: killing thread" means the the MonetDB
>>kernel (GDK) killed a thread still working at the time the process was
>>terminated. This is most likely an indication that the database was
>>still processing something at that time. Depending on what it was
>>actually doing, data corruption might occur. This can explain why
>>monetdbd reports an internal error why truing to start the mserver
>>process again.
>
>Data corruption should *never* occur.  If you have proof of (and a
>recipe for) data corruption, please file a bug report.
>Of course, if a thread was still working, the transaction it was working
>on will most likely not get committed.
>
>Otherwise the above is correct.
>
>> Check the merovingian.log file again for the events when monetdbd
>>reported the internal error. There should be more information logged on
>>why mserver cannot start up any more.
>> 
>> Best regards,
>> Dimitar
>> 
>>> On 2014-Oct-27, at 15:55 , Ashish Kumar Singh
>>><ashishkumar.singh at altair.com> wrote:
>>>
>>> Thanks Dimitar,
>>>
>>> Yes we are restarting the data base and as part of that we are seeing
>>>this
>>> error.
>>>
>>> I am more worried about the error below:
>>>
>>>
>>> 2014-10-27 09:01:26 ERR merovingian[23583]: unknown state: 42014-10-27
>>> 09:01:26 ERR pbsworksdb[6351]: #GDKexit: killing thread
>>>
>>>
>>> On java client side it says
>>>
>>>
>>> monetdbd: internal error while starting mserver, please refer to the
>>>logs
>>>
>>>
>>>
>>>
>>> Regards,
>>> Ashish
>>>
>>>
>>> -----Original Message-----
>>> From: Dimitar Nedev <D.G.Nedev at cwi.nl>
>>> Reply-To: "users-list at monetdb.org" <users-list at monetdb.org>
>>> Date: Monday, 27 October 2014 8:15 pm
>>> To: "users-list at monetdb.org" <users-list at monetdb.org>
>>> Subject: Re: MonetDB error while restarting the database
>>>
>>>> Hi Ashish,
>>>>
>>>> The exit timeout only tells monetdbd how long should it wait when
>>>>trying
>>>> to gracefully shut down a mserver process. According to the log, the
>>>> mserver process was shut down within the limit, since there are no
>>>> messages like "timeout of <exittimeout> seconds expired, sending
>>>>process
>>>> <PID> the KILL signal".
>>>>
>>>> Now, the interesting entry in the log is the control process message:
>>>> 2014-10-27 09:01:26 MSG control[23583]: (local): stopped database
>>>> 'pbsworksdb'
>>>>
>>>> This one would indicate that someone (or something) gracefully stopped
>>>> the database using the monetdbd client. Probably with the following
>>>> command line call like: 'monetdb stop pbsworksdb'.
>>>> Do not be confused by the order of logged events - monetdbd will first
>>>> stop the process and later log that the call to stop the a database
>>>>has
>>>> been successful.
>>>>
>>>> Best regards,
>>>> Dimitar
>>>>
>>>>
>>>>> On 2014-Oct-27, at 14:49 , Ashish Kumar Singh
>>>>> <ashishkumar.singh at altair.com> wrote:
>>>>>
>>>>> Thanks!
>>>>>
>>>>>
>>>>> Appreciate your response so there are few more people getting the
>>>>>same
>>>>> error :).
>>>>> Any suggestions from dev team on this issue? Can monetdbd exittimeout
>>>>> be also in picture for this?
>>>>>
>>>>>
>>>>> Regards,
>>>>> Ashish
>>>>>
>>>>>
>>>>>
>>>>> From: Pierre-Adrien Coustillas <pcoustillas at 1g6.biz>
>>>>> Date: Monday, 27 October 2014 6:51 pm
>>>>> To: "users-list at monetdb.org" <users-list at monetdb.org>
>>>>> Cc: "developers-list at monetdb.org" <developers-list at monetdb.org>,
>>>>>Ashish
>>>>> Singh <ashishkumar.singh at altair.com>
>>>>> Subject: Re: MonetDB error while restarting the database
>>>>>
>>>>>> Hello
>>>>>>
>>>>>>  (google translate)
>>>>>>
>>>>>> I receive this error when there is no specific spacedisk.
>>>>>> Monetdb sometimes uses a lot of temporary diskspace
>>>>>>(500GBisnormalfor
>>>>>> 1TBof data), which arereleasedat the end oftreatment or after acrash
>>>>>>
>>>>>> Error last week :
>>>>>> 2014-10-22 22:51:46 MSG merovingian[18865]: sending process 30608
>>>>>> (database 'lemonde') the TERM signal
>>>>>> 2014-10-22 22:51:46 ERR merovingian[18865]: unknown state:
>>>>>>42014-10-22
>>>>>> 22:51:46 ERR lemonde[30608]: #GDKexit: killing thread
>>>>>>
>>>>>> Pierre
>>>>>>
>>>>>> --
>>>>>> 1G6
>>>>>> 52 route de bischwiller
>>>>>> 67300 Schiltigheim
>>>>>> Société de Services et de Formations en Logiciels Libres
>>>>>> http://1g6.biz
>>>>>> Tél : 06 64 63 70 35
>>>>>>
>>>>>> De: "Ashish Kumar Singh" <ashishkumar.singh at altair.com>
>>>>>> À: "Ashish Kumar Singh" <ashishkumar.singh at altair.com>,
>>>>>> users-list at monetdb.org
>>>>>> Cc: developers-list at monetdb.org
>>>>>> Envoyé: Lundi 27 Octobre 2014 13:49:19
>>>>>> Objet: Re: MonetDB error while restarting the database
>>>>>>
>>>>>> Guys,
>>>>>>
>>>>>> Any help with this will be really helpful for us?
>>>>>>
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>> Ashish
>>>>>>
>>>>>> From: Ashish Singh <ashishkumar.singh at altair.com>
>>>>>> Date: Monday, 27 October 2014 4:06 pm
>>>>>> To: "users-list at monetdb.org" <users-list at monetdb.org>
>>>>>> Subject: MonetDB error while restarting the database
>>>>>>
>>>>>>>
>>>>>>> All,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> We are facing a new issue with monetdb where monetdb connections
>>>>>>>and
>>>>>>> process is being killed very frequently. Is this a known issue or
>>>>>>>any
>>>>>>> suggestion in getting more details will be helpful.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>> Ashish
>>>>>>>
>>>>>>> Mervogian.log:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: arguments:
>>>>>>> 
>>>>>>>/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/bin/mserver5
>>>>>>> 
>>>>>>>--dbpath=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pbs
>>>>>>>wor
>>>>>>> ksdata/pbsworksdb --set
>>>>>>> merovingian_uri=mapi:monetdb://blrec12-vm6:9200/pbsworksdb --set
>>>>>>> mapi_open=false --set mapi_port=0 --set
>>>>>>> 
>>>>>>>mapi_usock=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/p
>>>>>>>bsw
>>>>>>> orksdata/pbsworksdb/.mapi.sock --set
>>>>>>> 
>>>>>>>monet_vault_key=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/mone
>>>>>>>tdb
>>>>>>> /pbsworksdata/pbsworksdb/.vaultkey --set gdk_nr_threads=4 --set
>>>>>>> max_clients=64 --set sql_optimizer=default_pipe --set
>>>>>>>monet_daemon=yes
>>>>>>> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # MonetDB 5 server
>>>>>>> v11.17.17 "Jan2014-SP2"
>>>>>>> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Serving database
>>>>>>> 'pbsworksdb', using 4 threads
>>>>>>> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Compiled for
>>>>>>> x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked
>>>>>>> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Found 15.675 GiB
>>>>>>> available main-memory.
>>>>>>> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Copyright (c) 1993-July
>>>>>>> 2008 CWI.
>>>>>>> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Copyright (c) August
>>>>>>> 2008-2014 MonetDB B.V., all rights reserved
>>>>>>> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Visit
>>>>>>> http://www.monetdb.org/ for further information
>>>>>>> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Listening for UNIX
>>>>>>>domain
>>>>>>> connection requests on
>>>>>>> 
>>>>>>>mapi:monetdb:///opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monet
>>>>>>>db/
>>>>>>> pbsworksdata/pbsworksdb/.mapi.sock
>>>>>>> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # MonetDB/SQL module
>>>>>>>loaded
>>>>>>> 2014-10-27 09:01:00 MSG control[23583]: (local): started
>>>>>>>'pbsworksdb'
>>>>>>> 2014-10-27 09:01:00 MSG merovingian[23583]: proxying client
>>>>>>> localhost.localdomain:53844 for database 'pbsworksdb' to
>>>>>>> 
>>>>>>>mapi:monetdb:///opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monet
>>>>>>>db/
>>>>>>> pbsworksdata/pbsworksdb/.mapi.sock?database=pbsworksdb
>>>>>>> 2014-10-27 09:01:00 MSG merovingian[23583]: target connection is on
>>>>>>> local UNIX domain socket, passing on filedescriptor instead of
>>>>>>>proxying
>>>>>>> 2014-10-27 09:01:26 MSG control[23583]: (local): served status list
>>>>>>> 2014-10-27 09:01:26 MSG merovingian[23583]: sending process 6351
>>>>>>> (database 'pbsworksdb') the TERM signal
>>>>>>> 2014-10-27 09:01:26 ERR merovingian[23583]: unknown state:
>>>>>>> 42014-10-27 09:01:26 ERR pbsworksdb[6351]: #GDKexit: killing thread
>>>>>>> 2014-10-27 09:01:26 MSG merovingian[23583]: database 'pbsworksdb'
>>>>>>> (6351) has exited with exit status 0
>>>>>>> 2014-10-27 09:01:26 MSG merovingian[23583]: database 'pbsworksdb'
>>>>>>>has
>>>>>>> shut down
>>>>>>> 2014-10-27 09:01:26 MSG control[23583]: (local): stopped database
>>>>>>> 'pbsworksdb'
>>>>>>> 2014-10-27 09:01:26 MSG control[23583]: (local): served status list
>>>>>>> 2014-10-27 09:01:26 MSG merovingian[23583]: starting database
>>>>>>> 'pbsworksdb', up min/avg/max: 5s/14m/1h, crash average: 0.00 0.00
>>>>>>>0.00
>>>>>>> (648-648=0)
>>>>>>> 2014-10-27 09:01:27 MSG pbsworksdb[6575]: arguments:
>>>>>>> 
>>>>>>>/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/bin/mserver5
>>>>>>> 
>>>>>>>--dbpath=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pbs
>>>>>>>wor
>>>>>>> ksdata/pbsworksdb --set
>>>>>>> merovingian_uri=mapi:monetdb://blrec12-vm6:9200/pbsworksdb --set
>>>>>>> mapi_open=false --set mapi_port=0 --set
>>>>>>> 
>>>>>>>mapi_usock=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/p
>>>>>>>bsw
>>>>>>> orksdata/pbsworksdb/.mapi.sock --set
>>>>>>> 
>>>>>>>monet_vault_key=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/mone
>>>>>>>tdb
>>>>>>> /pbsworksdata/pbsworksdb/.vaultkey --set gdk_nr_threads=4 --set
>>>>>>> max_clients=64 --set sql_optimizer=default_pipe --set
>>>>>>>monet_daemon=yes
>>>>>>> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # MonetDB 5 server
>>>>>>> v11.17.17 "Jan2014-SP2"
>>>>>>> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Serving database
>>>>>>> 'pbsworksdb', using 4 threads
>>>>>>> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Compiled for
>>>>>>> x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked
>>>>>>> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Found 15.675 GiB
>>>>>>> available main-memory.
>>>>>>> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Copyright (c) 1993-July
>>>>>>> 2008 CWI.
>>>>>>> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Copyright (c) August
>>>>>>> 2008-2014 MonetDB B.V., all rights reserved
>>>>>>> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Visit
>>>>>>> http://www.monetdb.org/ for further information
>>>>>>> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Listening for UNIX
>>>>>>>domain
>>>>>>> connection requests on
>>>>>>> 
>>>>>>>mapi:monetdb:///opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monet
>>>>>>>db/
>>>>>>> pbsworksdata/pbsworksdb/.mapi.sock
>>>>>>> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # MonetDB/SQL module
>>>>>>>loaded
>>>>>>> 2014-10-27 09:01:32 MSG merovingian[23583]: proxying client
>>>>>>> localhost.localdomain:53853 for database 'pbsworksdb' to
>>>>>>> 
>>>>>>>mapi:monetdb:///opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monet
>>>>>>>db/
>>>>>>> pbsworksdata/pbsworksdb/.mapi.sock?database=pbsworksdb
>>>>>>> 2014-10-27 09:01:32 MSG merovingian[23583]: target connection is on
>>>>>>> local UNIX domain socket, passing on filedescriptor instead of
>>>>>>>proxying
>>>>>>> 2014-10-27 09:01:32 MSG c
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> users-list mailing list
>>>>>> users-list at monetdb.org
>>>>>> https://www.monetdb.org/mailman/listinfo/users-list
>>>>>>
>>>>> _______________________________________________
>>>>> developers-list mailing list
>>>>> developers-list at monetdb.org
>>>>> https://www.monetdb.org/mailman/listinfo/developers-list
>>>>
>>>> _______________________________________________
>>>> users-list mailing list
>>>> users-list at monetdb.org
>>>> https://www.monetdb.org/mailman/listinfo/users-list
>>>
>>> _______________________________________________
>>> developers-list mailing list
>>> developers-list at monetdb.org
>>> https://www.monetdb.org/mailman/listinfo/developers-list
>> 
>> _______________________________________________
>> developers-list mailing list
>> developers-list at monetdb.org
>> https://www.monetdb.org/mailman/listinfo/developers-list
>> 
>
>-- 
>Sjoerd Mullender
>




More information about the developers-list mailing list