RE: Details about MonetDB, data loss in restart server (Ying Zhang)
Hi,
- OS is Debian 7 64-bit, in a VM in VMware virtualization System. - Hard disk 100G with 26% usage. - 5 cores. - 6G RAM. - Version MonetDB: MonetDB 5 server v11.15.17 "Feb2013-SP5" - Table: create table (id int, dt date, product varchar (50) .... etc, and another 30 columns between int and varchar. - Query: select dt, count (id) from table where produtct in (....) group by dt - History: Long time MonetDB use with Pentaho (Saiku) for analysis of data cubes, never had problems, works great with millions of records. When the problem occurred, the steps taken were: 1- A couple of days before it occurred, I made a 32 million load records to the table. 2- I used this table for many analyzes via SQL client (without using cubes) and everything was fine. 3- I created a cube in Pentaho for this table and also used a lot and everything went well. 4- When I put the dt fields in the cube and the interface seems to have caught the Pentaho and mserver held all machine resources (memory and CPU), less disk. This lock may have been because the Pentaho, but specifically the mondrian, have had a problem with the date field 5- I tried to stop the Pentaho, but did not work, had to force a kill -9 pid. 6- In the case of MonetDB, the command MonetDB stop dw, worked properly and stopped the service. 7- When I run MonetDB start dw, it took a while longer than usual, but the service start usually without showing error on console. 8- With other tables there were no data loss problem, only that table lost data just the product column, which was used in the query, the remaining columns have not suffered problems. But but at no time update or delete command has been executed, only selects were executed 9- All though I forced Pentaho standing in the execution of the query, but MonetDB stopped normal, and a query should never change data, of course?
Log 2015-03-31 15:58:38 MSG control[1114]: (local): served status list 2015-03-31 15:58:38 MSG merovingian[1114]: sending process 14173 (database 'dw-zoombox') the TERM signal 2015-03-31 15:58:39 MSG merovingian[1114]: database 'dw-zoombox' (14173) has exited with exit status 0 2015-03-31 15:58:39 MSG merovingian[1114]: database 'dw-zoombox' has shut down 2015-03-31 15:58:39 MSG control[1114]: (local): stopped database 'dw-zoombox' 2015-03-31 15:58:44 MSG control[1114]: (local): served status list 2015-03-31 15:58:44 MSG merovingian[1114]: starting database 'dw-zoombox', up min/avg/max: 2d/9w/18w, crash average: 0.00 0.10 0.03 (3-2=1) 2015-03-31 15:58:45 MSG dw-zoombox[19366]: arguments: /usr/bin/mserver5 --dbpath=/dados/dbfarm/dw-zoombox --set merovingian_uri=mapi:monetdb://SRVBIL01:50000/dw-zoombox --set mapi_open=false --set mapi_port=0 --set mapi_usock=/dados/dbfarm/dw-zoombox/.mapi.sock --set monet_vault_key=/dados/dbfarm/dw-zoombox/.vaultkey --set gdk_nr_threads=5 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2015-03-31 15:59:34 MSG dw-zoombox[19366]: # MonetDB 5 server v11.15.17 "Feb2013-SP5" 2015-03-31 15:59:34 MSG dw-zoombox[19366]: # Serving database 'dw-zoombox', using 5 threads 2015-03-31 15:59:34 MSG dw-zoombox[19366]: # Compiled for x86_64-pc-linux-gnu/64bit with 64bit OIDs dynamically linked 2015-03-31 15:59:34 MSG dw-zoombox[19366]: # Found 11.765 GiB available main-memory. 2015-03-31 15:59:34 MSG dw-zoombox[19366]: # Copyright (c) 1993-July 2008 CWI. 2015-03-31 15:59:34 MSG dw-zoombox[19366]: # Copyright (c) August 2008-2013 MonetDB B.V., all rights reserved 2015-03-31 15:59:34 MSG dw-zoombox[19366]: # Visit http://www.monetdb.org/ for further information 2015-03-31 15:59:34 MSG dw-zoombox[19366]: # Listening for UNIX domain connection requests on mapi:monetdb:///dados/dbfarm/dw-zoombox/.mapi.sock 2015-03-31 15:59:34 MSG dw-zoombox[19366]: # MonetDB/SQL module loaded 2015-03-31 15:59:34 MSG control[1114]: (local): started 'dw-zoombox' 2015-03-31 15:59:34 MSG merovingian[1114]: proxying client 192.168.1.7:49105 for database 'dw-zoombox' to mapi:monetdb:///dados/dbfarm/dw-zoombox/.mapi.sock?database=dw-zoombox 2015-03-31 15:59:34 MSG merovingian[1114]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2015-03-31 16:05:00 MSG merovingian[1114]: proxying client 192.168.1.7:49117 for database 'dw-zoombox' to mapi:monetdb:///dados/dbfarm/dw-zoombox/.mapi.sock?database=dw-zoombox 2015-03-31 16:05:00 MSG merovingian[1114]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2015-03-31 16:05:20 MSG merovingian[1114]: proxying client 192.168.1.7:49119 for database 'dw-zoombox' to mapi:monetdb:///dados/dbfarm/dw-zoombox/.mapi.sock?database=dw-zoombox 2015-03-31 16:05:20 MSG merovingian[1114]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2015-03-31 16:06:05 ERR discovery[1114]: cannot retrieve name info: Temporary failure in name resolution 2015-03-31 16:16:05 ERR discovery[1114]: cannot retrieve name info: Temporary failure in name resolution 2015-03-31 16:16:16 MSG merovingian[1114]: proxying client 192.168.1.8:50220 for database 'dw-zoombox' to mapi:monetdb:///dados/dbfarm/dw-zoombox/.mapi.sock?database=dw-zoombox 2015-03-31 16:16:16 MSG merovingian[1114]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2015-03-31 16:17:04 MSG merovingian[1114]: proxying client 192.168.1.7:49159 for database 'dw-zoombox' to mapi:monetdb:///dados/dbfarm/dw-zoombox/.mapi.sock?database=dw-zoombox 2015-03-31 16:17:04 MSG merovingian[1114]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying
Message: 1 Date: Thu, 2 Apr 2015 23:16:39 +0200 From: Ying Zhang Y.Zhang@cwi.nl To: "Communication channel for developers of the MonetDB suite." developers-list@monetdb.org Cc: users-list@monetdb.org Subject: Re: MonetDB, data loss in restart server Message-ID: 6C2903C0-4399-462D-9D3F-822351DE1979@cwi.nl Content-Type: text/plain; charset=windows-1252
Hello Luciano,
Thanks for using monetdb, and sorry for the inconvenience. Would you please give us some background information about your setting:
- which OS?
- hardware spec. of your system
- which monetdb version?
- what does your query do?
- when and how did the database stop? If the monetdb server stopped, did it give any error message?
- what's at the end of the merovingian log?
With kind regards,
Jennie
On Apr 01, 2015, at 06:46, Luciano Sasso luciano@gsgroup.com.br wrote:
Hello,
I have used the MonetDB in production for some time, and I have nothing to complain about, are tables with more than 30 columns and 30 million records, do analysis through cubes in Pentaho / Saiku. Have restarted the server several times, and never had problems with loss of data.But today, after performing a query that was consuming a lot of machine, stopped the database, and when I started, data from some columns were blank. How can it, will the data were still uncommitted? when performing a query the MonetDB changes the data or reorganized? which may have occurred, I am concerned.
-- <Slice_01.png> <Slice_02.jpg> Luciano Sasso Vieira Gestor - Intelig?ncia de Neg?cios
luciano@gsgroup.com.br | tel: 17 3353-0833 | cel: 17 9706-9335 <Slice_04.png> www.gsgroup.com.br <Slice_06.png>
Este email foi escaneado pelo Avast antiv?rus. www.avast.com
developers-list mailing list developers-list@monetdb.org https://www.monetdb.org/mailman/listinfo/developers-list
Subject: Digest Footer
users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
End of users-list Digest, Vol 32, Issue 6
Hai Luciano,
Sorry that I didn't find time to reply you earlier. Thanks a lot for your background information.
The version of MonetDB you are using is very old. It's not supported any more. I'd like to recommend you to upgrade to the newest version of MonetDB Oct2014, and check if you still have the same problem.
With kind regards,
Jennie
On Apr 03, 2015, at 18:25 , Luciano Sasso luciano@gsgroup.com.br wrote:
Hi,
- OS is Debian 7 64-bit, in a VM in VMware virtualization System.
- Hard disk 100G with 26% usage.
- 5 cores.
- 6G RAM.
- Version MonetDB: MonetDB 5 server v11.15.17 "Feb2013-SP5"
- Table: create table (id int, dt date, product varchar (50) .... etc, and another 30 columns between int and varchar.
- Query: select dt, count (id) from table where produtct in (....) group by dt
- History: Long time MonetDB use with Pentaho (Saiku) for analysis of data cubes, never had problems, works great with millions of records. When the problem occurred, the steps taken were: 1- A couple of days before it occurred, I made a 32 million load records to the table. 2- I used this table for many analyzes via SQL client (without using cubes) and everything was fine. 3- I created a cube in Pentaho for this table and also used a lot and everything went well. 4- When I put the dt fields in the cube and the interface seems to have caught the Pentaho and mserver held all machine resources (memory and CPU), less disk. This lock may have been because the Pentaho, but specifically the mondrian, have had a problem with the date field 5- I tried to stop the Pentaho, but did not work, had to force a kill -9 pid. 6- In the case of MonetDB, the command MonetDB stop dw, worked properly and stopped the service. 7- When I run MonetDB start dw, it took a while longer than usual, but the service start usually without showing error on console. 8- With other tables there were no data loss problem, only that table lost data just the product column, which was used in the query, the remaining columns have not suffered problems. But but at no time update or delete command has been executed, only selects were executed 9- All though I forced Pentaho standing in the execution of the query, but MonetDB stopped normal, and a query should never change data, of course?
Log 2015-03-31 15:58:38 MSG control[1114]: (local): served status list 2015-03-31 15:58:38 MSG merovingian[1114]: sending process 14173 (database 'dw-zoombox') the TERM signal 2015-03-31 15:58:39 MSG merovingian[1114]: database 'dw-zoombox' (14173) has exited with exit status 0 2015-03-31 15:58:39 MSG merovingian[1114]: database 'dw-zoombox' has shut down 2015-03-31 15:58:39 MSG control[1114]: (local): stopped database 'dw-zoombox' 2015-03-31 15:58:44 MSG control[1114]: (local): served status list 2015-03-31 15:58:44 MSG merovingian[1114]: starting database 'dw-zoombox', up min/avg/max: 2d/9w/18w, crash average: 0.00 0.10 0.03 (3-2=1) 2015-03-31 15:58:45 MSG dw-zoombox[19366]: arguments: /usr/bin/mserver5 --dbpath=/dados/dbfarm/dw-zoombox --set merovingian_uri=mapi:monetdb://SRVBIL01:50000/dw-zoombox --set mapi_open=false --set mapi_port=0 --set mapi_usock=/dados/dbfarm/dw-zoombox/.mapi.sock --set monet_vault_key=/dados/dbfarm/dw-zoombox/.vaultkey --set gdk_nr_threads=5 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2015-03-31 15:59:34 MSG dw-zoombox[19366]: # MonetDB 5 server v11.15.17 "Feb2013-SP5" 2015-03-31 15:59:34 MSG dw-zoombox[19366]: # Serving database 'dw-zoombox', using 5 threads 2015-03-31 15:59:34 MSG dw-zoombox[19366]: # Compiled for x86_64-pc-linux-gnu/64bit with 64bit OIDs dynamically linked 2015-03-31 15:59:34 MSG dw-zoombox[19366]: # Found 11.765 GiB available main-memory. 2015-03-31 15:59:34 MSG dw-zoombox[19366]: # Copyright (c) 1993-July 2008 CWI. 2015-03-31 15:59:34 MSG dw-zoombox[19366]: # Copyright (c) August 2008-2013 MonetDB B.V., all rights reserved 2015-03-31 15:59:34 MSG dw-zoombox[19366]: # Visit http://www.monetdb.org/ for further information 2015-03-31 15:59:34 MSG dw-zoombox[19366]: # Listening for UNIX domain connection requests on mapi:monetdb:///dados/dbfarm/dw-zoombox/.mapi.sock 2015-03-31 15:59:34 MSG dw-zoombox[19366]: # MonetDB/SQL module loaded 2015-03-31 15:59:34 MSG control[1114]: (local): started 'dw-zoombox' 2015-03-31 15:59:34 MSG merovingian[1114]: proxying client 192.168.1.7:49105 for database 'dw-zoombox' to mapi:monetdb:///dados/dbfarm/dw-zoombox/.mapi.sock?database=dw-zoombox 2015-03-31 15:59:34 MSG merovingian[1114]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2015-03-31 16:05:00 MSG merovingian[1114]: proxying client 192.168.1.7:49117 for database 'dw-zoombox' to mapi:monetdb:///dados/dbfarm/dw-zoombox/.mapi.sock?database=dw-zoombox 2015-03-31 16:05:00 MSG merovingian[1114]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2015-03-31 16:05:20 MSG merovingian[1114]: proxying client 192.168.1.7:49119 for database 'dw-zoombox' to mapi:monetdb:///dados/dbfarm/dw-zoombox/.mapi.sock?database=dw-zoombox 2015-03-31 16:05:20 MSG merovingian[1114]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2015-03-31 16:06:05 ERR discovery[1114]: cannot retrieve name info: Temporary failure in name resolution 2015-03-31 16:16:05 ERR discovery[1114]: cannot retrieve name info: Temporary failure in name resolution 2015-03-31 16:16:16 MSG merovingian[1114]: proxying client 192.168.1.8:50220 for database 'dw-zoombox' to mapi:monetdb:///dados/dbfarm/dw-zoombox/.mapi.sock?database=dw-zoombox 2015-03-31 16:16:16 MSG merovingian[1114]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2015-03-31 16:17:04 MSG merovingian[1114]: proxying client 192.168.1.7:49159 for database 'dw-zoombox' to mapi:monetdb:///dados/dbfarm/dw-zoombox/.mapi.sock?database=dw-zoombox 2015-03-31 16:17:04 MSG merovingian[1114]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying
Message: 1 Date: Thu, 2 Apr 2015 23:16:39 +0200 From: Ying Zhang Y.Zhang@cwi.nl
To: "Communication channel for developers of the MonetDB suite."
Cc: users-list@monetdb.org
Subject: Re: MonetDB, data loss in restart server Message-ID: 6C2903C0-4399-462D-9D3F-822351DE1979@cwi.nl
Content-Type: text/plain; charset=windows-1252
Hello Luciano,
Thanks for using monetdb, and sorry for the inconvenience. Would you please give us some background information about your setting:
- which OS?
- hardware spec. of your system
- which monetdb version?
- what does your query do?
- when and how did the database stop? If the monetdb server stopped, did it give any error message?
- what's at the end of the merovingian log?
With kind regards,
Jennie
On Apr 01, 2015, at 06:46, Luciano Sasso luciano@gsgroup.com.br wrote:
Hello,
I have used the MonetDB in production for some time, and I have nothing to complain about, are tables with more than 30 columns and 30 million records, do analysis through cubes in Pentaho / Saiku. Have restarted the server several times, and never had problems with loss of data.But today, after performing a query that was consuming a lot of machine, stopped the database, and when I started, data from some columns were blank. How can it, will the data were still uncommitted? when performing a query the MonetDB changes the data or reorganized? which may have occurred, I am concerned.
-- <Slice_01.png> <Slice_02.jpg> Luciano Sasso Vieira Gestor - Intelig?ncia de Neg?cios
luciano@gsgroup.com.br | tel: 17 3353-0833 | cel: 17 9706-9335 <Slice_04.png> www.gsgroup.com.br
<Slice_06.png>
Este email foi escaneado pelo Avast antiv?rus.
www.avast.com
developers-list mailing list
developers-list@monetdb.org https://www.monetdb.org/mailman/listinfo/developers-list
Subject: Digest Footer
users-list mailing list
users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
End of users-list Digest, Vol 32, Issue 6
-- <Slice_01.png> <Slice_02.jpg> Luciano Sasso Vieira Gestor - Inteligência de Negócios
luciano@gsgroup.com.br | tel: 17 3353-0833 | cel: 17 9706-9335 <Slice_04.png> www.gsgroup.com.br <Slice_06.png>
Este email foi escaneado pelo Avast antivírus. www.avast.com
users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
participants (2)
-
Luciano Sasso -
Ying Zhang