users-list October 2014

users-list@monetdb.org

22 participants
22 discussions

Announcement: New Oct2014 Feature release of MonetDB suite

by Sjoerd Mullender

The MonetDB team at CWI/MonetDB BV is pleased to announce the Oct2014 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Important consideration when upgrading to this release. In this release we have fixed bug 3474. This bug made it possible that there were duplicates despite there being a unique or primary key constraint present. This could only happen for multi-column constraints. MonetDB maintains an internal column containing a hash for multi-column constraints, and there was a bug in the maintenance code of this column. During an upgrade, this column needs to be recreated. This is done by dropping and adding back the constraints. If there are no duplicates, adding back the constrains should work without problem, but if there are duplicates, adding the constraints will fail. There is unfortunately no easy way to tell the user, so after having made the first connection to the server (when the upgrade happens), check the merovingian log (if you use merovingian -- also known as monetdbd) or the server output window (if you don't use merovingian). Jaql support is removed. The experimental code for Jaql support has been removed. If you have RPM or DEB packages for Jaql installed from an earlier version (MonetDB-jaql and monetdb5_jaql respectively), you will need to uninstall them before the upgrade. Known issue when upgrading old versions. We have noticed that upgrading directly from a version older than the Jan2014 release gave an error. Upgrading to Jan2014 first and then on to Oct2014 should work. Backup. As always, make a backup of your databases before upgrading. Oct 2014 feature release MonetDB5 Server * Removed algebra.materialize. * Removed algebra.kunique and algebra.tunique. They were subsumed by algebra.subunique. * Remove algebra.antiuselect and algebra.thetauselect. They were subsumed by algebra.subselect. * Removed algebra.topN and its imlementation BATtopN. The function was not used. * Removed aggr.histogram and its implementation BAThistogram. The function was not used, and did not produce output in the "headless" (i.e. dense-headed) format. Histograms can be created as a by-product of group.subgroup. Build Environment * Jacqueline, the MonetDB/JAQL frontend, has been removed. The frontend never grew beyond being experimental, and there is no interest anymore to maintain the code. Merovingian * monetdb create: add -p flag to set monetdb user password on creation, and therefore allow creating the database in unlocked state mapilib * Changed mapi_timeout argument from seconds to milliseconds. stream * Changed mnstr_settimeout function so that the specified timeout is now in milliseconds (used to be seconds), and that it also needs an extra argument specifying a callback function (no arguments, int result) that should return TRUE if the timeout should cause the function to abort or continue what it was doing. MonetDB Common * Added "multifarm" capability. It is now possible to separate persistent and transient BATs into different directories (presumably on different disks). This can be done by using the --dbextra option of mserver5 (see the man page). SQL * Added PostgreSQL compatible string TRIM, LTRIM, RTRIM, LPAD and RPAD functions * Stop support for upgrading directly from a database created with a server from the Oct2012 release or older. You can upgrade via the Feb2013 or Jan2014 release. Bug Fixes * 2945: evaluation of SQL "between SYMMETRIC" requires MAL iterator because there is no (bulk) MIN/MAX(a,b) * 3204: monetdb create: allow setting admin password during creation * 3390: Missing definition for pushSht in monetdb5/mal/mal_builder.h * 3402: We should have a C implementation of mal.multiplex. * 3422: Segmentation fault after large table insert * 3459: incomplete implementation of JDBC driver getNumericFunctions(), getStringFunctions(), getSystemFunctions(), getTimeDateFunctions(), getSQLKeywords() methods in MonetDatabaseM * 3471: JDBC driver: incorrect output result of SQL query: SELECT 1 ; * 3474: bulk and scalar versions of mkey.rotate_xor_hash differ * 3484: COPY INTO on a file works fine on Linux/OSX, but not on Windows * 3488: Slow SQL execution for correlated scalar subquery * 3489: SQL query with ORDER BY does not order its result as requested * 3490: SQL query kills the mserver5 (Segmentation fault) * 3491: SQL query kills the mserver5 (Segmentation fault) * 3493: Test monetdb5/modules/mal/Tests/pqueue.mal fails since recent checkins * 3494: Tests monetdb5/modules/mal/Tests/pqueue[23].mal lack correct/expected/intended output * 3495: Test sql/test/centipede/Tests/olap.sql lacks correct/expected/intended output * 3497: monetdb start reports crash in merovingian.log * 3498: SQL throws TypeException if aggregations and limit statements are both present * 3502: Database was killed by signal SIGBUS * 3504: COPY INTO does not allow OFFSET without specifying amount of records * 3505: expression with <boolean> = NOT <boolean> returns a syntax error but NOT <boolean> = <boolean> not * 3506: conversion to varchar terminates mserver * 3508: conversion of string '0 ' to type smallint or integer fails * 3510: timestamp + month interval generates bogus MAL? * 3511: When having multiple selections combined with aliases not all of them seem to be evalauted. * 3512: auto-conversion of string to `sht` type no longer works * 3513: COPY BINARY INTO fails on 6gb file; works fine on 3gb * 3516: inserting '0' into a column of datatype numeric fails * 3518: UNION with subqueries * 3521: large results of function exp() are not automatically returned as double * 3522: SQL catalog table sys.columns lists columns for table ids which do not exist in sys.tables * 3523: Window function over union gives no result * 3524: wrong error on missing aggregation column * 3527: select distinct - order by - limit 2 results in one single result * 3528: segfault at mal_session.c:521 * 3532: several geom tests crash after manifold changes * 3534: missing table name with invalid column in join using (and problems after resolving it) * 3536: program contains error with join using integer and smallint * 3542: gdk/gdk_bat.c:2904: BATassertHeadProps: Assertion `!b->H->revsorted || cmp >= 0' failed. * 3543: invalid behavior and incorrect data results for SQL data type: numeric(4,4) * 3544: sys.reuse() corrupts data * 3546: Division by zero in CASE statement that should avoid it * 3547: Empty query when selecting a field from a view made of UNION ALL * 3551: Wrong ticks in TRACE * 3552: incorrect data results for "WHERE int_col <> 0" * 3554: Issue with subselect and ORDER BY * 3555: Order of evaluation inside CASE WHEN * 3558: numeric values (as strings) are incorrectly parsed/converted and invalid strings are accepted without error * 3560: Error "BATproject: does not match always" with subselect/groupby/having * 3562: mserver5: gdk_bat.c:2855: BATassertHeadProps: Assertion `!b->H->revsorted || cmp >= 0' failed. * 3563: incorrect results for scalar function locate(in_str, search_str, occurrence) * 3565: Wrong/confusing error message when trying to add a FK to a TEMP TABLE * 3572: Table names with escaped double quotes are rejected * 3573: alter table alter_not_null_test alter test set NOT NULL; is accepted when test contains null. This used to be restricted but isn't anymore * 3575: segmentation fault in mserver5 process * 3576: Dropping default value definitions from a table does not work as expected * 3577: SIGSEGV in BATins_kdiff * 3579: segmentation fault in mserver5 process * 3581: mserver5: rel_bin.c:2504: rel2bin_groupby: Assertion `0' failed. * 3582: mserver5: sql_mem.c:48: sql_ref_dec: Assertion `r->refcnt > 0' failed. * 3583: Possible buffer overflow in max(varchar) * 3585: Incorrect query terminates connection * 3586: mserver5: sql/storage/store.c:3610: sys_drop_func: Assertion `rid_func != oid_nil' failed. * 3592: SIGSEGV in MANIFOLDjob * 3593: delta_append_val: Assertion `!c || ((c)->S->count) == bat->ibase' failed. * 3594: gdk/gdk_bat.c:2855: BATassertHeadProps: Assertion `!b->H->revsorted || cmp >= 0' failed. * 3595: Race/heap corruption on thread exit * 3596: gdk_bat.c:2861: BATassertHeadProps: Assertion `!b->H->nonil || cmp != 0' failed. * 3597: SQL to MAL listing looses types * 3598: SQL bulk load should ignore leading/trailing spaces also with type decimal (as with integers & real/double) * 3599: Double-free of imprints * 3601: Trivial typo in debian/monetdb5-sql.init.d * 3603: "monetdb create -p" hangs monetdbd * 3604: Sys.queue ignored upon errors

7 years, 8 months

Improving performance of bulk loading data

by Dennis Pallett

Hi all, We're experiencing some problems during bulk loading of new data into our MonetDB database. We use COPY INTO to load several CSV files every day but this loading takes (too) long but more importantly the database is completely unresponsive with regards to any incoming queries therefore completely disabling our production environment. I'm currently looking into improving this proces and the hardware of our MonetDB server is obviously one of the starting points. Does anyone know which specs should be upgraded? Would adding more core's/CPU's have the most positive effect? Or perhaps more memory? Any advice or insight would be greatly appreciated! Best regards, Dennis Pallett

7 years, 8 months

Can't use ORDER BY in subqueries

by Kotschenreuther, Leo

Hi all, I tried to use some more complex queries, they contained subqueries. I will shortly desrcibe the situation: I have two tables (e.g. test and foo) and want to join them. Furthermore I want to preselect a subset of one of the two tables by adding some conditions (e.g. order the records by attribute c and only choosing the biggest one). A query could look like the this: sql>select * from test join (select * from foo order by c limit 1); The result I receive is the following: syntax error, unexpected ORDER, expecting INTERSECT or EXCEPT or UNION or ')' in: "select * from test join (select * from foo order” On the other hand, I can run the subquery without the main query and I receive a valid result. Is there a possibility to use subqueries with an ORDER BY statement without separating them into two independent queries? Regards Leo

7 years, 8 months

Re: MonetDB error while restarting the database

by Ashish Kumar Singh

Hi All, I have filled an issue Bug 3610: exittimeout property not being honoured against monetdb. It looks like GDKExit error is coming too frequently due to all this and its not even waiting for default value of 60 seconds. Regards, Ashish -----Original Message----- From: Ashish Singh <ashishkumar.singh(a)altair.com> Date: Thursday, 30 October 2014 8:24 pm To: "developers-list(a)monetdb.org" <developers-list(a)monetdb.org> Subject: Re: MonetDB error while restarting the database >Thanks All! > >we made sure now that database connection pool has released all >connections before we shutdown monetdb. > >But still infrequently we are seeing this error popping up and we have >double checked that client has returned all the connections. > >Before stopping the database a data load process happened from the client >to monetdb, can it be contributed to that?? > > > >Regards, >Ashish > >-----Original Message----- >From: Sjoerd Mullender <sjoerd(a)monetdb.org> >Reply-To: "developers-list(a)monetdb.org" <developers-list(a)monetdb.org> >Date: Monday, 27 October 2014 11:07 pm >To: "developers-list(a)monetdb.org" <developers-list(a)monetdb.org> >Subject: Re: MonetDB error while restarting the database > >> >> >>On 27/10/14 17:42, Dimitar Nedev wrote: >>> Hi Ashish, >>> >>> What happened there is monetdbd send a TERM signal to the mserver >>>process (the actual MonetDB database process), which killed all running >>>threads. The messages "#GDKexit: killing thread" means the the MonetDB >>>kernel (GDK) killed a thread still working at the time the process was >>>terminated. This is most likely an indication that the database was >>>still processing something at that time. Depending on what it was >>>actually doing, data corruption might occur. This can explain why >>>monetdbd reports an internal error why truing to start the mserver >>>process again. >> >>Data corruption should *never* occur. If you have proof of (and a >>recipe for) data corruption, please file a bug report. >>Of course, if a thread was still working, the transaction it was working >>on will most likely not get committed. >> >>Otherwise the above is correct. >> >>> Check the merovingian.log file again for the events when monetdbd >>>reported the internal error. There should be more information logged on >>>why mserver cannot start up any more. >>> >>> Best regards, >>> Dimitar >>> >>>> On 2014-Oct-27, at 15:55 , Ashish Kumar Singh >>>><ashishkumar.singh(a)altair.com> wrote: >>>> >>>> Thanks Dimitar, >>>> >>>> Yes we are restarting the data base and as part of that we are seeing >>>>this >>>> error. >>>> >>>> I am more worried about the error below: >>>> >>>> >>>> 2014-10-27 09:01:26 ERR merovingian[23583]: unknown state: 42014-10-27 >>>> 09:01:26 ERR pbsworksdb[6351]: #GDKexit: killing thread >>>> >>>> >>>> On java client side it says >>>> >>>> >>>> monetdbd: internal error while starting mserver, please refer to the >>>>logs >>>> >>>> >>>> >>>> >>>> Regards, >>>> Ashish >>>> >>>> >>>> -----Original Message----- >>>> From: Dimitar Nedev <D.G.Nedev(a)cwi.nl> >>>> Reply-To: "users-list(a)monetdb.org" <users-list(a)monetdb.org> >>>> Date: Monday, 27 October 2014 8:15 pm >>>> To: "users-list(a)monetdb.org" <users-list(a)monetdb.org> >>>> Subject: Re: MonetDB error while restarting the database >>>> >>>>> Hi Ashish, >>>>> >>>>> The exit timeout only tells monetdbd how long should it wait when >>>>>trying >>>>> to gracefully shut down a mserver process. According to the log, the >>>>> mserver process was shut down within the limit, since there are no >>>>> messages like "timeout of <exittimeout> seconds expired, sending >>>>>process >>>>> <PID> the KILL signal". >>>>> >>>>> Now, the interesting entry in the log is the control process message: >>>>> 2014-10-27 09:01:26 MSG control[23583]: (local): stopped database >>>>> 'pbsworksdb' >>>>> >>>>> This one would indicate that someone (or something) gracefully >>>>>stopped >>>>> the database using the monetdbd client. Probably with the following >>>>> command line call like: 'monetdb stop pbsworksdb'. >>>>> Do not be confused by the order of logged events - monetdbd will >>>>>first >>>>> stop the process and later log that the call to stop the a database >>>>>has >>>>> been successful. >>>>> >>>>> Best regards, >>>>> Dimitar >>>>> >>>>> >>>>>> On 2014-Oct-27, at 14:49 , Ashish Kumar Singh >>>>>> <ashishkumar.singh(a)altair.com> wrote: >>>>>> >>>>>> Thanks! >>>>>> >>>>>> >>>>>> Appreciate your response so there are few more people getting the >>>>>>same >>>>>> error :). >>>>>> Any suggestions from dev team on this issue? Can monetdbd >>>>>>exittimeout >>>>>> be also in picture for this? >>>>>> >>>>>> >>>>>> Regards, >>>>>> Ashish >>>>>> >>>>>> >>>>>> >>>>>> From: Pierre-Adrien Coustillas <pcoustillas(a)1g6.biz> >>>>>> Date: Monday, 27 October 2014 6:51 pm >>>>>> To: "users-list(a)monetdb.org" <users-list(a)monetdb.org> >>>>>> Cc: "developers-list(a)monetdb.org" <developers-list(a)monetdb.org>, >>>>>>Ashish >>>>>> Singh <ashishkumar.singh(a)altair.com> >>>>>> Subject: Re: MonetDB error while restarting the database >>>>>> >>>>>>> Hello >>>>>>> >>>>>>> (google translate) >>>>>>> >>>>>>> I receive this error when there is no specific spacedisk. >>>>>>> Monetdb sometimes uses a lot of temporary diskspace >>>>>>>(500GBisnormalfor >>>>>>> 1TBof data), which arereleasedat the end oftreatment or after >>>>>>>acrash >>>>>>> >>>>>>> Error last week : >>>>>>> 2014-10-22 22:51:46 MSG merovingian[18865]: sending process 30608 >>>>>>> (database 'lemonde') the TERM signal >>>>>>> 2014-10-22 22:51:46 ERR merovingian[18865]: unknown state: >>>>>>>42014-10-22 >>>>>>> 22:51:46 ERR lemonde[30608]: #GDKexit: killing thread >>>>>>> >>>>>>> Pierre >>>>>>> >>>>>>> -- >>>>>>> 1G6 >>>>>>> 52 route de bischwiller >>>>>>> 67300 Schiltigheim >>>>>>> Société de Services et de Formations en Logiciels Libres >>>>>>> http://1g6.biz >>>>>>> Tél : 06 64 63 70 35 >>>>>>> >>>>>>> De: "Ashish Kumar Singh" <ashishkumar.singh(a)altair.com> >>>>>>> À: "Ashish Kumar Singh" <ashishkumar.singh(a)altair.com>, >>>>>>> users-list(a)monetdb.org >>>>>>> Cc: developers-list(a)monetdb.org >>>>>>> Envoyé: Lundi 27 Octobre 2014 13:49:19 >>>>>>> Objet: Re: MonetDB error while restarting the database >>>>>>> >>>>>>> Guys, >>>>>>> >>>>>>> Any help with this will be really helpful for us? >>>>>>> >>>>>>> >>>>>>> >>>>>>> Regards, >>>>>>> Ashish >>>>>>> >>>>>>> From: Ashish Singh <ashishkumar.singh(a)altair.com> >>>>>>> Date: Monday, 27 October 2014 4:06 pm >>>>>>> To: "users-list(a)monetdb.org" <users-list(a)monetdb.org> >>>>>>> Subject: MonetDB error while restarting the database >>>>>>> >>>>>>>> >>>>>>>> All, >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> We are facing a new issue with monetdb where monetdb connections >>>>>>>>and >>>>>>>> process is being killed very frequently. Is this a known issue or >>>>>>>>any >>>>>>>> suggestion in getting more details will be helpful. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Regards, >>>>>>>> Ashish >>>>>>>> >>>>>>>> Mervogian.log: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: arguments: >>>>>>>> >>>>>>>>/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/bin/mserver >>>>>>>>5 >>>>>>>> >>>>>>>>--dbpath=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pb >>>>>>>>s >>>>>>>>wor >>>>>>>> ksdata/pbsworksdb --set >>>>>>>> merovingian_uri=mapi:monetdb://blrec12-vm6:9200/pbsworksdb --set >>>>>>>> mapi_open=false --set mapi_port=0 --set >>>>>>>> >>>>>>>>mapi_usock=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/ >>>>>>>>p >>>>>>>>bsw >>>>>>>> orksdata/pbsworksdb/.mapi.sock --set >>>>>>>> >>>>>>>>monet_vault_key=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/mon >>>>>>>>e >>>>>>>>tdb >>>>>>>> /pbsworksdata/pbsworksdb/.vaultkey --set gdk_nr_threads=4 --set >>>>>>>> max_clients=64 --set sql_optimizer=default_pipe --set >>>>>>>>monet_daemon=yes >>>>>>>> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # MonetDB 5 server >>>>>>>> v11.17.17 "Jan2014-SP2" >>>>>>>> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Serving database >>>>>>>> 'pbsworksdb', using 4 threads >>>>>>>> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Compiled for >>>>>>>> x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked >>>>>>>> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Found 15.675 GiB >>>>>>>> available main-memory. >>>>>>>> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Copyright (c) >>>>>>>>1993-July >>>>>>>> 2008 CWI. >>>>>>>> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Copyright (c) August >>>>>>>> 2008-2014 MonetDB B.V., all rights reserved >>>>>>>> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Visit >>>>>>>> http://www.monetdb.org/ for further information >>>>>>>> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Listening for UNIX >>>>>>>>domain >>>>>>>> connection requests on >>>>>>>> >>>>>>>>mapi:monetdb:///opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/mone >>>>>>>>t >>>>>>>>db/ >>>>>>>> pbsworksdata/pbsworksdb/.mapi.sock >>>>>>>> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # MonetDB/SQL module >>>>>>>>loaded >>>>>>>> 2014-10-27 09:01:00 MSG control[23583]: (local): started >>>>>>>>'pbsworksdb' >>>>>>>> 2014-10-27 09:01:00 MSG merovingian[23583]: proxying client >>>>>>>> localhost.localdomain:53844 for database 'pbsworksdb' to >>>>>>>> >>>>>>>>mapi:monetdb:///opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/mone >>>>>>>>t >>>>>>>>db/ >>>>>>>> pbsworksdata/pbsworksdb/.mapi.sock?database=pbsworksdb >>>>>>>> 2014-10-27 09:01:00 MSG merovingian[23583]: target connection is >>>>>>>>on >>>>>>>> local UNIX domain socket, passing on filedescriptor instead of >>>>>>>>proxying >>>>>>>> 2014-10-27 09:01:26 MSG control[23583]: (local): served status >>>>>>>>list >>>>>>>> 2014-10-27 09:01:26 MSG merovingian[23583]: sending process 6351 >>>>>>>> (database 'pbsworksdb') the TERM signal >>>>>>>> 2014-10-27 09:01:26 ERR merovingian[23583]: unknown state: >>>>>>>> 42014-10-27 09:01:26 ERR pbsworksdb[6351]: #GDKexit: killing >>>>>>>>thread >>>>>>>> 2014-10-27 09:01:26 MSG merovingian[23583]: database 'pbsworksdb' >>>>>>>> (6351) has exited with exit status 0 >>>>>>>> 2014-10-27 09:01:26 MSG merovingian[23583]: database 'pbsworksdb' >>>>>>>>has >>>>>>>> shut down >>>>>>>> 2014-10-27 09:01:26 MSG control[23583]: (local): stopped database >>>>>>>> 'pbsworksdb' >>>>>>>> 2014-10-27 09:01:26 MSG control[23583]: (local): served status >>>>>>>>list >>>>>>>> 2014-10-27 09:01:26 MSG merovingian[23583]: starting database >>>>>>>> 'pbsworksdb', up min/avg/max: 5s/14m/1h, crash average: 0.00 0.00 >>>>>>>>0.00 >>>>>>>> (648-648=0) >>>>>>>> 2014-10-27 09:01:27 MSG pbsworksdb[6575]: arguments: >>>>>>>> >>>>>>>>/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/bin/mserver >>>>>>>>5 >>>>>>>> >>>>>>>>--dbpath=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pb >>>>>>>>s >>>>>>>>wor >>>>>>>> ksdata/pbsworksdb --set >>>>>>>> merovingian_uri=mapi:monetdb://blrec12-vm6:9200/pbsworksdb --set >>>>>>>> mapi_open=false --set mapi_port=0 --set >>>>>>>> >>>>>>>>mapi_usock=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/ >>>>>>>>p >>>>>>>>bsw >>>>>>>> orksdata/pbsworksdb/.mapi.sock --set >>>>>>>> >>>>>>>>monet_vault_key=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/mon >>>>>>>>e >>>>>>>>tdb >>>>>>>> /pbsworksdata/pbsworksdb/.vaultkey --set gdk_nr_threads=4 --set >>>>>>>> max_clients=64 --set sql_optimizer=default_pipe --set >>>>>>>>monet_daemon=yes >>>>>>>> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # MonetDB 5 server >>>>>>>> v11.17.17 "Jan2014-SP2" >>>>>>>> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Serving database >>>>>>>> 'pbsworksdb', using 4 threads >>>>>>>> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Compiled for >>>>>>>> x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked >>>>>>>> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Found 15.675 GiB >>>>>>>> available main-memory. >>>>>>>> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Copyright (c) >>>>>>>>1993-July >>>>>>>> 2008 CWI. >>>>>>>> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Copyright (c) August >>>>>>>> 2008-2014 MonetDB B.V., all rights reserved >>>>>>>> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Visit >>>>>>>> http://www.monetdb.org/ for further information >>>>>>>> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Listening for UNIX >>>>>>>>domain >>>>>>>> connection requests on >>>>>>>> >>>>>>>>mapi:monetdb:///opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/mone >>>>>>>>t >>>>>>>>db/ >>>>>>>> pbsworksdata/pbsworksdb/.mapi.sock >>>>>>>> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # MonetDB/SQL module >>>>>>>>loaded >>>>>>>> 2014-10-27 09:01:32 MSG merovingian[23583]: proxying client >>>>>>>> localhost.localdomain:53853 for database 'pbsworksdb' to >>>>>>>> >>>>>>>>mapi:monetdb:///opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/mone >>>>>>>>t >>>>>>>>db/ >>>>>>>> pbsworksdata/pbsworksdb/.mapi.sock?database=pbsworksdb >>>>>>>> 2014-10-27 09:01:32 MSG merovingian[23583]: target connection is >>>>>>>>on >>>>>>>> local UNIX domain socket, passing on filedescriptor instead of >>>>>>>>proxying >>>>>>>> 2014-10-27 09:01:32 MSG c >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> users-list mailing list >>>>>>> users-list(a)monetdb.org >>>>>>> https://www.monetdb.org/mailman/listinfo/users-list >>>>>>> >>>>>> _______________________________________________ >>>>>> developers-list mailing list >>>>>> developers-list(a)monetdb.org >>>>>> https://www.monetdb.org/mailman/listinfo/developers-list >>>>> >>>>> _______________________________________________ >>>>> users-list mailing list >>>>> users-list(a)monetdb.org >>>>> https://www.monetdb.org/mailman/listinfo/users-list >>>> >>>> _______________________________________________ >>>> developers-list mailing list >>>> developers-list(a)monetdb.org >>>> https://www.monetdb.org/mailman/listinfo/developers-list >>> >>> _______________________________________________ >>> developers-list mailing list >>> developers-list(a)monetdb.org >>> https://www.monetdb.org/mailman/listinfo/developers-list >>> >> >>-- >>Sjoerd Mullender >> >

7 years, 9 months

SQL_NO_CACHE equivalent?

by Vijay Krishna

Hi, Is there any equivalent to SQL_NO_CACHE in monetdb?? I wish the queries I run to return results from disk in order to check performance. Any help much appreciated. Thanks in advance. With Regards, Vijayakrishna.P. Mobile : (+91) 9500402305.

7 years, 9 months

detect fragmented tables

by Pierre-Adrien Coustillas

Hello Before working with optimizer, Stethoscope and Tomograph , first I have to defragment the fragmented tables. How to detect fragmented tables ? Write a script that reads heapsize column ? select * , ( heapsize / 1021/1024/1024) as giga from sys.storage heapsize ORDER BY DESC LIMIT 20; What do you think ? or a more specific sql query by reading the column "count ", " typewidth ", " ColumnSize " and " heapsize " ? thank Pierre -- 1G6 52 route de bischwiller 67300 Schiltigheim Société de Services et de Formations en Logiciels Libres http://1g6.biz Tél : 06 64 63 70 35

7 years, 9 months

Transaction Failure: BATsubselect: invalid argument: b must have a dense head

by Tim Burress

Hello, I've been using MonetDB for a couple of weeks now with no problems, but today ran into this issue when doing a series of DELETE/INSERT commands to update a table. When I issue the commands individually (autocommit on), there doesn't seem to be any difficulty, but when I do them as part of a transaction, I get this error on the second insert: START TRANSACTION; DELETE... 1 affected rows (10.145ms) INSERT... 1 affected rows (11.464ms) DELETE... 1 affected rows (3.402ms) INSERT... GDK reported error. BATsubselect: invalid argument: b must have a dense head I'm not sure what this means or what to do about it. Similar transactions were done yesterday with no problem, so I thought perhaps the table had been corrupted, but if I drop the table and recreate it from the original data, the above error still occurs. Thanks for any help you can provide, and if there is anything I can do that would give you some hints about what's happening, just tell me what to do and I'll be happy to help. Tim

7 years, 9 months

MonetDB error while restarting the database

by Ashish Kumar Singh

All, We are facing a new issue with monetdb where monetdb connections and process is being killed very frequently. Is this a known issue or any suggestion in getting more details will be helpful. Regards, Ashish Mervogian.log: 2014-10-27 09:01:00 MSG pbsworksdb[6351]: arguments: /opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/bin/mserver5 --dbpath=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pbsworksdata/pbsworksdb --set merovingian_uri=mapi:monetdb://blrec12-vm6:9200/pbsworksdb --set mapi_open=false --set mapi_port=0 --set mapi_usock=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pbsworksdata/pbsworksdb/.mapi.sock --set monet_vault_key=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pbsworksdata/pbsworksdb/.vaultkey --set gdk_nr_threads=4 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # MonetDB 5 server v11.17.17 "Jan2014-SP2" 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Serving database 'pbsworksdb', using 4 threads 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Found 15.675 GiB available main-memory. 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Copyright (c) 1993-July 2008 CWI. 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Copyright (c) August 2008-2014 MonetDB B.V., all rights reserved 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Visit http://www.monetdb.org/ for further information 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Listening for UNIX domain connection requests on mapi:monetdb:///opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pbsworksdata/pbsworksdb/.mapi.sock 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # MonetDB/SQL module loaded 2014-10-27 09:01:00 MSG control[23583]: (local): started 'pbsworksdb' 2014-10-27 09:01:00 MSG merovingian[23583]: proxying client localhost.localdomain:53844 for database 'pbsworksdb' to mapi:monetdb:///opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pbsworksdata/pbsworksdb/.mapi.sock?database=pbsworksdb 2014-10-27 09:01:00 MSG merovingian[23583]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2014-10-27 09:01:26 MSG control[23583]: (local): served status list 2014-10-27 09:01:26 MSG merovingian[23583]: sending process 6351 (database 'pbsworksdb') the TERM signal 2014-10-27 09:01:26 ERR merovingian[23583]: unknown state: 42014-10-27 09:01:26 ERR pbsworksdb[6351]: #GDKexit: killing thread 2014-10-27 09:01:26 MSG merovingian[23583]: database 'pbsworksdb' (6351) has exited with exit status 0 2014-10-27 09:01:26 MSG merovingian[23583]: database 'pbsworksdb' has shut down 2014-10-27 09:01:26 MSG control[23583]: (local): stopped database 'pbsworksdb' 2014-10-27 09:01:26 MSG control[23583]: (local): served status list 2014-10-27 09:01:26 MSG merovingian[23583]: starting database 'pbsworksdb', up min/avg/max: 5s/14m/1h, crash average: 0.00 0.00 0.00 (648-648=0) 2014-10-27 09:01:27 MSG pbsworksdb[6575]: arguments: /opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/bin/mserver5 --dbpath=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pbsworksdata/pbsworksdb --set merovingian_uri=mapi:monetdb://blrec12-vm6:9200/pbsworksdb --set mapi_open=false --set mapi_port=0 --set mapi_usock=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pbsworksdata/pbsworksdb/.mapi.sock --set monet_vault_key=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pbsworksdata/pbsworksdb/.vaultkey --set gdk_nr_threads=4 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # MonetDB 5 server v11.17.17 "Jan2014-SP2" 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Serving database 'pbsworksdb', using 4 threads 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Found 15.675 GiB available main-memory. 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Copyright (c) 1993-July 2008 CWI. 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Copyright (c) August 2008-2014 MonetDB B.V., all rights reserved 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Visit http://www.monetdb.org/ for further information 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Listening for UNIX domain connection requests on mapi:monetdb:///opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pbsworksdata/pbsworksdb/.mapi.sock 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # MonetDB/SQL module loaded 2014-10-27 09:01:32 MSG merovingian[23583]: proxying client localhost.localdomain:53853 for database 'pbsworksdb' to mapi:monetdb:///opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pbsworksdata/pbsworksdb/.mapi.sock?database=pbsworksdb 2014-10-27 09:01:32 MSG merovingian[23583]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2014-10-27 09:01:32 MSG c

7 years, 9 months

set optimizer

by Pierre-Adrien Coustillas

Hello , (google translate) I am a little fun with the optimizer . sql>s elec t count (distinct idvisiteur ) from src_inbox__at__lm__visites ; + ---------- + | L1 | + + ========== | 75528080 | + ---------- + 1 tuple ( 3m 51s ) sql> set optimizer = ' minimal_pipe ; sql> select count (distinct idvisiteur ) from src_inbox__at__lm__visites ; + ---------- + | L1 | + + ========== | 75528080 | + ---------- + 1 tuple (1m 28s ) sql> set optimizer = ' default_pipe ; sql> select count (distinct idvisiteur ) from src_inbox__at__lm__visites ; + ---------- + | L1 | + + ========== | 75528080 | + ---------- + 1 tuple ( 3m 54s ) It's very interesting . Is there a risk to change optimizer ? What are the best practices ? thank you -- 1G6 52 route de bischwiller 67300 Schiltigheim Société de Services et de Formations en Logiciels Libres http://1g6.biz Tél : 06 64 63 70 35

7 years, 9 months

heapsize, speed, and vacuum

by Pierre-Adrien Coustillas

Hello, (google translate) I have a query that takes a long time : select count(distinct idvisiteur) from src_inbox__at__lm__visites; After two hours of running, it is not over , I'll kill her . if I look at the heapsize column storage ( ) sql>select * from storage() WHERE "table"='src_inbox__at__lm__visites'; +--------+----------------------------+---------------------+---------+-------------+-----------+-----------+------------+-------------+---------+--------+ | schema | table | column | type | location | count | typewidth | columnsize | heapsize | indices | sorted | +========+============================+=====================+=========+=============+===========+===========+============+=============+=========+========+ | sys | src_inbox__at__lm__visites | idvisite | varchar | 01/00/10025 | 315819936 | 5 | 2526559488 | 27755937792 | 0 | false | | sys | src_inbox__at__lm__visites | idvisiteur | varchar | 76/7667 | 315819936 | 19 | 2526559488 | 80449110016 | 0 | false | | sys | src_inbox__at__lm__visites | idvisiteuridentifie | varchar | 62/6201 | 315819936 | 2 | 1263279744 | 2222391296 | 0 | false | | sys | src_inbox__at__lm__visites | siteniveau1 | varchar | 77/7704 | 315819936 | 5 | 315819936 | 65536 | 0 | false | | sys | src_inbox__at__lm__visites | debutvisite | varchar | 30/3064 | 315819936 | 19 | 2526559488 | 10741678080 | 0 | false | | sys | src_inbox__at__lm__visites | catvisiteur | varchar | 31/3171 | 315819936 | 0 | 631639872 | 65536 | 0 | false | | sys | src_inbox__at__lm__visites | region | varchar | 67/6706 | 315819936 | 12 | 1263279744 | 2077556736 | 0 | false | | sys | src_inbox__at__lm__visites | ville | varchar | 40/4071 | 315819936 | 10 | 2526559488 | 22635741184 | 0 | false | | sys | src_inbox__at__lm__visites | os | varchar | 01/26/12673 | 315819936 | 8 | 631639872 | 65536 | 0 | false | | sys | src_inbox__at__lm__visites | browser | varchar | 01/44/14410 | 315819936 | 17 | 631639872 | 65536 | 0 | false | | sys | src_inbox__at__lm__visites | sourcevisite | varchar | 42/4234 | 315819936 | 1 | 631639872 | 65536 | 0 | false | | sys | src_inbox__at__lm__visites | idcampagne | varchar | 01/41/14100 | 315819936 | 0 | 631639872 | 65536 | 0 | false | | sys | src_inbox__at__lm__visites | prefixe | varchar | 01/22/12277 | 315819936 | 0 | 315819936 | 65536 | 0 | false | | sys | src_inbox__at__lm__visites | libcampagne | varchar | 65/6573 | 315819936 | 0 | 631639872 | 65536 | 0 | false | | sys | src_inbox__at__lm__visites | emailing | varchar | 01/04/10434 | 315819936 | 0 | 631639872 | 65536 | 0 | false | | sys | src_inbox__at__lm__visites | dateenvois | varchar | 01/22/12264 | 315819936 | 0 | 631639872 | 65536 | 0 | false | | sys | src_inbox__at__lm__visites | nbpages | int | 56/5615 | 315819936 | 4 | 1263279744 | 0 | 0 | false | | sys | src_inbox__at__lm__visites | dureevisite | time | 01/02/10211 | 315819936 | 4 | 1263279744 | 0 | 0 | false | | sys | src_inbox__at__lm__visites | id_unique | int | 01/23/12370 | 315819936 | 4 | 1263279744 | 0 | 0 | false | +--------+----------------------------+---------------------+---------+-------------+-----------+-----------+------------+-------------+---------+--------+ Is the heapsize column represents the amount of data to be placed in RAM ? I have 64GB of RAM on the server , so the idvisiteur column can not be held in RAM. Is this true? ( 64 000 000 000 < 80 449 110 016 ) I execute a vacuum (experimental command ?) : https://www.monetdb.org/Documentation/UserGuide/AutoloadingScript (Table vacuum commands (experimental!)) call vacuum('sys','src_inbox__at__lm__visites'); sql>select * from storage() WHERE "table"='src_inbox__at__lm__visites'; +--------+----------------------------+---------------------+---------+-------------+-----------+-----------+------------+------------+---------+--------+ | schema | table | column | type | location | count | typewidth | columnsize | heapsize | indices | sorted | +========+============================+=====================+=========+=============+===========+===========+============+============+=========+========+ | sys | src_inbox__at__lm__visites | idvisite | varchar | 36 | 315819936 | 5 | 2526559488 | 4906811392 | 0 | false | | sys | src_inbox__at__lm__visites | idvisiteur | varchar | 64/6435 | 315819936 | 19 | 1263279744 | 3667394560 | 0 | false | | sys | src_inbox__at__lm__visites | idvisiteuridentifie | varchar | 66/6640 | 315819936 | 2 | 1263279744 | 40042496 | 0 | false | | sys | src_inbox__at__lm__visites | siteniveau1 | varchar | 67/6743 | 315819936 | 5 | 315819936 | 73728 | 0 | false | | sys | src_inbox__at__lm__visites | debutvisite | varchar | 70/7045 | 315819936 | 19 | 2526559488 | 7576551424 | 0 | false | | sys | src_inbox__at__lm__visites | catvisiteur | varchar | 71/7152 | 315819936 | 0 | 631639872 | 73728 | 0 | false | | sys | src_inbox__at__lm__visites | region | varchar | 72/7252 | 315819936 | 12 | 1263279744 | 106561536 | 0 | false | | sys | src_inbox__at__lm__visites | ville | varchar | 73/7353 | 315819936 | 10 | 1263279744 | 1362165760 | 0 | false | | sys | src_inbox__at__lm__visites | os | varchar | 74/7453 | 315819936 | 8 | 631639872 | 73728 | 0 | false | | sys | src_inbox__at__lm__visites | browser | varchar | 75/7554 | 315819936 | 17 | 631639872 | 73728 | 0 | false | | sys | src_inbox__at__lm__visites | sourcevisite | varchar | 76/7655 | 315819936 | 1 | 631639872 | 73728 | 0 | false | | sys | src_inbox__at__lm__visites | idcampagne | varchar | 77/7761 | 315819936 | 0 | 631639872 | 73728 | 0 | false | | sys | src_inbox__at__lm__visites | prefixe | varchar | 01/00/10064 | 315819936 | 0 | 315819936 | 73728 | 0 | false | | sys | src_inbox__at__lm__visites | libcampagne | varchar | 01/01/10165 | 315819936 | 0 | 631639872 | 73728 | 0 | false | | sys | src_inbox__at__lm__visites | emailing | varchar | 01/02/10270 | 315819936 | 0 | 631639872 | 73728 | 0 | false | | sys | src_inbox__at__lm__visites | dateenvois | varchar | 01/03/10372 | 315819936 | 0 | 631639872 | 73728 | 0 | false | | sys | src_inbox__at__lm__visites | nbpages | int | 01/04/10476 | 315819936 | 4 | 1263279744 | 0 | 0 | false | | sys | src_inbox__at__lm__visites | dureevisite | time | 01/06/10600 | 315819936 | 4 | 1263279744 | 0 | 0 | false | | sys | src_inbox__at__lm__visites | id_unique | int | 01/07/10703 | 315819936 | 4 | 1263279744 | 0 | 0 | false | +--------+----------------------------+---------------------+---------+-------------+-----------+-----------+------------+------------+---------+--------+ heapsize of idvisteur = 3 667 394 560 (3,6GB) instead of 80 449 110 016 (80 GB) my query is present MUCH faster sql>select count(distinct idvisiteur) from src_inbox__at__lm__visites; +----------+ | L1 | +==========+ | 75528081 | +----------+ 1 tuple (4m 22s) HOURA !!! but WHY WHY WHY ? I want to understand Thank Pierre -- 1G6 52 route de bischwiller 67300 Schiltigheim Société de Services et de Formations en Logiciels Libres http://1g6.biz Tél : 06 64 63 70 35

7 years, 9 months

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

users-list October 2014