January 2016 - users-list

Multi-core join performance
by Alexandros Koliousis 20 Jan '16

20 Jan '16

Hi, I am joining two tables, each with 32K tuples. The query runs for ~23-24 seconds on my (multi-core) machine. During execution, however, only 1 core is utilised, while the rest of the CPU cores are idle. The cores are under-utilised although I have configured the MonetDB server to use all the physical cores of my machine (either by setting 'nthreads' or, in the case of mserver5, 'gdk_nr_threads'). I am running MonetDB server v11.21.13 "Jul2015-SP2", compiled from source. Is this due to some configuration setting I missed? Or, due to the fact that I am running a single-operator query? Alternatively, I was thinking of creating merge tables, thus partitioning my tables manually, hoping that this will allow for intra-operator parallelism. Thanks, Alexandros

2 1

create table uppercase with " "
by Pierre-Adrien Coustillas 20 Jan '16

20 Jan '16

Hello, bug or feature ? * PETER in uppercase : sql>CREATE TABLE "PETER" ( nom varchar(20) ); operation successful (2.545ms) sql>DROP TABLE PETER; DROP TABLE: no such table 'peter' sql>DROP TABLE "PETER"; operation successful (1.947ms) * peter in lowercase : sql>CREATE TABLE "peter" ( nom varchar(20) ); operation successful (2.917ms) sql>DROP TABLE peter; operation successful (1.882ms) * peter in lowercase and uppercase : sql>CREATE TABLE "peter" ( nom varchar(20) ); operation successful (2.900ms) sql>CREATE TABLE PETER ( nom varchar(20) ); CREATE TABLE: name 'peter' already in use sql>CREATE TABLE "PETER" ( nom varchar(20) ); operation successful (9.339ms) Conclusion : PETER is convert in peter "PETER" is not convert Pierre Coustillas -- 1G6 52 route de bischwiller 67300 Schiltigheim Société de Services et de Formations en Logiciels Libres http://1g6.biz Tél : 06 64 63 70 35 // 09 54 38 70 90

2 1

best practice use schema
by Pierre-Adrien Coustillas 20 Jan '16

20 Jan '16

Hello, What is the best practice on the use of schema ? S hould we always create a schema , or is can I use the " sys" schema by default ? Pierre Coustillas -- 1G6 52 route de bischwiller 67300 Schiltigheim Société de Services et de Formations en Logiciels Libres http://1g6.biz Tél : 06 64 63 70 35 // 09 54 38 70 90

1 0

creating a binary file for "COPY binary ..."
by Lynn Carol Johnson 19 Jan '16

19 Jan '16

Hi all - I'm creating a large db table with several columns. To speed it up, I'm using monetdb's "COPY BINARY" command. I have a question on creating the binary files. What I've done is use java's DataOutputStream to create the binary file - using code that looks as below (sometimes an int, sometimes a real or byte): writerChr = new DataOutputStream(new BufferedOutputStream(new FileOutputStream(chrFile))); writerChr.writeInt(intChrom); When I read this file using a test program and java's DataInputStream, it correctly shows me the integer values I've written. In the "writerChr" case, all values in this particular column are "10". However, when I load this into monetdb the value is way off - it shows a large number. The command I am using to copy this file and one other into a table with 2 "int" columns is: COPY binary into justchrpos from ('/home/lcj34/chrFileC10.bin','/home/lcj34/posFileC10.bin'); It copies in fine, but when I cheek the values I see this: (not 10!) sql>\d justchrpos CREATE TABLE "testbig"."justchrpos" ( "chr" INTEGER, "pos" INTEGER ); sql>select * from justchrpos limit 10; +-----------+-----------+ | chr | pos | +===========+===========+ | 167772160 | 16777216 | | 167772160 | 33554432 | | 167772160 | 50331648 | | 167772160 | 67108864 | | 167772160 | 83886080 | | 167772160 | 100663296 | | 167772160 | 117440512 | | 167772160 | 134217728 | | 167772160 | 150994944 | | 167772160 | 167772160 | +-----------+-----------+ 10 tuples (3.601ms) sql> QUESTION: Is using java's DataOutputStream a valid way to create the binary files for monetdb? If not, what should I be using? As I mentioned above, when I read back the created file using java's DataInputStream, I see the values "10" as I expect. Thanks - Lynn

3 3

Monet way of handling string columns
by bhuvan Mitra 18 Jan '16

18 Jan '16

Hello, I have recently started looking into Monetdb and want to contribute to it. My complete focus is on string columns as numeric data can already be processed with maximum speed. I know it uses BATs to store every single column, but I read some where that it uses dictionary for string columns. How is it actually using dictionary? And are there any other special handling techniques for strings? When a query like 'contains' comes, how will monet handle that? I found there is no inverted index in monet to deal queries like this. (Please tell a rough model of how query execution is done incase of string columns) Hope this forum gives me some insight of what's happening during string column processing. Thanks in advance, -Mitra

1 0

MonetDB-11.21.13 (Jul2015-SP2) can't start in CentOS 6.3
by chenkaijiang＠bigknow.com.cn 18 Jan '16

18 Jan '16

I downloaded the source codes at https://www.monetdb.org/downloads/sources/Jul2015-SP2/MonetDB-11.21… ./configure --prefix=/usr/local/MonetDBmakemake install No error is reported. But can't start:monetdbd create ./ycmonetdbd: error while loading shared libraries: libstream.so.7: cannot open shared object file: No such file or directory It looks like the libstream.so is lib on debian linux.  How can I fix it on CentOS 6.3?

2 1

How to set query timeout in MonetDB Server
by Luciano Sasso 16 Jan '16

16 Jan '16

How to set query timeout in MonetDB Server? -- Luciano Sasso Vieira Data Scientist & Solutions Architect luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 99706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. https://www.avast.com/antivirus

1 0

Announcement: New Jul2015-SP2 Bugfix release of MonetDB suite
by Sjoerd Mullender 16 Jan '16

16 Jan '16

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jul2015-SP2 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jul 2015-SP2 bugfix release Bug Fixes * 2014: 'null' from copy into gets wrong * 3817: opt_pushselect stuck with multi-table UDF * 3835: windows does not release ram after operations * 3836: rand() only gets evaluated once when used as an expression * 3838: Update column with or without parenthesis produce different results * 3840: savepoints may crash the database * 3841: mclient fails with response "Challenge string is not valid" * 3842: SQL execution fails to finish and reports bogus error messages * 3845: Too many VALUES in INSERT freeze mserver5 * 3847: Wrong SQL results for a certain combination of GROUP BY / ORDER BY / LIMIT * 3848: mserver segfault during bulk loading/updating * 3849: HUGEINT incorrect value * 3850: DEL character not escaped * 3851: expression that should evaluate to FALSE evaluates to TRUE in SELECT query * 3852: CASE statement produces GDK error on multithreaded database: BATproject does not match always * 3854: Complex expression with comparison evaluates incorrectly in WHERE clause * 3855: Foreign key referencing table in a different schema - not allowed. * 3857: Large LIMIT in SELECT may abort the query * 3861: Using window functions cause a crash * 3864: Error in bulk import for chinese character * 3871: NOT x LIKE triggers "too many nested operators" * 3872: mserver crashes under specific combination of JOIN and WHERE conditions * 3873: mserver5: gdk_bat.c:1015: setcolprops: Assertion `x != ((void *)0) || col->type == 0' failed. * 3879: Database crashes when querying with several UNION ALLs. * 3887: Querying "sys"."tracelog" causes assertion violation and crash of mserver5 process * 3889: read only does not protect empty tables * 3895: read only does not protect this table

2 1

error opening DB on External drive
by Lynn Carol Johnson 14 Jan '16

14 Jan '16

Hello - I am running monetdb with the July2015 service pack on MAC OSX. I have no problem running and connecting to a dbfarm that lives on my MAC. But I want to test a very large data set so have created a dbfarm on a Samsung T1 SSD external drive that is USB connected to my MAC. I have successfully started monetdbd server and have attached to my dbfarm. I created/released a new database. Yet when I attempt to run mClient it fails. Here is the "get all" command: rs-btlcj34lt:testMapiSock lcj34$ monetdbd get all /Volumes/Samsung_T1/alt-dbfarm property value hostname rs-btlcj34lt.maize.cornell.edu dbfarm /Volumes/Samsung_T1/alt-dbfarm status monetdbd[740] 1.7 (Jul2015-SP1) is serving this dbfarm mserver /Users/lcj34/development/usr/local/monetdb/bin/mserver5 logfile /Volumes/Samsung_T1/alt-dbfarm/merovingian.log pidfile /Volumes/Samsung_T1/alt-dbfarm/merovingian.pid sockdir /tmp port 50000 exittimeout 60 forward proxy discovery true discoveryttl 600 control no passphrase <unknown> mapisock /tmp/.s.monetdb.50000 controlsock /tmp/.s.merovingian.50000 rs-btlcj34lt:testMapiSock lcj34$ Here is the command line error: rs-btlcj34lt:alt-dbfarm lcj34$ mclient -u monetdb -d maizeFullGenomeDB password: monetdbd: internal error while starting mserver, please refer to the logs And here is the error from the log file. 2016-01-14 10:33:26 MSG maizeFullGenomeDB[749]: # Visit http://www.monetdb.org/ for further information 2016-01-14 10:33:26 MSG maizeFullGenomeDB[749]: !IOException:mal_mapi.listen:operation failed: binding to UNIX socket file /Volumes/Samsung_T1/alt-dbfarm/maizeFullGenomeDB/.mapi.sock failed: No such file or directory If I switch back to a dbfarm local to the MAC, it works fine. Any ideas why I can't connect to a dbfarm on the external drive? Is there something else that needs to be setup for the socket creation to work? Thanks - Lynn

3 3

Error on database startup
by Robin Cijvat 13 Jan '16

13 Jan '16

Hello everyone! Every now and then my laptop fails to wake up from standby mode and I have to do a hard reset, while monetdb is running in the background. I have had the following problem multiple times lately when trying to start the database that was stopped during the hard reset: monetdb start tpch starting database 'tpch'... FAILED start: starting 'tpch' failed: database 'tpch' did not initialise the sql scenario This is from the log: 2016-01-13 07:22:38 MSG tpch[7085]: # Start processing logs sql/sql_logs version 52200 2016-01-13 07:22:38 ERR tpch[7085]: !mvc_init: unable to create system tables 2016-01-13 07:22:38 MSG tpch[7085]: !SQLException:SQLinit:Catalogue initialization failed 2016-01-13 07:22:38 MSG tpch[7085]: !ERROR: Could not read the version number from the file 'sql_logs/sql//log'. 2016-01-13 07:22:38 MSG merovingian[7066]: database 'tpch' (7085) has exited with exit status 0 2016-01-13 07:22:38 MSG merovingian[7066]: database 'tpch' has shut down 2016-01-13 07:22:38 ERR control[7066]: (local): failed to fork mserver: database 'tpch' did not initialise the sql scenario Now I don't keep any important information in this database, so for me it is fine to reload everything. However, since this is a recurring problem, I wondered if anyone has any suggestions as to how to fix such a 'broken' database.

1 0