August 2016 - users-list

synchronize data from mysql performance
by wang.tong＠shomop.com 19 Aug '16

19 Aug '16

Hi, I try to synchronize data from mysql to monetdb. Steps: 1. Read the modified rows from mysql table periodically. 2. Delete the rows in monetdb by primary key. (To avoid 'PRIMARY KEY constraint violated' error.) The sql is like 'delete from a where id in (1, 2, 3)'. 3. Insert the rows into monetdb by 'copy into ... from stdin'. I find step 2 may be too slow. When there are 2 million rows, delete 10,000 rows takes about 1 minute. When there are 30 million rows, delete 10,000 rows takes about 20 minute. I have to do join on the 30 million rows table, so I cannot split it. I try to upgrade the server from 2 core 8 G memory to 8 core 32 G memory, but it has no effect. It seems the limitation is CPU, and the deleted operation can only use 1 core. How can I optimize the performance? Is there any better approach to synchronize data? Thanks in advance! OS version: centos-release-7-0.1406.el7.centos.2.5.x86_64 mserver5 --version MonetDB 5 server v11.23.7 "Jun2016-SP1" (64-bit, 64-bit oids, 128-bit integers) Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008-2016 MonetDB B.V., all rights reserved Visit http://www.monetdb.org/ for further information Found 31.0GiB available memory, 8 available cpu cores Libraries: libpcre: 8.32 2012-11-30 (compiled with 8.32) openssl: OpenSSL 1.0.1e 11 Feb 2013 (compiled with OpenSSL 1.0.1e-fips 11 Feb 2013) libxml2: 2.9.1 (compiled with 2.9.1) Compiled by: root@jstu87d9w6zj4u (x86_64-unknown-linux-gnu) Compilation: gcc -g -O2 Linking : /usr/bin/ld -m elf_x86_64 wang.tong(a)shomop.com

2 2

database was killed by signal SIGSEGV
by 王彤 19 Aug '16

19 Aug '16

Hi, I try to synchronize data from mysql to monetdb. Total rows is about 15 million. I insert the rows in batch. Each batch has about 10000 rows. I use 'copy into from stdin' as in http://dev.monetdb.org/hg/MonetDB/file/tip/java/example/SQLcopyinto.java . The virtual memory continuously grows. When it grows to about 80G, the server is killed by signal SIGSEGV. The database size on disk is about 30GB. Below is the server infomation and error log. Any help would be greatly appreciated! OS version: centos-release-7-0.1406.el7.centos.2.5.x86_64 mserver5 --version MonetDB 5 server v11.23.7 "Jun2016-SP1" (64-bit, 64-bit oids, 128-bit integers) Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008-2016 MonetDB B.V., all rights reserved Visit http://www.monetdb.org/ for further information Found 7.4GiB available memory, 2 available cpu cores Libraries: libpcre: 8.32 2012-11-30 (compiled with 8.32) openssl: OpenSSL 1.0.1e 11 Feb 2013 (compiled with OpenSSL 1.0.1e-fips 11 Feb 2013) libxml2: 2.9.1 (compiled with 2.9.1) Compiled by: root@jstu87d9w6zj4u (x86_64-unknown-linux-gnu) Compilation: gcc -g -O2 Linking : /usr/bin/ld -m elf_x86_64 in /data/dbfarm/merovingian.log 2016-08-18 17:29:04 ERR shomopcrm[10246]: #GDKmmap(38862848) fails, try to free up space [memory in use=98280928,virtual memory in use=3779503584] 2016-08-18 17:29:04 ERR shomopcrm[10246]: #GDKmmap(38862848) result [mem=98280928,vm=3779503584] 2016-08-18 17:29:04 MSG merovingian[10194]: proxying client 10.173.87.201:46953 for database 'shomopcrm' to mapi:monetdb:///data/dbfarm/shomopcrm/.mapi.sock?database=shomopcrm 2016-08-18 17:29:04 MSG merovingian[10194]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2016-08-18 17:29:04 MSG merovingian[10194]: proxying client 10.153.193.249:55915 for database 'shomopcrm' to mapi:monetdb:///data/dbfarm/shomopcrm/.mapi.sock?database=shomopcrm 2016-08-18 17:29:04 MSG merovingian[10194]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2016-08-18 17:29:04 ERR shomopcrm[10246]: #GDKmmap(38862848) fails, try to free up space [memory in use=185008352,virtual memory in use=3840344288] 2016-08-18 17:29:04 ERR shomopcrm[10246]: #GDKmmap(38862848) result [mem=185008352,vm=3840344288] 2016-08-18 17:29:04 MSG merovingian[10194]: proxying client 10.173.87.201:46963 for database 'shomopcrm' to mapi:monetdb:///data/dbfarm/shomopcrm/.mapi.sock?database=shomopcrm 2016-08-18 17:29:04 MSG merovingian[10194]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2016-08-18 17:29:04 ERR shomopcrm[10246]: #GDKmmap(51838976) fails, try to free up space [memory in use=99455712,virtual memory in use=3766391520] 2016-08-18 17:29:04 ERR shomopcrm[10246]: #GDKmmap(51838976) result [mem=99455712,vm=3766391520] 2016-08-18 17:29:04 MSG merovingian[10194]: database 'shomopcrm' (10246) was killed by signal SIGSEGV 2016-08-18 17:29:04 MSG merovingian[10194]: starting database 'shomopcrm', up min/avg/max: 17s/15h/5d, crash average: 1.00 0.90 0.40 (23-11=12) 2016-08-18 17:29:05 MSG shomopcrm[984]: arguments: /usr/local/bin/mserver5 --dbpath=/data/dbfarm/shomopcrm --set merovingian_uri=mapi:monetdb://jstu87d9w6zj4u:30001/shomopcrm --set mapi_open=false --set mapi_port=0 --set mapi_usock=/data/dbfarm/shomopcrm/.mapi.sock --set monet_vault_key=/data/dbfarm/shomopcrm/.vaultkey --set gdk_nr_threads=2 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2016-08-18 17:29:05 MSG shomopcrm[984]: # MonetDB 5 server v11.23.7 "Jun2016-SP1" 2016-08-18 17:29:05 MSG shomopcrm[984]: # Serving database 'shomopcrm', using 2 threads 2016-08-18 17:29:05 MSG shomopcrm[984]: # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs and 128bit integers dynamically linked 2016-08-18 17:29:05 MSG shomopcrm[984]: # Found 7.390 GiB available main-memory. 2016-08-18 17:29:05 MSG shomopcrm[984]: # Copyright (c) 1993-July 2008 CWI. 2016-08-18 17:29:05 MSG shomopcrm[984]: # Copyright (c) August 2008-2016 MonetDB B.V., all rights reserved 2016-08-18 17:29:05 MSG shomopcrm[984]: # Visit http://www.monetdb.org/ for further information 2016-08-18 17:29:05 MSG shomopcrm[984]: # Listening for UNIX domain connection requests on mapi:monetdb:///data/dbfarm/shomopcrm/.mapi.sock 2016-08-18 17:29:05 MSG shomopcrm[984]: # MonetDB/SQL module loaded in /var/log/messages, there is an error: kernel: mserver5[981]: segfault at 18 ip 00007fe10f845a89 sp 00007fcf38e79760 error 4 in lib_sql.so[7fe10f817000+175000] wang.tong(a)shomop.com

2 1

Distributed Query Processing questions
by Anton Kravchenko 18 Aug '16

18 Aug '16

Hi there, I have few questions on merge/remote tables. Are there known MonetDB users that use(d) merge/remote tables in production? Are there official MonetDB performance benchmarks for merge/remote vs local tables? Thank you, Anton

2 1

distributed query processing example
by Anton Kravchenko 16 Aug '16

16 Aug '16

Hi there, I am trying to follow a distributed query processing example from https://www.monetdb.org/Documentation/Cookbooks/SQLrecipes/DistributedQuery… and here is my experience: Welcome to mclient, the MonetDB/SQL interactive terminal (Jun2016-SP1) Database: MonetDB v11.23.7 (Jun2016-SP1), 'mapi:monetdb://hnode3.secdev.local:52018/db3' Type \q to quit, \? for a list of available commands auto commit mode: on sql>CREATE MERGE TABLE mt1 (t int); operation successful (2.367ms) sql>CREATE REMOTE TABLE t1 (i int) on 'mapi:monetdb://hnode1.secdev.local:52016/db1'; operation successful (1.287ms) sql>select * from t1; +------+ | i | +======+ | 11 | | 13 | +------+ 2 tuples (4.694ms) *sql>select count(*) from t1;* *Connection lost* *sql>select count(*) from t1;* *illegal input, not a JSON header (got '')* *sql>select count(*) from t1;* *Connection lost* *sql>select count(*) from t1;* *Connection lost* *sql>select count(*) from t1;* *illegal input, not a JSON header (got '')* *sql>select count(*) from t1;* *illegal input, not a JSON header (got '')* *sql>select count(*) from t1;* *an error occurred on connection: Connection lost* *sql>select count(*) from t1;* *Connection lost* sql>select * from t1; +------+ | i | +======+ | 11 | | 13 | +------+ 2 tuples (508.682ms) sql>select * from t1; +------+ | i | +======+ | 11 | | 13 | +------+ 2 tuples (6.227ms) sql>select * from t1; +------+ | i | +======+ | 11 | | 13 | +------+ 2 tuples (5.065ms) sql>select * from t1; +------+ | i | +======+ | 11 | | 13 | +------+ 2 tuples (4.178ms) sql> P.S. Is there something wrong with my monetdb cluster setup or is it a real bug? On hnode3 (CentOS 6.8 machine): [akravchenko@hnode3 ~]$ monetdbd -v MonetDB Database Server v1.7 (Jun2016-SP1) [akravchenko@hnode3 ~]$ monetdb discover location hnode3.secdev.local:52018* mapi:monetdb://hnode3.secdev.local:52018/db3 [akravchenko@hnode3 ~]$ monetdb status name state health remarks db3 R 20h 100% 11m mapi:monetdb://hnode3.secdev.local:52018/db3 [akravchenko@hnode3 ~]$ monetdbd get all hnode3-dbfarm property value hostname hnode3.secdev.local dbfarm hnode3-dbfarm status monetdbd[18052] 1.7 (Jun2016-SP1) is serving this dbfarm mserver /usr/bin/mserver5 logfile hnode3-dbfarm/merovingian.log pidfile hnode3-dbfarm/merovingian.pid sockdir /tmp listenaddr 0.0.0.0 port 52018 exittimeout 60 forward proxy discovery yes discoveryttl 600 control yes passphrase {SHA512}a73f1d86383446438ac64f56e15ada38b41fbb18f029d2181723aeb2acac6a831f60e5fdbd64ac2c8c70e035dd44cbbe3b45565ef2d58feb2821a2078c7fad35 mapisock /tmp/.s.monetdb.52018 controlsock /tmp/.s.merovingian.52018 and on hnode1 (CentOS 6.8 machine): [akravchenko@hnode1 ~]$ monetdb -v MonetDB Database Server Toolkit v1.1 (Jun2016-SP1) [akravchenko@hnode1 ~]$ monetdb discover location hnode1.secdev.local:52016* mapi:monetdb://hnode1.secdev.local:52016/db1 [akravchenko@hnode1 ~]$ monetdb status name state health remarks db1 R 4m 15% 32m mapi:monetdb://hnode1.secdev.local:52016/db1 [akravchenko@hnode1 ~]$ monetdbd get all hnode1-dbfarm/ property value hostname hnode1.secdev.local dbfarm hnode1-dbfarm/ status monetdbd[1355] 1.7 (Jun2016-SP1) is serving this dbfarm mserver /usr/bin/mserver5 logfile hnode1-dbfarm//merovingian.log pidfile hnode1-dbfarm//merovingian.pid sockdir /tmp listenaddr 0.0.0.0 port 52016 exittimeout 60 forward proxy discovery yes discoveryttl 600 control yes passphrase {SHA512}a73f1d86383446438ac64f56e15ada38b41fbb18f029d2181723aeb2acac6a831f60e5fdbd64ac2c8c70e035dd44cbbe3b45565ef2d58feb2821a2078c7fad35 mapisock /tmp/.s.monetdb.52016 controlsock /tmp/.s.merovingian.52016 [akravchenko@hnode1 ~]$ Thank you, Anton

1 0

Add a new optimizer pipeline.
by Xu,Wenjian 10 Aug '16

10 Aug '16

Hi all, MonetDB version: 11.23.7 Ubuntu version: 14.04 I created a new optimizer pipeline by adding an additional optimization step after 'default_pipe', i.e.: '*my_customized_pipe*'='optimizer.inline();optimizer.candidates(); optimizer.remap();optimizer.costModel();optimizer.coercions(); optimizer.evaluate();optimizer.pushselect();optimizer.aliases(); optimizer.mitosis();optimizer.mergetable();optimizer.deadcode(); optimizer.aliases();optimizer.constants();optimizer.commonTerms(); optimizer.projectionpath();optimizer.deadcode();optimizer.reorder(); optimizer.reduce();optimizer.matpack();optimizer.dataflow(); optimizer.querylog();optimizer.multiplex();optimizer.generator(); optimizer.profiler();optimizer.garbageCollector(); *optimizer.customizedstep()*' What I have done is: 1) Create an empty optimizer step: -- 1.1 add *opt_customizedstep.c/h* in package *monetdb/optimizer/**, leave function body *OPTcustomizedstepImplementation() *empty*, *i.e. with one statement **return 0** * -- *1.2 register the step by modifying *opt_wrapper.c *and* optimizer.mal* 2) Add the definition of the new pipeline in *opt_pipes.c* 3) Modify the configuration file to compile the added source files. After re-build the system, I can enable the new optimizer pipeline by *set optimizer='my_customized_pipe'*. However, I have two issues: i) using EXPLAIN statement, I found the MAL plan different from the one generated by 'default_pipe' (which should be the same since I did nothing in the added step?). Specifically, the plan seems to be not affected by *optimizer.reorder() *step, i.e., bat.new() and bat.append() instructions appear in the end of the plan. ii) then I entered mdb mode by DEBUG statement. But it seems that some command did not work as before. For example, *help* command does not display all commands available; *optimizer* command does not display actions performed by each optimizer step. What's wrong with my procedure to add a new optimizer pipeline? Any comment would be helpful. Best regards, Wenjian

3 3

[Low level] Understanding duplicate entries in delta BAT (delete BAT)
by Frédéric Jolliton 08 Aug '16

08 Aug '16

Hi, As per MonetDB Solution suggestion, we are asking here about this topic. We are studying how MonetDB store its database on disk. So far we were able to understand pretty much all the files (BAT, journal, BBP.dir, heap, hash, imprints, and so on). We have custom tools to decode and dump them (experimental at this point). There is one thing however that is unclear: we thought that "delta BAT", which are the BAT assigned to an SQL tables to list the OIDs of rows that were deleted (named "D_<schema>_<table>" internaly), would contains only *uniques* values (since you can never delete twice the same row). But on several databases that we are running, we found that some of theses BATs contain duplicates. Lot of duplicates actually. Especially on D_sys__columns and D_sys__tables BATs (respectively for the sys._columns and sys._tables system tables). Some databases do not have this "issue" (while they all have the same schema, and process the same kind of data as the other ones). Can someone explain if duplicates are expected in theses BATs ? Here is an excerpt of D_sys__tables: # hexdump -C 02/210.tail 00000000 2e 00 00 00 00 00 00 00 2f 00 00 00 00 00 00 00 |......../.......| 00000010 30 00 00 00 00 00 00 00 31 00 00 00 00 00 00 00 |0.......1.......| 00000020 32 00 00 00 00 00 00 00 33 00 00 00 00 00 00 00 |2.......3.......| 00000030 34 00 00 00 00 00 00 00 35 00 00 00 00 00 00 00 |4.......5.......| 00000040 36 00 00 00 00 00 00 00 37 00 00 00 00 00 00 00 |6.......7.......| 00000050 42 00 00 00 00 00 00 00 43 00 00 00 00 00 00 00 |B.......C.......| 00000060 44 00 00 00 00 00 00 00 45 00 00 00 00 00 00 00 |D.......E.......| 00000070 46 00 00 00 00 00 00 00 47 00 00 00 00 00 00 00 |F.......G.......| 00000080 48 00 00 00 00 00 00 00 49 00 00 00 00 00 00 00 |H.......I.......| 00000090 4a 00 00 00 00 00 00 00 4b 00 00 00 00 00 00 00 |J.......K.......| 000000a0 4c 00 00 00 00 00 00 00 4d 00 00 00 00 00 00 00 |L.......M.......| 000000b0 4e 00 00 00 00 00 00 00 4f 00 00 00 00 00 00 00 |N.......O.......| 000000c0 50 00 00 00 00 00 00 00 50 00 00 00 00 00 00 00 |P.......P.......| 000000d0 51 00 00 00 00 00 00 00 50 00 00 00 00 00 00 00 |Q.......P.......| 000000e0 51 00 00 00 00 00 00 00 52 00 00 00 00 00 00 00 |Q.......R.......| 000000f0 50 00 00 00 00 00 00 00 51 00 00 00 00 00 00 00 |P.......Q.......| 00000100 52 00 00 00 00 00 00 00 53 00 00 00 00 00 00 00 |R.......S.......| 00000110 50 00 00 00 00 00 00 00 51 00 00 00 00 00 00 00 |P.......Q.......| 00000120 52 00 00 00 00 00 00 00 53 00 00 00 00 00 00 00 |R.......S.......| ... You can clearly see duplicates OID (the first one being 0x50 at 0xc8). Its entry in BBP.dir (split into sections): 136 32 tmp_210 tmpr_210 02/210 610523782 2 171807 0 0 171807 172032 0 0 0 0 void 0 1 1793 0 0 0 0 0 1000651 0 0 0 oid 8 0 1024 24 25 27 1 46 773603235 1374456 1376256 1 We are using: MonetDB 5 server v11.21.14 (64-bit, 64-bit oids, 128-bit integers). -- Frédéric Jolliton Sécuractive

4 3

Does R run in a sandbox in MonetDB/R?
by Pablo S 07 Aug '16

07 Aug '16

MonetDB allows to create R functions that run embedded in the database. For example, you can do CREATE FUNCTION rapi00() RETURNS TABLE (d INTEGER) LANGUAGE R { seq(1,10); }; Now, I need my users to create dynamically these user defined functions by entering R statements on a screen; the application will create the functions in the database and run them. Problem is that R allows to run operating system level commands, such as list.files(path=".") or even file.remove("some_file") Will MonetDB allow these external interactions with the operating system, or it runs in a sandbox?

1 0

How could I obtain the running time information in the MAL interface of mclient?
by Xu,Wenjian 07 Aug '16

07 Aug '16

Hi all, MonetDB version: 11.23.7 Ubuntu version: 14.04 How could I obtain the running time information in the MAL interface of mclient? I tried to use the *--interactive* option but it did not work. Specifically, I do the following: =============================================== $ *mclient -dnba2009 -lmsql --interactive=ms* Welcome to mclient, the MonetDB interactive terminal (Jun2016-SP1) Type \q to quit, \? for a list of available commands msql>\<query.txt // I save the MAL instructions of a query (obtained from EXPLAIN statement) in this file msql>user.s1_3("1998"); // call the function of that query % sys.player, sys.player, sys.player # table_name % p_lastname, p_lastseason, p_weight # name % char, int, int # type % 100, 4, 3 # length [ "Wheeler", 1998, 165 ] [ "Baker", 1998, 175 ] [ "Garnett", 1998, 186 ] ...... ======================================================== I would like to obtain the running time information after getting the result of that query. How can I do it? Any help would be appreciated. Best, Wenjian

4 6

Strange behaviour with optimizer.pushselect and batalgebra.like
by Moritz Bruder 05 Aug '16

05 Aug '16

Hi, I get a really strange behaviour with the *pushselect* optimizer for this piece of MAL code: http://codepad.org/bqulPEtV . The problem I'm having is, that depending on whether the *pushselect* optimizer is used or not, the query will yield different results. Context info: 1. MonetDB version number: 11.23.3 2. Operating system platform: Linux 4.6.4-1-ARCH #1 SMP PREEMPT Mon Jul 11 19:12:32 CEST 2016 x86_64 GNU/Linux 3. Compiled from source tarball 4. http://codepad.org/bqulPEtV Regards, Moritz

1 0

Using MonetDB with Scala/Slick
by Pablo S 05 Aug '16

05 Aug '16

I develop in Scala, and need to access a MonetDB database. I use the Scala standard, Slick, however Slick doesn't have a driver to access MonetDB. I'm interested in developing a Slick/MonetDB driver (it's a JDBC wrapper) but I'd like to know first if someone already started the process (we could join forces). Please let me know. Thanks.

1 0