March 2015 - users-list

Re: Multi threaded (2 threads) Java program for copy command
by Shivanandan Gupta 13 Mar '15

13 Mar '15

> > Hi Martin, > Thanks for your prompt reply , I am using two different tables here but still I am getting "Exception in thread "main" java.sql.SQLException: COMMIT: failed" exception. > Message: 2 > Date: Thu, 12 Mar 2015 08:32:31 +0100 > From: Martin Kersten <martin(a)monetdb.org> > To: users-list(a)monetdb.org > Subject: Re: Multi threaded (2 threads) Java program for copy command > commit fails for one thread > Message-ID: <5501410F.1070508(a)monetdb.org> > Content-Type: text/plain; charset=windows-1252; format=flowed > > Hi, > > Yes. Insertion into the same table would be invalidated due to the > transaction semantics. > See > https://www.monetdb.org/Documentation/Manuals/SQLreference/Transactions > > On 12/03/15 08:18, Shivanandan Gupta wrote: > > Hi All, > > > > I am evaluating monetdb for our dwh and I am trying to load data from > two different csv file to 2 different tables using multi-threaded java > program in parallel, for one thread the load is successful and other one is > failing with exception given below. > > > > Exception in thread "main" java.sql.SQLException: COMMIT: failed > > at > nl.cwi.monetdb.jdbc.MonetConnection$ResponseList.executeQuery(MonetConnection.java:2535) > > at > nl.cwi.monetdb.jdbc.MonetConnection$ResponseList.processQuery(MonetConnection.java:2284) > > at nl.cwi.monetdb.jdbc.MonetConnection.commit(MonetConnection.java:374) > > at connect_monetdb.runETL(connect_monetdb.java:78) > > at connect_monetdb.main(connect_monetdb.java:14) > > > > > > command used is : > > > > String csql = "copy into s_asset_"+filename+" from '"+filepath+""+ > filename +".csv' using delimiters',';"; > > st.execute(csql); > > con.commit(); > > > > > > I am using monetdb jdbc driver and monetdb server is also on the same > machine. > > > > Any help will be really great. > > > > Thank you. > > > > Cheers, > > Shivanandan Gupta > > > > > > > > _______________________________________________ > > users-list mailing list > > users-list(a)monetdb.org > > https://www.monetdb.org/mailman/listinfo/users-list > > > > > >

2 2

Multi threaded (2 threads) Java program for copy command commit fails for one thread
by Shivanandan Gupta 12 Mar '15

12 Mar '15

Hi All, I am evaluating monetdb for our dwh and I am trying to load data from two different csv file to 2 different tables using multi-threaded java program in parallel, for one thread the load is successful and other one is failing with exception given below. Exception in thread "main" java.sql.SQLException: COMMIT: failed at nl.cwi.monetdb.jdbc.MonetConnection$ResponseList.executeQuery(MonetConnection.java:2535) at nl.cwi.monetdb.jdbc.MonetConnection$ResponseList.processQuery(MonetConnection.java:2284) at nl.cwi.monetdb.jdbc.MonetConnection.commit(MonetConnection.java:374) at connect_monetdb.runETL(connect_monetdb.java:78) at connect_monetdb.main(connect_monetdb.java:14) command used is : String csql = "copy into s_asset_"+filename+" from '"+filepath+""+ filename +".csv' using delimiters',';"; st.execute(csql); con.commit(); I am using monetdb jdbc driver and monetdb server is also on the same machine. Any help will be really great. Thank you. Cheers, Shivanandan Gupta

2 1

Re: Problem with Datacell
by Julien Marie 10 Mar '15

10 Mar '15

I managed to make it work a bit more by installing Jan-2014 version of MonetDB. I can send data into datacell, I can send data out of datacell if I insert directly into the bsktout. But the queries are not working : +----------------+---------+---------------------------+--------+------+----------+----------------------------+----------------------------+ | nme | status | lastrun | cycles | even | time | error | def | : : : : : ts : : : : +================+=========+===========================+========+======+==========+============================+============================+ | datacell.passd | running | null | 980245 | 0 | 17780876 | Query datacell.passd faile | insert into datacell.bout | : : : : : : : d:SQLException:mvc:SQL mod : select * from datacell.bbi : : : : : : : : ule not initialized : n; : +----------------+---------+---------------------------+--------+------+----------+----------------------------+----------------------------+ Is there a version or release that was working ? Julien Marie > Message: 1 > Date: Mon, 9 Mar 2015 09:55:06 +0800 > From: Julien Marie <jm(a)producture.com> > To: users-list(a)monetdb.org > Subject: Problem with Datacell > Message-ID: > < > CABmP2VKm0csQTzcUcgwiusw9o52qxmbmrZ7PM+auxoj7skdrCw(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > I know the feature is deemed experimental, but I have troubles just to > start the example and I keep on having transaction error messages : > > sql>call datacell.receptor('datacell.bsktin', 'localhost', 50500); > sql>call datacell.emitter('datacell.bsktout', 'localhost', 50600); > START TRANSACTION: cannot start a transaction within a transaction > > What is going on ? > > Thank you very much > > -- > *Julien Marie* > > jm(a)producture.com > skype : thewiseup > +63 917 511 8258 > > >

1 0

Horizontal partitioning
by Bart Lenoir 09 Mar '15

09 Mar '15

Hi All, Does MonetDB besides load balancing (round robin) in its clustering also support horizontal partitioning? If so, how? Thanks. Cheers, John

1 0

Problem with Datacell
by Julien Marie 09 Mar '15

09 Mar '15

I know the feature is deemed experimental, but I have troubles just to start the example and I keep on having transaction error messages : sql>call datacell.receptor('datacell.bsktin', 'localhost', 50500); sql>call datacell.emitter('datacell.bsktout', 'localhost', 50600); START TRANSACTION: cannot start a transaction within a transaction What is going on ? Thank you very much -- *Julien Marie* jm(a)producture.com skype : thewiseup +63 917 511 8258 -----BEGIN PGP PUBLIC KEY BLOCK----- xsBNBFNmiRIBCAC77MQjKyaDaRo9bQhfGNXIDK3SldtI+A56EAY2xh8/WCrJ HYwhdwW/2aConIa5eL92WLf0W6FSmjk0rqSp1bkkDt8RW7JFiftCACZkSwHF Rg3zpUeMgFG3BBRpwCiOxSygDza61AmMAQ+8LuLZOETEFddT/DQNCBouZLYN KE9uCs+lYkrpc3YbPfx+kGUDj5rT2SYnHwt4r352OjT1U8ABQdxBigymmzgT rZh/4oyVXjMtBCRwAqwmVtzwoNoarAcgsm6qQgNwTdy5NtmKlgs2hqQtIkQK 3uL+e3zyWb1B8e6gUhpn5cooeJG1hvlJ65XUTHgHOCCz7Ue4q1rl/5kZABEB AAHNIEp1bGllbiBNYXJpZSA8am1AcHJvZHVjdHVyZS5jb20+wsByBBABCAAm BQJTZokZBgsJCAcDAgkQ7nj49Sxgl2gEFQgCCgMWAgECGwMCHgEAANulB/4u c+UeSeo8Wn8COAKSoMyWh2m5zBEdRdTlELVNcQIH3ssqu1M3TEpfBixVofp2 NPYKNF7FrQPl7z9+w0azlVJVIDF6DlGWjVlKm1nDTlVr4gW8J6mgRYivV3Tp HlBHIFmelOIcBpJAhUfc73H5a9BAsJ/a2Iekpc0TERsnarVuA3sqeTqPkwBY komURJWx0yxiYj91Be1PDRRi+DyGBKa8AeapXiCmCtpGAxksB43iwiiFnLev iZCWp3bKeBCLZy5IkONNU6mNzTcljrgHzRjs9P8OSiEIJ9T8tubHUkkA0LsR 9l7vmBk3aESNF8w1LmRTRN+gBkXsMSjzNilGj7zBzsBNBFNmiRkBCACOsST4 AuZffy8Ny7w1Qhvo1Okt9bBoh5zIJKE0tEo1L4YmDYALKnagVHNUwiAM26Bv ynK+pAvRopt62Wu+s1hQgHNuwnTMuuL0OW4Y8daOIN9pUAqnT/etskSwH24W niP445IEa09RyA1b2Mwhy8hEN/eCFHmnY36U5+qiiUAjN6Khsflzkq4I2Qy9 wwMVhNC8/J1QtmODcUekMgnGG0XcXNm22N1EU3llr5yG/oYD0dAKnIIv4rf8 3s6LU7ND/SOjmaLNtzwLuGawcCBPGfaBpB77ozJz0jJ3RaEwuY/Kqt4Zfp5T 0xLFgr+fXxSjHku0qU6PGT3iH+eHkWZXGG2pABEBAAHCwF8EGAEIABMFAlNm iRwJEO54+PUsYJdoAhsMAADWtwf9HJiym4Oi2zwWiDksmLc7699iVY1pvuic RlqwMw2bmFd+PUUFupU6CR8jOTcYPJm3NzZ9gwPM3yzjd/BReWmd4mfZJfvf EIYHzJoDRmnT5t8b9Wp6jllWtVAHbN9FTUW3uJODDKhffhYNbmAhxpotao+a XQJcSrcWE1Gl030ZQryOfkz2l/eUfYYSK38gsR7ksz22Gb1CKS4gmG8jgxQs vmDTSRagUYEXOdTaX5HKMdHrMw1ll1Z8bMb/y0N0kgDmIF0pAX2i+bk5UpnU 6mbeLCwo0OoIPYpC4GvGPBglull+BRJlwvf9Kpzlip5KHYSTnb19yyjbs1qC rpsqrnUA6g== =o6AX -----END PGP PUBLIC KEY BLOCK-----

1 0

Clustering
by Jiří Nádvorník 03 Mar '15

03 Mar '15

Hi All, I'm trying to solve a clustering problem on my astronomical data. It is simply repeated observations of stars on images with random errors, effectively creating clusters with Gaussian distributions. These clusters have from 1 to cca 1000 points and are approximately of the same size (~1 arcsec). So much for the introduction. I already tried to solve this problem with simple methods but with poor results. Our regions are very dense and the size of the clusters is often (cca 20 % of the observations) comparable to their distances. Overlaps are, of course, happening too. So I decided to use a more robust algorithm and compare the results. Due to the size of our dataset (cca 4e8 observations, cca 8e6 real objects), I am also counting on massive parallelization of the algorithm, focusing on saving real time. I have one simple question for you. What algorithm would you use for the clustering? I would like to use K-means or EM Gaussian mixtures, because the data is naturally Gaussian distributed. But then I would have to foretell the number of clusters - what algorithm would you use there? Or should I use some more complicated Bayesian algorithm which does not need the number of clusters? Thank you very much for your answers. Jiri Nadvornik Astronomical Institute AV CR Stellar department Czech Republic <mailto:nadvornik.ji@gmail.com> nadvornik.ji(a)gmail.com

2 1

Converting Bigint field (containing unixtime in millisecs) to TIMESTAMP
by Davide Magatti 02 Mar '15

02 Mar '15

Hi all, Database: MonetDB v11.19.9 (Oct2014-SP2) I am trying to convert a big string fiel in a table containing unixtimes in millisecs to timestamps but i cannot find the right way to do it. So my table descriptor: TABLE "table" ( "user" VARCHAR(512), "time" BIGINT, [...] ); i tried to do > select CAST( CAST(time/1000 AS INTERVAL SECOND) AS TIMESTAMP) from table ; >>types sec_interval(13,0) and timestamp(7,0) are not equal And did not work, also using time in millisec the thing does not change. I know i am doing something wrong but i cannot get it. Could you please advise me? :) thx! Davide

2 1

SQL scenario initialization error
by Doug Service 02 Mar '15

02 Mar '15

A monet database we have created will not start with the following error: sudo monetdb start dbprod starting database 'dbprod'... FAILED start: starting 'dbprod' failed: database 'dbprod' did not initialise the sql scenario Is it possible to recover from an error like this? Here is the relevant log file portion. 2015-03-01 16:11:36 MSG dbprod[5743]: arguments: /usr/bin/mserver5 --dbpath=/var/lib/monetdb/dbprod --set merovingian_uri=mapi:monetdb://qbdata:42000/dbprod --set mapi_open=false --set mapi_port=0 --set mapi_usock=/var/lib/monetdb/dbprod/.mapi.sock --set monet_vault_key=/var/lib/monetdb/dbprod/.vaultkey --set gdk_nr_threads=2 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2015-03-01 16:11:36 ERR dbprod[5743]: !mvc_init: unable to create system tables 2015-03-01 16:11:36 MSG dbprod[5743]: # MonetDB 5 server v11.19.7 "Oct2014-SP1" 2015-03-01 16:11:36 MSG dbprod[5743]: # Serving database 'dbprod', using 2 threads 2015-03-01 16:11:36 MSG dbprod[5743]: # Compiled for x86_64-pc-linux-gnu/64bit with 64bit OIDs dynamically linked 2015-03-01 16:11:36 MSG dbprod[5743]: # Found 1.955 GiB available main-memory. 2015-03-01 16:11:36 MSG dbprod[5743]: # Copyright (c) 1993-July 2008 CWI. 2015-03-01 16:11:36 MSG dbprod[5743]: # Copyright (c) August 2008-2014 MonetDB B.V., all rights reserved 2015-03-01 16:11:36 MSG dbprod[5743]: # Visit http://www.monetdb.org/ for further information 2015-03-01 16:11:36 MSG dbprod[5743]: # Listening for UNIX domain connection requests on mapi:monetdb:///var/lib/monetdb/dbprod/.mapi.sock 2015-03-01 16:11:36 MSG dbprod[5743]: !SQLException:SQLinit:Catalogue initialization failed 2015-03-01 16:11:36 MSG dbprod[5743]: !ERROR: Could not read the version number from the file 'sql_logs/sql//log'. 2015-03-01 16:11:36 MSG merovingian[1568]: database 'dbprod' (5743) has exited with exit status 0 2015-03-01 16:11:36 MSG merovingian[1568]: database 'dbprod' has shut down 2015-03-01 16:11:36 ERR control[1568]: (local): failed to fork mserver: database 'dbprod' did not initialise the sql scenario 2015-03-01 16:12:45 MSG control[1568]: (local): served status list 2015-03-01 16:12:45 MSG control[1568]: (local): locked database 'dbprod' 2015-03-01 16:13:09 MSG control[1568]: (local): served status list 2015-03-01 16:13:09 MSG control[1568]: (local): released database 'dbprod' 2015-03-01 16:13:19 MSG control[1568]: (local): served status list 2015-03-01 16:13:19 MSG merovingian[1568]: starting database 'dbprod', up min/avg/max: 0s/2d/1w, crash average: 0.00 0.50 0.17 (15-10=5) 2015-03-01 16:13:20 MSG merovingian[1568]: sending process 5769 (database 'dbprod') the TERM signal

3 3