March 2015 - users-list

Better understand those MAL functions
by Leon-Constantin Fopa 29 Mar '15

29 Mar '15

hello dear all, I'm trying to understand an execution plan generated in MAL by monetDB. I would like to how exactlly the following functions works : - sql.mvc - algebra.subselect - sql.bind - sql.subdelta - algebra.leftfetchjoin Best regards,

1 0

Arrays in MonetDB
by Jiří Nádvorník 27 Mar '15

27 Mar '15

Hi, Do I understand correctly that MonetDB does not support array types? What's the best possibility to use instead of them? I have a table of pixels and table of points. For each pixel.ID I have two different arrays of points to store in the DB. Thank you for clarification. Best regards, Jiri Nadvornik Astronomical Institute AV CR Stellar department Czech Republic <mailto:nadvornik.ji@gmail.com> nadvornik.ji(a)gmail.com

1 0

MonetDB user defined function grouping
by Jiří Nádvorník 26 Mar '15

26 Mar '15

Hi all, I am trying to do a user defined function written in C++ which takes the whole table as a parameter, loads it into a vector structure, and then divides the data in some groups (duplicities will occur). How can I return the results of this grouping from the C function? Probably I'd like to store the group IDs in a separate table and create and then create an M:N table between the group IDs and the original records. Any help appreciated. Best regards, Jiri Nadvornik Astronomical Institute AV CR Stellar department Czech Republic <mailto:nadvornik.ji@gmail.com> nadvornik.ji(a)gmail.com

1 0

HEALPix with monetDB
by Jiří Nádvorník 25 Mar '15

25 Mar '15

Hi all, I'm planning to index my spherical data in MonetDB byHEALPix. I will need it for: 1. Data partitioning - distribute the data uniformly on the DB nodes, but keep them sorted (keep points close on the sphere on the same node). 2. Neighbor lookup - can I actually index my spherical coordinates stored in MonetDB by some kind of a BTree of HEALPix pixels? I understand how computing a hash for the coordinates and indexing on it helps in relational DB, but how do the indices function in an array DB like MonetDB? I'd like to benefit from the array storage of the data, because I will often look for nearest neighbors and compute distances between these neighbors. I will also divide the data in the beginning for parallel tasks by some calculation of their pixels, but that will be done only once in the beginning, the main computation will be around clustering these objects, which means lots and lots of neighbor searches and distance computation. Has somebody already tried to use HEALPix, or some other spherical index in MonetDB? How can I control the distribution of data across the nodes (e.g. sort them by this index)? How can I use and benefit from this kind of indexing in MonetDB? Thank you very much for even some questions answered. Best regards, Jiri Nadvornik Astronomical Institute AV CR Stellar department Czech Republic nadvornik.ji(a)gmail.com

1 0

Case insensitive LIKE ??
by Vijay Krishna 23 Mar '15

23 Mar '15

Hi, By default the LIKE criteria of MonetDB returns results by matching the exact case of the search word (MonetDB Jan 2014 SP1 Mac). How to make the LIKE criteria to do a case insensitive search by default?? (Something like the BINARY keyword that does case sensitive search in MySQL??) I don't wish to add a workaround like "LOWER(column) like LOWER(searchword)" here.. Any help much appreciated. Thanks in advance. Thanks & Regards, Vijayakrishna.P. Mobile : (+91) 9500402305.

2 2

How to tell MonetDB that data are sorted in the table?
by Leon-Constantin Fopa 19 Mar '15

19 Mar '15

Hi Given a table `data` (as described below) and given the insurance that that values imported into this table are sorted. CREATE TABLE data (INT value) CREATE INDEX index_value ON data (value) Since MonetDB can neglect the `CREATE INDEX` statement (see details here : https://www.monetdb.org/Documentation/Manuals/SQLreference/Indices). How do I tell MonetDB that data are sorted in order to let him improves lookup operations ? Best regards, Fopa Léon Constantin

2 3

Datacell issue on Jan2014-SP3 build
by Tsai Li Ming 18 Mar '15

18 Mar '15

Hi, We are trying data cell on the Jan2014-SP3 build. There is a “Query datacell.pass failed:SQLException:mvc:SQL module not initialised” error. Please see below: Welcome to mclient, the MonetDB/SQL interactive terminal (Jan2014-SP3) Database: MonetDB v11.17.21 (Jan2014-SP3), 'mapi:monetdb://localhost.localdomain:50000/foo' Type \q to quit, \? for a list of available commands auto commit mode: on sql>set optimizer='datacell_pipe'; operation successful (1.465ms) sql> sql>create table datacell.bsktin( more> id integer, more> tag integer, more> payload integer more>); operation successful (325.852ms) sql>create table datacell.bsktout (like datacell.bsktin); operation successful (106.356ms) sql> sql>call datacell.receptor('datacell.bsktin','localhost',50500); sql> sql>call datacell.emitter('datacell.bsktout','localhost',50600); sql> sql>call datacell.query('datacell.pass', 'insert into datacell.bsktout select * from datacell.bsktin;'); sql> sql>select * from datacell.receptors(); select * from datacell.emitters(); select * from datacell.queries(); select * from datacell.baskets(); +-----------------+-----------+-------+----------+---------+--------+----------------------------+--------+----------+---------+ | nme | host | port | protocol | mode | status | lastseen | cycles | received | pending | +=================+===========+=======+==========+=========+========+============================+========+==========+=========+ | datacell.bsktin | localhost | 50500 | TCP | passive | paused | null | 0 | 0 | 0 | +-----------------+-----------+-------+----------+---------+--------+----------------------------+--------+----------+---------+ 1 tuple (10.546ms) +------------------+-----------+-------+----------+--------+--------+----------------------------+--------+------+---------+ | nme | host | port | protocol | mode | status | lastsent | cycles | sent | pending | +==================+===========+=======+==========+========+========+============================+========+======+=========+ | datacell.bsktout | localhost | 50600 | UDP | active | paused | null | 0 | 0 | 0 | +------------------+-----------+-------+----------+--------+--------+----------------------------+--------+------+---------+ 1 tuple (10.780ms) +---------------+--------+----------------------------+--------+--------+------+-------+-------------------------------------------------------------+ | nme | status | lastrun | cycles | events | time | error | def | +===============+========+============================+========+========+======+=======+=============================================================+ | datacell.pass | paused | null | 0 | 0 | 0 | | insert into datacell.bsktout select * from datacell.bsktin; | +---------------+--------+----------------------------+--------+--------+------+-------+-------------------------------------------------------------+ 1 tuple (15.348ms) +------------------+-----------+---------+-----------+-----------+------------+------+----------------------------+--------+ | nme | threshold | winsize | winstride | timeslice | timestride | beat | seen | events | +==================+===========+=========+===========+===========+============+======+============================+========+ | datacell.bsktin | 0 | 0 | 0 | 0 | 0 | 0 | null | 0 | | datacell.bsktout | 0 | 0 | 0 | 0 | 0 | 0 | null | 0 | +------------------+-----------+---------+-----------+-----------+------------+------+----------------------------+--------+ 2 tuples (15.704ms) sql> sql>call datacell.resume(); sql>call datacell.dump(); invalid/unknown response from server, ignoring output ? #baskets[ 1] datacell.bsktin columns 3 threshold 0 window=[0,0] time window=[0,0] beat 0 milliseconds events 0 ? #baskets[ 2] datacell.bsktout columns 3 threshold 0 window=[0,0] time window=[0,0] beat 0 milliseconds events 0 ? #receptor datacell.bsktin at localhost:50500 protocol=TCP mode=passive status=running delay=1000 ? #emitter datacell.bsktout at localhost:50600 protocol=UDP mode=active status=running delay=1000 ? #scheduler status running (remaining output omitted, use \fraw to examine in detail) We then ran nc and sensor program but get nothing on the nc output: sql>select * from datacell.receptors(); select * from datacell.emitters(); select * from datacell.queries(); select * from datacell.baskets(); +-----------------+-----------+-------+----------+---------+---------+----------------------------+--------+----------+---------+ | nme | host | port | protocol | mode | status | lastseen | cycles | received | pending | +=================+===========+=======+==========+=========+=========+============================+========+==========+=========+ | datacell.bsktin | localhost | 50500 | TCP | passive | running | 2015-03-12 09:58:36.000000 | 1 | 1000 | 1000 | +-----------------+-----------+-------+----------+---------+---------+----------------------------+--------+----------+---------+ 1 tuple (5.729ms) +------------------+-----------+-------+----------+--------+---------+----------------------------+--------+------+---------+ | nme | host | port | protocol | mode | status | lastsent | cycles | sent | pending | +==================+===========+=======+==========+========+=========+============================+========+======+=========+ | datacell.bsktout | localhost | 50600 | UDP | active | running | null | 0 | 0 | 0 | +------------------+-----------+-------+----------+--------+---------+----------------------------+--------+------+---------+ 1 tuple (6.183ms) +---------------+---------+----------------------------+--------+------+----------+------------------------------------------------+------------------------------------------------+ | nme | status | lastrun | cycles | even | time | error | def | : : : : : ts : : : : +===============+=========+============================+========+======+==========+================================================+================================================+ | datacell.pass | running | null | 105797 | 0 | 12205937 | Query datacell.pass failed:SQLException:mvc:SQ | insert into datacell.bsktout select * from dat | : : : : : : : L module not initialized : acell.bsktin; : +---------------+---------+----------------------------+--------+------+----------+------------------------------------------------+------------------------------------------------+ 1 tuple (6.299ms) +------------------+-----------+---------+-----------+-----------+------------+------+----------------------------+--------+ | nme | threshold | winsize | winstride | timeslice | timestride | beat | seen | events | +==================+===========+=========+===========+===========+============+======+============================+========+ | datacell.bsktin | 0 | 0 | 0 | 0 | 0 | 0 | 2015-03-12 10:04:22.000000 | 1000 | | datacell.bsktout | 0 | 0 | 0 | 0 | 0 | 0 | null | 0 | +------------------+-----------+---------+-----------+-----------+------------+------+----------------------------+--------+ 2 tuples (6.487ms) Can someone help? Our MonetDB build is faulty? Regards, Liming

4 5

Geometrical grid for spherical data
by Jiří Nádvorník 17 Mar '15

17 Mar '15

Hi All, I have spherical coordinates (right ascension, declination) and I'd like to iterate over areas in this 2-dimensional space. Easiest that comes to mind is just create a grid of squares (say 5 arcsecs wide) and iterate over them. Is it easier to do this in MonetDB with just spherical coordinates (some kind of spherical index), or do I have to convert them to x,y,z coordinates and create the grid in 3D? Thank you for your insight. Jiri Nadvornik Astronomical Institute AV CR Stellar department Czech Republic <mailto:nadvornik.ji@gmail.com> nadvornik.ji(a)gmail.com

3 4

Discussion on Concurrency Insert Issue "insert into A from B table and insert into C from D "
by Shivanandan Gupta 17 Mar '15

17 Mar '15

Hi All, I have written a java program when runs a two threads in parallel , copy command in parallel is working fine using batch files but the insert into using batch file is getting hanged , I have given the commands below which I have used in the batch file. Copy command Batch. mclient.bat demo -s "copy into A FROM 'D:\ETL_Source\abcd.csv' using delimiters ',';" Insert into Batch mclient.bat demo -s "insert into B select * from A;" One more problem I am facing is that while running concurrent insert into A select * from B and insert into C select * from D. I have read a lot about concurrent issue but still I am a bit confuse , my understanding is that for any insert into table monetdb do a db lock and rollback all other queries running in parallel , if this is true then in real time scenario how we can use this db for a DWH , as any time we will have multiple facts and dimensions and for facts we will be doing parallel loads, data volume will be very high. Please help me out if my understanding is wrong or if I can change some settings to accept parallel insert command. Please let me know if you need more information from my side for better understanding. -- Cheers, Shivanandan Gupta

1 0

Multi threaded (2 threads) Java program for copy command
by Shivanandan Gupta 13 Mar '15

13 Mar '15

Hi Jennie, I am getting below given exception while running in parallel. Exception in thread "main" java.sql.SQLException: COMMIT: transaction is aborted because of concurency conflicts, will ROLLBACK instead at nl.cwi.monetdb.jdbc.MonetConnection$ResponseList.executeQuery(MonetConnection.java:2535) at nl.cwi.monetdb.jdbc.MonetConnection$ResponseList.processQuery(MonetConnection.java:2284) at nl.cwi.monetdb.jdbc.MonetStatement.internalExecute(MonetStatement.java:508) at nl.cwi.monetdb.jdbc.MonetStatement.execute(MonetStatement.java:349) at connect_monetdb.runETL2(connect_monetdb.java:79) at connect_monetdb.main(connect_monetdb.java:14) Thank you, Shivanandan Gupta. On Fri, Mar 13, 2015 at 8:09 PM, Shivanandan Gupta < shivnandangupta(a)gmail.com> wrote: > Hi Jennie, > > Thanks for your reply. please find my replies below: > > Jennie Commented: Concurrent insertion into two different tables should > work. Do you have anymore information about why the COMMIT has failed? > Maybe the problem lies somewhere else than concurrent insertions? > > Shiva's Comment: yes when I run two batch file by clicking it with > difference of second ( as it takes few seconds to click the second batch > file) and it runs perfectly, even I have tried running copy command > directly on the monetsql client and on two clients in parallel it works > perfectly. Is there some configuration change required on monetdb server? > > Jennie Commented: - Have you tried the two COPY INTO separately? To know > if the problem is indeed caused by concurrent insertions. > > Shiva's Comment: When I am trying with two separate java program running > concurrently its throwing same exception. > > > Jennie Comment: - Does the problem occur one-time, or every time you > repeatedly execute your code? > Shiva's Comment: Yes the same error message is coming every time I run > concurrent java programs with copy command in it ( here copying data to > different tables ). > > Any help will be really appreciated. > > Thank you. > > Shivanandan Gupta. > > > Date: Fri, 13 Mar 2015 15:36:23 +0800 > From: Ying Zhang <Y.Zhang(a)cwi.nl> > To: Communication channel for MonetDB users <users-list(a)monetdb.org> > Cc: martin(a)monetdb.org > Subject: Re: Multi threaded (2 threads) Java program for copy command > Message-ID: <89A66109-B083-45A4-B59E-62B7AF0C9D86(a)cwi.nl> > Content-Type: text/plain; charset=us-ascii > > > > On Mar 12, 2015, at 19:27, Shivanandan Gupta <shivnandangupta(a)gmail.com> > wrote: > > > > Hi Martin, > > > > Thanks for your prompt reply , I am using two different tables here but > still I am getting "Exception in thread "main" java.sql.SQLException: > COMMIT: failed" exception. > > Hello Shivanandan, > > Concurrent insertion into two different tables should work. Do you have > anymore information about why the COMMIT has failed? Maybe the problem > lies somewhere else than concurrent insertions? > > - Have you tried the two COPY INTO separately? To know if the problem is > indeed caused by concurrent insertions. > - Does the problem occur one-time, or every time you repeatedly execute > your code? > > Regards, > > Jennie > > > > On Thu, Mar 12, 2015 at 4:57 PM, Shivanandan Gupta < > shivnandangupta(a)gmail.com> wrote: > >> Hi Martin, >>> >> >> Thanks for your prompt reply , I am using two different tables here but >> still I am getting "Exception in thread "main" java.sql.SQLException: >> COMMIT: failed" exception. >> >> >> >>> Message: 2 >>> Date: Thu, 12 Mar 2015 08:32:31 +0100 >>> From: Martin Kersten <martin(a)monetdb.org> >>> To: users-list(a)monetdb.org >>> Subject: Re: Multi threaded (2 threads) Java program for copy command >>> commit fails for one thread >>> Message-ID: <5501410F.1070508(a)monetdb.org> >>> Content-Type: text/plain; charset=windows-1252; format=flowed >>> >>> Hi, >>> >>> Yes. Insertion into the same table would be invalidated due to the >>> transaction semantics. >>> See >>> https://www.monetdb.org/Documentation/Manuals/SQLreference/Transactions >>> >>>

1 0