March 2012 - users-list

[MonetDB-users] Query ignored
by Mints, Alexey 15 Mar '12

15 Mar '12

Hello! I have an issue with MonetDB query. This one select count(*) from catalogedsources c1 left outer join catalogedsources c2 on ( c2.cat_id = c1.cat_id + 2 AND ((c1.decl - c2.decl) * (c1.decl - c2.decl))< 1e-06) where c1.cat_id = 4 AND c1.decl BETWEEN 77.0 AND 83.0 AND c1.ra BETWEEN 42.0 AND 48.0 ; works fine. But if I replace "c2.cat_id = c1.cat_id + 2" with "c2.cat_id = 6" - i receive nothing. Not zero, not empty dataset, but just nothing at all. "EXPLAIN" does not work as well. Also, with more complex join conditions I have no result as well. I am using: Welcome to mclient, the MonetDB/SQL interactive terminal (Dec2011-SP1) Database: MonetDB v11.7.7 (Dec2011-SP1), 'mapi:monetdb://linux-6d2s.site:50000/*****' What can this be? Regards, Alexey

1 0

[MonetDB-users] MonetDB out of memory errors
by Stathis Gkotsis 14 Mar '12

14 Mar '12

Hi all, I am using MonetDB v11.7.7 (Dec2011-SP1) in an application which constantly feeds data in tables (daily tables). The communication with the server is done through the mapi API. This is accomplished through the "COPY INTO" command. From time to time (every 1-2 days) I see errors in the application logs such as: exception at 00000000004C0109: Connect error (Error code -4): monetdbd: an internal error has occurred, refer to the logs for details, please try again later . exception at 00000000004C0109: Connect error (Error code -4): monetdbd: there are no available connections for 'mydatabase' . exception at 00000000004C0109: Connect error (Error code -4): monetdbd: there are no available connections for 'mydatabase' Then looking at dmesg log: [964932.242080] 251025 pages non-shared [964932.242083] Out of memory: kill process 11585 (monetdbd) score 787911 or a child [964932.302785] Killed process 11781 (mserver5) Sometimes, monetdb can recover from this error and then the loading continues normally, some others I have to manually restart monetdb. Is there anything I can do to troubleshoot this issue? Is there a possibility for me to fall back to an earlier version of MonetDB with potentially less features but more stability? Thanks for all the help, Stathis

2 2

[MonetDB-users] rdf support
by Jürgen Jakobitsch 14 Mar '12

14 Mar '12

hi, first of all congrats to this great project! i came to monetDB via a search for "column stores", since i know that one of the big players in the field of linked data and rdf storage switched to usage of a column store in one of their latest releases. i just built from source without any hassle on openSuse 11.4 with support for rdf since this is my primary interest at the moment. i didn't find much of an info regarding monetDB and rdf besides an old nabble-mail and a small-meeting wiki. is there some more info available? any pointer really appreciated. wkr jürgen -- | Jürgen Jakobitsch, | Software Developer | Semantic Web Company GmbH | Mariahilfer Straße 70 / Neubaugasse 1, Top 8 | A - 1070 Wien, Austria | Mob +43 676 62 12 710 | Fax +43.1.402 12 35 - 22 COMPANY INFORMATION | http://www.semantic-web.at/ PERSONAL INFORMATION | web : http://www.turnguard.com | foaf : http://www.turnguard.com/turnguard | skype : jakobitsch-punkt | xmlns:tg = "http://www.turnguard.com/turnguard#"

1 0

[MonetDB-users] Replication/Redundancy
by Joseph Brower 13 Mar '12

13 Mar '12

We're using MonetDB quite heavily and are really enjoying it. We're now wanting to add replication/redundancy to ensure we have as little downtime as possible. We're also thinking of balancing reads if that is possible. I've heard this is possible, but I'm unsure on the best way to approach it. I've not been able to find many docs that talk about this either. Could someone point me in the right direction? Thanks, Joseph Brower

4 10

[MonetDB-users] monetdbd: binding to datagram socket port 50000 failed: no available address
by Charalampos Nikolaou 09 Mar '12

09 Mar '12

When I start monetdbd I get the following error "monetdbd: binding to datagram socket port 50000 failed: no available address" Prior to this error I had properly terminated the server, so there is no server running. How can I overcome that problem? Also, netstat gives the following: $ netstat -p udp | grep 50000 udp4 1562 0 *.50000 *.* Thanks, Babis P.S.: I am on a Mac if that plays any role.

2 2

[MonetDB-users] some questions regarding queries in MonetDB
by Freddy Priyatna 06 Mar '12

06 Mar '12

Dear all, I've been using MonetDB version 5 on Ubuntu machine for evaluating BSBM queries. The dataset and queries can be found here[1]. It seems that there are some queries that are not supported on MonetDB : 1) SELECT p.nr, p.label FROM product p, producttypeproduct ptp WHERE p.nr=ptp.product AND "productType"=35 AND "propertyNum1">500 AND "propertyNum3"<2000 AND 168 IN (SELECT "productFeature" FROM productfeatureproduct WHERE productfeatureproduct.product=p.nr) AND 147 NOT IN (SELECT "productFeature" FROM productfeatureproduct WHERE product=p.nr) >From the error, the subquery seems can't refer to the table alias "p". 2) SELECT distinct p.nr, p.label FROM product p, product po, (Select distinct pfp1.product FROM productfeatureproduct pfp1, (SELECT "productFeature" FROM productfeatureproduct WHERE product=2) pfp2 WHERE pfp2."productFeature"=pfp1."productFeature") pfp WHERE p.nr=pfp.product AND po.nr=2 AND p.nr <> po.nr AND p."propertyNum1" < (po."propertyNum1"+120) AND p."propertyNum1" > (po."propertyNum1"-120) AND p."propertyNum2" < (po."propertyNum2"+170) AND p."propertyNum2" > (po."propertyNum2"-170) This is strange, because if I just comment one of the last 2 lines, the query works. But the last two lines can't be used together. 3) SELECT nr, label FROM product WHERE label like "%countries%"; It seems that MonetDB doesn't support regex. 4) SELECT * FROM (select label from product where nr=2) p left join ((select o.nr as onr, o.price, v.nr as vnr, v.label from offer o, vendor v where 2=o.product AND o.vendor=v.nr AND v.country='GB' AND o."validTo">'2008-07-01') ov right join (select r.nr as rnr, r.title, pn.nr as pnnr, pn.name, r.rating1, r.rating2 from review r, person pn where r.product=2 AND r.person=pn.nr) rpn on (1=1)) on (1=1); It seems that MonetDB can't do join with true condition because when I tried to run a simple query : SELECT * FROM product p RIGHT JOIN offer o ON (1=1) Then MonetDB gives an error. Could anyone please help me how to run the queries above? Many thanks in advance. Freddy [1] http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/ExploreUse…

5 11

[MonetDB-users] duplicate tables in schema
by Bas Kaptijn 05 Mar '12

05 Mar '12

Hi, repeatedly we get in a situation where tables end up duplicate in the sys.tables table: SELECT count(*) FROM tables WHERE name='mytable'; returns: 2 We think this should never be the case. It seems this gives problems with dumping the database and in other situations. This is an older version of MonetDB (Apr2011 SP1). We want to know whether someone knows this was/is a symptom of any known issue Sincerely, Bas

3 2

[MonetDB-users] Slow range join performance / regression with Dec2011-SP1
by Viktor Rosenfeld 02 Mar '12

02 Mar '12

Hi, I have a few queries that join a table with itself using a ranged predicate on the primary key. The simplest query is: SELECT count(*) AS count FROM rank_copy rank1, rank_copy rank2 WHERE rank1.cat_s = true AND rank2.pos_vvfin = true AND rank1.pre < rank2.pre AND rank2.pre < rank1.post As you can see, the range lookup on rank2.pre is restricted on both sides by rank1.pre and rank1.post. The selectivity of the attributes and the complete query is: - row count : 2431307 - query : 95977 (0.039) - cat_s : 148799 (0.061) - pos_vvfin : 71965 (0.030) PostgreSQL is able to evaluate the query in 2 seconds using an index on pre. (This index is automatically generated because pre is the primary key.) MonetDB Dec2011 requires about 35 seconds. The MAL plan is posted below. I've also tried sorting the table on pre, and created an (advisory) index on pre but this makes no difference. +---------------------------------------------------------------------------+ | mal | +===========================================================================+ | function user.s1_1{autoCommit=true}():void; | | X_2 := sql.mvc(); | | X_9:bat[:oid,:bit] := sql.bind(X_2,"sys","rank_copy","pos_vvfin",0); | | X_10 := algebra.uselect(X_9,true:bit); | | X_11 := algebra.markT(X_10,0@0:oid); | | X_12 := bat.reverse(X_11); | | X_4:bat[:oid,:int] := sql.bind(X_2,"sys","rank_copy","pre",0); | | X_13 := algebra.leftjoin(X_12,X_4); | | X_17:bat[:oid,:bit] := sql.bind(X_2,"sys","rank_copy","cat_s",0); | | X_18 := algebra.uselect(X_17,true:bit); | | X_19 := algebra.markT(X_18,0@0:oid); | | X_20 := bat.reverse(X_19); | | X_23 := algebra.leftjoin(X_20,X_4); | | X_15:bat[:oid,:int] := sql.bind(X_2,"sys","rank_copy","post",0); | | X_21 := algebra.leftjoin(X_20,X_15); | | X_24 := algebra.join(X_13,X_23,X_21,false,false); | | X_26 := algebra.markT(X_24,0@0:oid); | | X_27 := bat.reverse(X_26); | | X_28 := algebra.leftjoin(X_27,X_13); | | X_29 := aggr.count(X_28); | | sql.exportValue(1,"sys.rank2","count":str,"wrd",64,0,6,X_29,""); | | end s1_1; | +---------------------------------------------------------------------------+ 22 tuples If I use a slightly more complicated query that has the boolean attributes pos_vvfin and cat_s on another table that is joined by rank, the situation is even worse. SELECT count(*) AS count FROM node_copy node1 JOIN rank_copy rank1 ON (node1.id = rank1.node_ref), node_copy node2 JOIN rank_copy rank2 ON (node2.id = rank2.node_ref) WHERE node1.cat_s = true AND node2.pos_vvfin = true AND rank1.pre < rank2.pre AND rank2.pre < rank1.post PostgreSQL evaluates the query in 9 seconds, again using the primary key indexes on node.id and rank.pre. MonetDB Dec2011 requires 14 minutes. The MAL plan follows: +-----------------------------------------------------------------------------------------------+ | mal | +===============================================================================================+ | function user.s1_1{autoCommit=true}():void; | | X_2 := sql.mvc(); | | X_19:bat[:oid,:oid] := sql.bind_idxbat(X_2,"sys","rank_copy","FK_rank_copy_node_ref",0); | | X_13:bat[:oid,:bit] := sql.bind(X_2,"sys","node_copy","pos_vvfin",0); | | X_14 := algebra.uselect(X_13,true:bit); | | X_15 := algebra.markT(X_14,0@0:oid); | | X_16 := bat.reverse(X_15); | | X_9:bat[:oid,:int] := sql.bind(X_2,"sys","node_copy","id",0); | | X_11 := bat.mirror(X_9); | | X_17 := algebra.leftjoin(X_16,X_11); | | X_18 := bat.reverse(X_17); | | X_37:bat[:oid,:bit] := sql.bind(X_2,"sys","node_copy","cat_s",0); | | X_39 := algebra.uselect(X_37,true:bit); | | X_40 := algebra.markT(X_39,0@0:oid); | | X_41 := bat.reverse(X_40); | | X_42 := algebra.leftjoin(X_41,X_11); | | X_43 := bat.reverse(X_42); | | X_24 := algebra.join(X_19,X_18); | | X_25 := algebra.markT(X_24,0@0:oid); | | X_26 := bat.reverse(X_25); | | X_4:bat[:oid,:int] := sql.bind(X_2,"sys","rank_copy","pre",0); | | X_27 := algebra.leftjoin(X_26,X_4); | | X_29:bat[:oid,:int] := sql.bind(X_2,"sys","rank_copy","post",0); | | X_30 := algebra.join(X_27,X_4,X_29,false,false); | | X_32 := algebra.markT(X_30,0@0:oid); | | X_33 := bat.reverse(X_32); | | X_44 := bat.reverse(X_30); | | X_45 := algebra.markT(X_44,0@0:oid); | | X_46 := bat.reverse(X_45); | | X_47 := algebra.leftjoin(X_46,X_19); | | X_48 := algebra.join(X_47,X_43); | | X_49 := algebra.markT(X_48,0@0:oid); | | X_50 := bat.reverse(X_49); | | X_51:bat[:oid,:int] := algebra.leftjoinPath(X_50,X_33,X_27); | | X_52 := aggr.count(X_51); | | sql.exportValue(1,"sys.rank2","count","wrd",64,0,6,X_52,""); | | end s1_1; | +-----------------------------------------------------------------------------------------------+ 37 tuples The Dec2011-SP1 release takes even longer, about 18 minutes. The MAL plan is unchanged. The culprit appears to be the ranged lookup because if I replace it with an equi-join on another attribute MonetDB evaluates it in about 100 ms. SELECT count(*) AS count FROM node_copy node1 JOIN rank_copy rank1 ON (node1.id = rank1.node_ref), node_copy node2 JOIN rank_copy rank2 ON (node2.id = rank2.node_ref) WHERE node1.cat_s = true AND node2.pos_vvfin = true AND rank1.pre = rank2.parent I've tried rewriting the query using WITH and had some success. The first query again takes 35 seconds: WITH rank1 AS (SELECT * FROM rank_copy WHERE cat_s = true), rank2 AS (SELECT * FROM rank_copy WHERE pos_vvfin = true) SELECT count(*) FROM rank1, rank2 WHERE rank1.pre < rank2.pre AND rank2.pre < rank1.post The second query also takes 35 seconds when rewritten: WITH rank1 AS (SELECT * FROM rank_copy JOIN node_copy ON (rank_copy.node_ref = node_copy.id) WHERE node_copy.cat_s = true), rank2 AS (SELECT * FROM rank_copy JOIN node_copy ON (rank_copy.node_ref = node_copy.id) WHERE node_copy.pos_vvfin = true) SELECT count(*) FROM rank1, rank2 WHERE rank1.pre < rank2.pre AND rank2.pre < rank1.post MAL plan: +-----------------------------------------------------------------------------------------------+ | mal | +===============================================================================================+ | function user.s1_2{autoCommit=true}():void; | | X_2 := sql.mvc(); | | X_20:bat[:oid,:oid] := sql.bind_idxbat(X_2,"sys","rank_copy","FK_rank_copy_node_ref",0); | | X_13:bat[:oid,:bit] := sql.bind(X_2,"sys","node_copy","pos_vvfin",0); | | X_14 := algebra.uselect(X_13,true:bit); | | X_15 := algebra.markT(X_14,0@0:oid); | | X_16 := bat.reverse(X_15); | | X_9:bat[:oid,:int] := sql.bind(X_2,"sys","node_copy","id",0); | | X_11 := bat.mirror(X_9); | | X_17 := algebra.leftjoin(X_16,X_11); | | X_18 := bat.reverse(X_17); | | X_32:bat[:oid,:bit] := sql.bind(X_2,"sys","node_copy","cat_s",0); | | X_33 := algebra.uselect(X_32,true:bit); | | X_34 := algebra.markT(X_33,0@0:oid); | | X_35 := bat.reverse(X_34); | | X_36 := algebra.leftjoin(X_35,X_11); | | X_37 := bat.reverse(X_36); | | X_22 := algebra.join(X_20,X_18); | | X_23 := algebra.markT(X_22,0@0:oid); | | X_24 := bat.reverse(X_23); | | X_5:bat[:oid,:int] := sql.bind(X_2,"sys","rank_copy","pre",0); | | X_25 := algebra.leftjoin(X_24,X_5); | | X_38 := algebra.join(X_20,X_37); | | X_39 := algebra.markT(X_38,0@0:oid); | | X_40 := bat.reverse(X_39); | | X_43 := algebra.leftjoin(X_40,X_5); | | X_27:bat[:oid,:int] := sql.bind(X_2,"sys","rank_copy","post",0); | | X_41 := algebra.leftjoin(X_40,X_27); | | X_44 := algebra.join(X_25,X_43,X_41,false,false); | | X_46 := algebra.markT(X_44,0@0:oid); | | X_47 := bat.reverse(X_46); | | X_48 := algebra.leftjoin(X_47,X_25); | | X_49 := aggr.count(X_48); | | sql.exportValue(1,"sys.rank2","L1","wrd",64,0,6,X_49,""); | | end s1_2; | +-----------------------------------------------------------------------------------------------+ 35 tuples Is there anything I can do to further speed up the execution of these queries? Thanks, Viktor

1 0

[MonetDB-users] "ERROR = !Connection terminated" for some SQL queries
by Viktor Rosenfeld 01 Mar '12

01 Mar '12

Hi, the following query results in a "Connection terminated error": SELECT count(*) FROM ( SELECT DISTINCT node1.id AS id1, node2.id AS id2, node1.toplevel_corpus FROM node AS node1, rank AS rank1, component AS component1, node AS node2, rank AS rank2, component AS component2 WHERE component1.id = component2.id AND component1.name IS NULL AND component1.type = 'd' AND component2.name IS NULL AND component2.type = 'd' AND rank1.component_ref = component1.id AND rank1.node_ref = node1.id AND rank1.pre = rank2.parent AND rank2.component_ref = component2.id AND rank2.node_ref = node2.id ) AS solutions; The full error message is: MAPI = annis@moos.local:50000 ACTION= read_line QUERY = SELECT count(*) FROM ( SELECT DISTINCT node1.id AS id1, node2.id AS id2, node1.toplevel_corpus FROM node AS node1, rank AS rank1, component AS component1, node AS node2, rank AS rank2, component AS component2 WHERE component1.id = component2.id AND component1.name IS NULL AND component1.type = 'd' AND component2.name IS NULL AND component2.type = 'd' AND rank1.component_ref = component1.id AND rank1.node_ref = node1.id AND rank1.pre = rank2.parent AND rank2.component_ref = component2.id AND rank2.node_ref = node2.id ) AS solutions; ERROR = !Connection terminated I can get it to run if I remove the following line: component1.id = component2.id AND or if I remove the following 4 lines: component1.name IS NULL AND component1.type = 'd' AND component2.name IS NULL AND component2.type = 'd' AND However, if I include all constraints on component, the error appears. Let me know, if you need more information. Cheers, Viktor

2 3