March 2008 - developers-list

[Monetdb-developers] blobs performance
by Henrik Vendelbo 30 Mar '08

30 Mar '08

I am working through the options for clustered file management. Normally I would expect fairly poor performance of binary file storage in a database. Hence I would favor using the file system, and sync across the machines in the cluster. Being quite different from common relational implementations, how well does Monet handle blobs of 1-10Mb? As a side question is there a support for multi-master replication? The technology looks really impressive, I hope to be able to base my future projects on MonetDB Thanks, Henrik

2 1

Re: [Monetdb-developers] Monetdb-developers Digest, Vol 22, Issue 4
by Peter Boncz 28 Mar '08

28 Mar '08

Hi, I read the digests of the monetdb mailing list, which in this case took more than a month to send me this discussion. That's pretty useless.. I will switch to immediate mails. Anyway, Jennie is currently working on getting the repeatable reads transactions working. This means that by passing a request-ID you will be guaranteed that all queries and updates execute on the same database image, i.e. a transaction. In case of update requests in the same transaction, these will be executed atomically using a 2PC protocol. While this functionality is designed to be triggerered from XQuery queries, you client API could also generate the appropriate messages. It may not answer you needs regarding performance (you would send a read-request, update-request, prepare-request and commit-request), but it would answer your functional requirements AFAIS. Peter > Message: 1 > Date: Wed, 19 Mar 2008 12:30:20 +0100 > From: "Keulen, M. van (Maurice)" <m.vankeulen(a)utwente.nl> > Subject: Re: [Monetdb-developers] [XQ] Mixing update/docmgmt/query > To: monetdb-developers(a)lists.sourceforge.net > Message-ID: <47E0F94C.4000709(a)utwente.nl> > Content-Type: text/plain; charset=windows-1252; format=flowed > > Ying Zhang wrote: > > > >> Jennie: my SOAP-messages do not come from MonetDB/XQuery, but from > >> outside (i.e., some scripting language). > > > > Then, an order can only be guaranteed if you, in your > scripts, send a > > message, wait for result, and send the next message. In all other > > cases, unfortunately, no guarantee, as an xrpc server immediately > > start handling a request when it's received. > > > > Jennie > > I thought so already. This is why I don't like to have to > send multiple > requests to do one thing (transaction). I'd rather be able to > send one > request to both do, say, an update and return a result, or add a > document and return a result. The system so far doesn't allow > me to do > this, because I cannot define one query (or function in a > module) that > does both because of typing restrictions, and I also cannot send two > separate queries in one request because of restrictions in the XRPC > protocol. > > Any thoughts about achieving what I want? > > Maurice. > > -- > ---------------------------------------------------------------------- > Dr.Ir. M. van Keulen - Assistant Professor, Data Management Technology > Univ. of Twente, Dept of EEMCS, POBox 217, 7500 AE Enschede, > Netherlands > Email: m.vankeulen(a)utwente.nl, Phone: +31 534893688, Fax: +31 > 534892927 > Room: ZI 3039, WWW: http://www.cs.utwente.nl/~keulen *********************************

3 8

[Monetdb-developers] MonetDB testing problems
by Stefan Manegold 27 Mar '08

27 Mar '08

Dear fellow MonetDB developers, the recently introduced archiving of testing result summaries seems to require more storage than expected --- I don't know, yet, whether this is indeed inherent in the approach, or "just" some bug --- filling up the disk space on our testing server, and thus preventing testing from running properly. I will not have any time to investigate (let alone fix) the problem this week --- in fact, given other obligations, I'm not sure whether I will find any time before May 1st ... Hence, there might be some time ahead without proper/reliable MonetDB testing. I'll keep you posted, in case/once I managed to solve the problem... Stefan -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 2

[Monetdb-developers] [PF] Question about timings
by Keulen, M. van (Maurice) 25 Mar '08

25 Mar '08

Hi, A student of mine is trying to speed up some queries and is puzzled about the timing-readings he gets, for example: Trans 19681.811 msec Shred 0.000 msec Query 16384.143 msec Print 81.180 msec Timer 55087.438 msec The total of the first four 19.7+0+16.4+0.08 is approximately 36.2 seconds. "Timer" gives 55.1 seconds. Where are the remaining 18.9 seconds going? By the way, what do these measures precisely indicate? "Query", for example, can be interpreted as query compilation or query execution. Maurice. -- ---------------------------------------------------------------------- Dr.Ir. M. van Keulen - Assistant Professor, Data Management Technology Univ. of Twente, Dept of EEMCS, POBox 217, 7500 AE Enschede, Netherlands Email: m.vankeulen(a)utwente.nl, Phone: +31 534893688, Fax: +31 534892927 Room: ZI 3039, WWW: http://www.cs.utwente.nl/~keulen

2 3

[Monetdb-developers] Experimental testweb history available
by Fabian Groffen 20 Mar '08

20 Mar '08

Hi all, This is just to inform you all of a recently added "feature" (to both Stable and Current ;) ) in our beloved TestWeb. We added a "history" function to parts of TestWeb's output, which can come in handy if you're chasing pavem^Wbugs, or just interested in (since) when happened what. First off, a nice, nifty and sexy GUI is missing for sure, but who knows what the future will bring :D The MonetDB PermaStore (it is *not* powered by any MonetDB technology) http://www.ins.cwi.nl/~monet/permastore/ At this URL you will find all the output of the compilation check script, that some of you may already have been confronted with. In particular, here you can find per day the skeletons of the email messages sent out, and the annotated HTML versions of the build logs. Improvements in this area are low hanging fruit, but they just require time. Note that the aforementioned URL is still sort of experimental only. The MonetDB TestWeb 30-day archive http://monetdb.cwi.nl/testing/projects/monetdb/<branch>/<module>/.<thing>..<german_date>/ where <german_date> = yyyy.mm.dd If you know what branch (Stable, Current), module (buildtools, MonetDB, sql, ...) and thing (Makes3, Int.32.32.d-Fedora8, etc.) should be then you can inspect the TestWeb output till 30 days back in time. Note that the full 30 days are not yet generated, so only partially available.

3 2

[Monetdb-developers] [XQ] Mixing update/docmgmt/query
by Keulen, M. van (Maurice) 19 Mar '08

19 Mar '08

Hi, I know that the XQUF prohibits it, but for writing robust functions, it is often necessary to mix query and update/docmgmt functionality. Some examples: * Expression that queries/updates document, but if the document does not exist, adds an empty version of it: let $name := "foo.xml" let $emptydoc := <root/> let $doc := if (empty(pf:documents()[.=$name])) then (put($emptydoc,$name),$emptydoc) else doc("foo.xml") return $doc//bar * Expression that adds to a log-file: let $log := doc("log.xml") let $expr := ...some expression... return (do insert <entry>...{$expr}...<entry> as last into $log,$expr) Each of these patterns gives you a "!type error: err:XUST0101: illegal mix of updating and non-updating expressions" error. Isn't is possible to somehow allow such expressions? It is against the standard, so it probably requires some server configuration parameter, but it would be really really useful to have it. Mixing queries with update/docmgmt is known to cause many semantical problems, but only if these updates are visible within the query. Looking at the implementation based on pending update lists, then each of these patterns is harmless. If you declare that no updates are visible within the transaction (which is against the standard), then allowing these queries is I guess not hard: it simply means that with updating expressions and document management expressions you do return a result. Another thing I've been thinking about in this context: If I send two SOAP-messages to a server on its XRPC-port, am I guaranteed that they are served in the same order? I could perhaps, with a lot of duplication of code, separate update/docmgmt from a query, i.e., implement these things with two requests to the server. I really dislike such a solution, though. Looking forward to comments :-) Maurice. -- ---------------------------------------------------------------------- Dr.Ir. M. van Keulen - Assistant Professor, Data Management Technology Univ. of Twente, Dept of EEMCS, POBox 217, 7500 AE Enschede, Netherlands Email: m.vankeulen(a)utwente.nl, Phone: +31 534893688, Fax: +31 534892927 Room: ZI 3039, WWW: http://www.cs.utwente.nl/~keulen

3 5

[Monetdb-developers] Bug in Python interface converters.py Feb2008-SuperBall
by Andrei Tovtchigretchko 15 Mar '08

15 Mar '08

MonetSQLdb.converters.escape(x, conv=None) should return 'tuple' for any input sequence type (currently it returns tuple only if it receives tuple, and returns list if it receives list). The return result is used as a second argument in expression 'query = query % self.connection.literal(args)' in cursors.py. The '%' operator raises an exception on anything but tuple. Also, when 'escape()' cannot convert some argument, it should immediately raise a descriptive exception. Currently it silently drops the argument from the result sequence, which causes an exception to be raised later in the calling code. That exception reads like this: """ProgrammingError: not enough arguments for format string""" This unnecessarily complicates debugging. Thank you, Andrey _________________________________________________________________ Watch “Cause Effect,” a show about real people making a real difference. Learn more. http://im.live.com/Messenger/IM/MTV/?source=text_watchcause

2 1

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/mal mal_recycle.mx, , 1.49, 1.50
by Stefan Manegold 15 Mar '08

15 Mar '08

On Sat, Mar 15, 2008 at 08:41:35AM +0000, Martin Kersten wrote: > Update of /cvsroot/monetdb/MonetDB5/src/mal > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv17899 > > Modified Files: > mal_recycle.mx > Log Message: > Silence Windows compiler FYI: `man bzero` says: ======== CONFORMING TO 4.3BSD. This function is deprecated (marked as LEGACY in POSIX.1-2001): use memset(3) in new programs. ======== Stefan > Index: mal_recycle.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/mal/mal_recycle.mx,v > retrieving revision 1.49 > retrieving revision 1.50 > diff -u -d -r1.49 -r1.50 > --- mal_recycle.mx 14 Mar 2008 09:43:07 -0000 1.49 > +++ mal_recycle.mx 15 Mar 2008 08:41:33 -0000 1.50 > @@ -260,7 +260,7 @@ > newMalBlkStmt(recycleBlk,recycleBlk->ssize); > > used= alloca(recycleBlk->vtop); > - bzero(used, recycleBlk->vtop); > + memset(used,0, recycleBlk->vtop); > > /* set all used variables */ > for (i = 0; i < pc; i++){ > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/modules/mal recycle.mx, , 1.9, 1.10
by Stefan Manegold 10 Mar '08

10 Mar '08

On Mon, Mar 10, 2008 at 11:21:59AM +0000, Romulo Goncalves wrote: > Update of /cvsroot/monetdb/MonetDB5/src/modules/mal > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv24521 > > Modified Files: > recycle.mx > Log Message: > Check in the MALblk exist to avoid a mserver crash when we to > recycle.dump() before any query execution why do you check the loop-invariant condition with every iteration of the loop? Stefan > Index: recycle.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/modules/mal/recycle.mx,v > retrieving revision 1.9 > retrieving revision 1.10 > diff -u -d -r1.9 -r1.10 > --- recycle.mx 9 Mar 2008 15:47:30 -0000 1.9 > +++ recycle.mx 10 Mar 2008 11:21:47 -0000 1.10 > @@ -147,7 +147,7 @@ > stream_printf(GDKout,"Recycler catalog \n"); > > /* and dump the statistics */ > - for(i=0; i< recycleBlk->stop; i++){ > + for(i=0; recycleBlk && i< recycleBlk->stop; i++){ > stream_printf(GDKout,"#%4d count=%3d ticks=%4d %s\n", i, > recycleBlk->profiler[i].counter, > recycleBlk->profiler[i].ticks, > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 1

Re: [Monetdb-developers] [Monetdb-pf-checkins] pathfinder/runtime xrpc_client.mx, xquery-decomposition, 1.41.2.8, 1.41.2.9
by Stefan Manegold 10 Mar '08

10 Mar '08

On Mon, Mar 10, 2008 at 02:01:16AM +0000, Ying Zhang wrote: > Update of /cvsroot/monetdb/pathfinder/runtime > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv3350 > > Modified Files: > Tag: xquery-decomposition > xrpc_client.mx > Log Message: > - adopt Stefan's tips: used rangesplit to split item|kind per container > (Thanks!). > > - a lot of corrections to get this thing working > > > > Index: xrpc_client.mx > =================================================================== > RCS file: /cvsroot/monetdb/pathfinder/runtime/xrpc_client.mx,v > retrieving revision 1.41.2.8 > retrieving revision 1.41.2.9 > diff -u -d -r1.41.2.8 -r1.41.2.9 > --- xrpc_client.mx 8 Mar 2008 21:17:27 -0000 1.41.2.8 > +++ xrpc_client.mx 10 Mar 2008 02:01:13 -0000 1.41.2.9 > @@ -94,94 +94,40 @@ > BAT[void, dbl] dbl_values, > BAT[void, dbl] dec_values, > BAT[void, str] str_values) : oid { > - > var maxnconts := ws.fetch(0).count(); > # split used_<item|kind> into multiple BATs according to their container IDs vvvvvvvvv > var uitems := bat(oid, bat, maxnconts); # container ID in head > var ukinds := bat(oid, bat, maxnconts); ^^^^^^^^^ These two BATs are IMHO overwritten below (see "vvvvvvvvv"/"^^^^^^^^^"); hence, no (more) need to make them, here!?? > - var conts := used_kind.[int]().[>>](6).tsort(); # [oid,int]: container IDs in tail > - var nconts := conts.tunique().count(); > - if(nconts = 1){ # all items are from the same container > - uitems.insert(conts.fetch(0).oid(), used_item); > - ukinds.insert(conts.fetch(0).oid(), used_kind); > - } else { > - # sort used_<item|kind> on container ID > - var itemcont := used_item.reverse().join(conts).reverse().tmark(0@0); > - var kindcont := used_kind.reverse().join(conts).reverse().tmark(0@0); > - > - var curcont := conts.fetch(0); > - var tmpitem := bat(void, oid).seqbase(0@0); > - var tmpkind := bat(void, int).seqbase(0@0); > - > - conts := conts.tmark(0@0); # conts[void,int] > - conts@batloop(){ > - # start of next container, save items|kinds of previous container > - if($t != curcont) { > - curcont := $t; > - uitems.insert(curcont, tmpitem); > - ukinds.insert(curcont, tmpkind); > - tmpitem := bat(void, oid).seqbase(0@0); > - tmpkind := bat(void, int).seqbase(0@0); > - } > - > - tmpitem.append(itemcont.fetch($h)); > - tmpkind.append(kindcont.fetch($h)); > - } > - uitems.insert(curcont, tmpitem); > - ukinds.insert(curcont, tmpkind); > - } > + var conts := get_container([int](used_kind)); # [oid,int]: container IDs in tail > + var unq_conts := tunique(conts).hmark(0@0); > + var nconts := count(unq_conts); > + var rs := rangesplit(conts, nconts).tmark(0(a)0).reverse().join(unq_conts).reverse(); This will do want you want (have rs's head contain the conts from the BATs in its tail) ONLY if tunique(conts) & rangesplit(conts, nconts) happen to produce the same order! To be sure, you'd need to sort the unq_conts before marking them (result of rangesplit shoiuld hopefully be sorted): var unq_conts := tunique(conts).sort().hmark(0@0); Was there something wrong with my proposal: var rs := rangesplit(conts, nconts); rs := rs.reverse().mirror().[fetch](0).reverse(); ? vvvvvvvvv > + uitems := rs.[mirror]().[leftfetchjoin](const used_item); > + uitems := [tmark](uitems, const 0@0); > + ukinds := rs.[mirror]().[leftfetchjoin](const used_kind); > + ukinds := [tmark](ukinds, const 0@0); ^^^^^^^^^ (see "vvvvvvvvv"/^^^^^^^^^" above) Plus: It might be more efficient to get rid of the head before the join, since you "overwrite" them afterwards, anyway; i.e.: uitems := rs.[tmark](oid(nil)).[leftfetchjoin](const used_item).[tmark](uitems, const 0@0); > > # split returned_<item|kind> into multiple BATs according to their container IDs vvvvvvvvv > var ritems := bat(oid, bat, maxnconts); # container ID in head > var rkinds := bat(oid, bat, maxnconts); ^^^^^^^^^ Same as with uitems/ukinds above. > - conts := returned_kind.[int]().[>>](6).tsort(); # [oid,int]: container IDs in tail > - nconts := conts.tunique().count(); > - if(nconts = 1){ # all items are from the same container > - ritems.insert(conts.fetch(0).oid(), returned_item); > - rkinds.insert(conts.fetch(0).oid(), returned_kind); > - } else { > - # sort used_<item|kind> on container ID > - var itemcont := returned_item.reverse().join(conts).reverse().tmark(0@0); > - var kindcont := returned_kind.reverse().join(conts).reverse().tmark(0@0); > - > - var curcont := conts.fetch(0); > - var tmpitem := bat(void, oid).seqbase(0@0); > - var tmpkind := bat(void, int).seqbase(0@0); > - > - conts := conts.tmark(0@0); # conts[void,int] > - conts@batloop(){ > - # start of next container, save items|kinds of previous container > - if($t != curcont) { > - curcont := $t; > - ritems.insert(curcont, tmpitem); > - rkinds.insert(curcont, tmpkind); > - tmpitem := bat(void, oid).seqbase(0@0); > - tmpkind := bat(void, int).seqbase(0@0); > - } > - > - tmpitem.append(itemcont.fetch($h)); > - tmpkind.append(kindcont.fetch($h)); > - } > - ritems.insert(curcont, tmpitem); > - rkinds.insert(curcont, tmpkind); > - } > + conts := get_container([int](returned_kind)); # [oid,int]: container IDs in tail > + unq_conts := tunique(conts).hmark(0@0); > + nconts := count(unq_conts); > + rs := rangesplit(conts, nconts).tmark(0(a)0).reverse().join(unq_conts).reverse(); (see above) vvvvvvvvv > + ritems := rs.[mirror]().[leftfetchjoin](const returned_item); > + ritems := [tmark](ritems, const 0@0); > + rkinds := rs.[mirror]().[leftfetchjoin](const returned_kind); > + rkinds := [tmark](rkinds, const 0@0); ^^^^^^^^^ (see above) > > # align uitems|kinds and ritems|kinds, > # fill missing container with emtpybat > var emptybat := bat(void,void,0); > - uitems@batloop(){ > - var err := CATCH({ritems.find($h);}); > - if(err.search("not found") >= 0){ > - ritems.insert($h, emptybat); > - } > - } > - ritems@batloop(){ > - var err := CATCH({uitems.find($h);}); > - if(err.search("not found") >= 0){ > - uitems.insert($h, emptybat); > - } > - } > - > + var usedonly := kdiff(uitems,ritems).project(emptybat); > + ritems.access(BAT_WRITE).insert(usedonly); > + rkinds.access(BAT_WRITE).insert(usedonly); > + var returnedonly := kdiff(ritems,uitems).project(emptybat); > + uitems.access(BAT_WRITE).insert(returnedonly); > + ukinds.access(BAT_WRITE).insert(returnedonly); a) Do you need to modify these BATs later on? If not, you might want the set them read-only, again, after the insert: ritems.access(BAT_WRITE).insert(usedonly).access(BAT_READ); b) To be sure that your variables till point to the correct BATs (access(BAT_WRITE) *might* return a copy!), you should asign the result: ritems := ritems.access(BAT_WRITE).insert(usedonly)[.access(BAT_READ)]; Stefan -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 1