June 2008 - developers-list

[Monetdb-developers] New default configure options for MonetDB
by Stefan Manegold 09 Jun '08

09 Jun '08

Dear MonetDB developer and users, For convenience of both developers and users as well as to comply even more with open source standards, we now set/use the following defaults for the configure options --enable-strict, --enable-assert, --enable-debug, --enable-optimize (Unless specified differently, the same defaults are used for both the "Stable" release branches and the "Current" development trunk.) When compiling from CVS sources (as mainly done by developers): strict=yes assert=yes debug=yes optimize=no (*) When compiling from packaged/distributed sources (i.e., tarballs) (as mainly done by users): strict=no assert=no debug=no optimize=no (*) For building binary distributions (RPMs, MSIs): strict=no assert=no debug=no optimize=yes With nightly testing for the "Stable" release branches: strict=yes assert=no debug=no optimize=yes With nightly testing for the "Current" development trunk: strict=yes assert=yes debug=no optimize=yes (*) IMPORTANT NOTE: Since "--enable-optimize=yes" is no longer the default for any case except binary packages, it is *strongly recommended* to (re)compile everything from scratch, *explicitly configured* with "--enable-debug=no --enable-optimize=yes", in case you want/need to run any performance experiments with MonetDB! (Please note: "--enable-X=yes" is equivalent to "--enable-X", and "--enable-X=no" is equivalent to "--disable-X".) Don't hesitate to ask in case you have any questions! Enjoy! Stefan -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0

[Monetdb-developers] script postgresSQL to Monetdb
by Roberto De Virgilio 09 Jun '08

09 Jun '08

Hi i have this problem.. i have to convert a dump file (.sql) of a PostgreSQL Db into a dump file for MonetDB to create the same DB of postgres into MonetDB... do you help me with a script? thanks a lot ____________________________________________________________________________________________________________ Roberto De Virgilio Ph.D Università Roma Tre Dipartimento di Informatica e Automazione Via della Vasca Navale 79, 00146 Roma Phone: +39-06-55173229 Fax: +39-06-5573030 E-mail: rde79(a)yahoo.com Home Page: http://mais.dia.uniroma3.it/rodevirg Room: 219

1 0

[Monetdb-developers] windows problem...
by Romulo Goncalves 09 Jun '08

09 Jun '08

I removed all the dbfarm, sql_logs and all the lib_* files.... Clean installation and I continue to get: E:\MonetDB\Current\sql\NT>RunMserver5.bat !WARNING: GDKlockHome: created directory E:\MonetDB\Current\\sql\NT\dbfarm\demo\ !WARNING: GDKlockHome: ignoring empty or invalid .gdk_lock. !WARNING: BBPdir: initializing BBP. # MonetDB server v5.6.0, based on kernel v1.24.0 # Serving database 'demo' # Compiled for i686-pc-win32/64bit with 64bit OIDs dynamically linked # Copyright (c) 1993-2008 CWI, all rights reserved # Visit http://monetdb.cwi.nl/ for further information #warning: please don't forget to set your vault key! #(see E:\MonetDB\Current\MonetDB5\NT\etc\monetdb5.conf) !FATAL: MALException:addUser:user 'monetdb' already exists E:\MonetDB\Current\sql\NT> It seems the bats are somewhere, the problem is where. The search of windows does not detect any dbfarm... Any suggestion? Romulo

2 1

[Monetdb-developers] Fastest way to calculate median?
by Rt Ibmer 09 Jun '08

09 Jun '08

Hi - I have an int column with several million rows of data. What is the fastest way to write a query to get the median value of the column? Apparently there is no median() function in SQL. A web search revealed lots of various talk about using stored procedures and fancy functions. None of this looked like it would be very efficient. I searched these archives but didn't see a single mention of median. Thank you!

2 1

[Monetdb-developers] Bug report - COPY RECORDS and encoded data
by Rt Ibmer 09 Jun '08

09 Jun '08

I was going crazy trying to figure out why my COPY RECORDS command wasn't working. Turns out it is caused from funny (encoded?) characters in the string for one of the fields. For instance one of the fields contained this "?a?as?e??, 18 ?a??? 2001 10:24:00 pµ" How did the data get this way? Well I exported it from SQL Server and I guess that db is set up to store encoded characters. When I did the export I told SQL Server to make the file ASCII since I figured that is what COPY RECORDS expects. So apparently SQL Server is writing the characters this way into the string. Then MonetDB (latest version) cannot handle this. So what happens is that the COPY x RECORDS command just returns immediately after I try this with this data in it. After that all my SQL statements in mclient return immediately. So I can say "select count(*) from foo;" and it just comes immediately back with the sql> prompt. So something with these funny characters sends mclient into a funk. I have to exit and restart it to get it going again. Any advice on how to handle this? Thank you.

3 2

[Monetdb-developers] Calculate disk space for table?
by Rt Ibmer 08 Jun '08

08 Jun '08

Hi - how can I tell how much disk storage space on of my tables is using? I tried one rough way but not sure if it is accurate. Basically I did a linux du command and looked at the farmdb dir root, then imported about 2M rows, then looked a tthe du number again. But it only went up about 1.5MB for 2M rows of data? There are about 150 columns but only about 30-40 columns for any given row are populated. Certainly this would be fantastic if the compression is really this good! Seems possible, but I wasn't sure if this technique was accurate and if there was a better way. Also on a related note - is there any detailed information on what types of compression and encoding MonetDB uses. For instance does it use Run Length Encoding or deltas or other such methods? And are there query plans / optimizers of any sort that you can feed MonetDB and it will optimize the layout or create internal subtables to help speed the types of queries we will be making? Thank you!

3 3

[Monetdb-developers] Division by zero on this query - why?
by Rt Ibmer 08 Jun '08

08 Jun '08

Any ideas why the following query would result in a division by zero exception? sql> select prodcode, avg(price) from sales group by prodcode; !ArithmeticException:batcalc./:Division by zero Timer 24326.296 msec 0 rows Thank you.

1 0

Re: [Monetdb-developers] [Monetdb-pf-checkins] pathfinder/runtime pf_support.mx, XQuery_0-24, 1.299.2.3, 1.299.2.4
by Stefan Manegold 06 Jun '08

06 Jun '08

On Fri, Jun 06, 2008 at 09:00:53PM +0000, Lefteris Sidirourgos wrote: > Update of /cvsroot/monetdb/pathfinder/runtime > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv9671/runtime > > Modified Files: > Tag: XQuery_0-24 > pf_support.mx > Log Message: > A rather aggresive fix for the order of attributes and nodes in updates. > > Now we reverse in algebra the order of insert as last and insert before at runtime like the MPS. This fix the bug [1981844] PF/alg: result order differs from mps. Also it fix the test Update/update.xq (which makes unnecessary the algebra specific output). Also other failing tests of updates are fixed in tests/BugTrack. > > However, now the test insert_test_order_seq fails:( However, this is a different error that should be addressed with a diffrent fix. > > To be continued... > > > U pf_support.mx > Index: pf_support.mx > =================================================================== > RCS file: /cvsroot/monetdb/pathfinder/runtime/pf_support.mx,v > retrieving revision 1.299.2.3 > retrieving revision 1.299.2.4 > diff -u -d -r1.299.2.3 -r1.299.2.4 > --- pf_support.mx 29 May 2008 16:45:15 -0000 1.299.2.3 > +++ pf_support.mx 6 Jun 2008 21:00:46 -0000 1.299.2.4 > @@ -3321,9 +3321,27 @@ > { > var inserts := command.ord_select(UPDATE_INSERT_FIRST, UPDATE_REPLACENODE); # [i,insertcommand] > if (inserts.count() > 0) { > + > + # UPDATE_INSERT_LAST and UPDATE_INSERT_BEFORE commands must be executed in reverse order > + var order := inserts.ord_uselect(UPDATE_INSERT_LAST, UPDATE_INSERT_BEFORE).copy().mark(0@0); > + if (order.count() > 1) { > + pre_tgt.access(BAT_WRITE).replace(order.leftjoin(order.mirror().leftjoin(pre_tgt).access(BAT_WRITE).revert().tmark(0@0))); > + pre_cont_tgt.access(BAT_WRITE).replace(order.leftjoin(order.mirror().leftjoin(pre_cont_tgt).access(BAT_WRITE).revert().tmark(0@0))); > + attr_tgt.access(BAT_WRITE).replace(order.leftjoin(order.mirror().leftjoin(attr_tgt).access(BAT_WRITE).revert().tmark(0@0))); > + attr_cont_tgt.access(BAT_WRITE).replace(order.leftjoin(order.mirror().leftjoin(attr_cont_tgt).access(BAT_WRITE).revert().tmark(0@0))); > + pre_ins.access(BAT_WRITE).replace(order.leftjoin(order.mirror().leftjoin(pre_ins).access(BAT_WRITE).revert().tmark(0@0))); > + pre_cont_ins.access(BAT_WRITE).replace(order.leftjoin(order.mirror().leftjoin(pre_cont_ins).access(BAT_WRITE).revert().tmark(0@0))); > + attr_ins.access(BAT_WRITE).replace(order.leftjoin(order.mirror().leftjoin(attr_ins).access(BAT_WRITE).revert().tmark(0@0))); > + attr_cont_ins.access(BAT_WRITE).replace(order.leftjoin(order.mirror().leftjoin(attr_cont_ins).access(BAT_WRITE).revert().tmark(0@0))); > + inserts.access(BAT_WRITE).replace(order.leftjoin(order.mirror().leftjoin(inserts).access(BAT_WRITE).revert().tmark(0@0))); > + command.access(BAT_WRITE).replace(order.leftjoin(order.mirror().leftjoin(command).access(BAT_WRITE).revert().tmark(0@0))); > + } Lefteris, thanks for the fix! I'm not completely sure, whether I understand what your above code does, or better, whether I understand how is does what *I think* it is supposed to do, i.e. revert the order of BUNs that are related to UPDATE_INSERT_LAST and UPDATE_INSERT_BEFORE commands in all BATs. However given my idea of what it is supposed to do, wouldn't the following solution look "simpler" and be more efficient? ======== var order := inserts.ord_uselect(UPDATE_INSERT_LAST, UPDATE_INSERT_BEFORE); if (order.count() > 1) { var redro := order.copy().access(BAT_WRITE).revert().access(BAT_READ); var order_redro := order.mark(0@0).leftfetchjoin(redro.tmark(0@0)); pre_tgt .access(BAT_WRITE).replace( order_redro.leftjoin(pre_tgt) ).access(BAT_READ); pre_cont_tgt .access(BAT_WRITE).replace( order_redro.leftjoin(pre_cont_tgt) ).access(BAT_READ); attr_tgt .access(BAT_WRITE).replace( order_redro.leftjoin(attr_tgt) ).access(BAT_READ); attr_cont_tgt.access(BAT_WRITE).replace( order_redro.leftjoin(attr_cont_tgt) ).access(BAT_READ); pre_ins .access(BAT_WRITE).replace( order_redro.leftjoin(pre_ins) ).access(BAT_READ); pre_cont_ins .access(BAT_WRITE).replace( order_redro.leftjoin(pre_cont_ins) ).access(BAT_READ); attr_ins .access(BAT_WRITE).replace( order_redro.leftjoin(attr_ins) ).access(BAT_READ); attr_cont_ins.access(BAT_WRITE).replace( order_redro.leftjoin(attr_cont_ins) ).access(BAT_READ); inserts .access(BAT_WRITE).replace( order_redro.leftjoin(inserts) ).access(BAT_READ); command .access(BAT_WRITE).replace( order_redro.leftjoin(command) ).access(BAT_READ); } ======== ... just an idea; might be completely wrong ... let's think about anc check it tomorrow or "live" on Monday ... Good night > + > var inserts_pre_ins := inserts.mirror().leftfetchjoin(pre_ins); # [i,PRE] > var inserts_elems := inserts_pre_ins.ord_uselect(oid_nil, oid_nil).hmark(0@0); # [j,i] inserts on elements > var inserts_attrs := inserts_pre_ins.ord_uselect(oid_nil).hmark(0@0); # [k,i] inserts on attributes > + > + > if (inserts_attrs.count() > 0) { > do_update_insert_attributes(ws, inserts_attrs.leftjoin(pre_tgt), inserts_attrs.leftjoin(pre_cont_tgt), inserts_attrs.leftjoin(attr_ins), inserts_attrs.leftjoin(attr_cont_ins)); > } > > > ------------------------------------------------------------------------- > Check out the new SourceForge.net Marketplace. > It's the best place to buy or sell services for > just about anything Open Source. > http://sourceforge.net/services/buy/index.php > _______________________________________________ > Monetdb-pf-checkins mailing list > Monetdb-pf-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-pf-checkins > > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 1

Re: [Monetdb-developers] [Monetdb-sql-checkins] sql/src/backends/monet5 sql_optimizer.mx, , 1.182, 1.183 sql_scenario.mx, , 1.308, 1.309
by Stefan Manegold 06 Jun '08

06 Jun '08

On Fri, Jun 06, 2008 at 08:25:18PM +0000, Martin Kersten wrote: > Update of /cvsroot/monetdb/sql/src/backends/monet5 > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv27738 > > Modified Files: > sql_optimizer.mx sql_scenario.mx > Log Message: > The first nightly test of the recycler ... will have to wait until Sunday: To have more time for (hopefully) migrating MonetDB testing to a new "server", I decided to run only Stable testing tonight, and "skip" Current testing for one night, assuming that Stable testing is currently more "mission-critical" than Current testing. Sorry for the inconveniences this might cause ... Stefan -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/modules/mal recycle.mx, , 1.27, 1.28
by Romulo Goncalves 06 Jun '08

06 Jun '08

Milena Ivanova wrote: > Update of /cvsroot/monetdb/MonetDB5/src/modules/mal > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv27013/mal > > Modified Files: > recycle.mx > Log Message: > Reset is called after an update instruction to clean the recycle cache. > > > U recycle.mx > Index: recycle.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/modules/mal/recycle.mx,v > retrieving revision 1.27 > retrieving revision 1.28 > diff -u -d -r1.27 -r1.28 > --- recycle.mx 5 Jun 2008 08:37:43 -0000 1.27 > +++ recycle.mx 6 Jun 2008 14:00:22 -0000 1.28 > @@ -70,7 +70,7 @@ > command getCachePolicy():int > address RECYCLEgetCache; > > -pattern reset():void > +pattern reset(bid:bat):void Is the optimized aware of this change? It might call for the command with wrong number of arguments and it might break several sql tests... Romulo > address RECYCLEresetCMD > comment "Reset off all recycled variables"; > > @@ -78,11 +78,6 @@ > address RECYCLEshutdownWrap > comment "Clear the recycle cache"; > > -command restore():void > -address RECYCLErestore > -comment "Restore the recycle cache to the start of the call"; > - > - > command monitor(f:int):void > address RECYCLEmonitor > comment "start/stop the monitoring (printing) of the recycler info (storage size used and number of statements retained)"; > @@ -207,7 +202,7 @@ > > /* and dump the statistics */ > if (recycleBlk) > - for(i=0; i< recycleBlk->stop; i++){ > + for(i=1; i< recycleBlk->stop; i++){ > stream_printf(cntxt->fdout,"# %4d lru= %8d cnt= %8d ticks= %9d rd= %9d wr= %9d %s\n", i, > recycleBlk->profiler[i].clk, > recycleBlk->profiler[i].counter, > > > ------------------------------------------------------------------------- > Check out the new SourceForge.net Marketplace. > It's the best place to buy or sell services for > just about anything Open Source. > http://sourceforge.net/services/buy/index.php > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins

1 0