February 2010 - developers-list

[Monetdb-developers] cracking pipeline
by Isidor Zeuner 13 Feb '10

13 Feb '10

Dear MonetDB developers, I'd like to write some words of gratitude for your work on the cracking code. I have read your paper describing the approach, which looks promising. And current code already seems to be somehow stable (I could get it to segfault using integer equality, and got wrong results using integer comparisons, but both were easy to work around .. nothing serious compared with the MALexception whack-a-mole I experienced with mitosis_pipe). Is there a roadmap somewhere for this part of the code, so I could see what to expect, or where to test/patch/contribute most appropriately? Best regards, Isidor Zeuner

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB/src/gdk gdk.mx, Feb2010, 1.301.2.9, 1.301.2.10 gdk_bat.mx, Feb2010, 1.236.2.4, 1.236.2.5 gdk_batop.mx, Feb2010, 1.180.2.15, 1.180.2.16 gdk_bbp.mx, Feb2010, 1.269.2.5, 1.269.2.6 gdk_posix.mx, Feb2010, 1.176.2.11, 1.176.2.12 gdk_relop.mx, Feb2010, 1.179.2.13, 1.179.2.14 gdk_storage.mx, Feb2010, 1.149.2.21, 1.149.2.22 gdk_utils.mx, Feb2010, 1.262.2.10, 1.262.2.11
by Stefan Manegold 12 Feb '10

12 Feb '10

Peter, > +/* a thread informs it is goin to (preload==1) or stops (preload==-1) using a range of memory */ > void > -MT_mmap_pin(void *base, size_t len) > +MT_mmap_inform(void *base, size_t len, int preload, int advise, int writable) > { > int i; > > (void) pthread_mutex_lock(&MT_mmap_lock); > i = MT_mmap_idx(base, len); > if (i >= 0) { > - MT_mmap_tab[i].pincnt++; > + if (writable) MT_mmap_tab[i].writable = (writable > 0); > + MT_mmap_tab[i].random += (advise == BUF_WILLNEED); /* done as a counter to keep track of multiple threads */ ^^^^^^^^^^^^^^^^^^^^^^^^ shouldn't this rather be something like MT_mmap_tab[i].random += preload * (advise == BUF_WILLNEED); ? Otherwise (i.e., in the current case), both BATpreload_() & _BATpreload() only increment MT_mmap_tab[i].random, but never decrement it, right? Stefan > + MT_mmap_tab[i].usecnt += preload; /* active thread count */ > + if (preload > 0) (void) posix_madvise(base, len, advise); > } > (void) pthread_mutex_unlock(&MT_mmap_lock); > > } -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4199 |

2 2

Re: [Monetdb-developers] [Monetdb-checkins] clients/src/mapiclient MapiClient.mx, , 1.194, 1.195
by Stefan de Konink 12 Feb '10

12 Feb '10

Op 12-02-10 18:01, Fabian Groffen schreef: > On 12-02-2010 16:37:28 +0000, Stefan de Konink wrote: >> Update of /cvsroot/monetdb/clients/src/mapiclient >> In directory sfp-cvsdas-1.v30.ch3.sourceforge.com:/tmp/cvs-serv30000 >> >> Modified Files: >> MapiClient.mx >> Log Message: >> Minor issue with save_history = false. > > you forgot a changelog entry for this feature. I didn't know there was a changelog ;) But it looks that that is branch related? And I just checked it into head? Stefan

2 1

[Monetdb-developers] @= atomtostr memory usage
by Stefan de Konink 11 Feb '10

11 Feb '10

Hi, While creating some new code and seeing unexpected, but interesting, behavior I stumbled upon the @= atomtostr code, some simple questions regarding this. It seems that the memory requested in @= atommem is rather pessimistic. It seems that for a regular integer 24 characters are allocated. Now since this integer seems to go up to 2^31 (10 chars) + sign + \0 (2 chars), why is the double amount needed if a per type approach is available in MAL? The actual toStr code finishes with a strlen, now this is nice behavior, but if we now know what the length of the string actually is, why not shoot a GDKrealloc on it, and make the *len the actual size? Regarding to memory management and memory function calling, the code isn't very efficient. Effectively all the functions seem to introduce their own buffers. If we look from a perspective of for example strConcat, the value is copied several times from serveral position. We could overcome this bahavior if we would return a maximum length for a type, the concat function would add the length of the second string alloc this buffer, forwards the pointer to the typical atomToStr call, and instead of allocing a new string, using the prealloced buffer. (...and yes, doing a GDKrealloc after that function wouldn't be good.) I took the liberty to come up with a patch for the second issue; Since snprintf will return the amount of bytes written, and since we have established that there is no numeric type overflowing the string space, we can ommit the strlen, use the amount of bytes + 1 from snprintf, realloc, and return the actual amount of bytes representing the string. Catches; 1) Old implementation resulted bestcase in an off by one return vs len 2) Is there any code dependent on the *len or that length will include the \0 value? If there are no objections, I'll check in that patch later today in head. >From benchmarking point of view, it might be worthwhile to test with and without the realloc. Stefan

2 2

[Monetdb-developers] confused by Timer/QueryHistory/CallHistory
by Mo Zhou 08 Feb '10

08 Feb '10

Hi, Sorry for bothering you again... I am confused by the time shown in Timer/queryHistory/callHistory. In queryHistory, there are two time: parse time and optimization time. In callHistory, there are two important time: xtime and rtime. These times should not overlap each other. but when I sum up these times, the number is different from the one returned by Timer. I wonder which time should I use for performance test. Thank you very much. -- Thanks, Mo

1 1

[Monetdb-developers] \f none not working
by Mo Zhou 08 Feb '10

08 Feb '10

Hi, I found "\f none" not working in XQuery. here is my commands: >\f none >\f Current formatter: none >for $t in doc("...")//Product return <pid> {pf:nid($t)} <XQueryResult>.....</pid><pid>1711511</pid></XQueryResult> >\f >Current formatter: none The results should not be printed according to the manual. "none: Finally, the none mode suppresses any output. This mode is mostly useful for performance measurements, if one wants to exclude the result serialization time." Could you please let me know if there are any solutions? Thanks! -- Thanks, Mo

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/modules/kernel batcast.mx, Feb2010, 1.31, 1.31.2.1 batstr.mx, Feb2010, 1.68, 1.68.2.1
by Stefan Manegold 06 Feb '10

06 Feb '10

On Sat, Feb 06, 2010 at 10:17:54AM +0000, Martin Kersten wrote: > Update of /cvsroot/monetdb/MonetDB5/src/modules/kernel > In directory sfp-cvsdas-1.v30.ch3.sourceforge.com:/tmp/cvs-serv685 > > Modified Files: > Tag: Feb2010 > batcast.mx batstr.mx > Log Message: > All string BATs should be preloaded to avoid random access to disk. Preloading their string heap(s) is indeed a good idea (unless they exceeds memory size, anyway) due to their inherently random access. However, preloading sequentially traversed BUN heaps might just create an unnecessarily huge memory footprint, in particular with large BATs. Hence, we might want to consider extending BATpreload to not uncoditionally preload all heaps of a BAT (up to four right now, since we ignore hash tables), but rather allow selective preload of those heaps that are (a) accessed at all, and (b) access randomly (or not accessed sequentially). I'll look into this later ... Stefan > > Index: batstr.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/modules/kernel/batstr.mx,v > retrieving revision 1.68 > retrieving revision 1.68.2.1 > diff -u -d -r1.68 -r1.68.2.1 > --- batstr.mx 7 Jan 2010 15:24:26 -0000 1.68 > +++ batstr.mx 6 Feb 2010 10:17:49 -0000 1.68.2.1 > @@ -239,6 +239,7 @@ > > bi = bat_iterator(b); > > + BATpreload(b); > BATloop(b, p, q) { > ptr h = BUNhead(bi,p); > x = (@3) BUNtail(bi,p); > @@ -275,6 +276,7 @@ > > bi = bat_iterator(b); > > + BATpreload(b); > BATloop(b, p, q) { > ptr h = BUNhead(bi,p); > str y = (str)str_nil, *yp = &y; > @@ -322,6 +324,7 @@ > > if (strcmp(*cst, str_nil) == 0) > isnil = 1; > + BATpreload(b); > BATloop(b, p, q) { > ptr h = BUNhead(bi, p); > > @@ -369,6 +372,8 @@ > lefti = bat_iterator(left); > righti = bat_iterator(right); > > + BATpreload(left); > + BATpreload(right); > BATloop(left, p, q) { > bit y = TRUE; > ptr h = BUNhead(lefti, p); > @@ -411,6 +416,8 @@ > lefti = bat_iterator(left); > righti = bat_iterator(right); > > + BATpreload(left); > + BATpreload(right); > BATloop(left, p, q) { > ptr h = BUNhead(lefti,p); > ptr tl = BUNtail(lefti,p); > @@ -444,6 +451,7 @@ > > lefti = bat_iterator(left); > > + BATpreload(left); > BATloop(left, p, q) { > ptr h = BUNhead(lefti,p); > ptr tl = BUNtail(lefti,p); > @@ -476,6 +484,8 @@ > lefti = bat_iterator(left); > righti = bat_iterator(right); > > + BATpreload(left); > + BATpreload(right); > BATloop(left, p, q) { > ptr h = BUNhead(lefti,p); > str tl = (str) BUNtail(lefti,p); > @@ -509,6 +519,7 @@ > > lefti = bat_iterator(left); > > + BATpreload(left); > BATloop(left, p, q) { > ptr h = BUNhead(lefti,p); > str tl = (str) BUNtail(lefti,p); > @@ -539,6 +550,7 @@ > > righti = bat_iterator(right); > > + BATpreload(right); > BATloop(right, p, q) { > ptr h = BUNhead(righti,p); > str tr = (str) BUNtail(righti,p); > @@ -579,6 +591,7 @@ > > bi = bat_iterator(b); > > + BATpreload(b); > BATloop(b, p, q) { > ptr h = BUNhead(bi, p); > str y = (str)str_nil, *yp = &y; > @@ -622,6 +635,7 @@ > > bi = bat_iterator(b); > > + BATpreload(b); > BATloop(b, p, q) { > ptr h = BUNhead(bi, p); > ptr t = BUNtail(bi, p); > @@ -666,6 +680,7 @@ > bn->tsorted = b->tsorted; > > bi = bat_iterator(b); > + BATpreload(b); > BATloop(b, p, q) { > ptr h = BUNhead(bi, p); > str t = (str) BUNtail(bi, p); > @@ -720,6 +735,7 @@ > lefti = bat_iterator(left); > starti = bat_iterator(start); > lengthi = bat_iterator(length); > + BATpreload(left); > BATloop(left, p, q) { > ptr h = BUNhead(lefti,p); > str tl = (str) BUNtail(lefti,p); > @@ -764,6 +780,7 @@ > bn->tsorted=0; > > li = bat_iterator(left); > + BATpreload(left); > BATloop(left, p, q) { > ptr h = BUNhead(li,p); > str tl = (str) BUNtail(li,p); > > Index: batcast.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/modules/kernel/batcast.mx,v > retrieving revision 1.31 > retrieving revision 1.31.2.1 > diff -u -d -r1.31 -r1.31.2.1 > --- batcast.mx 7 Jan 2010 15:24:26 -0000 1.31 > +++ batcast.mx 6 Feb 2010 10:17:47 -0000 1.31.2.1 > @@ -253,6 +253,7 @@ > @:voidresultBAT(TYPE_@1,"batcalc.@1")@ > bi = bat_iterator(b); > > + BATpreload(b); > BATloop(b, p, q) { > ptr v = NULL; > int len = 0; > > > ------------------------------------------------------------------------------ > The Planet: dedicated and managed hosting, cloud storage, colocation > Stay online with enterprise data centers and the best network in the business > Choose flexible plans and management services without long-term contracts > Personal 24x7 support from experience hosting pros just a phone call away. > http://p.sf.net/sfu/theplanet-com > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4199 |

2 1

[Monetdb-developers] Build MonetDB on Linux
by zhangyuan 06 Feb '10

06 Feb '10

I'm trying to build MonetDB default branch(HEAD) on Linux, and I have built buildtools, but when I go on with MonetDB, I met some trouble. When run the ./bootstrap under /MonetDB directory, it produces the following errors: [root@node1 MonetDB]# ./bootstrap automake 1.5 is 1.5 or newer. OK, unless you want to compile with Intel's icc (on Linux), which requires automake 1.7 or newer. autoconf 2.60 is 2.57 or newer. Good. libtool 1.5 is 1.4 or newer. Good. Python 2.5.1 is 2.0.0 or newer. Good. /root/monetDB/MonetDB/MonetDB /root/monetDB/MonetDB/MonetDB /root/monetDB/MonetDB/MonetDB/src /root/monetDB/MonetDB/MonetDB/src /root/monetDB/MonetDB/MonetDB /root/monetDB/MonetDB/MonetDB/src/common /root/monetDB/MonetDB/MonetDB/src/common /root/monetDB/MonetDB/MonetDB /root/monetDB/MonetDB/MonetDB/src/gdk /root/monetDB/MonetDB/MonetDB/src/gdk /root/monetDB/MonetDB/MonetDB /root/monetDB/MonetDB/MonetDB/conf /root/monetDB/MonetDB/MonetDB/conf /root/monetDB/MonetDB/MonetDB You should add the contents of `/usr/local/share/aclocal/libtool.m4' to `aclocal.m4'. Putting files in AC_CONFIG_AUX_DIR, `conf'. aclocal: configure.in: 63: macro `AM_MONETDB_XQ_VARS_1' not found in library aclocal: configure.in: 66: macro `AM_MONETDB_DEFAULTS' not found in library aclocal: configure.in: 69: macro `AM_MONETDB_COMPILER' not found in library aclocal: configure.in: 70: macro `AM_MONETDB_TOOLS' not found in library aclocal: configure.in: 71: macro `AM_MONETDB_OPTIONS' not found in library aclocal: configure.in: 72: macro `AM_MONETDB_UTILS' not found in library aclocal: configure.in: 85: macro `AM_MONETDB_LIBS' not found in library aclocal: configure.in: 119: macro `AM_MONETDB_XQ_VARS_2' not found in library [root@node1 MonetDB]# I must be doing something wrong, but I can't figure out the problem. Any help would be appreciated. 2010-02-06 zhangyuan

1 0

Re: [Monetdb-developers] [Monetdb-sql-checkins] sql/src/server rel_schema.mx, Feb2010, 1.2, 1.2.2.1
by Wouter Alink 05 Feb '10

05 Feb '10

fantastisch! 2010/2/5 Niels Nes <nielsnes(a)users.sourceforge.net>: > Update of /cvsroot/monetdb/sql/src/server > In directory sfp-cvsdas-1.v30.ch3.sourceforge.com:/tmp/cvs-serv25790/src/server > > Modified Files: > Tag: Feb2010 > rel_schema.mx > Log Message: > fixed bug in handeling topn in create table as select with data. > > > > Index: rel_schema.mx > =================================================================== > RCS file: /cvsroot/monetdb/sql/src/server/rel_schema.mx,v > retrieving revision 1.2 > retrieving revision 1.2.2.1 > diff -u -d -r1.2 -r1.2.2.1 > --- rel_schema.mx 11 Jan 2010 10:29:17 -0000 1.2 > +++ rel_schema.mx 5 Feb 2010 10:18:16 -0000 1.2.2.1 > @@ -127,9 +127,14 @@ > static char * > as_subquery( mvc *sql, sql_table *t, sql_rel *sq, dlist *column_spec ) > { > + sql_rel *r = sq; > + > + if (is_topn(r->op)) > + r = sq->l; > + > if (column_spec) { > dnode *n = column_spec->h; > - node *m = sq->exps->h; > + node *m = r->exps->h; > > for (; n; n = n->next, m = m->next) { > char *cname = n->data.sval; > @@ -143,7 +148,7 @@ > } else { > node *m; > > - for (m = sq->exps->h; m; m = m->next) { > + for (m = r->exps->h; m; m = m->next) { > sql_exp *e = m->data; > char *cname = exp_name(e); > sql_subtype *tp = exp_subtype(e); > > > ------------------------------------------------------------------------------ > The Planet: dedicated and managed hosting, cloud storage, colocation > Stay online with enterprise data centers and the best network in the business > Choose flexible plans and management services without long-term contracts > Personal 24x7 support from experience hosting pros just a phone call away. > http://p.sf.net/sfu/theplanet-com > _______________________________________________ > Monetdb-sql-checkins mailing list > Monetdb-sql-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-sql-checkins > >

2 1

[Monetdb-developers] How to know the database size and use indexes?
by Mo Zhou 04 Feb '10

04 Feb '10

Hi, I wonder if there is a way to get the size of a database (SQL/XQuery)? And I wonder if I can ask MonetDB (SQL/XQuery) to use a specific index explicitly? -- Thanks, Mo

2 3