January 2010 - developers-list

[Monetdb-developers] encoding of strings returned by Python client
by Mark Bucciarelli 28 Jun '10

28 Jun '10

Any chance the Python client could be modified to return strings in Unicode instead of UTF-8? Seems like a better fit for Python; for example, that's how Python's sqlite3 module behaves. An alternative approach is to expose a set_encoding method on the connection that let's you specify the encoding of all text fields returned. If backward compatibility is an issue, the latter idea seems better even though it adds a knob. Any comments? (Is the maintainer of the Python adapter active even on this list? I didn't see many Python-specific posts in the archives.) Thanks, m

3 12

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB/src/gdk gdk_storage.mx, Feb2010, 1.149.2.5, 1.149.2.6
by Stefan Manegold 31 Jan '10

31 Jan '10

On Sun, Jan 31, 2010 at 04:43:29PM +0000, Martin Kersten wrote: > Update of /cvsroot/monetdb/MonetDB/src/gdk > In directory sfp-cvsdas-1.v30.ch3.sourceforge.com:/tmp/cvs-serv28035 > > Modified Files: > Tag: Feb2010 > gdk_storage.mx > Log Message: > A pity that the first code was not properly checked, because it did not > even tried to force the preload by memory touching properly. > As per request (though admittedly belated ...): Wouldn't it be sufficient (and faster and less memory consuming) to preload only that part of the heaps that is indeed used (i.e., holds useful/relevant data) instead of preloading the whole (possibly consciously over-allocated) heaps? I.e., replacing ...heap->size by ...heap->free (as it was in the original code, only that the original code incorrecly assumed ...heap->free to be an absolute address instead of a relative byte offset ...) Stefan > > Index: gdk_storage.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_storage.mx,v > retrieving revision 1.149.2.5 > retrieving revision 1.149.2.6 > diff -u -d -r1.149.2.5 -r1.149.2.6 > --- gdk_storage.mx 31 Jan 2010 15:50:19 -0000 1.149.2.5 > +++ gdk_storage.mx 31 Jan 2010 16:43:27 -0000 1.149.2.6 > @@ -703,28 +703,32 @@ > @c > size_t > BATpreload_(BAT *b) { > - size_t dummy = 0, *i; > + size_t dummy = 0, *i, *limit; > size_t step = MT_pagesize()/sizeof(size_t); > size_t pages = (size_t) (0.8 * MT_npages()); > > - if ( b->T->vheap && b->T->vheap->base ){ > - IODEBUG THRprintf(GDKout,"#BATpreload(%s): T->vheap\n", BATgetId(b)); > - for ( i = (size_t *) b->T->vheap->base; i < (size_t *) b->T->vheap->free && pages > 0; i+= step, pages--) > + if ( b->T->vheap && b->T->vheap->base && pages){ > + IODEBUG THRprintf(GDKout,"#BATpreloaded(%s): T->vheap\n", BATgetId(b)); > + limit = (size_t *) (b->T->vheap->base + b->T->vheap->size); > + for ( i = (size_t *) b->T->vheap->base; i < limit && pages > 0; i+= step, pages--) > dummy += *i; > } > - if ( b->H->vheap && b->H->vheap->base ){ > - IODEBUG THRprintf(GDKout,"#BATpreload(%s): H->vheap\n", BATgetId(b)); > - for ( i = (size_t *) b->H->vheap->base; i < (size_t *) b->H->vheap->free && pages > 0 ; i+= step, pages--) > + if ( b->T->heap.base && pages ){ > + IODEBUG THRprintf(GDKout,"#BATpreload(%s): T->heap\n", BATgetId(b)); > + limit = (size_t *) (b->T->heap.base + b->T->heap.size); > + for ( i = (size_t *) b->T->heap.base; i < limit && pages > 0; i+= step, pages--) > dummy += *i; > } > - if ( b->hheap && b->hheap->base ){ > - IODEBUG THRprintf(GDKout,"#BATpreload(%s): hheap\n", BATgetId(b)); > - for ( i = (size_t *) b->hheap->base; i < (size_t *)b->hheap->free && pages > 0 ; i+= step, pages--) > + if ( b->H->heap.base && pages ){ > + IODEBUG THRprintf(GDKout,"#BATpreload(%s): H->heap\n", BATgetId(b)); > + limit = (size_t *) (b->H->heap.base + b->H->heap.size); > + for ( i = (size_t *) b->H->heap.base; i < limit && pages > 0 ; i+= step, pages--) > dummy += *i; > } > - if ( b->theap && b->theap->base ){ > - IODEBUG THRprintf(GDKout,"#BATpreload(%s): theap\n", BATgetId(b)); > - for ( i = (size_t *) b->theap->base; i < (size_t *) b->theap->free && pages > 0; i+= step, pages--) > + if ( b->H->vheap && b->H->vheap->base && pages ){ > + IODEBUG THRprintf(GDKout,"#BATpreload(%s): H->vheap\n", BATgetId(b)); > + limit = (size_t *) (b->H->vheap->base + b->H->vheap->size); > + for ( i = (size_t *) b->H->vheap->base; i < limit && pages > 0 ; i+= step, pages--) > dummy += *i; > } > return dummy; > > > ------------------------------------------------------------------------------ > The Planet: dedicated and managed hosting, cloud storage, colocation > Stay online with enterprise data centers and the best network in the business > Choose flexible plans and management services without long-term contracts > Personal 24x7 support from experience hosting pros just a phone call away. > http://p.sf.net/sfu/theplanet-com > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins > > > > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4199 |

2 1

[Monetdb-developers] How to not print on screen?
by Mo Zhou 29 Jan '10

29 Jan '10

Hi, I wonder if there is a way to not print query results on screen? -- Thanks, Mo

4 5

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/modules/kernel aggr.mx, , 1.41, 1.42
by Stefan Manegold 29 Jan '10

29 Jan '10

On Fri, Jan 29, 2010 at 04:41:00AM +0000, Stefan de Konink wrote: > Update of /cvsroot/monetdb/MonetDB5/src/modules/kernel > In directory sfp-cvsdas-1.v30.ch3.sourceforge.com:/tmp/cvs-serv5117 > > Modified Files: > aggr.mx > Log Message: > Fixes typo's and/or inconsistent throw references to function names. > Please propagate to stable. The standard policy is to check in bug fixes to the latest release branch; propagation will than happen automa[tgn]ically. Back-porting such changes by hand creates quite some overhead. Stefan > > > Index: aggr.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/modules/kernel/aggr.mx,v > retrieving revision 1.41 > retrieving revision 1.42 > diff -u -d -r1.41 -r1.42 > --- aggr.mx 7 Jan 2010 15:24:26 -0000 1.41 > +++ aggr.mx 29 Jan 2010 04:40:58 -0000 1.42 > @@ -276,7 +276,7 @@ > } > BBPreleaseref(b->batCacheid); > BBPreleaseref(e->batCacheid); > - throw(MAL, "aggrX3_sum.@4",GDK_EXCEPTION); > + throw(MAL, "aggrX3_prod.@4",GDK_EXCEPTION); > } > @- > @= AX3sumprod > @@ -548,7 +548,7 @@ > BBPreleaseref(b->batCacheid); > BBPreleaseref(e->batCacheid); > BBPreleaseref(g->batCacheid); > - throw(MAL, "aggrX3_prod3",GDK_EXCEPTION); > + throw(MAL, "aggrX3_avg3",GDK_EXCEPTION); > } > @c > @:AX3arithavg3(bte,loc)@ > @@ -590,7 +590,7 @@ > BBPreleaseref(b->batCacheid); > BBPreleaseref(e->batCacheid); > BBPreleaseref(g->batCacheid); > - throw(MAL, "aggrX3_prod3",GDK_EXCEPTION); > + throw(MAL, "aggrX3_@13",GDK_EXCEPTION); > } > @c > @:AX3extreme3(min,<)@ > @@ -627,7 +627,7 @@ > BBPreleaseref(b->batCacheid); > BBPreleaseref(e->batCacheid); > BBPreleaseref(g->batCacheid); > - throw(MAL, "aggrX3_prod3", GDK_EXCEPTION); > + throw(MAL, "aggrX3_count3", GDK_EXCEPTION); > } > @- > The remainder contains the few MIL proc translations > > > ------------------------------------------------------------------------------ > The Planet: dedicated and managed hosting, cloud storage, colocation > Stay online with enterprise data centers and the best network in the business > Choose flexible plans and management services without long-term contracts > Personal 24x7 support from experience hosting pros just a phone call away. > http://p.sf.net/sfu/theplanet-com > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins > > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4199 |

1 0

[Monetdb-developers] Announcement: New Nov2009-SP2 bugfix release of MonetDB suite
by Sjoerd Mullender 28 Jan '10

28 Jan '10

The MonetDB team at CWI/MonetDB BV is pleased to announce the Nov2009-SP2 bug fix release of the MonetDB suite of programs. More information on this release will be available at <http://monetdb.cwi.nl/Development/Releases/Nov2009/>. Fixes include: - No need to require bash for running the *-config scripts. (SF bug #2914563) - MonetDB5 development package now contains all development include files. (SF RFE #2929558) - Added support for compiling on OpenBSD (contributed, i.e. not tested by us). - General code improvements, partly inspired by running Coverity Scan. - mclient -D will exit with a non-zero exit code if dumping failed. (SF bug #2925674) - The Python interface now uses the same socket options as the C version, giving a boost in performance. (SF bug #2925750) - Fixed a bug in the Python interface to deal with fields with embedded newline characters. (SF bug #2917219) - Added initial support to the Python client interface for specifying the encoding. - Fixed a bug in execute argument parsing in Python interface. - Added missing .sql files to the Windows installer. - In MonetDB/SQL, fixed bug in not(invalidtable). (SF bug #2927174) -- Sjoerd Mullender

1 0

[Monetdb-developers] Regarding indexes in MonetDB XQuery
by Mo Zhou 27 Jan '10

27 Jan '10

Hi, Does MonetDB XQuery support any type of structural indexes? Or caching results of a frequently queried path (with branches)? -- Thanks, M

1 0

[Monetdb-developers] mmap region resizing
by Isidor Zeuner 26 Jan '10

26 Jan '10

While figuring out some things related to memory management, I stumbled upon a line in MonetDB/src/gdk/gdk_posix.mx which I don't fully understand: munmap(p + oldsize, oldsize - newsize); ..which is called from MT_vmrealloc when the old size is larger than the new size. This would unmap a region as large as the size difference between both sizes, but after the currently mapped region [p; p + oldsize[ (which is probably not mapped, or is mapped to something unrelated). The attached patch changes this behavior so the resulting mapped region will be [p; p + newsize[, with the now unneeded [p + newsize; p + oldsize[ region being unmapped (which is a behavior I find more intuitive). Please review and use as it fits. Best regards, Isidor Zeuner

1 0

[Monetdb-developers] GDKupgradevarheap allocate/copy inconsistency
by Isidor Zeuner 23 Jan '10

23 Jan '10

Dear MonetDB developers, please see the newly created tracker item [1]. I haven't seen relevant design documentation which would have allowed me to deduce if BATs from VIEWcreate_ are actually supposed to end up in GDKupgradevarheap, as it happened to me. Still, I could imagine the suggested fix to be a step in the right direction as it helps modularity by making the function working with less preassumptions on the inputs. However, if it fits into the global design, the capacity argument can (should?) be removed from the function signature. Best regards, Isidor Zeuner [1] https://sourceforge.net/tracker/?func=detail&aid=2936616&group_id=56967&ati…

3 6

[Monetdb-developers] sql_delta ibase issue
by Isidor Zeuner 21 Jan '10

21 Jan '10

Dear MonetDB developers, using MonetDB-SQL Feb2010, I occasionally encounter this error message: bat_storage.mx:1274: tr_update_delta: Assertion `obat->ibase == cbat->ibase' failed It occurs only seldomly, so I'll probably not have an isolated test case all too soon. I wonder if someone with more insight into the SQL engine has some pointers on where to look at. Many thanks, Isidor Zeuner

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/modules/mal groupby.mx, , 1.2, 1.3
by Lefteris 17 Jan '10

17 Jan '10

it still calls GROUPmin and not GROUPavg is that the intention? On Sun, Jan 17, 2010 at 8:32 PM, Niels Nes <nielsnes(a)users.sourceforge.net> wrote: > Update of /cvsroot/monetdb/MonetDB5/src/modules/mal > In directory sfp-cvsdas-1.v30.ch3.sourceforge.com:/tmp/cvs-serv30908/src/modules/mal > > Modified Files: > groupby.mx > Log Message: > fix cut/paste bug > > > Index: groupby.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/modules/mal/groupby.mx,v > retrieving revision 1.2 > retrieving revision 1.3 > diff -u -d -r1.2 -r1.3 > --- groupby.mx 7 Jan 2010 15:24:26 -0000 1.2 > +++ groupby.mx 17 Jan 2010 19:32:16 -0000 1.3 > @@ -74,7 +74,7 @@ > > pattern avg(pivot:bat[:oid,:any_1],b:bat[:oid,:any_2]...) :bat[:oid,:any_2] > address GROUPmin > -comment "Derive the grouping of all tuples in the BATs and determine minimum value. > +comment "Derive the grouping of all tuples in the BATs and determine average value. > The group oid is materialized."; > > @- > > > ------------------------------------------------------------------------------ > Throughout its 18-year history, RSA Conference consistently attracts the > world's best and brightest in the field, creating opportunities for Conference > attendees to learn about information security's most important issues through > interactions with peers, luminaries and emerging and established companies. > http://p.sf.net/sfu/rsaconf-dev2dev > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins >

1 0