April 2010 - developers-list

[Monetdb-developers] encoding of strings returned by Python client
by Mark Bucciarelli 28 Jun '10

28 Jun '10

Any chance the Python client could be modified to return strings in Unicode instead of UTF-8? Seems like a better fit for Python; for example, that's how Python's sqlite3 module behaves. An alternative approach is to expose a set_encoding method on the connection that let's you specify the encoding of all text fields returned. If backward compatibility is an issue, the latter idea seems better even though it adds a knob. Any comments? (Is the maintainer of the Python adapter active even on this list? I didn't see many Python-specific posts in the archives.) Thanks, m

3 12

[Monetdb-developers] IMPORTANT: move of MonetDB repository imminent
by Sjoerd Mullender 04 May '10

04 May '10

On Monday, May 3, 2010, we are going to move the MonetDB repository over from the Sourceforge-hosted CVS repository to a Mercurial repository hosted on dev.monetdb.org. On March 18 I sent a pre announcement to the monetdb-developers list (IMPORTANT: changes are afoot). [1] In particular, the following CVS modules will be converted and moved: MonetDB, MonetDB4, MonetDB5, buildtools, clients, geom, java, pathfinder, sql, template, testing. The other CVS modules that are hosted at Sourceforge (pf-haskell, pf-tools, xml) will remain there, at least for now. This means that as of Monday, the affected modules in the CVS repository will be made read only (it will remain readable for a long time, just not writable). When the move is complete, the repository will open at the following location: http://dev.monetdb.org/hg/MonetDB/ This URL can be used to browse the repository using a web browser, and it can be used as the URL to make a Mercurial clone (hg clone http://dev.monetdb.org/hg/MonetDB/). This will be a read-only version of the repository. Core developers will get read-write access at the URL ssh://hg@dev.monetdb.org/MonetDB/ I wrote a document to help with the conversion to Mercurial, both describing the way to use Mercurial, and how to convert your CVS working set to a Mercurial clone. For now (until we find a better place), this document can be accessed at: http://homepages.cwi.nl/~sjoerd/downloads/monetdb/MonetDB-Mercurial.html http://homepages.cwi.nl/~sjoerd/downloads/monetdb/MonetDB-Mercurial.pdf As part of the move, I will subscribe subscribers to the various monetdb checkin mailing lists to the new mailing list checkin-list(a)monetdb.org. If you are subscribed to one or more of monetdb-checkins(a)lists.sourceforge.net monetdb-sql-checkins(a)lists.sourceforge.net monetdb-pf-checkins(a)lists.sourcefoge.net I will subscribe you to the list checkin-list(a)monetdb.org. This I will do on Monday as well, so you should get a message to that effect on Monday. [1] http://sourceforge.net/mailarchive/forum.php?thread_name=4BA239BF.4090307%4… -- Sjoerd Mullender

3 2

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/modules/mal tablet_sql.mx, Feb2010, 1.44, 1.44.2.1
by Sjoerd Mullender 25 Apr '10

25 Apr '10

Should I build a new release candidate with this fix included? On 2010-04-25 12:07, Niels Nes wrote: > Update of /cvsroot/monetdb/MonetDB5/src/modules/mal > In directory sfp-cvsdas-4.v30.ch3.sourceforge.com:/tmp/cvs-serv6881/src/modules/mal > > Modified Files: > Tag: Feb2010 > tablet_sql.mx > Log Message: > fixed bug in loader, now (again) we handle embedded seperators properly > > > Index: tablet_sql.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/modules/mal/tablet_sql.mx,v > retrieving revision 1.44 > retrieving revision 1.44.2.1 > diff -u -d -r1.44 -r1.44.2.1 > --- tablet_sql.mx 10 Jan 2010 14:16:36 -0000 1.44 > +++ tablet_sql.mx 25 Apr 2010 10:07:29 -0000 1.44.2.1 > @@ -448,11 +448,10 @@ > int res = 0; /* < 0: error, > 0: success, == 0: continue processing */ > int j; > BUN i; > - char q=0; > size_t rseplen; > READERtask *task= (READERtask*) GDKzalloc(sizeof(READERtask)); > READERtask ptask[16]; > - int threads= GDKnr_threads < 16? GDKnr_threads: 16; > + int threads= (!maxrow || maxrow > (1<<16)) ? (GDKnr_threads < 16? GDKnr_threads: 16):1; > lng lio =0, tio, t1 = 0, total=0, iototal=0; > int vmtrim = GDK_vm_trim; > > @@ -603,6 +602,7 @@ > rethink the matter. */ > e = s; > while (s < end && (maxrow < 0 || cnt < (BUN) maxrow) ) { > + char q=0; > /* tokenize the record completely > the format of the input should comply to > the following grammar rule > @@ -617,55 +617,50 @@ > */ > if ( quote == 0) { > if (rseplen == 1) > - for ( ; *e ; e++) > - { > + for ( ; *e ; e++) { > if ( *e == '\\' ){ > e++; > continue; > } > if (*e == *rsep ) > - break; > + break; > } > else > - for ( ; *e ; e++) > - { > + for ( ; *e ; e++) { > if ( *e == '\\' ) { > e++; > continue; > } > if (*e == *rsep && strncmp(e, rsep, rseplen) == 0) > - break; > + break; > } > if ( *e == 0) > e = 0; /* nonterminated record, we need more */ > - } else > - if ( rseplen == 1 ) { > - for ( ; *e ; e++) > - { > + } else if ( rseplen == 1 ) { > + for ( ; *e ; e++) { > if (*e == q) > - q = 0; > - else if (*e == quote) > - q = *e; > - else if (*e == '\\') { > - if (e[1]) > - e++; > - } else if (*e == *rsep ) > - break; > + q = 0; > + else if (*e == quote) > + q = *e; > + else if (*e == '\\') { > + if (e[1]) > + e++; > + } else if (!q && *e == *rsep ) > + break; > } > if ( *e == 0) > e = 0; /* nonterminated record, we need more */ > } else { > - for ( ; *e ; e++) > - { > + for ( ; *e ; e++) { > if (*e == q) > - q = 0; > - else if (*e == quote) > - q = *e; > - else if (*e == '\\') { > - if (e[1]) > - e++; > - } else if (*e == *rsep && strncmp(e, rsep, rseplen) == 0 ) > - break; > + q = 0; > + else if (*e == quote) > + q = *e; > + else if (*e == '\\') { > + if (e[1]) > + e++; > + } else if (!q && *e == *rsep && strncmp(e, rsep, rseplen) == 0 ) > + break; > } > if (*e == 0 ) > e = 0; /* nonterminated record, we need more */ > > > ------------------------------------------------------------------------------ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins -- Sjoerd Mullender

1 0

[Monetdb-developers] MonetDB memory management
by Xiao Lei 23 Apr '10

23 Apr '10

Hi, Does anyone has the same memory management problem as me? When I inserted many records like: Insert into tbl1 (id,dt) values(1,blob ‘…………………………………………’); Memory is used up to 1G. That's OK. But when I delete the whole table like: Delete from tbl1; None memory is released, still 1G is used. And when i insert again, memory used keeps increasing. Is this a problem or is it the way MonetDB works? Thx. -- Best Regards, Xiao Lei

3 3

[Monetdb-developers] Nov2009 segmentation fault (invalid read)
by Isidor Zeuner 23 Apr '10

23 Apr '10

Dear MonetDB developers, I am currently experiencing segmentation faults on SQL-based data processing using the MonetDB Nov2009 branch. As the processing is rather complex, I don't know when I will be able to isolate a test case. Based on the clues I gathered (see below), could you give me some pointers on how to investigate further? It always crashes on this query: DELETE FROM progress WHERE EXISTS ( SELECT true FROM finalization WHERE finalization.id=progress.id AND finalization.category=active ); The related tables are defined like this: CREATE TABLE progress ( id VARCHAR(32), attempt INTEGER, clearance INTEGER, PRIMARY KEY (id) ); CREATE TABLE finalization ( category VARCHAR(32), id VARCHAR(32), governing VARCHAR(64), sub_governing VARCHAR(64), domain VARCHAR(64), reason VARCHAR(16), epoch INTEGER, size INTEGER, PRIMARY KEY (category,id,governing,sub_governing,domain,reason,epoch) ); On the crash, the stack backtrace always looks like this: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x42006940 (LWP 17625)] 0x00002b8e691fdad8 in strCmp (l=0x2aaab97c6a20 "06EED104D173", r=0xdbdbdbdbe98848f3 <Address 0xdbdbdbdbe98848f3 out of bounds>) at gdk_atoms.mx:1855 1855 return GDK_STRCMP(l, r); (gdb) bt #0 0x00002b8e691fdad8 in strCmp (l=0x2aaab97c6a20 "06EED104D173", r=0xdbdbdbdbe98848f3 <Address 0xdbdbdbdbe98848f3 out of bounds>) at gdk_atoms.mx:1855 #1 0x00002b8e693d7322 in BAThashjoin (l=0x2aaab9417188, r=0xd614ba0, estimate=9223372036854775807) at gdk_relop.mx:452 #2 0x00002b8e6940e151 in batjoin (l=0x2aaab9417188, r=0xd614ba0, estimate=9223372036854775807, swap=1 '\001') at gdk_relop.mx:1244 #3 0x00002b8e6940e193 in BATjoin (l=0x2aaab9417188, r=0xd614ba0, estimate=9223372036854775807) at gdk_relop.mx:1251 #4 0x00002aaaaafd105e in CMDjoin () from /opt/monetdb-nov2009/lib64/MonetDB5/lib/lib_algebra.so.5 #5 0x00002aaaaaffcef5 in ALGjoinestimate () from /opt/monetdb-nov2009/lib64/MonetDB5/lib/lib_algebra.so.5 #6 0x00002aaaaaffcfc1 in ALGjoin () from /opt/monetdb-nov2009/lib64/MonetDB5/lib/lib_algebra.so.5 #7 0x00002b8e6868b106 in DFLOWstep () from /opt/monetdb-nov2009/lib64/libmonetdb5.so.5 #8 0x00002b8e6868ed7d in runDFLOWworker () from /opt/monetdb-nov2009/lib64/libmonetdb5.so.5 #9 0x00000039d04064a7 in start_thread () from /lib64/libpthread.so.0 #10 0x00000039cf8d3c2d in clone () from /lib64/libc.so.6 The second argument's address to the comparison at the top has always one or more 0xdb memory deletion markers, so it looks like it is at least overlapping with deleted space. Please note that I could not create a backtrace with full symbols as the error does not occur with the MonetDB5 module being compiled with debugging enabled. Best regards, Isidor

1 1

[Monetdb-developers] Fw: BATcacheid
by anshul gangwar 22 Apr '10

22 Apr '10

Does anybody have any idea ............... hello Are BATcacheid's real column id's i.e those BATid's which are given at the time of storage of the document(static id's). If no than from where we can print those static BATid's

1 0

[Monetdb-developers] BATcacheid
by anshul gangwar 21 Apr '10

21 Apr '10

hello Are BATcacheid's real column id's i.e those BATid's which are given at the time of storage of the document(static id's). If no than from where we can print those static BATid's

1 0

[Monetdb-developers] ~~Hi~~
by Erin's Ghost 21 Apr '10

21 Apr '10

http://template09.mydevnet.ca/NepK4RsZ0U.htm

1 0

[Monetdb-developers] flow diagram
by anshul gangwar 21 Apr '10

21 Apr '10

hello Where I can find the full flow diagram of execution of monetdb. Apart from that We are printing the BATcacheid of the columns passed to the BATfetchjoin function ,We are getting the different BATCacheids . So what is the reason behind that.

2 1

[Monetdb-developers] flow diagram
by anshul gangwar 21 Apr '10

21 Apr '10

hello I was talking about flow diagram image, so that i can visualize that.

1 0