developers-list April 2010

developers-list@monetdb.org

20 participants
40 discussions

[Monetdb-developers] encoding of strings returned by Python client

by Mark Bucciarelli

Any chance the Python client could be modified to return strings in Unicode instead of UTF-8? Seems like a better fit for Python; for example, that's how Python's sqlite3 module behaves. An alternative approach is to expose a set_encoding method on the connection that let's you specify the encoding of all text fields returned. If backward compatibility is an issue, the latter idea seems better even though it adds a knob. Any comments? (Is the maintainer of the Python adapter active even on this list? I didn't see many Python-specific posts in the archives.) Thanks, m

12 years, 1 month

[Monetdb-developers] IMPORTANT: move of MonetDB repository imminent

by Sjoerd Mullender

On Monday, May 3, 2010, we are going to move the MonetDB repository over from the Sourceforge-hosted CVS repository to a Mercurial repository hosted on dev.monetdb.org. On March 18 I sent a pre announcement to the monetdb-developers list (IMPORTANT: changes are afoot). [1] In particular, the following CVS modules will be converted and moved: MonetDB, MonetDB4, MonetDB5, buildtools, clients, geom, java, pathfinder, sql, template, testing. The other CVS modules that are hosted at Sourceforge (pf-haskell, pf-tools, xml) will remain there, at least for now. This means that as of Monday, the affected modules in the CVS repository will be made read only (it will remain readable for a long time, just not writable). When the move is complete, the repository will open at the following location: http://dev.monetdb.org/hg/MonetDB/ This URL can be used to browse the repository using a web browser, and it can be used as the URL to make a Mercurial clone (hg clone http://dev.monetdb.org/hg/MonetDB/). This will be a read-only version of the repository. Core developers will get read-write access at the URL ssh://hg@dev.monetdb.org/MonetDB/ I wrote a document to help with the conversion to Mercurial, both describing the way to use Mercurial, and how to convert your CVS working set to a Mercurial clone. For now (until we find a better place), this document can be accessed at: http://homepages.cwi.nl/~sjoerd/downloads/monetdb/MonetDB-Mercurial.html http://homepages.cwi.nl/~sjoerd/downloads/monetdb/MonetDB-Mercurial.pdf As part of the move, I will subscribe subscribers to the various monetdb checkin mailing lists to the new mailing list checkin-list(a)monetdb.org. If you are subscribed to one or more of monetdb-checkins(a)lists.sourceforge.net monetdb-sql-checkins(a)lists.sourceforge.net monetdb-pf-checkins(a)lists.sourcefoge.net I will subscribe you to the list checkin-list(a)monetdb.org. This I will do on Monday as well, so you should get a message to that effect on Monday. [1] http://sourceforge.net/mailarchive/forum.php?thread_name=4BA239BF.4090307%4… -- Sjoerd Mullender

12 years, 2 months

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/modules/mal tablet_sql.mx, Feb2010, 1.44, 1.44.2.1

by Sjoerd Mullender

Should I build a new release candidate with this fix included? On 2010-04-25 12:07, Niels Nes wrote: > Update of /cvsroot/monetdb/MonetDB5/src/modules/mal > In directory sfp-cvsdas-4.v30.ch3.sourceforge.com:/tmp/cvs-serv6881/src/modules/mal > > Modified Files: > Tag: Feb2010 > tablet_sql.mx > Log Message: > fixed bug in loader, now (again) we handle embedded seperators properly > > > Index: tablet_sql.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/modules/mal/tablet_sql.mx,v > retrieving revision 1.44 > retrieving revision 1.44.2.1 > diff -u -d -r1.44 -r1.44.2.1 > --- tablet_sql.mx 10 Jan 2010 14:16:36 -0000 1.44 > +++ tablet_sql.mx 25 Apr 2010 10:07:29 -0000 1.44.2.1 > @@ -448,11 +448,10 @@ > int res = 0; /* < 0: error, > 0: success, == 0: continue processing */ > int j; > BUN i; > - char q=0; > size_t rseplen; > READERtask *task= (READERtask*) GDKzalloc(sizeof(READERtask)); > READERtask ptask[16]; > - int threads= GDKnr_threads < 16? GDKnr_threads: 16; > + int threads= (!maxrow || maxrow > (1<<16)) ? (GDKnr_threads < 16? GDKnr_threads: 16):1; > lng lio =0, tio, t1 = 0, total=0, iototal=0; > int vmtrim = GDK_vm_trim; > > @@ -603,6 +602,7 @@ > rethink the matter. */ > e = s; > while (s < end && (maxrow < 0 || cnt < (BUN) maxrow) ) { > + char q=0; > /* tokenize the record completely > the format of the input should comply to > the following grammar rule > @@ -617,55 +617,50 @@ > */ > if ( quote == 0) { > if (rseplen == 1) > - for ( ; *e ; e++) > - { > + for ( ; *e ; e++) { > if ( *e == '\\' ){ > e++; > continue; > } > if (*e == *rsep ) > - break; > + break; > } > else > - for ( ; *e ; e++) > - { > + for ( ; *e ; e++) { > if ( *e == '\\' ) { > e++; > continue; > } > if (*e == *rsep && strncmp(e, rsep, rseplen) == 0) > - break; > + break; > } > if ( *e == 0) > e = 0; /* nonterminated record, we need more */ > - } else > - if ( rseplen == 1 ) { > - for ( ; *e ; e++) > - { > + } else if ( rseplen == 1 ) { > + for ( ; *e ; e++) { > if (*e == q) > - q = 0; > - else if (*e == quote) > - q = *e; > - else if (*e == '\\') { > - if (e[1]) > - e++; > - } else if (*e == *rsep ) > - break; > + q = 0; > + else if (*e == quote) > + q = *e; > + else if (*e == '\\') { > + if (e[1]) > + e++; > + } else if (!q && *e == *rsep ) > + break; > } > if ( *e == 0) > e = 0; /* nonterminated record, we need more */ > } else { > - for ( ; *e ; e++) > - { > + for ( ; *e ; e++) { > if (*e == q) > - q = 0; > - else if (*e == quote) > - q = *e; > - else if (*e == '\\') { > - if (e[1]) > - e++; > - } else if (*e == *rsep && strncmp(e, rsep, rseplen) == 0 ) > - break; > + q = 0; > + else if (*e == quote) > + q = *e; > + else if (*e == '\\') { > + if (e[1]) > + e++; > + } else if (!q && *e == *rsep && strncmp(e, rsep, rseplen) == 0 ) > + break; > } > if (*e == 0 ) > e = 0; /* nonterminated record, we need more */ > > > ------------------------------------------------------------------------------ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins -- Sjoerd Mullender

12 years, 3 months

[Monetdb-developers] MonetDB memory management

by Xiao Lei

Hi, Does anyone has the same memory management problem as me? When I inserted many records like: Insert into tbl1 (id,dt) values(1,blob ‘…………………………………………’); Memory is used up to 1G. That's OK. But when I delete the whole table like: Delete from tbl1; None memory is released, still 1G is used. And when i insert again, memory used keeps increasing. Is this a problem or is it the way MonetDB works? Thx. -- Best Regards, Xiao Lei

12 years, 3 months

[Monetdb-developers] Nov2009 segmentation fault (invalid read)

by Isidor Zeuner

Dear MonetDB developers, I am currently experiencing segmentation faults on SQL-based data processing using the MonetDB Nov2009 branch. As the processing is rather complex, I don't know when I will be able to isolate a test case. Based on the clues I gathered (see below), could you give me some pointers on how to investigate further? It always crashes on this query: DELETE FROM progress WHERE EXISTS ( SELECT true FROM finalization WHERE finalization.id=progress.id AND finalization.category=active ); The related tables are defined like this: CREATE TABLE progress ( id VARCHAR(32), attempt INTEGER, clearance INTEGER, PRIMARY KEY (id) ); CREATE TABLE finalization ( category VARCHAR(32), id VARCHAR(32), governing VARCHAR(64), sub_governing VARCHAR(64), domain VARCHAR(64), reason VARCHAR(16), epoch INTEGER, size INTEGER, PRIMARY KEY (category,id,governing,sub_governing,domain,reason,epoch) ); On the crash, the stack backtrace always looks like this: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x42006940 (LWP 17625)] 0x00002b8e691fdad8 in strCmp (l=0x2aaab97c6a20 "06EED104D173", r=0xdbdbdbdbe98848f3 <Address 0xdbdbdbdbe98848f3 out of bounds>) at gdk_atoms.mx:1855 1855 return GDK_STRCMP(l, r); (gdb) bt #0 0x00002b8e691fdad8 in strCmp (l=0x2aaab97c6a20 "06EED104D173", r=0xdbdbdbdbe98848f3 <Address 0xdbdbdbdbe98848f3 out of bounds>) at gdk_atoms.mx:1855 #1 0x00002b8e693d7322 in BAThashjoin (l=0x2aaab9417188, r=0xd614ba0, estimate=9223372036854775807) at gdk_relop.mx:452 #2 0x00002b8e6940e151 in batjoin (l=0x2aaab9417188, r=0xd614ba0, estimate=9223372036854775807, swap=1 '\001') at gdk_relop.mx:1244 #3 0x00002b8e6940e193 in BATjoin (l=0x2aaab9417188, r=0xd614ba0, estimate=9223372036854775807) at gdk_relop.mx:1251 #4 0x00002aaaaafd105e in CMDjoin () from /opt/monetdb-nov2009/lib64/MonetDB5/lib/lib_algebra.so.5 #5 0x00002aaaaaffcef5 in ALGjoinestimate () from /opt/monetdb-nov2009/lib64/MonetDB5/lib/lib_algebra.so.5 #6 0x00002aaaaaffcfc1 in ALGjoin () from /opt/monetdb-nov2009/lib64/MonetDB5/lib/lib_algebra.so.5 #7 0x00002b8e6868b106 in DFLOWstep () from /opt/monetdb-nov2009/lib64/libmonetdb5.so.5 #8 0x00002b8e6868ed7d in runDFLOWworker () from /opt/monetdb-nov2009/lib64/libmonetdb5.so.5 #9 0x00000039d04064a7 in start_thread () from /lib64/libpthread.so.0 #10 0x00000039cf8d3c2d in clone () from /lib64/libc.so.6 The second argument's address to the comparison at the top has always one or more 0xdb memory deletion markers, so it looks like it is at least overlapping with deleted space. Please note that I could not create a backtrace with full symbols as the error does not occur with the MonetDB5 module being compiled with debugging enabled. Best regards, Isidor

12 years, 3 months

[Monetdb-developers] Fw: BATcacheid

by anshul gangwar

Does anybody have any idea ............... hello Are BATcacheid's real column id's i.e those BATid's which are given at the time of storage of the document(static id's). If no than from where we can print those static BATid's

12 years, 3 months

[Monetdb-developers] BATcacheid

by anshul gangwar

hello Are BATcacheid's real column id's i.e those BATid's which are given at the time of storage of the document(static id's). If no than from where we can print those static BATid's

12 years, 3 months

[Monetdb-developers] ~~Hi~~

by Erin's Ghost

http://template09.mydevnet.ca/NepK4RsZ0U.htm

12 years, 3 months

[Monetdb-developers] flow diagram

by anshul gangwar

hello Where I can find the full flow diagram of execution of monetdb. Apart from that We are printing the BATcacheid of the columns passed to the BATfetchjoin function ,We are getting the different BATCacheids . So what is the reason behind that.

12 years, 3 months

[Monetdb-developers] flow diagram

by anshul gangwar

hello I was talking about flow diagram image, so that i can visualize that.

12 years, 3 months

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

developers-list April 2010