May 2009 - developers-list

[Monetdb-developers] new release branch
by Sjoerd Mullender 15 May '09

15 May '09

I am planning to create a new "stable" branch today or tomorrow for the upcoming May feature release. -- Sjoerd Mullender

1 0

Re: [Monetdb-developers] [Monetdb-pf-checkins] pathfinder/modules/pftijah pftijah.mx, , 1.228, 1.229
by Sjoerd Mullender 14 May '09

14 May '09

It's slightly more complex than this. At the MIL level you're right, you need to use wrd instead of int for bat counts. On the C level however, you should use the BUN type and also check for overflow when converting a wrd parameter to BUN (if sizeof(wrd) > sizeof(BUN) which is entirely possible). And then the real bug in your checkin: you use OIDFMT to print some value of type wrd. This is incorrect. You must use LLFMT together with a cast to lng for that. (Better would be to use BUN and BUNFMT.) I guess we could introduce a WRDFMT for printing wrd values, but we don't have that currently. On 2009-05-14 19:42, Roberto Cornacchia wrote: > Update of /cvsroot/monetdb/pathfinder/modules/pftijah > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv25786 > > Modified Files: > pftijah.mx > Log Message: > This commit is a temporary fix for the incorrect definition of count() mil proc: > count(BAT b) : int > > This should be instead: > > count(BAT b) : wrd > > !! Note that the problem should be solved properly in algebra.mx, changing the definition of count() and fixing all related code. > > In addition, this commit fixes some more incorrect usage of the int type, for example all bat offsets should be wrd and not int. > > > > U pftijah.mx > Index: pftijah.mx > =================================================================== > RCS file: /cvsroot/monetdb/pathfinder/modules/pftijah/pftijah.mx,v > retrieving revision 1.228 > retrieving revision 1.229 > diff -u -d -r1.228 -r1.229 > --- pftijah.mx 13 May 2009 07:35:03 -0000 1.228 > +++ pftijah.mx 14 May 2009 17:42:09 -0000 1.229 > @@ -117,7 +117,7 @@ > : BAT[oid,oid] = CMDpf2tijah_node; > "Translate Pathfinder node sequence to tijah node sequence" > > -.COMMAND offsetindex( BAT[void,oid] offset_tid, int res_size) > +.COMMAND offsetindex( BAT[void,oid] offset_tid, wrd res_size) > : BAT[void,oid] = CMDoffsetindex; > > "PARAMETERS: > @@ -127,7 +127,7 @@ > creates an offset index. > [...972 lines suppressed...] > BATiter tidprei, tidsizei, oldindexi, oldprei, oldsizei; > oid *s_newindex, *lst_newindex, *s_newpre, *lst_newpre; > @@ -5238,7 +5249,7 @@ > newsize = BATnew(TYPE_void, TYPE_int, ressize); > if (newsize == NULL) > { > - GDKerror("%s: could not allocate a result BAT[void,int] of size %d.\n", name, ressize); > + GDKerror("%s: could not allocate a result BAT[void,int] of size "OIDFMT".\n", name, ressize); > BATfree(res); > BATfree(newpre); > return(GDK_FAIL); > @@ -5248,7 +5259,7 @@ > newindex = BATnew(TYPE_void, TYPE_oid, ressize); > if (newindex == NULL) > { > - GDKerror("%s: could not allocate a result BAT[void,oid] of size %d.\n", name, ressize); > + GDKerror("%s: could not allocate a result BAT[void,oid] of size "OIDFMT".\n", name, ressize); > BATfree(res); > BATfree(newpre); > BATfree(newsize); > > > ------------------------------------------------------------------------------ > The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your > production scanning environment may not be a perfect world - but thanks to > Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700 > Series Scanner you'll get full speed at 300 dpi even with all image > processing features enabled. http://p.sf.net/sfu/kodak-com > _______________________________________________ > Monetdb-pf-checkins mailing list > Monetdb-pf-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-pf-checkins -- Sjoerd Mullender

2 1

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/modules/mal mal_mapi.mx, , 1.27, 1.28
by Stefan Manegold 12 May '09

12 May '09

On Sun, May 10, 2009 at 06:59:05PM +0000, Fabian wrote: > Update of /cvsroot/monetdb/MonetDB5/src/modules/mal > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv28294/src/modules/mal > > Modified Files: > mal_mapi.mx > Log Message: > refine protocol v9 to use the hash the server told the client > > The hash algorithm can be chosen at configure time, defaulting to > SHA512, mainly because conversion between hashes is impossible, and this > way we are forward compatible for a long time, by allowing the DBA to > keep on using an older/weaker hash algorithm in favour of dumping and > restoring the database. > > !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! > !!! TRUNK USERS, BEWARE: !!! > !!! !!! > !!! If you want to keep on using an existing trunk database, you !!! > !!! have to configure MonetDB5 with --with-password-backend=SHA1 !!! > !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! > > > U mal_mapi.mx > Index: mal_mapi.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/modules/mal/mal_mapi.mx,v > retrieving revision 1.27 > retrieving revision 1.28 > diff -u -d -r1.27 -r1.28 > --- mal_mapi.mx 6 May 2009 19:59:51 -0000 1.27 > +++ mal_mapi.mx 10 May 2009 18:59:03 -0000 1.28 > @@ -440,16 +440,17 @@ > /* generate the challenge string */ > challenge = alloca(sizeof(char) * (12 + 1)); > generateChallenge(challenge, 8, 12); > - if (AUTHgetHashAlgorithms(&algos) != MAL_SUCCEED) assert (0); > - /* note that we claim to speak proto 9 here for SHA-1 passwords */ > - stream_printf(fdout, "%s:mserver:9:%s:%s", > + assert(AUTHgetHashAlgorithms(&algos) == MAL_SUCCEED); ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This call will only be executed (and hence variable algos initialized, in case assertions are enabled. Maybe, this is not was you intended... (Assertions are (by default) disabled in released binary distributions and source tarballs.) Stefan > + /* note that we claim to speak proto 9 here for hashed passwords */ > + stream_printf(fdout, "%s:mserver:9:%s:%s:%s:", > challenge, > algos, > #ifdef WORDS_BIGENDIAN > - "BIG" > + "BIG", > #else > - "LIT" > + "LIT", > #endif > + MONETDB5_PASSWDHASH > ); > GDKfree(algos); > stream_flush(fdout); > > > ------------------------------------------------------------------------------ > The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your > production scanning environment may not be a perfect world - but thanks to > Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700 > Series Scanner you'll get full speed at 300 dpi even with all image > processing features enabled. http://p.sf.net/sfu/kodak-com > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 1

[Monetdb-developers] Mapi wire protocol level 9
by Fabian.Groffen＠cwi.nl 11 May '09

11 May '09

Hi, As I'm working on changing the password backend of M5 to use hashes instead of cypher3d clear text passwords[1], I have to bump the wire protocol level from 8 to 9, since the client will have to construct the challenge response differently. Since this is a bump, this is a good time to also incorporate (or prepara) any other changes that we would like to see in the wire protocol for the next feature release. [1] to allow sql to dump users and passwords in a safe manner, the dump needs to be able restore passwords from a hash, which are then stored in the backend. Hence the backend needs to work with those hashes instead of clear text passwords, as it currently does.

2 1

Re: [Monetdb-developers] [Monetdb-sql-checkins] sql/src/backends/monet5 sql_user.mx, , 1.42, 1.43
by Stefan de Konink 11 May '09

11 May '09

Fabian wrote: > int len = strlen(passwd); > - if ((ret = AUTHSHA1Sum(&pwd, &passwd, &len)) != MAL_SUCCEED) { > + if ((ret = AUTHBackendSum(&pwd, &passwd, &len)) != MAL_SUCCEED) { A sum of 0 will not work. Stefan

2 1

Re: [Monetdb-developers] [Monetdb-pf-checkins] pathfinder/compiler/include Makefile.ag, , 1.52, 1.53 logdebug.h, , 1.9, 1.10 properties.h, , 1.51, 1.52 xml2lalg.h, , 1.6, 1.7
by Stefan Manegold 07 May '09

07 May '09

Jan, could you please also check in the below included header file "plan_bundle.h" ? Thanks! Stefan On Thu, May 07, 2009 at 08:00:19PM +0000, Jan Rittinger wrote: > Update of /cvsroot/monetdb/pathfinder/compiler/include > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv23317/include > > Modified Files: > Makefile.ag logdebug.h properties.h xml2lalg.h > Log Message: > -- Extend Pathfinder (and the binaries pfopt and pfsql) to cope > with query plan bundles (generated by ferry). Multiple algebraic > plans bundled together form a query with a nested result type. > > We can now import the XML representation of a query plan bundle > and optimize all plans in the bundle. Furthermore we can generate > a DOT, an XML, and a SQL representation of the query plan bundle. > > NOTE: The XML import for 'normal' query plans as well as the code > and output generation still works as before. > > > U xml2lalg.h > Index: xml2lalg.h > =================================================================== > RCS file: /cvsroot/monetdb/pathfinder/compiler/include/xml2lalg.h,v > retrieving revision 1.6 > retrieving revision 1.7 > diff -u -d -r1.6 -r1.7 > --- xml2lalg.h 8 Jan 2009 16:54:12 -0000 1.6 > +++ xml2lalg.h 7 May 2009 20:00:16 -0000 1.7 > @@ -39,6 +39,7 @@ > #define XML2ALG_H > > #include "logical.h" > +#include "plan_bundle.h" > > #include "array.h" > > @@ -53,10 +54,10 @@ > XML2LALGContext* > PFxml2la_xml2lalgContext(void); > > -PFla_op_t* > +PFla_pb_t* > PFxml2la_importXMLFromFile(XML2LALGContext* ctx, const char *filename); > > -PFla_op_t* > +PFla_pb_t* > PFxml2la_importXMLFromMemory(XML2LALGContext* ctx, const char* xml, int size); > > #endif > > U logdebug.h > Index: logdebug.h > =================================================================== > RCS file: /cvsroot/monetdb/pathfinder/compiler/include/logdebug.h,v > retrieving revision 1.9 > retrieving revision 1.10 > diff -u -d -r1.9 -r1.10 > --- logdebug.h 8 Jan 2009 16:54:12 -0000 1.9 > +++ logdebug.h 7 May 2009 20:00:16 -0000 1.10 > @@ -4,7 +4,7 @@ > * @file > * > * Declarations for logdebug.c; dump Algebra > - * tree in AY&T dot format. > + * tree in AT&T dot or XML format. > * > * > * Copyright Notice: > @@ -43,10 +43,15 @@ > #include <stdio.h> > > #include "logical.h" > +#include "plan_bundle.h" > > void PFla_dot (FILE *, PFla_op_t *, char *prop_args); > void PFla_xml (FILE *, PFla_op_t *, char *prop_args); > > +/* Declaration for debug printing of plan bundles */ > +void PFla_dot_bundle (FILE *, PFla_pb_t *, char *prop_args); > +void PFla_xml_bundle (FILE *, PFla_pb_t *, char *prop_args); > + > #endif /* LOGDEBUG_H */ > > /* vim:set shiftwidth=4 expandtab: */ > > U properties.h > Index: properties.h > =================================================================== > RCS file: /cvsroot/monetdb/pathfinder/compiler/include/properties.h,v > retrieving revision 1.51 > retrieving revision 1.52 > diff -u -d -r1.51 -r1.52 > --- properties.h 31 Jan 2009 00:01:15 -0000 1.51 > +++ properties.h 7 May 2009 20:00:16 -0000 1.52 > @@ -374,7 +374,7 @@ > * Write domain-subdomain relationships of property container @a prop > * to in AT&T dot notation to character array @a f. > */ > -void PFprop_write_dom_rel_dot (PFarray_t *f, const PFprop_t *prop); > +void PFprop_write_dom_rel_dot (PFarray_t *f, const PFprop_t *prop, int id); > > /** > * Write domain-subdomain relationships of property container @a prop > > U Makefile.ag > Index: Makefile.ag > =================================================================== > RCS file: /cvsroot/monetdb/pathfinder/compiler/include/Makefile.ag,v > retrieving revision 1.52 > retrieving revision 1.53 > diff -u -d -r1.52 -r1.53 > --- Makefile.ag 8 Jan 2009 16:54:12 -0000 1.52 > +++ Makefile.ag 7 May 2009 20:00:16 -0000 1.53 > @@ -82,6 +82,7 @@ > physical.h \ > physical_mnemonic.h \ > planner.h \ > + plan_bundle.h \ > prettyp.h \ > properties.h \ > qname.h \ > > > ------------------------------------------------------------------------------ > The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your > production scanning environment may not be a perfect world - but thanks to > Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700 > Series Scanner you'll get full speed at 300 dpi even with all image > processing features enabled. http://p.sf.net/sfu/kodak-com > _______________________________________________ > Monetdb-pf-checkins mailing list > Monetdb-pf-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-pf-checkins -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0

[Monetdb-developers] Workaround for JDK6 Bug 6795561
by Bram Van Dam 07 May '09

07 May '09

Howdy, When using the MonetDB JDBC driver on JDK6 (or JRE6) > 6.10 you will get IndexOutOfBoundsExceptions when using transactions (possibly in other cases as well). This happens because there is a bug in CharBuffer.subSequence(), which is used in StartOfHeaderParser.java The following patch remedies this problem, without any regressions as far as I can tell. Recompiling the JDBC driver is all that's required. You'll still have to compile with the 1.5 option because the JDBC interface changed quite a bit between 5 and 6, but if I have time left I might fix that as well. Hope this'll help someone. --- MonetDB-java/java/src/nl/cwi/monetdb/mcl/parser/StartOfHeaderParser.java Wed Mar 04 12:21:50 2009 +++ MonetDB-java/java/src/nl/cwi/monetdb/mcl/parser/StartOfHeaderParser.java Thu May 07 11:57:47 2009 @@ -141,8 +141,7 @@ while (soh.hasRemaining() && soh.get() != ' ') { cnt++; } - soh.reset(); - return(soh.subSequence(0, cnt).toString()); + return(soh.slice().subSequence(0, cnt).toString()); } }

3 2

Re: [Monetdb-developers] Compiling pathfinder (Stable) failed on GNU.32.32.d.1-Debian4.0
by Stefan Manegold 07 May '09

07 May '09

On Fri, May 01, 2009 at 05:56:16AM +0200, Testweb Compilation wrote: > from lex.pf.c:19: > /usr/include/features.h:150:1: error: this is the location of the previous definition > make[5]: *** [libparser_la-scanner.yy.lo] Error 1 > > http://www.ins.cwi.nl/~monet/permastore/2009-05-01/Stable/make-pathfinder-G… > > CVS committers: fabian(a)cwi.nl manegold(a)cwi.nl > > module maintainers: lsidir(a)cwi.nl zhang(a)cwi.nl henning(a)cwi.nl The problem here is that in pathfinder, pf_config.h is in most C files (possibly at least all C files in pathfinder/compiler/ not included directly, but via pathfinder/compilier/include/pathfinder.h . Hence, our "trick" in buildtools/conf/rules.mk, to push the inclusion of pf_config.h (also) for flex, bison, swig, etc. -generated C files to the veru front of the file, does not work properly, i.e., pf_config.h get includes twice, once "forced" directly at the beginning of the (generated) C file, and once via pathfinder.h . I see three solutions: 1) Remove '#include "pf_config.h"' from pathfinder/compilier/include/pathfinder.h and add '#include "pf_config.h"' as first include in all source files in pathfinder// (e.g., just before the existing '#include "pathfinder.h"'. 2) Make buildtools/conf/rules.mk, aware of pathfinder.h and handle is just like $(CONFIG_H) 3) (Try to) make bootstrap (i.e., auto*) a conf/config.h.in (or make configure create the <pkg>_config.h from conf/config.h.in) in such a way that it is resistent against multiple inclusion, e.g., using the usual #ifndef #define ... #endif mechanism. I have a strong preference for solution one, as that is what we do the the remainder of the code base. Any comments? Any volunteers? Stefan -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 1

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/modules/mal remote.mx, , 1.95, 1.96
by Sjoerd Mullender 07 May '09

07 May '09

Martin Kersten wrote: > Update of /cvsroot/monetdb/MonetDB5/src/modules/mal > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv8081 > > Modified Files: > remote.mx > Log Message: > remote - simplified the RMT connection setup > in RMTquery first check for connnection errors, otherwise the handler > seems rubish. Question, is mapi_query() thread safe? I seriously doubt it. Use multiple connection handles if you want to use multiple threads. We could look into thread safety in mapilib sometime. -- Sjoerd Mullender

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/modules/mal pcre.mx, , 1.77, 1.78
by Stefan Manegold 06 May '09

06 May '09

Peter, I do not like this change very much. To ensure portability, we require that <PKG>_config.h (here: mal_config.h) is the very first include in each C file. Moving '#include "mal_config.h"' behind '#include "mal_config.h"' might cause unintended/unexpected problems now or in the future on other systems than Red Hat 4.3.2-7. Likewise, a hardwired and unconditional '#define _GNU_SOURCE 1' will not work on all systems. If Red Hat 4.3.2-7 indeed needs some special treatment, we should better try to fix our configure script(s) accordingly than have these hardcode fixes in the code itself. Stefan On Wed, May 06, 2009 at 01:10:59PM +0000, Peter Boncz wrote: > Update of /cvsroot/monetdb/MonetDB5/src/modules/mal > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv1809 > > Modified Files: > pcre.mx > Log Message: > make it compile on Red Hat 4.3.2-7 > > > > U pcre.mx > Index: pcre.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/modules/mal/pcre.mx,v > retrieving revision 1.77 > retrieving revision 1.78 > diff -u -d -r1.77 -r1.78 > --- pcre.mx 4 May 2009 13:53:25 -0000 1.77 > +++ pcre.mx 6 May 2009 13:10:53 -0000 1.78 > @@ -123,9 +123,10 @@ > @+ Implementation > @include prelude.mx > @c > -#include "mal_config.h" > +#define _GNU_SOURCE 1 > #include <string.h> > > +#include "mal_config.h" > #include "mal.h" > #include "mal_exception.h" > > @@ -142,6 +143,7 @@ > > #include <pcre.h> > > + > pcre_export str PCREquote(str *r, str *v); > pcre_export str PCREselect(int *res, str *pattern, int *bid); > pcre_export str PCREuselect(int *res, str *pattern, int *bid); > > > ------------------------------------------------------------------------------ > The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your > production scanning environment may not be a perfect world - but thanks to > Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700 > Series Scanner you'll get full speed at 300 dpi even with all image > processing features enabled. http://p.sf.net/sfu/kodak-com > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 1