developers-list March 2015

developers-list@monetdb.org

5 participants
6 discussions

Re: MonetDB: default - Defense against non-existing MAL primitives

by Stefan Manegold

FYI, test clients/R/Tests/survey.R still fails with "Server says '!TypeException:user.s94_2[40]:'batcalc.*' undefined in: X_168:bat[:oid,:hge] := batcalc.*(X_146,X_166);'." i.e., the check whether the requested new signature exists does not seem to work, yet. cf., http://monetdb.cwi.nl/testweb/web/testgrid.php?serial=55075:ca15cab457b8&or… Stefan ----- Original Message ----- > Changeset: 4165dc34b2bb for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=4165dc34b2bb > Modified Files: > monetdb5/mal/mal_resolve.c > monetdb5/mal/mal_resolve.h > monetdb5/optimizer/opt_coercion.c > Branch: default > Log Message: > > Defense against non-existing MAL primitives > and further limit the rewrite to the simpliest > and most expensive case encountered. > > > diffs (244 lines): > > diff --git a/monetdb5/mal/mal_resolve.c b/monetdb5/mal/mal_resolve.c > --- a/monetdb5/mal/mal_resolve.c > +++ b/monetdb5/mal/mal_resolve.c > @@ -115,7 +115,6 @@ findFunctionType(stream *out, Module sco > Symbol s; > InstrPtr sig; > int i, k, unmatched = 0, s1; > - /* int foundbutwrong=0; */ > int polytype[MAXTYPEVAR]; > int returns[256]; > int *returntype = NULL; > @@ -396,7 +395,6 @@ findFunctionType(stream *out, Module sco > } > } > if (s1 < 0) { > - /* if(getSignature(s)->token !=PATTERNsymbol) foundbutwrong++; */ > s = s->peer; > continue; > } > @@ -445,7 +443,6 @@ findFunctionType(stream *out, Module sco > * typed. This should be reflected in the symbol table. > */ > s1 = returntype[0]; /* for those interested */ > - /* foundbutwrong = 0; */ > /* > * If the call refers to a polymorphic function, we clone it > * to arrive at a bounded instance. Polymorphic patterns and > @@ -485,12 +482,6 @@ findFunctionType(stream *out, Module sco > * arguments, but that clashes with one of the target variables. > */ > wrapup: > - /* foundbutwrong has not been changed, commented out code above > - if (foundbutwrong && !silent) { > - showScriptException(out, mb, getPC(mb, p), TYPE, > - "type conflict in assignment"); > - } > - */ > if (returntype && returntype != returns) > GDKfree(returntype); > return -3; > @@ -819,13 +810,17 @@ chkTypes(stream *out, Module s, MalBlkPt > > /* > * Type checking an individual instruction is dangerous, > - * because it ignores data flow and declarations issues. > - * It is only to be used in isolated cases. > + * because it ignores data flow and variable declarations. > */ > -void > +int > chkInstruction(stream *out, Module s, MalBlkPtr mb, InstrPtr p) > { > - typeChecker(out, s, mb, p, FALSE); > + int olderrors= mb->errors; > + int error; > + typeChecker(out, s, mb, p, TRUE); > + error = mb->errors; > + mb->errors = olderrors; > + return error; > } > > void > diff --git a/monetdb5/mal/mal_resolve.h b/monetdb5/mal/mal_resolve.h > --- a/monetdb5/mal/mal_resolve.h > +++ b/monetdb5/mal/mal_resolve.h > @@ -20,7 +20,7 @@ > #define MAXTYPEVAR 10 > > mal_export void chkProgram(stream *out, Module s, MalBlkPtr mb); > -mal_export void chkInstruction(stream *out, Module s, MalBlkPtr mb, InstrPtr > p); > +mal_export int chkInstruction(stream *out, Module s, MalBlkPtr mb, InstrPtr > p); > mal_export void chkTypes(stream *out, Module s, MalBlkPtr mb, int silent); > mal_export void typeChecker(stream *out, Module scope, MalBlkPtr mb, > InstrPtr p, int silent); > mal_export int fcnBinder(stream *out, Module scope, MalBlkPtr mb, InstrPtr > p); > diff --git a/monetdb5/optimizer/opt_coercion.c > b/monetdb5/optimizer/opt_coercion.c > --- a/monetdb5/optimizer/opt_coercion.c > +++ b/monetdb5/optimizer/opt_coercion.c > @@ -18,8 +18,9 @@ typedef struct{ > int fromtype; > int totype; > int src; > -/* not used, yet !?? > +/* not used, yet !?? Indeed > int digits; > + int fromscale; > int scale; > */ > } Coercion; > @@ -47,58 +48,60 @@ coercionOptimizerStep(MalBlkPtr mb, int > return 0; > } > > -/* Check coercions for numeric types that can be handled with smaller ones. > +/* Check coercions for numeric types towards :hge that can be handled with > smaller ones. > * For now, limit to +,-,/,*,% hge expressions > - * To be extended to deal with math calls as well. > + * Not every combination may be available in the MAL layer, which calls > + * for a separate type check before fixing it. > + * Superflous coercion statements will be garbagecollected later on in the > pipeline > */ > static void > -coercionOptimizerCalcStep(MalBlkPtr mb, int i, Coercion *coerce) > +coercionOptimizerCalcStep(Client cntxt, MalBlkPtr mb, int i, Coercion > *coerce) > { > InstrPtr p = getInstrPtr(mb,i); > - int r, a, b; > + int r, a, b, varid; > > + r = getColumnType(getVarType(mb, getArg(p,0))); > +#ifdef HAVE_HGE > + if ( r != TYPE_hge) > + return; > +#endif > if( getModuleId(p) != batcalcRef || getFunctionId(p) == 0) return; > if( ! (getFunctionId(p) == plusRef || getFunctionId(p) == minusRef || > getFunctionId(p) == mulRef || getFunctionId(p) == divRef || > *getFunctionId(p) =='%') || p->argc !=3) > return; > > - r = getColumnType(getVarType(mb, getArg(p,0))); > - switch(r){ > - case TYPE_bte: > - case TYPE_sht: > - case TYPE_int: > - case TYPE_lng: > -#ifdef HAVE_HGE > - case TYPE_hge: > -#endif > - break; > - case TYPE_dbl: > - case TYPE_flt: > - /* to be determined */ > - default: > - return; > - } > a = getColumnType(getVarType(mb, getArg(p,1))); > b = getColumnType(getVarType(mb, getArg(p,2))); > - if ( a == r && coerce[getArg(p,1)].src && coerce[getArg(p,1)].fromtype < r > ) /*digit/scale test as well*/ > + if ( a == r && coerce[getArg(p,1)].src && coerce[getArg(p,1)].fromtype < r > ) > + { > + varid = getArg(p,1); > getArg(p,1) = coerce[getArg(p,1)].src; > - if ( b == r && coerce[getArg(p,2)].src && coerce[getArg(p,2)].fromtype < r > ) /*digit/scale test as well*/ > + if ( chkInstruction(NULL, cntxt->nspace, mb, p)) > + p->argv[1] = varid; > + } > + if ( b == r && coerce[getArg(p,2)].src && coerce[getArg(p,2)].fromtype < r > ) > + { > + varid = getArg(p,2); > getArg(p,2) = coerce[getArg(p,2)].src; > + if ( chkInstruction(NULL, cntxt->nspace, mb, p)) > + p->argv[2] = varid; > + } > return; > } > > static void > -coercionOptimizerAggrStep(MalBlkPtr mb, int i, Coercion *coerce) > +coercionOptimizerAggrStep(Client cntxt, MalBlkPtr mb, int i, Coercion > *coerce) > { > InstrPtr p = getInstrPtr(mb,i); > int r, k; > > + (void) cntxt; > + > if( getModuleId(p) != aggrRef || getFunctionId(p) == 0) return; > if( ! (getFunctionId(p) == subavgRef ) || p->argc !=6) > return; > > r = getColumnType(getVarType(mb, getArg(p,0))); > k = getArg(p,1); > - // check the digits/scale > if( r == TYPE_dbl && coerce[k].src ) > getArg(p,1) = coerce[getArg(p,1)].src; > return; > @@ -124,35 +127,33 @@ OPTcoercionImplementation(Client cntxt,M > if (getModuleId(p) == NULL) > continue; > /* Downscale the type, avoiding hge storage when lng would be sufficient. > - * The code template can be extended to handle other downscale options as > well > */ > #ifdef HAVE_HGE > if ( getModuleId(p) == batcalcRef > && getFunctionId(p) == hgeRef > && p->retc == 1 > - && ( p->argc == 2 > - || ( p->argc == 5 > - && isVarConstant(mb,getArg(p,1)) > - && getArgType(mb,p,1) == TYPE_int > - && isVarConstant(mb,getArg(p,3)) > - && getArgType(mb,p,3) == TYPE_int > - && isVarConstant(mb,getArg(p,4)) > - && getArgType(mb,p,4) == TYPE_int > - /* from-scale == to-scale, i.e., no scale change */ > - && *(int*) getVarValue(mb, getArg(p,1)) == *(int*) > getVarValue(mb, getArg(p,4)) ) ) ){ > + && ( p->argc == 5 > + && isVarConstant(mb,getArg(p,1)) > + && getArgType(mb,p,1) == TYPE_int > + && isVarConstant(mb,getArg(p,3)) > + && getArgType(mb,p,3) == TYPE_int > + && isVarConstant(mb,getArg(p,4)) > + && getArgType(mb,p,4) == TYPE_int > + /* from-scale == to-scale, i.e., no scale change */ > + && *(int*) getVarValue(mb, getArg(p,1)) == *(int*) getVarValue(mb, > getArg(p,4)) ) ){ > k = getArg(p,0); > coerce[k].pc= i; > coerce[k].totype= TYPE_hge; > coerce[k].src= getArg(p,2); > coerce[k].fromtype= getColumnType(getArgType(mb,p,2)); > -/* not used, yet !?? > - if (p->argc == 5) { > - coerce[k].digits= getVarConstant(mb,getArg(p,3)).val.ival; > - coerce[k].scale= getVarConstant(mb,getArg(p,4)).val.ival; > - } > +/* not used, yet !?? indeed > + coerce[k].fromscale= getVarConstant(mb,getArg(p,1)).val.ival; > + coerce[k].digits= getVarConstant(mb,getArg(p,3)).val.ival; > + coerce[k].scale= getVarConstant(mb,getArg(p,4)).val.ival; > */ > } > #endif > +/* > if ( getModuleId(p) == batcalcRef > && getFunctionId(p) == dblRef > && p->retc == 1 > @@ -160,7 +161,7 @@ OPTcoercionImplementation(Client cntxt,M > || ( p->argc == 3 > && isVarConstant(mb,getArg(p,1)) > && getArgType(mb,p,1) == TYPE_int > - /* to-scale == 0, i.e., no scale change */ > + //to-scale == 0, i.e., no scale change > && *(int*) getVarValue(mb, getArg(p,1)) == 0 ) ) ) { > k = getArg(p,0); > coerce[k].pc= i; > @@ -168,8 +169,9 @@ OPTcoercionImplementation(Client cntxt,M > coerce[k].src= getArg(p,1 + (p->argc ==3)); > coerce[k].fromtype= getColumnType(getArgType(mb,p,1 + (p->argc ==3))); > } > - coercionOptimizerAggrStep(mb, i, coerce); > - coercionOptimizerCalcStep(mb, i, coerce); > +*/ > + coercionOptimizerAggrStep(cntxt,mb, i, coerce); > + coercionOptimizerCalcStep(cntxt,mb, i, coerce); > if (getModuleId(p)==calcRef && p->argc == 2) { > k= coercionOptimizerStep(mb, i, p); > actions += k; > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

7 years, 4 months

Re: MonetDB: default - Minor textual changes to remote tests

by Stefan Manegold

> Changeset: ca15cab457b8 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=ca15cab457b8 > Modified Files: > monetdb5/modules/mal/Tests/All > monetdb5/modules/mal/Tests/remote07.mal > monetdb5/modules/mal/Tests/remote88.mal > Branch: default > Log Message: > > Minor textual changes to remote tests > > > diffs (104 lines): > > diff --git a/monetdb5/modules/mal/Tests/All b/monetdb5/modules/mal/Tests/All > --- a/monetdb5/modules/mal/Tests/All > +++ b/monetdb5/modules/mal/Tests/All > @@ -36,6 +36,7 @@ remote02 > HAVE_GEOM?remote03 > remote04 > remote06 > +# should be run as 'mclient -d demo -l mal remote07.mal For what it's worth: MAL tests with extension ".malC" (instead of ".mal") are automatically run trough "mclient -lmal ..." (instead of "mserver5 ...") by Mtest.py; cf., dev.monetdb.org/hg/MonetDB/file/tip/testing/README i.e., `hg rename monetdb5/modules/mal/Tests/remote07.mal monetdb5/modules/mal/Tests/remote07.malC` might be considered ... Stefan > remote08 > remote09 > remote11 [...] -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

7 years, 4 months

Re: MonetDB: default - Avoid coercion in aggr.subavg

by Stefan Manegold

According to Mtest, this makes TPCh Q1 return a wrong result. I suspect because this now indeed removes a scale-changing cast. Stefan ----- Original Message ----- > Changeset: 9779801a1482 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=9779801a1482 > Modified Files: > monetdb5/optimizer/opt_coercion.c > Branch: default > Log Message: > > Avoid coercion in aggr.subavg > Another example where we can avoid coercions in most cases > > > diffs (43 lines): > > diff --git a/monetdb5/optimizer/opt_coercion.c > b/monetdb5/optimizer/opt_coercion.c > --- a/monetdb5/optimizer/opt_coercion.c > +++ b/monetdb5/optimizer/opt_coercion.c > @@ -84,6 +84,24 @@ coercionOptimizerCalcStep(MalBlkPtr mb, > return; > } > > +static void > +coercionOptimizerAggrStep(MalBlkPtr mb, int i, Coercion *coerce) > +{ > + InstrPtr p = getInstrPtr(mb,i); > + int r, k; > + > + if( getModuleId(p) != aggrRef || getFunctionId(p) == 0) return; > + if( ! (getFunctionId(p) == subavgRef ) || p->argc !=6) > + return; > + > + r = getColumnType(getVarType(mb, getArg(p,0))); > + k = getArg(p,1); > + // check the digits/scale > + if( r == TYPE_dbl && coerce[k].src ) > + getArg(p,1) = coerce[getArg(p,1)].src; > + return; > +} > + > int > OPTcoercionImplementation(Client cntxt,MalBlkPtr mb, MalStkPtr stk, InstrPtr > pci) > { > @@ -117,6 +135,14 @@ OPTcoercionImplementation(Client cntxt,M > coerce[k].scale= getVarConstant(mb,getArg(p,4)).val.ival; > } > #endif > + if ( getModuleId(p) == batcalcRef && getFunctionId(p) == dblRef && p->retc > == 1 && ((p->argc == 3 && isVarConstant(mb,getArg(p,1))) || p->argc ==2) ){ > + k = getArg(p,0); > + coerce[k].pc= i; > + coerce[k].totype= TYPE_dbl; > + coerce[k].src= getArg(p,1 + (p->argc ==3)); > + coerce[k].fromtype= getColumnType(getArgType(mb,p,1 + (p->argc ==3))); > + } > + coercionOptimizerAggrStep(mb, i, coerce); > coercionOptimizerCalcStep(mb, i, coerce); > if (getModuleId(p)==calcRef && p->argc == 2) { > k= coercionOptimizerStep(mb, i, p); > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

7 years, 4 months

Re: MonetDB: default - coercionOptimizerCalcStep(): fixing changeset...

by Stefan Manegold

IMHO this new coersion optimization as introduced in changeset 5786f1be12bb only works correctly for decimal casts (coersions) that do not change the scale (number of decimals), i.e., when the first and last argument (i.e., input and output number of decimals, respectively) are identical. This sanity check is currently missing in the code. Stefan ----- Original Message ----- > Changeset: 289a5a038774 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=289a5a038774 > Modified Files: > monetdb5/optimizer/opt_coercion.c > Branch: default > Log Message: > > coercionOptimizerCalcStep(): fixing changeset 71bfaf7e9841: also handle > TYPE_hge > to re-activate the intended behavior of changeset 5786f1be12bb > > OPEN ISSUE / QUESTION: > > This optimizer now (only) handles (i.e., removes) decimal (up-)casts, > but no plain/pure integer casts at all. > > (a) Shouldn't we also handle (remove) plain/pure integer casts? > > (b) Can we always safely remove decimal (up-)casts without altering > semantics? > > > diffs (13 lines): > > diff --git a/monetdb5/optimizer/opt_coercion.c > b/monetdb5/optimizer/opt_coercion.c > --- a/monetdb5/optimizer/opt_coercion.c > +++ b/monetdb5/optimizer/opt_coercion.c > @@ -65,6 +65,9 @@ coercionOptimizerCalcStep(MalBlkPtr mb, > case TYPE_sht: > case TYPE_int: > case TYPE_lng: > +#ifdef HAVE_HGE > + case TYPE_hge: > +#endif > break; > case TYPE_dbl: > case TYPE_flt: > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

7 years, 4 months

About the Join Algorithm in MonetDB

by Zhiwei Fan

Dear Developers of MonetDB, ?I really appreciate your excellent work on MonetDB. I am a student in University of Wisconsin Madison and now I am doing research on processing machine learning in the column-oriented Database since we have found a way that could possibly improve the performance of learning process if we use data stored in column rather than data stored in row. For the research purpose, we need to know the detail of the join algorithm in column-store database in order to analyze the cost of the process. And monetDB, as we know, has done a very good job in this part, so I hope you could tell us about the details of the join algorithms implemented in the monetDB, that would be really helpful and we would really appreciate it! Thanks. Best Regards Zhiwei Fan ?

7 years, 4 months

MAL property for thread-unsafe function

by Roberto Cornacchia

Hello, I have a function definition command batstemmer.stem(terms:bat[:oid,:str], stemmer_name:str):bat[:oid,:str] address CMDbatstem comment "Wrapper for snowball stemmer"; which internally uses the snowball stemmer (http://snowball.tartarus.org/). When the bat to be stemmed is large enough, mitosis will split it into chunks and call the function "stem" on each chunk, possibly in parallel. Problem is, the snowball stemmer implementation appears to be thread-unsafe, which causes a SIGSEGV. Indeed, using the no_mitosis_pipe solves the issue. However, this solution is suboptimal. Another solution I found is to mark the mal signature as {unsafe}. This works, although it does something a bit silly: it splits the table into chunks, then repacks everything, and finally runs my function on the re-packed bat (basically wasting effort on a useless split + repack). Now, my question is: is there a more focussed property to use? {unsafe} implies thread-unsafe, but it is actually stronger than that. For example, it also implies that there might be side-effects. Therefore, the result cannot be recycled. In my case, instead, the result is perfectly safe to be reused. Thanks for any tip. Roberto

7 years, 4 months

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

developers-list March 2015