[Monetdb-developers] [Monetdb-checkins] MonetDB4/src/modules/contrib malalgebra.mx, MonetDB_4-24, 1.8.2.1, 1.8.2.2

Peter Boncz P.Boncz at cwi.nl
Wed Oct 29 10:59:36 CET 2008


Hi Stefan,

I engineered the changes to be MonetDB4 and even MonetDB5 friendly, and though
PF_ROX is a proof of concept branch, the concepts will at a later stage
re-appear in a main branch (certainly those on a GDK level).

However, the changes in MonetDB to support it are minimal; it is not a sacrifice
to have them in a HEAD branch. 

I actually think we should just make that little bit of effort to port PF_ROX to
the MonetDB[4] HEAD, and take it from there. Part of that step is actually the
propagation of my branch changes to the HEAD (as I was assuming would happen
semi-automatically).

Peter

> -----Original Message-----
> From: Stefan Manegold [mailto:Stefan.Manegold at cwi.nl]
> Sent: Wednesday, October 29, 2008 10:35 AM
> To: monetdb-developers at lists.sourceforge.net
> Cc: P.Boncz at cwi.nl
> Subject: Re: [Monetdb-checkins] MonetDB4/src/modules/contrib malalgebra.mx,
> MonetDB_4-24, 1.8.2.1, 1.8.2.2
> 
> On Wed, Oct 29, 2008 at 10:27:35AM +0100, Stefan Manegold wrote:
> > Gents,
> >
> > given (1) the inherent incompatibility between the PF_ROX branch (forked
> of
> > the XQuery_0-24 branch of pathfinder) and the pathfinder development
> trunk
> 
> and (hence) the incompatibility between the PF_ROX branch and the
> development trunks of MonetDB & MonetDB4
> 
> > and the facts that (2) PF_ROX (at least for now) is purely for internal
> > research purposes and (3) cahnges ("features") in PF_ROX do (e.g.) not
> > support updates, i.e., are far from becoming "main stream",
> >
> > I guess the (only?) proper solution would (have been?) to also fork
> PF_ROX
> > branches of the MonetDB_1-24 branch (of MonetDB) and the MonetDB_4-24
> branch
> > (of MonetDB4) to accomodate Peter's yesterday's changes.
> >
> > Note though, that this would / will exclude PF_ROX-specific changes in
> > MonetDB & MonetDB$ from being tested --- this is already the case for the
> > pathfinder PF_ROX branch and will be the case for the MonetDB_1-24 &
> > MonetDB_4-24 branches, too, once we start prparing for the next release
> no
> > later than next week.
> 
> Also, changes from the PF_ROX branches of MonetDB & MonetDB4 would/will
> than
> NOT be propagated to the respective development trunks; the same will be
> the
> case for (potential0 changes to the MonetDB_1-24 & MonetDB_4-24 branches
> as soon as we start prparing for the next release no later than next week.
> 
> In fact, I would even suggest to not propagate Peter's yesterday's changes
> to the development trunk since they are "PF_ROX-specific" (at least for
> now).
> 
> Stefan
> 
> > See the results of "Stable" testing (once they are ready later today) to
> > judged whether testing is "desired"/"required" or not ...
> >
> > Stefan
> >
> > On Tue, Oct 28, 2008 at 11:36:17PM +0100, Sjoerd Mullender wrote:
> > > On 2008-10-28 22:37, Peter Boncz wrote:
> > > > Hi Sjoerd,
> > > >
> > > > The one problem that could occur is people trying to check out
> MonetDB_4-24
> > > > now, compiling it and trying to use with a precompiled MonetDB_1-24,
> without
> > > > updating/recompiling that one. So, I agree with you that a problem
> could
> > > > occur, but I would be surprised if any of our users would really hit
> this.
> > >
> > > Because we are going to abandon this branch I was willing to let the
> > > change pass.  But in general I am opposed.
> > >
> > > I agree that in this particular case, it is unlikely that people will
> > > encounter any problems.  But again, that is because we are going to
> > > abandon this branch.
> > >
> > > > With excuses to pick nits, but contrary to you I believe that API
> appends
> > > > are not "just as bad" as API modifications. An API modification in
> MonetDB,
> > > > would force a release in MonetDB5, for instance. An API append does
> not do
> > > > that (so it is "less bad").
> > >
> > > It *is* bad.  See the disaster involving the addition of True and False
> > > in Python in a minor release.  They have sworn to never make that
> > > mistake again.  People expect API stability in minor releases and
> adding
> > > something to the API is *not* stability.
> > >
> > > > PF_ROX is quite different from the HEAD in the pathfinder code due to
> bot
> > > > branches making different changes to the indexing scheme; afaik this
> has
> > > > prevented us from keeping it in sync with the HEAD. Various API
> changes (not
> > > > appends) and even buildtool changes now force it to depend on the
> stable
> > > > MonetDB/MonetDB4. For PF_ROX itself this is not a problem as it has
> no
> > > > outside users. However, it might occur that this scenario will repeat
> > > > (PF_ROX triggering a change/addition in MonetDB/MonetDB4 that would
> trickle
> > > > towards the head via MonetDB_[14]-24).
> > >
> > > I am not talking about the PF_ROX branch.  It is not a "stable" branch
> > > and you can do with it what you want.  But please do consider whether
> it
> > > is possible to use the development branches of MonetDB and MonetDB4
> with
> > > PF_ROX.  If you can do that, you can make needed changes to the API
> there.
> > >
> > > > Peter
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Sjoerd Mullender [mailto:sjoerd at acm.org]
> > > > Sent: Tuesday, October 28, 2008 9:50 PM
> > > > To: monetdb-developers at lists.sourceforge.net
> > > > Cc: monetdb-checkins at lists.sourceforge.net
> > > > Subject: Re: [Monetdb-checkins] MonetDB4/src/modules/contrib
> malalgebra.mx,
> > > > MonetDB_4-24, 1.8.2.1, 1.8.2.2
> > > >
> > > > On 2008-10-28 20:20, Peter Boncz wrote:
> > > >> Hi Sjoerd,
> > > >>
> > > >> Thanks for keeping an eye on developer behavior and effort to
> educate. My
> > > >> change, however, only introduces new functions in GDK and commands
> in MIL;
> > > > it
> > > >> is only an API append.
> > > >
> > > > API appends are just as bad as any other API change.  If new code
> > > > depends on the extra calls, it cannot run with the older version of
> the
> > > > release.  That is bad.  It means that the branch is *not* "stable"
> anymore.
> > > >
> > > >> It is checked into the stable branch because afaik PF_ROX only works
> with
> > > >> that. But I may as often be mistaken.
> > > >
> > > > I noticed it was for PF_ROX, and that does pose a problem.  It seems
> > > > that the PF_ROX branch should depend on the current version of
> MonetDB
> > > > and not on the stable version.
> > > >
> > > >> Peter
> > > >>
> > > >>> This type of change is not allowed on a stable branch.  Changes of
> API
> > > >>> are *not* allowed.
> > > >>>
> > > >>> Since we're going to abandon this stable branch soon, I'll let it
> pass.
> > > >>>
> > > >>> On 2008-10-28 18:49, Peter Boncz wrote:
> > > >>>> Update of /cvsroot/monetdb/MonetDB4/src/modules/contrib
> > > >>>> In directory
> > > >>>> 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv901/src/modules/contrib
> > > >>>>
> > > >>>> Modified Files:
> > > >>>>       Tag: MonetDB_4-24
> > > >>>>  malalgebra.mx
> > > >>>> Log Message:
> > > >>>> changes to minimize sampling overhead (mostly in XML text index)
> > > >>>>
> > > >>>> - introduce bandmergejoin(), and leftmergejoin()/bandmergejoin()
> with a
> > > >>>> cutoff
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> U malalgebra.mx
> > > >>>> Index: malalgebra.mx
> > > >>>>
> ===================================================================
> > > >>>> RCS file:
> /cvsroot/monetdb/MonetDB4/src/modules/contrib/malalgebra.mx,v
> > > >>>> retrieving revision 1.8.2.1
> > > >>>> retrieving revision 1.8.2.2
> > > >>>> diff -u -d -r1.8.2.1 -r1.8.2.2
> > > >>>> --- malalgebra.mx 10 Oct 2008 08:42:22 -0000 1.8.2.1
> > > >>>> +++ malalgebra.mx 28 Oct 2008 17:49:34 -0000 1.8.2.2
> > > >>>> @@ -43,7 +43,6 @@
> > > >>>>  @:leftjoin(fetch,\nCAUTION: positional matches are assumed not to
> be
> > > >>>> out-of-bounds!!)@
> > > >>>>  @:leftjoin(merge)@
> > > >>>>  @:leftjoin(hash)@
> > > >>>> -
> > > >>>>    .COMMAND leftthetajoin ( BAT[any::1,any::2] left,
> BAT[any::2,any::3]
> > > >>>> right, int mode) :
> > > >>>>                      BAT[any::1,any::3] = CMDleftthetajoin;
> > > >>>>    "Hook directly into the 'leftthetajoin' implementation of the
> join.
> > > >>>> @@ -58,6 +57,22 @@
> > > >>>>     Also, for each left tuple, all matching right tuples will
> appear in
> > > >>>> their order
> > > >>>>     of appearrance in the right BAT. This property is handy for
> XQuery
> > > >>>> processing."
> > > >>>>
> > > >>>> +  .COMMAND bandmergejoin ( BAT[any::1,any::2] outer,
> BAT[any::2,any::3]
> > > >>>> inner,
> > > >>>> +                               any::2 minus, any::2 plus, bit
> l_in, bit
> > > >>>> h_in) :
> > > >>>> +                                BAT[any::1,any::3] =
> CMDbandmergejoin;
> > > >>>> +  "The bandjoin algorithm, but forced to use (left) mergejoin
> rather
> > > > than
> > > >>>> nested loop"
> > > >>>> +
> > > >>>> +  .COMMAND bandmergejoin ( BAT[any::1,any::2] outer,
> BAT[any::2,any::3]
> > > >>>> inner,
> > > >>>> +                               any::2 minus, any::2 plus, bit
> l_in, bit
> > > >>>> h_in, lng limit) :
> > > >>>> +                                BAT[lng,BAT] =
> CMDbandmergejoin_limit;
> > > >>>> +  "A bandjoin that uses the merge algorithm and limits the result
> size.
> > > >>>> +   Result generation is cut off after this point and the
> > > >>>> +   result [lng,BAT] holds
> [estimated_full_resultsize,limited_result]"
> > > >>>> +
> > > >>>> +  .COMMAND leftmergejoin ( BAT[any::1,any::2] left,
> BAT[any::2,any::3]
> > > >>>> right, lng limit) :
> > > >>>> +                    BAT[lng,BAT] = CMDleftmergejoin_limit;
> > > >>>> +  "leftmergejoin, but limit the result (by cutting off result
> > > > generation)
> > > >>>> +   return a BAT[lng,BAT] containing
> > > > [full_result_estimate,limited_result]"
> > > >>>>  @= select
> > > >>>>  .COMMAND ord_ at 1select ( BAT[any::1,any::2] b, any::2 low, any::2
> high)
> > > > :
> > > >>>> @@ -119,6 +134,7 @@
> > > >>>>  #ifndef __MALALGEBRA_H__
> > > >>>>  #define __MALALGEBRA_H__
> > > >>>>  #include "gdk.h"
> > > >>>> +#include "gdk_rangejoin.h"
> > > >>>>
> > > >>>>  /* nothing much */
> > > >>>>
> > > >>>> @@ -240,6 +256,40 @@
> > > >>>>   return (*result = BATnlthetajoin(l, r, *mode, (size_t) *
> estimate)) ?
> > > >>>> GDK_SUCCEED : GDK_FAIL;
> > > >>>>  }
> > > >>>>
> > > >>>> +int
> > > >>>> +CMDbandmergejoin(BAT **result, BAT *left, BAT *right, ptr minus,
> ptr
> > > > plus,
> > > >>>> bit *li, bit *hi)
> > > >>>> +{
> > > >>>> +        return (*result = BATbandmergejoin(left, right, minus,
> plus,
> > > > *li,
> > > >>>> *hi)) ? GDK_SUCCEED : GDK_FAIL;
> > > >>>> +}
> > > >>>> +
> > > >>>> +int
> > > >>>> +CMDbandmergejoin_limit(BAT **result, BAT *left, BAT *right, ptr
> minus,
> > > > ptr
> > > >>>> plus, bit *li, bit *hi, lng* limit)
> > > >>>> +{
> > > >>>> + size_t cutoff = *limit;
> > > >>>> +        *result = BATbandmergejoin_limit(left, right, minus,
> plus, *li,
> > > >>>> *hi, &cutoff);
> > > >>>> +        if (*result) {
> > > >>>> +                bat bid = (*result)->batCacheid;
> > > >>>> +                *result = BATnew(TYPE_lng, TYPE_bat, 1);
> > > >>>> +                if (*result) BUNins(*result, &cutoff, &bid,
> FALSE);
> > > >>>> +                BBPunfix(bid);
> > > >>>> +        }
> > > >>>> +        return (*result)?GDK_SUCCEED:GDK_FAIL;
> > > >>>> +}
> > > >>>> +
> > > >>>> +int
> > > >>>> +CMDleftmergejoin_limit(BAT **result, BAT *left, BAT *right, lng*
> limit)
> > > >>>> +{
> > > >>>> + size_t cutoff = *limit;
> > > >>>> +        *result = BATleftmergejoin_limit(left, right, cutoff,
> &cutoff);
> > > >>>> +        if (*result) {
> > > >>>> +                bat bid = (*result)->batCacheid;
> > > >>>> +                *result = BATnew(TYPE_lng, TYPE_bat, 1);
> > > >>>> +                if (*result) BUNins(*result, &cutoff, &bid,
> FALSE);
> > > >>>> +                BBPunfix(bid);
> > > >>>> +        }
> > > >>>> +        return (*result)?GDK_SUCCEED:GDK_FAIL;
> > > >>>> +}
> > > >>>> +
> > > >>>>  @= selectcmd
> > > >>>>  int CMDord_ at 1select1(BAT **result, BAT* b, ptr value) {
> > > >>>>   return (*result = BAT_select_(b, value, 0, TRUE, TRUE, @2,
> FALSE,
> > > >>>> TRUE))?GDK_SUCCEED:GDK_FAIL;
> > > >>>>
> > > >>>>
> > > >>>>
> > > > ---------------------------------------------------------------------
> ----
> > > >>>> This SF.Net email is sponsored by the Moblin Your Move Developer's
> > > > challenge
> > > >>>> Build the coolest Linux based applications with Moblin SDK & win
> great
> > > >>>> prizes
> > > >>>> Grand prize is a trip for two to an Open Source event anywhere in
> the
> > > > world
> > > >>>> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> > > >>>> _______________________________________________
> > > >>>> Monetdb-checkins mailing list
> > > >>>> Monetdb-checkins at lists.sourceforge.net
> > > >>>> https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
> > > >>> --
> > > >>> Sjoerd Mullender
> > > >>>
> > > >>> -------------------------------------------------------------------
> ------
> > > >>> This SF.Net email is sponsored by the Moblin Your Move Developer's
> > > > challenge
> > > >>> Build the coolest Linux based applications with Moblin SDK & win
> great
> > > > prizes
> > > >>> Grand prize is a trip for two to an Open Source event anywhere in
> the
> > > > world
> > > >>> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> > > >>> _______________________________________________
> > > >>> Monetdb-checkins mailing list
> > > >>> Monetdb-checkins at lists.sourceforge.net
> > > >>> https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
> > > >>>
> > > >>
> > > >> --------------------------------------------------------------------
> -----
> > > >> This SF.Net email is sponsored by the Moblin Your Move Developer's
> > > > challenge
> > > >> Build the coolest Linux based applications with Moblin SDK & win
> great
> > > > prizes
> > > >> Grand prize is a trip for two to an Open Source event anywhere in
> the
> > > > world
> > > >> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> > > >> _______________________________________________
> > > >> Monetdb-checkins mailing list
> > > >> Monetdb-checkins at lists.sourceforge.net
> > > >> https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
> > > >
> > > >
> > > > --
> > > > Sjoerd Mullender
> > > >
> > > > ---------------------------------------------------------------------
> ----
> > > > This SF.Net email is sponsored by the Moblin Your Move Developer's
> challenge
> > > > Build the coolest Linux based applications with Moblin SDK & win
> great
> > > > prizes
> > > > Grand prize is a trip for two to an Open Source event anywhere in the
> world
> > > > http://moblin-contest.org/redirect.php?banner_id=100&url=/
> > > > _______________________________________________
> > > > Monetdb-checkins mailing list
> > > > Monetdb-checkins at lists.sourceforge.net
> > > > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> ----
> > > > This SF.Net email is sponsored by the Moblin Your Move Developer's
> challenge
> > > > Build the coolest Linux based applications with Moblin SDK & win
> great prizes
> > > > Grand prize is a trip for two to an Open Source event anywhere in the
> world
> > > > http://moblin-contest.org/redirect.php?banner_id=100&url=/
> > > > _______________________________________________
> > > > Monetdb-checkins mailing list
> > > > Monetdb-checkins at lists.sourceforge.net
> > > > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
> > >
> > >
> > > --
> > > Sjoerd Mullender
> > >
> > > -----------------------------------------------------------------------
> --
> > > This SF.Net email is sponsored by the Moblin Your Move Developer's
> challenge
> > > Build the coolest Linux based applications with Moblin SDK & win great
> prizes
> > > Grand prize is a trip for two to an Open Source event anywhere in the
> world
> > > http://moblin-contest.org/redirect.php?banner_id=100&url=/
> > > _______________________________________________
> > > Monetdb-checkins mailing list
> > > Monetdb-checkins at lists.sourceforge.net
> > > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
> > >
> >
> > --
> > | Dr. Stefan Manegold | mailto:Stefan.Manegold at cwi.nl |
> > | CWI,  P.O.Box 94079 | http://www.cwi.nl/~manegold/  |
> > | 1090 GB Amsterdam   | Tel.: +31 (20) 592-4212       |
> > | The Netherlands     | Fax : +31 (20) 592-4312       |
> 
> --
> | Dr. Stefan Manegold | mailto:Stefan.Manegold at cwi.nl |
> | CWI,  P.O.Box 94079 | http://www.cwi.nl/~manegold/  |
> | 1090 GB Amsterdam   | Tel.: +31 (20) 592-4212       |
> | The Netherlands     | Fax : +31 (20) 592-4312       |





More information about the developers-list mailing list