Re: [Monetdb-developers] [Monetdb-checkins] MonetDB4/src/modules/contrib malalgebra.mx, MonetDB_4-24, 1.8.2.1, 1.8.2.2
This type of change is not allowed on a stable branch. Changes of API are *not* allowed.
Since we're going to abandon this stable branch soon, I'll let it pass.
On 2008-10-28 18:49, Peter Boncz wrote:
Update of /cvsroot/monetdb/MonetDB4/src/modules/contrib In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv901/src/modules/contrib
Modified Files: Tag: MonetDB_4-24 malalgebra.mx Log Message: changes to minimize sampling overhead (mostly in XML text index)
- introduce bandmergejoin(), and leftmergejoin()/bandmergejoin() with a cutoff
U malalgebra.mx Index: malalgebra.mx =================================================================== RCS file: /cvsroot/monetdb/MonetDB4/src/modules/contrib/malalgebra.mx,v retrieving revision 1.8.2.1 retrieving revision 1.8.2.2 diff -u -d -r1.8.2.1 -r1.8.2.2 --- malalgebra.mx 10 Oct 2008 08:42:22 -0000 1.8.2.1 +++ malalgebra.mx 28 Oct 2008 17:49:34 -0000 1.8.2.2 @@ -43,7 +43,6 @@ @:leftjoin(fetch,\nCAUTION: positional matches are assumed not to be out-of-bounds!!)@ @:leftjoin(merge)@ @:leftjoin(hash)@
- .COMMAND leftthetajoin ( BAT[any::1,any::2] left, BAT[any::2,any::3] right, int mode) : BAT[any::1,any::3] = CMDleftthetajoin; "Hook directly into the 'leftthetajoin' implementation of the join.
@@ -58,6 +57,22 @@ Also, for each left tuple, all matching right tuples will appear in their order of appearrance in the right BAT. This property is handy for XQuery processing."
- .COMMAND bandmergejoin ( BAT[any::1,any::2] outer, BAT[any::2,any::3] inner,
any::2 minus, any::2 plus, bit l_in, bit h_in) :BAT[any::1,any::3] = CMDbandmergejoin;- "The bandjoin algorithm, but forced to use (left) mergejoin rather than nested loop"
- .COMMAND bandmergejoin ( BAT[any::1,any::2] outer, BAT[any::2,any::3] inner,
any::2 minus, any::2 plus, bit l_in, bit h_in, lng limit) :BAT[lng,BAT] = CMDbandmergejoin_limit;- "A bandjoin that uses the merge algorithm and limits the result size.
- Result generation is cut off after this point and the
- result [lng,BAT] holds [estimated_full_resultsize,limited_result]"
- .COMMAND leftmergejoin ( BAT[any::1,any::2] left, BAT[any::2,any::3] right, lng limit) :
BAT[lng,BAT] = CMDleftmergejoin_limit;- "leftmergejoin, but limit the result (by cutting off result generation)
- return a BAT[lng,BAT] containing [full_result_estimate,limited_result]"
@= select .COMMAND ord_@1select ( BAT[any::1,any::2] b, any::2 low, any::2 high) : @@ -119,6 +134,7 @@ #ifndef __MALALGEBRA_H__ #define __MALALGEBRA_H__ #include "gdk.h" +#include "gdk_rangejoin.h"
/* nothing much */
@@ -240,6 +256,40 @@ return (*result = BATnlthetajoin(l, r, *mode, (size_t) * estimate)) ? GDK_SUCCEED : GDK_FAIL; }
+int +CMDbandmergejoin(BAT **result, BAT *left, BAT *right, ptr minus, ptr plus, bit *li, bit *hi) +{
return (*result = BATbandmergejoin(left, right, minus, plus, *li, *hi)) ? GDK_SUCCEED : GDK_FAIL;+}
+int +CMDbandmergejoin_limit(BAT **result, BAT *left, BAT *right, ptr minus, ptr plus, bit *li, bit *hi, lng* limit) +{
- size_t cutoff = *limit;
*result = BATbandmergejoin_limit(left, right, minus, plus, *li, *hi, &cutoff);if (*result) {bat bid = (*result)->batCacheid;*result = BATnew(TYPE_lng, TYPE_bat, 1);if (*result) BUNins(*result, &cutoff, &bid, FALSE);BBPunfix(bid);}return (*result)?GDK_SUCCEED:GDK_FAIL;+}
+int +CMDleftmergejoin_limit(BAT **result, BAT *left, BAT *right, lng* limit) +{
- size_t cutoff = *limit;
*result = BATleftmergejoin_limit(left, right, cutoff, &cutoff);if (*result) {bat bid = (*result)->batCacheid;*result = BATnew(TYPE_lng, TYPE_bat, 1);if (*result) BUNins(*result, &cutoff, &bid, FALSE);BBPunfix(bid);}return (*result)?GDK_SUCCEED:GDK_FAIL;+}
@= selectcmd int CMDord_@1select1(BAT **result, BAT* b, ptr value) { return (*result = BAT_select_(b, value, 0, TRUE, TRUE, @2, FALSE, TRUE))?GDK_SUCCEED:GDK_FAIL;
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
Hi Sjoerd,
Thanks for keeping an eye on developer behavior and effort to educate. My change, however, only introduces new functions in GDK and commands in MIL; it is only an API append.
It is checked into the stable branch because afaik PF_ROX only works with that. But I may as often be mistaken.
Peter
This type of change is not allowed on a stable branch. Changes of API are *not* allowed.
Since we're going to abandon this stable branch soon, I'll let it pass.
On 2008-10-28 18:49, Peter Boncz wrote:
Update of /cvsroot/monetdb/MonetDB4/src/modules/contrib In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv901/src/modules/contrib
Modified Files: Tag: MonetDB_4-24 malalgebra.mx Log Message: changes to minimize sampling overhead (mostly in XML text index)
- introduce bandmergejoin(), and leftmergejoin()/bandmergejoin() with a
cutoff
U malalgebra.mx Index: malalgebra.mx =================================================================== RCS file: /cvsroot/monetdb/MonetDB4/src/modules/contrib/malalgebra.mx,v retrieving revision 1.8.2.1 retrieving revision 1.8.2.2 diff -u -d -r1.8.2.1 -r1.8.2.2 --- malalgebra.mx 10 Oct 2008 08:42:22 -0000 1.8.2.1 +++ malalgebra.mx 28 Oct 2008 17:49:34 -0000 1.8.2.2 @@ -43,7 +43,6 @@ @:leftjoin(fetch,\nCAUTION: positional matches are assumed not to be out-of-bounds!!)@ @:leftjoin(merge)@ @:leftjoin(hash)@
- .COMMAND leftthetajoin ( BAT[any::1,any::2] left, BAT[any::2,any::3]
right, int mode) : BAT[any::1,any::3] = CMDleftthetajoin; "Hook directly into the 'leftthetajoin' implementation of the join. @@ -58,6 +57,22 @@ Also, for each left tuple, all matching right tuples will appear in their order of appearrance in the right BAT. This property is handy for XQuery processing."
- .COMMAND bandmergejoin ( BAT[any::1,any::2] outer, BAT[any::2,any::3]
inner,
any::2 minus, any::2 plus, bit l_in, bith_in) :
BAT[any::1,any::3] = CMDbandmergejoin;- "The bandjoin algorithm, but forced to use (left) mergejoin rather than
nested loop"
- .COMMAND bandmergejoin ( BAT[any::1,any::2] outer, BAT[any::2,any::3]
inner,
any::2 minus, any::2 plus, bit l_in, bith_in, lng limit) :
BAT[lng,BAT] = CMDbandmergejoin_limit;- "A bandjoin that uses the merge algorithm and limits the result size.
- Result generation is cut off after this point and the
- result [lng,BAT] holds [estimated_full_resultsize,limited_result]"
- .COMMAND leftmergejoin ( BAT[any::1,any::2] left, BAT[any::2,any::3]
right, lng limit) :
BAT[lng,BAT] = CMDleftmergejoin_limit;- "leftmergejoin, but limit the result (by cutting off result generation)
- return a BAT[lng,BAT] containing [full_result_estimate,limited_result]"
@= select .COMMAND ord_@1select ( BAT[any::1,any::2] b, any::2 low, any::2 high) : @@ -119,6 +134,7 @@ #ifndef __MALALGEBRA_H__ #define __MALALGEBRA_H__ #include "gdk.h" +#include "gdk_rangejoin.h"
/* nothing much */
@@ -240,6 +256,40 @@ return (*result = BATnlthetajoin(l, r, *mode, (size_t) * estimate)) ? GDK_SUCCEED : GDK_FAIL; }
+int +CMDbandmergejoin(BAT **result, BAT *left, BAT *right, ptr minus, ptr plus, bit *li, bit *hi) +{
return (*result = BATbandmergejoin(left, right, minus, plus, *li,*hi)) ? GDK_SUCCEED : GDK_FAIL; +}
+int +CMDbandmergejoin_limit(BAT **result, BAT *left, BAT *right, ptr minus, ptr plus, bit *li, bit *hi, lng* limit) +{
- size_t cutoff = *limit;
*result = BATbandmergejoin_limit(left, right, minus, plus, *li,*hi, &cutoff);
if (*result) {bat bid = (*result)->batCacheid;*result = BATnew(TYPE_lng, TYPE_bat, 1);if (*result) BUNins(*result, &cutoff, &bid, FALSE);BBPunfix(bid);}return (*result)?GDK_SUCCEED:GDK_FAIL;+}
+int +CMDleftmergejoin_limit(BAT **result, BAT *left, BAT *right, lng* limit) +{
- size_t cutoff = *limit;
*result = BATleftmergejoin_limit(left, right, cutoff, &cutoff);if (*result) {bat bid = (*result)->batCacheid;*result = BATnew(TYPE_lng, TYPE_bat, 1);if (*result) BUNins(*result, &cutoff, &bid, FALSE);BBPunfix(bid);}return (*result)?GDK_SUCCEED:GDK_FAIL;+}
@= selectcmd int CMDord_@1select1(BAT **result, BAT* b, ptr value) { return (*result = BAT_select_(b, value, 0, TRUE, TRUE, @2, FALSE, TRUE))?GDK_SUCCEED:GDK_FAIL;
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
-- Sjoerd Mullender
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
On 2008-10-28 20:20, Peter Boncz wrote:
Hi Sjoerd,
Thanks for keeping an eye on developer behavior and effort to educate. My change, however, only introduces new functions in GDK and commands in MIL; it is only an API append.
API appends are just as bad as any other API change. If new code depends on the extra calls, it cannot run with the older version of the release. That is bad. It means that the branch is *not* "stable" anymore.
It is checked into the stable branch because afaik PF_ROX only works with that. But I may as often be mistaken.
I noticed it was for PF_ROX, and that does pose a problem. It seems that the PF_ROX branch should depend on the current version of MonetDB and not on the stable version.
Peter
This type of change is not allowed on a stable branch. Changes of API are *not* allowed.
Since we're going to abandon this stable branch soon, I'll let it pass.
On 2008-10-28 18:49, Peter Boncz wrote:
Update of /cvsroot/monetdb/MonetDB4/src/modules/contrib In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv901/src/modules/contrib
Modified Files: Tag: MonetDB_4-24 malalgebra.mx Log Message: changes to minimize sampling overhead (mostly in XML text index)
- introduce bandmergejoin(), and leftmergejoin()/bandmergejoin() with a
cutoff
U malalgebra.mx Index: malalgebra.mx =================================================================== RCS file: /cvsroot/monetdb/MonetDB4/src/modules/contrib/malalgebra.mx,v retrieving revision 1.8.2.1 retrieving revision 1.8.2.2 diff -u -d -r1.8.2.1 -r1.8.2.2 --- malalgebra.mx 10 Oct 2008 08:42:22 -0000 1.8.2.1 +++ malalgebra.mx 28 Oct 2008 17:49:34 -0000 1.8.2.2 @@ -43,7 +43,6 @@ @:leftjoin(fetch,\nCAUTION: positional matches are assumed not to be out-of-bounds!!)@ @:leftjoin(merge)@ @:leftjoin(hash)@
- .COMMAND leftthetajoin ( BAT[any::1,any::2] left, BAT[any::2,any::3]
right, int mode) : BAT[any::1,any::3] = CMDleftthetajoin; "Hook directly into the 'leftthetajoin' implementation of the join. @@ -58,6 +57,22 @@ Also, for each left tuple, all matching right tuples will appear in their order of appearrance in the right BAT. This property is handy for XQuery processing."
- .COMMAND bandmergejoin ( BAT[any::1,any::2] outer, BAT[any::2,any::3]
inner,
any::2 minus, any::2 plus, bit l_in, bith_in) :
BAT[any::1,any::3] = CMDbandmergejoin;- "The bandjoin algorithm, but forced to use (left) mergejoin rather than
nested loop"
- .COMMAND bandmergejoin ( BAT[any::1,any::2] outer, BAT[any::2,any::3]
inner,
any::2 minus, any::2 plus, bit l_in, bith_in, lng limit) :
BAT[lng,BAT] = CMDbandmergejoin_limit;- "A bandjoin that uses the merge algorithm and limits the result size.
- Result generation is cut off after this point and the
- result [lng,BAT] holds [estimated_full_resultsize,limited_result]"
- .COMMAND leftmergejoin ( BAT[any::1,any::2] left, BAT[any::2,any::3]
right, lng limit) :
BAT[lng,BAT] = CMDleftmergejoin_limit;- "leftmergejoin, but limit the result (by cutting off result generation)
- return a BAT[lng,BAT] containing [full_result_estimate,limited_result]"
@= select .COMMAND ord_@1select ( BAT[any::1,any::2] b, any::2 low, any::2 high) : @@ -119,6 +134,7 @@ #ifndef __MALALGEBRA_H__ #define __MALALGEBRA_H__ #include "gdk.h" +#include "gdk_rangejoin.h"
/* nothing much */
@@ -240,6 +256,40 @@ return (*result = BATnlthetajoin(l, r, *mode, (size_t) * estimate)) ? GDK_SUCCEED : GDK_FAIL; }
+int +CMDbandmergejoin(BAT **result, BAT *left, BAT *right, ptr minus, ptr plus, bit *li, bit *hi) +{
return (*result = BATbandmergejoin(left, right, minus, plus, *li,*hi)) ? GDK_SUCCEED : GDK_FAIL; +}
+int +CMDbandmergejoin_limit(BAT **result, BAT *left, BAT *right, ptr minus, ptr plus, bit *li, bit *hi, lng* limit) +{
- size_t cutoff = *limit;
*result = BATbandmergejoin_limit(left, right, minus, plus, *li,*hi, &cutoff);
if (*result) {bat bid = (*result)->batCacheid;*result = BATnew(TYPE_lng, TYPE_bat, 1);if (*result) BUNins(*result, &cutoff, &bid, FALSE);BBPunfix(bid);}return (*result)?GDK_SUCCEED:GDK_FAIL;+}
+int +CMDleftmergejoin_limit(BAT **result, BAT *left, BAT *right, lng* limit) +{
- size_t cutoff = *limit;
*result = BATleftmergejoin_limit(left, right, cutoff, &cutoff);if (*result) {bat bid = (*result)->batCacheid;*result = BATnew(TYPE_lng, TYPE_bat, 1);if (*result) BUNins(*result, &cutoff, &bid, FALSE);BBPunfix(bid);}return (*result)?GDK_SUCCEED:GDK_FAIL;+}
@= selectcmd int CMDord_@1select1(BAT **result, BAT* b, ptr value) { return (*result = BAT_select_(b, value, 0, TRUE, TRUE, @2, FALSE, TRUE))?GDK_SUCCEED:GDK_FAIL;
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
-- Sjoerd Mullender
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
Hi Sjoerd,
The one problem that could occur is people trying to check out MonetDB_4-24 now, compiling it and trying to use with a precompiled MonetDB_1-24, without updating/recompiling that one. So, I agree with you that a problem could occur, but I would be surprised if any of our users would really hit this.
With excuses to pick nits, but contrary to you I believe that API appends are not "just as bad" as API modifications. An API modification in MonetDB, would force a release in MonetDB5, for instance. An API append does not do that (so it is "less bad").
PF_ROX is quite different from the HEAD in the pathfinder code due to bot branches making different changes to the indexing scheme; afaik this has prevented us from keeping it in sync with the HEAD. Various API changes (not appends) and even buildtool changes now force it to depend on the stable MonetDB/MonetDB4. For PF_ROX itself this is not a problem as it has no outside users. However, it might occur that this scenario will repeat (PF_ROX triggering a change/addition in MonetDB/MonetDB4 that would trickle towards the head via MonetDB_[14]-24).
Peter
-----Original Message----- From: Sjoerd Mullender [mailto:sjoerd@acm.org] Sent: Tuesday, October 28, 2008 9:50 PM To: monetdb-developers@lists.sourceforge.net Cc: monetdb-checkins@lists.sourceforge.net Subject: Re: [Monetdb-checkins] MonetDB4/src/modules/contrib malalgebra.mx, MonetDB_4-24, 1.8.2.1, 1.8.2.2
On 2008-10-28 20:20, Peter Boncz wrote:
Hi Sjoerd,
Thanks for keeping an eye on developer behavior and effort to educate. My change, however, only introduces new functions in GDK and commands in MIL;
it
is only an API append.
API appends are just as bad as any other API change. If new code depends on the extra calls, it cannot run with the older version of the release. That is bad. It means that the branch is *not* "stable" anymore.
It is checked into the stable branch because afaik PF_ROX only works with that. But I may as often be mistaken.
I noticed it was for PF_ROX, and that does pose a problem. It seems that the PF_ROX branch should depend on the current version of MonetDB and not on the stable version.
Peter
This type of change is not allowed on a stable branch. Changes of API are *not* allowed.
Since we're going to abandon this stable branch soon, I'll let it pass.
On 2008-10-28 18:49, Peter Boncz wrote:
Update of /cvsroot/monetdb/MonetDB4/src/modules/contrib In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv901/src/modules/contrib
Modified Files: Tag: MonetDB_4-24 malalgebra.mx Log Message: changes to minimize sampling overhead (mostly in XML text index)
- introduce bandmergejoin(), and leftmergejoin()/bandmergejoin() with a
cutoff
U malalgebra.mx Index: malalgebra.mx =================================================================== RCS file: /cvsroot/monetdb/MonetDB4/src/modules/contrib/malalgebra.mx,v retrieving revision 1.8.2.1 retrieving revision 1.8.2.2 diff -u -d -r1.8.2.1 -r1.8.2.2 --- malalgebra.mx 10 Oct 2008 08:42:22 -0000 1.8.2.1 +++ malalgebra.mx 28 Oct 2008 17:49:34 -0000 1.8.2.2 @@ -43,7 +43,6 @@ @:leftjoin(fetch,\nCAUTION: positional matches are assumed not to be out-of-bounds!!)@ @:leftjoin(merge)@ @:leftjoin(hash)@
- .COMMAND leftthetajoin ( BAT[any::1,any::2] left, BAT[any::2,any::3]
right, int mode) : BAT[any::1,any::3] = CMDleftthetajoin; "Hook directly into the 'leftthetajoin' implementation of the join. @@ -58,6 +57,22 @@ Also, for each left tuple, all matching right tuples will appear in their order of appearrance in the right BAT. This property is handy for XQuery processing."
- .COMMAND bandmergejoin ( BAT[any::1,any::2] outer, BAT[any::2,any::3]
inner,
any::2 minus, any::2 plus, bit l_in, bith_in) :
BAT[any::1,any::3] = CMDbandmergejoin;- "The bandjoin algorithm, but forced to use (left) mergejoin rather
than
nested loop"
- .COMMAND bandmergejoin ( BAT[any::1,any::2] outer, BAT[any::2,any::3]
inner,
any::2 minus, any::2 plus, bit l_in, bith_in, lng limit) :
BAT[lng,BAT] = CMDbandmergejoin_limit;- "A bandjoin that uses the merge algorithm and limits the result size.
- Result generation is cut off after this point and the
- result [lng,BAT] holds [estimated_full_resultsize,limited_result]"
- .COMMAND leftmergejoin ( BAT[any::1,any::2] left, BAT[any::2,any::3]
right, lng limit) :
BAT[lng,BAT] = CMDleftmergejoin_limit;- "leftmergejoin, but limit the result (by cutting off result
generation)
- return a BAT[lng,BAT] containing
[full_result_estimate,limited_result]"
@= select .COMMAND ord_@1select ( BAT[any::1,any::2] b, any::2 low, any::2 high)
:
@@ -119,6 +134,7 @@ #ifndef __MALALGEBRA_H__ #define __MALALGEBRA_H__ #include "gdk.h" +#include "gdk_rangejoin.h"
/* nothing much */
@@ -240,6 +256,40 @@ return (*result = BATnlthetajoin(l, r, *mode, (size_t) * estimate)) ? GDK_SUCCEED : GDK_FAIL; }
+int +CMDbandmergejoin(BAT **result, BAT *left, BAT *right, ptr minus, ptr
plus,
bit *li, bit *hi) +{
return (*result = BATbandmergejoin(left, right, minus, plus,
*li,
*hi)) ? GDK_SUCCEED : GDK_FAIL; +}
+int +CMDbandmergejoin_limit(BAT **result, BAT *left, BAT *right, ptr minus,
ptr
plus, bit *li, bit *hi, lng* limit) +{
- size_t cutoff = *limit;
*result = BATbandmergejoin_limit(left, right, minus, plus, *li,*hi, &cutoff);
if (*result) {bat bid = (*result)->batCacheid;*result = BATnew(TYPE_lng, TYPE_bat, 1);if (*result) BUNins(*result, &cutoff, &bid, FALSE);BBPunfix(bid);}return (*result)?GDK_SUCCEED:GDK_FAIL;+}
+int +CMDleftmergejoin_limit(BAT **result, BAT *left, BAT *right, lng* limit) +{
- size_t cutoff = *limit;
*result = BATleftmergejoin_limit(left, right, cutoff, &cutoff);if (*result) {bat bid = (*result)->batCacheid;*result = BATnew(TYPE_lng, TYPE_bat, 1);if (*result) BUNins(*result, &cutoff, &bid, FALSE);BBPunfix(bid);}return (*result)?GDK_SUCCEED:GDK_FAIL;+}
@= selectcmd int CMDord_@1select1(BAT **result, BAT* b, ptr value) { return (*result = BAT_select_(b, value, 0, TRUE, TRUE, @2, FALSE, TRUE))?GDK_SUCCEED:GDK_FAIL;
-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the
world
http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
-- Sjoerd Mullender
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win great
prizes
Grand prize is a trip for two to an Open Source event anywhere in the
world
http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win great
prizes
Grand prize is a trip for two to an Open Source event anywhere in the
world
http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
-- Sjoerd Mullender
------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
On 2008-10-28 22:37, Peter Boncz wrote:
Hi Sjoerd,
The one problem that could occur is people trying to check out MonetDB_4-24 now, compiling it and trying to use with a precompiled MonetDB_1-24, without updating/recompiling that one. So, I agree with you that a problem could occur, but I would be surprised if any of our users would really hit this.
Because we are going to abandon this branch I was willing to let the change pass. But in general I am opposed.
I agree that in this particular case, it is unlikely that people will encounter any problems. But again, that is because we are going to abandon this branch.
With excuses to pick nits, but contrary to you I believe that API appends are not "just as bad" as API modifications. An API modification in MonetDB, would force a release in MonetDB5, for instance. An API append does not do that (so it is "less bad").
It *is* bad. See the disaster involving the addition of True and False in Python in a minor release. They have sworn to never make that mistake again. People expect API stability in minor releases and adding something to the API is *not* stability.
PF_ROX is quite different from the HEAD in the pathfinder code due to bot branches making different changes to the indexing scheme; afaik this has prevented us from keeping it in sync with the HEAD. Various API changes (not appends) and even buildtool changes now force it to depend on the stable MonetDB/MonetDB4. For PF_ROX itself this is not a problem as it has no outside users. However, it might occur that this scenario will repeat (PF_ROX triggering a change/addition in MonetDB/MonetDB4 that would trickle towards the head via MonetDB_[14]-24).
I am not talking about the PF_ROX branch. It is not a "stable" branch and you can do with it what you want. But please do consider whether it is possible to use the development branches of MonetDB and MonetDB4 with PF_ROX. If you can do that, you can make needed changes to the API there.
Peter
-----Original Message----- From: Sjoerd Mullender [mailto:sjoerd@acm.org] Sent: Tuesday, October 28, 2008 9:50 PM To: monetdb-developers@lists.sourceforge.net Cc: monetdb-checkins@lists.sourceforge.net Subject: Re: [Monetdb-checkins] MonetDB4/src/modules/contrib malalgebra.mx, MonetDB_4-24, 1.8.2.1, 1.8.2.2
On 2008-10-28 20:20, Peter Boncz wrote:
Hi Sjoerd,
Thanks for keeping an eye on developer behavior and effort to educate. My change, however, only introduces new functions in GDK and commands in MIL;
it
is only an API append.
API appends are just as bad as any other API change. If new code depends on the extra calls, it cannot run with the older version of the release. That is bad. It means that the branch is *not* "stable" anymore.
It is checked into the stable branch because afaik PF_ROX only works with that. But I may as often be mistaken.
I noticed it was for PF_ROX, and that does pose a problem. It seems that the PF_ROX branch should depend on the current version of MonetDB and not on the stable version.
Peter
This type of change is not allowed on a stable branch. Changes of API are *not* allowed.
Since we're going to abandon this stable branch soon, I'll let it pass.
On 2008-10-28 18:49, Peter Boncz wrote:
Update of /cvsroot/monetdb/MonetDB4/src/modules/contrib In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv901/src/modules/contrib
Modified Files: Tag: MonetDB_4-24 malalgebra.mx Log Message: changes to minimize sampling overhead (mostly in XML text index)
- introduce bandmergejoin(), and leftmergejoin()/bandmergejoin() with a
cutoff
U malalgebra.mx Index: malalgebra.mx =================================================================== RCS file: /cvsroot/monetdb/MonetDB4/src/modules/contrib/malalgebra.mx,v retrieving revision 1.8.2.1 retrieving revision 1.8.2.2 diff -u -d -r1.8.2.1 -r1.8.2.2 --- malalgebra.mx 10 Oct 2008 08:42:22 -0000 1.8.2.1 +++ malalgebra.mx 28 Oct 2008 17:49:34 -0000 1.8.2.2 @@ -43,7 +43,6 @@ @:leftjoin(fetch,\nCAUTION: positional matches are assumed not to be out-of-bounds!!)@ @:leftjoin(merge)@ @:leftjoin(hash)@
- .COMMAND leftthetajoin ( BAT[any::1,any::2] left, BAT[any::2,any::3]
right, int mode) : BAT[any::1,any::3] = CMDleftthetajoin; "Hook directly into the 'leftthetajoin' implementation of the join. @@ -58,6 +57,22 @@ Also, for each left tuple, all matching right tuples will appear in their order of appearrance in the right BAT. This property is handy for XQuery processing."
- .COMMAND bandmergejoin ( BAT[any::1,any::2] outer, BAT[any::2,any::3]
inner,
any::2 minus, any::2 plus, bit l_in, bith_in) :
BAT[any::1,any::3] = CMDbandmergejoin;- "The bandjoin algorithm, but forced to use (left) mergejoin rather
than
nested loop"
- .COMMAND bandmergejoin ( BAT[any::1,any::2] outer, BAT[any::2,any::3]
inner,
any::2 minus, any::2 plus, bit l_in, bith_in, lng limit) :
BAT[lng,BAT] = CMDbandmergejoin_limit;- "A bandjoin that uses the merge algorithm and limits the result size.
- Result generation is cut off after this point and the
- result [lng,BAT] holds [estimated_full_resultsize,limited_result]"
- .COMMAND leftmergejoin ( BAT[any::1,any::2] left, BAT[any::2,any::3]
right, lng limit) :
BAT[lng,BAT] = CMDleftmergejoin_limit;- "leftmergejoin, but limit the result (by cutting off result
generation)
- return a BAT[lng,BAT] containing
[full_result_estimate,limited_result]"
@= select .COMMAND ord_@1select ( BAT[any::1,any::2] b, any::2 low, any::2 high)
:
@@ -119,6 +134,7 @@ #ifndef __MALALGEBRA_H__ #define __MALALGEBRA_H__ #include "gdk.h" +#include "gdk_rangejoin.h"
/* nothing much */
@@ -240,6 +256,40 @@ return (*result = BATnlthetajoin(l, r, *mode, (size_t) * estimate)) ? GDK_SUCCEED : GDK_FAIL; }
+int +CMDbandmergejoin(BAT **result, BAT *left, BAT *right, ptr minus, ptr
plus,
bit *li, bit *hi) +{
return (*result = BATbandmergejoin(left, right, minus, plus,*li,
*hi)) ? GDK_SUCCEED : GDK_FAIL; +}
+int +CMDbandmergejoin_limit(BAT **result, BAT *left, BAT *right, ptr minus,
ptr
plus, bit *li, bit *hi, lng* limit) +{
- size_t cutoff = *limit;
*result = BATbandmergejoin_limit(left, right, minus, plus, *li,*hi, &cutoff);
if (*result) {bat bid = (*result)->batCacheid;*result = BATnew(TYPE_lng, TYPE_bat, 1);if (*result) BUNins(*result, &cutoff, &bid, FALSE);BBPunfix(bid);}return (*result)?GDK_SUCCEED:GDK_FAIL;+}
+int +CMDleftmergejoin_limit(BAT **result, BAT *left, BAT *right, lng* limit) +{
- size_t cutoff = *limit;
*result = BATleftmergejoin_limit(left, right, cutoff, &cutoff);if (*result) {bat bid = (*result)->batCacheid;*result = BATnew(TYPE_lng, TYPE_bat, 1);if (*result) BUNins(*result, &cutoff, &bid, FALSE);BBPunfix(bid);}return (*result)?GDK_SUCCEED:GDK_FAIL;+}
@= selectcmd int CMDord_@1select1(BAT **result, BAT* b, ptr value) { return (*result = BAT_select_(b, value, 0, TRUE, TRUE, @2, FALSE, TRUE))?GDK_SUCCEED:GDK_FAIL;
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the
world
http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
-- Sjoerd Mullender
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win great
prizes
Grand prize is a trip for two to an Open Source event anywhere in the
world
http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win great
prizes
Grand prize is a trip for two to an Open Source event anywhere in the
world
http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
-- Sjoerd Mullender
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
Gents,
given (1) the inherent incompatibility between the PF_ROX branch (forked of the XQuery_0-24 branch of pathfinder) and the pathfinder development trunk and the facts that (2) PF_ROX (at least for now) is purely for internal research purposes and (3) cahnges ("features") in PF_ROX do (e.g.) not support updates, i.e., are far from becoming "main stream",
I guess the (only?) proper solution would (have been?) to also fork PF_ROX branches of the MonetDB_1-24 branch (of MonetDB) and the MonetDB_4-24 branch (of MonetDB4) to accomodate Peter's yesterday's changes.
Note though, that this would / will exclude PF_ROX-specific changes in MonetDB & MonetDB$ from being tested --- this is already the case for the pathfinder PF_ROX branch and will be the case for the MonetDB_2-4 & MonetDB_4-24 branches, too, once we start prparing for the next release no later than next week.
See the results of "Stable" testing (once they are ready later today) to judged whether testing is "desired"/"required" or not ...
Stefan
On Tue, Oct 28, 2008 at 11:36:17PM +0100, Sjoerd Mullender wrote:
On 2008-10-28 22:37, Peter Boncz wrote:
Hi Sjoerd,
The one problem that could occur is people trying to check out MonetDB_4-24 now, compiling it and trying to use with a precompiled MonetDB_1-24, without updating/recompiling that one. So, I agree with you that a problem could occur, but I would be surprised if any of our users would really hit this.
Because we are going to abandon this branch I was willing to let the change pass. But in general I am opposed.
I agree that in this particular case, it is unlikely that people will encounter any problems. But again, that is because we are going to abandon this branch.
With excuses to pick nits, but contrary to you I believe that API appends are not "just as bad" as API modifications. An API modification in MonetDB, would force a release in MonetDB5, for instance. An API append does not do that (so it is "less bad").
It *is* bad. See the disaster involving the addition of True and False in Python in a minor release. They have sworn to never make that mistake again. People expect API stability in minor releases and adding something to the API is *not* stability.
PF_ROX is quite different from the HEAD in the pathfinder code due to bot branches making different changes to the indexing scheme; afaik this has prevented us from keeping it in sync with the HEAD. Various API changes (not appends) and even buildtool changes now force it to depend on the stable MonetDB/MonetDB4. For PF_ROX itself this is not a problem as it has no outside users. However, it might occur that this scenario will repeat (PF_ROX triggering a change/addition in MonetDB/MonetDB4 that would trickle towards the head via MonetDB_[14]-24).
I am not talking about the PF_ROX branch. It is not a "stable" branch and you can do with it what you want. But please do consider whether it is possible to use the development branches of MonetDB and MonetDB4 with PF_ROX. If you can do that, you can make needed changes to the API there.
Peter
-----Original Message----- From: Sjoerd Mullender [mailto:sjoerd@acm.org] Sent: Tuesday, October 28, 2008 9:50 PM To: monetdb-developers@lists.sourceforge.net Cc: monetdb-checkins@lists.sourceforge.net Subject: Re: [Monetdb-checkins] MonetDB4/src/modules/contrib malalgebra.mx, MonetDB_4-24, 1.8.2.1, 1.8.2.2
On 2008-10-28 20:20, Peter Boncz wrote:
Hi Sjoerd,
Thanks for keeping an eye on developer behavior and effort to educate. My change, however, only introduces new functions in GDK and commands in MIL;
it
is only an API append.
API appends are just as bad as any other API change. If new code depends on the extra calls, it cannot run with the older version of the release. That is bad. It means that the branch is *not* "stable" anymore.
It is checked into the stable branch because afaik PF_ROX only works with that. But I may as often be mistaken.
I noticed it was for PF_ROX, and that does pose a problem. It seems that the PF_ROX branch should depend on the current version of MonetDB and not on the stable version.
Peter
This type of change is not allowed on a stable branch. Changes of API are *not* allowed.
Since we're going to abandon this stable branch soon, I'll let it pass.
On 2008-10-28 18:49, Peter Boncz wrote:
Update of /cvsroot/monetdb/MonetDB4/src/modules/contrib In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv901/src/modules/contrib
Modified Files: Tag: MonetDB_4-24 malalgebra.mx Log Message: changes to minimize sampling overhead (mostly in XML text index)
- introduce bandmergejoin(), and leftmergejoin()/bandmergejoin() with a
cutoff
U malalgebra.mx Index: malalgebra.mx =================================================================== RCS file: /cvsroot/monetdb/MonetDB4/src/modules/contrib/malalgebra.mx,v retrieving revision 1.8.2.1 retrieving revision 1.8.2.2 diff -u -d -r1.8.2.1 -r1.8.2.2 --- malalgebra.mx 10 Oct 2008 08:42:22 -0000 1.8.2.1 +++ malalgebra.mx 28 Oct 2008 17:49:34 -0000 1.8.2.2 @@ -43,7 +43,6 @@ @:leftjoin(fetch,\nCAUTION: positional matches are assumed not to be out-of-bounds!!)@ @:leftjoin(merge)@ @:leftjoin(hash)@
- .COMMAND leftthetajoin ( BAT[any::1,any::2] left, BAT[any::2,any::3]
right, int mode) : BAT[any::1,any::3] = CMDleftthetajoin; "Hook directly into the 'leftthetajoin' implementation of the join. @@ -58,6 +57,22 @@ Also, for each left tuple, all matching right tuples will appear in their order of appearrance in the right BAT. This property is handy for XQuery processing."
- .COMMAND bandmergejoin ( BAT[any::1,any::2] outer, BAT[any::2,any::3]
inner,
any::2 minus, any::2 plus, bit l_in, bith_in) :
BAT[any::1,any::3] = CMDbandmergejoin;- "The bandjoin algorithm, but forced to use (left) mergejoin rather
than
nested loop"
- .COMMAND bandmergejoin ( BAT[any::1,any::2] outer, BAT[any::2,any::3]
inner,
any::2 minus, any::2 plus, bit l_in, bith_in, lng limit) :
BAT[lng,BAT] = CMDbandmergejoin_limit;- "A bandjoin that uses the merge algorithm and limits the result size.
- Result generation is cut off after this point and the
- result [lng,BAT] holds [estimated_full_resultsize,limited_result]"
- .COMMAND leftmergejoin ( BAT[any::1,any::2] left, BAT[any::2,any::3]
right, lng limit) :
BAT[lng,BAT] = CMDleftmergejoin_limit;- "leftmergejoin, but limit the result (by cutting off result
generation)
- return a BAT[lng,BAT] containing
[full_result_estimate,limited_result]"
@= select .COMMAND ord_@1select ( BAT[any::1,any::2] b, any::2 low, any::2 high)
:
@@ -119,6 +134,7 @@ #ifndef __MALALGEBRA_H__ #define __MALALGEBRA_H__ #include "gdk.h" +#include "gdk_rangejoin.h"
/* nothing much */
@@ -240,6 +256,40 @@ return (*result = BATnlthetajoin(l, r, *mode, (size_t) * estimate)) ? GDK_SUCCEED : GDK_FAIL; }
+int +CMDbandmergejoin(BAT **result, BAT *left, BAT *right, ptr minus, ptr
plus,
bit *li, bit *hi) +{
return (*result = BATbandmergejoin(left, right, minus, plus,*li,
*hi)) ? GDK_SUCCEED : GDK_FAIL; +}
+int +CMDbandmergejoin_limit(BAT **result, BAT *left, BAT *right, ptr minus,
ptr
plus, bit *li, bit *hi, lng* limit) +{
- size_t cutoff = *limit;
*result = BATbandmergejoin_limit(left, right, minus, plus, *li,*hi, &cutoff);
if (*result) {bat bid = (*result)->batCacheid;*result = BATnew(TYPE_lng, TYPE_bat, 1);if (*result) BUNins(*result, &cutoff, &bid, FALSE);BBPunfix(bid);}return (*result)?GDK_SUCCEED:GDK_FAIL;+}
+int +CMDleftmergejoin_limit(BAT **result, BAT *left, BAT *right, lng* limit) +{
- size_t cutoff = *limit;
*result = BATleftmergejoin_limit(left, right, cutoff, &cutoff);if (*result) {bat bid = (*result)->batCacheid;*result = BATnew(TYPE_lng, TYPE_bat, 1);if (*result) BUNins(*result, &cutoff, &bid, FALSE);BBPunfix(bid);}return (*result)?GDK_SUCCEED:GDK_FAIL;+}
@= selectcmd int CMDord_@1select1(BAT **result, BAT* b, ptr value) { return (*result = BAT_select_(b, value, 0, TRUE, TRUE, @2, FALSE, TRUE))?GDK_SUCCEED:GDK_FAIL;
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the
world
http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
-- Sjoerd Mullender
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win great
prizes
Grand prize is a trip for two to an Open Source event anywhere in the
world
http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win great
prizes
Grand prize is a trip for two to an Open Source event anywhere in the
world
http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
-- Sjoerd Mullender
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
-- Sjoerd Mullender
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
On Wed, Oct 29, 2008 at 10:27:35AM +0100, Stefan Manegold wrote:
Gents,
given (1) the inherent incompatibility between the PF_ROX branch (forked of the XQuery_0-24 branch of pathfinder) and the pathfinder development trunk
and (hence) the incompatibility between the PF_ROX branch and the development trunks of MonetDB & MonetDB4
and the facts that (2) PF_ROX (at least for now) is purely for internal research purposes and (3) cahnges ("features") in PF_ROX do (e.g.) not support updates, i.e., are far from becoming "main stream",
I guess the (only?) proper solution would (have been?) to also fork PF_ROX branches of the MonetDB_1-24 branch (of MonetDB) and the MonetDB_4-24 branch (of MonetDB4) to accomodate Peter's yesterday's changes.
Note though, that this would / will exclude PF_ROX-specific changes in MonetDB & MonetDB$ from being tested --- this is already the case for the pathfinder PF_ROX branch and will be the case for the MonetDB_1-24 & MonetDB_4-24 branches, too, once we start prparing for the next release no later than next week.
Also, changes from the PF_ROX branches of MonetDB & MonetDB4 would/will than NOT be propagated to the respective development trunks; the same will be the case for (potential0 changes to the MonetDB_1-24 & MonetDB_4-24 branches as soon as we start prparing for the next release no later than next week.
In fact, I would even suggest to not propagate Peter's yesterday's changes to the development trunk since they are "PF_ROX-specific" (at least for now).
Stefan
See the results of "Stable" testing (once they are ready later today) to judged whether testing is "desired"/"required" or not ...
Stefan
On Tue, Oct 28, 2008 at 11:36:17PM +0100, Sjoerd Mullender wrote:
On 2008-10-28 22:37, Peter Boncz wrote:
Hi Sjoerd,
The one problem that could occur is people trying to check out MonetDB_4-24 now, compiling it and trying to use with a precompiled MonetDB_1-24, without updating/recompiling that one. So, I agree with you that a problem could occur, but I would be surprised if any of our users would really hit this.
Because we are going to abandon this branch I was willing to let the change pass. But in general I am opposed.
I agree that in this particular case, it is unlikely that people will encounter any problems. But again, that is because we are going to abandon this branch.
With excuses to pick nits, but contrary to you I believe that API appends are not "just as bad" as API modifications. An API modification in MonetDB, would force a release in MonetDB5, for instance. An API append does not do that (so it is "less bad").
It *is* bad. See the disaster involving the addition of True and False in Python in a minor release. They have sworn to never make that mistake again. People expect API stability in minor releases and adding something to the API is *not* stability.
PF_ROX is quite different from the HEAD in the pathfinder code due to bot branches making different changes to the indexing scheme; afaik this has prevented us from keeping it in sync with the HEAD. Various API changes (not appends) and even buildtool changes now force it to depend on the stable MonetDB/MonetDB4. For PF_ROX itself this is not a problem as it has no outside users. However, it might occur that this scenario will repeat (PF_ROX triggering a change/addition in MonetDB/MonetDB4 that would trickle towards the head via MonetDB_[14]-24).
I am not talking about the PF_ROX branch. It is not a "stable" branch and you can do with it what you want. But please do consider whether it is possible to use the development branches of MonetDB and MonetDB4 with PF_ROX. If you can do that, you can make needed changes to the API there.
Peter
-----Original Message----- From: Sjoerd Mullender [mailto:sjoerd@acm.org] Sent: Tuesday, October 28, 2008 9:50 PM To: monetdb-developers@lists.sourceforge.net Cc: monetdb-checkins@lists.sourceforge.net Subject: Re: [Monetdb-checkins] MonetDB4/src/modules/contrib malalgebra.mx, MonetDB_4-24, 1.8.2.1, 1.8.2.2
On 2008-10-28 20:20, Peter Boncz wrote:
Hi Sjoerd,
Thanks for keeping an eye on developer behavior and effort to educate. My change, however, only introduces new functions in GDK and commands in MIL;
it
is only an API append.
API appends are just as bad as any other API change. If new code depends on the extra calls, it cannot run with the older version of the release. That is bad. It means that the branch is *not* "stable" anymore.
It is checked into the stable branch because afaik PF_ROX only works with that. But I may as often be mistaken.
I noticed it was for PF_ROX, and that does pose a problem. It seems that the PF_ROX branch should depend on the current version of MonetDB and not on the stable version.
Peter
This type of change is not allowed on a stable branch. Changes of API are *not* allowed.
Since we're going to abandon this stable branch soon, I'll let it pass.
On 2008-10-28 18:49, Peter Boncz wrote:
Update of /cvsroot/monetdb/MonetDB4/src/modules/contrib In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv901/src/modules/contrib
Modified Files: Tag: MonetDB_4-24 malalgebra.mx Log Message: changes to minimize sampling overhead (mostly in XML text index)
- introduce bandmergejoin(), and leftmergejoin()/bandmergejoin() with a
cutoff
U malalgebra.mx Index: malalgebra.mx =================================================================== RCS file: /cvsroot/monetdb/MonetDB4/src/modules/contrib/malalgebra.mx,v retrieving revision 1.8.2.1 retrieving revision 1.8.2.2 diff -u -d -r1.8.2.1 -r1.8.2.2 --- malalgebra.mx 10 Oct 2008 08:42:22 -0000 1.8.2.1 +++ malalgebra.mx 28 Oct 2008 17:49:34 -0000 1.8.2.2 @@ -43,7 +43,6 @@ @:leftjoin(fetch,\nCAUTION: positional matches are assumed not to be out-of-bounds!!)@ @:leftjoin(merge)@ @:leftjoin(hash)@
- .COMMAND leftthetajoin ( BAT[any::1,any::2] left, BAT[any::2,any::3]
right, int mode) : BAT[any::1,any::3] = CMDleftthetajoin; "Hook directly into the 'leftthetajoin' implementation of the join. @@ -58,6 +57,22 @@ Also, for each left tuple, all matching right tuples will appear in their order of appearrance in the right BAT. This property is handy for XQuery processing."
- .COMMAND bandmergejoin ( BAT[any::1,any::2] outer, BAT[any::2,any::3]
inner,
any::2 minus, any::2 plus, bit l_in, bith_in) :
BAT[any::1,any::3] = CMDbandmergejoin;- "The bandjoin algorithm, but forced to use (left) mergejoin rather
than
nested loop"
- .COMMAND bandmergejoin ( BAT[any::1,any::2] outer, BAT[any::2,any::3]
inner,
any::2 minus, any::2 plus, bit l_in, bith_in, lng limit) :
BAT[lng,BAT] = CMDbandmergejoin_limit;- "A bandjoin that uses the merge algorithm and limits the result size.
- Result generation is cut off after this point and the
- result [lng,BAT] holds [estimated_full_resultsize,limited_result]"
- .COMMAND leftmergejoin ( BAT[any::1,any::2] left, BAT[any::2,any::3]
right, lng limit) :
BAT[lng,BAT] = CMDleftmergejoin_limit;- "leftmergejoin, but limit the result (by cutting off result
generation)
- return a BAT[lng,BAT] containing
[full_result_estimate,limited_result]"
@= select .COMMAND ord_@1select ( BAT[any::1,any::2] b, any::2 low, any::2 high)
:
@@ -119,6 +134,7 @@ #ifndef __MALALGEBRA_H__ #define __MALALGEBRA_H__ #include "gdk.h" +#include "gdk_rangejoin.h"
/* nothing much */
@@ -240,6 +256,40 @@ return (*result = BATnlthetajoin(l, r, *mode, (size_t) * estimate)) ? GDK_SUCCEED : GDK_FAIL; }
+int +CMDbandmergejoin(BAT **result, BAT *left, BAT *right, ptr minus, ptr
plus,
bit *li, bit *hi) +{
return (*result = BATbandmergejoin(left, right, minus, plus,*li,
*hi)) ? GDK_SUCCEED : GDK_FAIL; +}
+int +CMDbandmergejoin_limit(BAT **result, BAT *left, BAT *right, ptr minus,
ptr
plus, bit *li, bit *hi, lng* limit) +{
- size_t cutoff = *limit;
*result = BATbandmergejoin_limit(left, right, minus, plus, *li,*hi, &cutoff);
if (*result) {bat bid = (*result)->batCacheid;*result = BATnew(TYPE_lng, TYPE_bat, 1);if (*result) BUNins(*result, &cutoff, &bid, FALSE);BBPunfix(bid);}return (*result)?GDK_SUCCEED:GDK_FAIL;+}
+int +CMDleftmergejoin_limit(BAT **result, BAT *left, BAT *right, lng* limit) +{
- size_t cutoff = *limit;
*result = BATleftmergejoin_limit(left, right, cutoff, &cutoff);if (*result) {bat bid = (*result)->batCacheid;*result = BATnew(TYPE_lng, TYPE_bat, 1);if (*result) BUNins(*result, &cutoff, &bid, FALSE);BBPunfix(bid);}return (*result)?GDK_SUCCEED:GDK_FAIL;+}
@= selectcmd int CMDord_@1select1(BAT **result, BAT* b, ptr value) { return (*result = BAT_select_(b, value, 0, TRUE, TRUE, @2, FALSE, TRUE))?GDK_SUCCEED:GDK_FAIL;
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the
world
http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
-- Sjoerd Mullender
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win great
prizes
Grand prize is a trip for two to an Open Source event anywhere in the
world
http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win great
prizes
Grand prize is a trip for two to an Open Source event anywhere in the
world
http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
-- Sjoerd Mullender
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
-- Sjoerd Mullender
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
I would only move items into the development trunk that are part of GDK.
Stefan Manegold wrote:
On Wed, Oct 29, 2008 at 10:27:35AM +0100, Stefan Manegold wrote:
Gents,
given (1) the inherent incompatibility between the PF_ROX branch (forked of the XQuery_0-24 branch of pathfinder) and the pathfinder development trunk
and (hence) the incompatibility between the PF_ROX branch and the development trunks of MonetDB & MonetDB4
and the facts that (2) PF_ROX (at least for now) is purely for internal research purposes and (3) cahnges ("features") in PF_ROX do (e.g.) not support updates, i.e., are far from becoming "main stream",
I guess the (only?) proper solution would (have been?) to also fork PF_ROX branches of the MonetDB_1-24 branch (of MonetDB) and the MonetDB_4-24 branch (of MonetDB4) to accomodate Peter's yesterday's changes.
Note though, that this would / will exclude PF_ROX-specific changes in MonetDB & MonetDB$ from being tested --- this is already the case for the pathfinder PF_ROX branch and will be the case for the MonetDB_1-24 & MonetDB_4-24 branches, too, once we start prparing for the next release no later than next week.
Also, changes from the PF_ROX branches of MonetDB & MonetDB4 would/will than NOT be propagated to the respective development trunks; the same will be the case for (potential0 changes to the MonetDB_1-24 & MonetDB_4-24 branches as soon as we start prparing for the next release no later than next week.
In fact, I would even suggest to not propagate Peter's yesterday's changes to the development trunk since they are "PF_ROX-specific" (at least for now).
Stefan
See the results of "Stable" testing (once they are ready later today) to judged whether testing is "desired"/"required" or not ...
Stefan
On Tue, Oct 28, 2008 at 11:36:17PM +0100, Sjoerd Mullender wrote:
On 2008-10-28 22:37, Peter Boncz wrote:
Hi Sjoerd,
The one problem that could occur is people trying to check out MonetDB_4-24 now, compiling it and trying to use with a precompiled MonetDB_1-24, without updating/recompiling that one. So, I agree with you that a problem could occur, but I would be surprised if any of our users would really hit this.
Because we are going to abandon this branch I was willing to let the change pass. But in general I am opposed.
I agree that in this particular case, it is unlikely that people will encounter any problems. But again, that is because we are going to abandon this branch.
With excuses to pick nits, but contrary to you I believe that API appends are not "just as bad" as API modifications. An API modification in MonetDB, would force a release in MonetDB5, for instance. An API append does not do that (so it is "less bad").
It *is* bad. See the disaster involving the addition of True and False in Python in a minor release. They have sworn to never make that mistake again. People expect API stability in minor releases and adding something to the API is *not* stability.
PF_ROX is quite different from the HEAD in the pathfinder code due to bot branches making different changes to the indexing scheme; afaik this has prevented us from keeping it in sync with the HEAD. Various API changes (not appends) and even buildtool changes now force it to depend on the stable MonetDB/MonetDB4. For PF_ROX itself this is not a problem as it has no outside users. However, it might occur that this scenario will repeat (PF_ROX triggering a change/addition in MonetDB/MonetDB4 that would trickle towards the head via MonetDB_[14]-24).
I am not talking about the PF_ROX branch. It is not a "stable" branch and you can do with it what you want. But please do consider whether it is possible to use the development branches of MonetDB and MonetDB4 with PF_ROX. If you can do that, you can make needed changes to the API there.
Peter
-----Original Message----- From: Sjoerd Mullender [mailto:sjoerd@acm.org] Sent: Tuesday, October 28, 2008 9:50 PM To: monetdb-developers@lists.sourceforge.net Cc: monetdb-checkins@lists.sourceforge.net Subject: Re: [Monetdb-checkins] MonetDB4/src/modules/contrib malalgebra.mx, MonetDB_4-24, 1.8.2.1, 1.8.2.2
On 2008-10-28 20:20, Peter Boncz wrote:
Hi Sjoerd,
Thanks for keeping an eye on developer behavior and effort to educate. My change, however, only introduces new functions in GDK and commands in MIL;
it
is only an API append.
API appends are just as bad as any other API change. If new code depends on the extra calls, it cannot run with the older version of the release. That is bad. It means that the branch is *not* "stable" anymore.
It is checked into the stable branch because afaik PF_ROX only works with that. But I may as often be mistaken.
I noticed it was for PF_ROX, and that does pose a problem. It seems that the PF_ROX branch should depend on the current version of MonetDB and not on the stable version.
Peter
This type of change is not allowed on a stable branch. Changes of API are *not* allowed.
Since we're going to abandon this stable branch soon, I'll let it pass.
On 2008-10-28 18:49, Peter Boncz wrote:
> Update of /cvsroot/monetdb/MonetDB4/src/modules/contrib > In directory > 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv901/src/modules/contrib > > Modified Files: > Tag: MonetDB_4-24 > malalgebra.mx > Log Message: > changes to minimize sampling overhead (mostly in XML text index) > > - introduce bandmergejoin(), and leftmergejoin()/bandmergejoin() with a > cutoff > > > > U malalgebra.mx > Index: malalgebra.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB4/src/modules/contrib/malalgebra.mx,v > retrieving revision 1.8.2.1 > retrieving revision 1.8.2.2 > diff -u -d -r1.8.2.1 -r1.8.2.2 > --- malalgebra.mx 10 Oct 2008 08:42:22 -0000 1.8.2.1 > +++ malalgebra.mx 28 Oct 2008 17:49:34 -0000 1.8.2.2 > @@ -43,7 +43,6 @@ > @:leftjoin(fetch,\nCAUTION: positional matches are assumed not to be > out-of-bounds!!)@ > @:leftjoin(merge)@ > @:leftjoin(hash)@ > - > .COMMAND leftthetajoin ( BAT[any::1,any::2] left, BAT[any::2,any::3] > right, int mode) : > BAT[any::1,any::3] = CMDleftthetajoin; > "Hook directly into the 'leftthetajoin' implementation of the join. > @@ -58,6 +57,22 @@ > Also, for each left tuple, all matching right tuples will appear in > their order > of appearrance in the right BAT. This property is handy for XQuery > processing." > > + .COMMAND bandmergejoin ( BAT[any::1,any::2] outer, BAT[any::2,any::3] > inner, > + any::2 minus, any::2 plus, bit l_in, bit > h_in) : > + BAT[any::1,any::3] = CMDbandmergejoin; > + "The bandjoin algorithm, but forced to use (left) mergejoin rather >
than
> nested loop" > + > + .COMMAND bandmergejoin ( BAT[any::1,any::2] outer, BAT[any::2,any::3] > inner, > + any::2 minus, any::2 plus, bit l_in, bit > h_in, lng limit) : > + BAT[lng,BAT] = CMDbandmergejoin_limit; > + "A bandjoin that uses the merge algorithm and limits the result size. > + Result generation is cut off after this point and the > + result [lng,BAT] holds [estimated_full_resultsize,limited_result]" > + > + .COMMAND leftmergejoin ( BAT[any::1,any::2] left, BAT[any::2,any::3] > right, lng limit) : > + BAT[lng,BAT] = CMDleftmergejoin_limit; > + "leftmergejoin, but limit the result (by cutting off result >
generation)
> + return a BAT[lng,BAT] containing >
[full_result_estimate,limited_result]"
> @= select > .COMMAND ord_@1select ( BAT[any::1,any::2] b, any::2 low, any::2 high) >
:
> @@ -119,6 +134,7 @@ > #ifndef __MALALGEBRA_H__ > #define __MALALGEBRA_H__ > #include "gdk.h" > +#include "gdk_rangejoin.h" > > /* nothing much */ > > @@ -240,6 +256,40 @@ > return (*result = BATnlthetajoin(l, r, *mode, (size_t) * estimate)) ? > GDK_SUCCEED : GDK_FAIL; > } > > +int > +CMDbandmergejoin(BAT **result, BAT *left, BAT *right, ptr minus, ptr >
plus,
> bit *li, bit *hi) > +{ > + return (*result = BATbandmergejoin(left, right, minus, plus, >
*li,
> *hi)) ? GDK_SUCCEED : GDK_FAIL; > +} > + > +int > +CMDbandmergejoin_limit(BAT **result, BAT *left, BAT *right, ptr minus, >
ptr
> plus, bit *li, bit *hi, lng* limit) > +{ > + size_t cutoff = *limit; > + *result = BATbandmergejoin_limit(left, right, minus, plus, *li, > *hi, &cutoff); > + if (*result) { > + bat bid = (*result)->batCacheid; > + *result = BATnew(TYPE_lng, TYPE_bat, 1); > + if (*result) BUNins(*result, &cutoff, &bid, FALSE); > + BBPunfix(bid); > + } > + return (*result)?GDK_SUCCEED:GDK_FAIL; > +} > + > +int > +CMDleftmergejoin_limit(BAT **result, BAT *left, BAT *right, lng* limit) > +{ > + size_t cutoff = *limit; > + *result = BATleftmergejoin_limit(left, right, cutoff, &cutoff); > + if (*result) { > + bat bid = (*result)->batCacheid; > + *result = BATnew(TYPE_lng, TYPE_bat, 1); > + if (*result) BUNins(*result, &cutoff, &bid, FALSE); > + BBPunfix(bid); > + } > + return (*result)?GDK_SUCCEED:GDK_FAIL; > +} > + > @= selectcmd > int CMDord_@1select1(BAT **result, BAT* b, ptr value) { > return (*result = BAT_select_(b, value, 0, TRUE, TRUE, @2, FALSE, > TRUE))?GDK_SUCCEED:GDK_FAIL; > > > >
> This SF.Net email is sponsored by the Moblin Your Move Developer's >
challenge
> Build the coolest Linux based applications with Moblin SDK & win great > prizes > Grand prize is a trip for two to an Open Source event anywhere in the >
world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
>
Sjoerd Mullender
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win great
prizes
Grand prize is a trip for two to an Open Source event anywhere in the
world
http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win great
prizes
Grand prize is a trip for two to an Open Source event anywhere in the
world
http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
-- Sjoerd Mullender
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
-- Sjoerd Mullender
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
Hi Stefan,
I engineered the changes to be MonetDB4 and even MonetDB5 friendly, and though PF_ROX is a proof of concept branch, the concepts will at a later stage re-appear in a main branch (certainly those on a GDK level).
However, the changes in MonetDB to support it are minimal; it is not a sacrifice to have them in a HEAD branch.
I actually think we should just make that little bit of effort to port PF_ROX to the MonetDB[4] HEAD, and take it from there. Part of that step is actually the propagation of my branch changes to the HEAD (as I was assuming would happen semi-automatically).
Peter
-----Original Message----- From: Stefan Manegold [mailto:Stefan.Manegold@cwi.nl] Sent: Wednesday, October 29, 2008 10:35 AM To: monetdb-developers@lists.sourceforge.net Cc: P.Boncz@cwi.nl Subject: Re: [Monetdb-checkins] MonetDB4/src/modules/contrib malalgebra.mx, MonetDB_4-24, 1.8.2.1, 1.8.2.2
On Wed, Oct 29, 2008 at 10:27:35AM +0100, Stefan Manegold wrote:
Gents,
given (1) the inherent incompatibility between the PF_ROX branch (forked
of
the XQuery_0-24 branch of pathfinder) and the pathfinder development
trunk
and (hence) the incompatibility between the PF_ROX branch and the development trunks of MonetDB & MonetDB4
and the facts that (2) PF_ROX (at least for now) is purely for internal research purposes and (3) cahnges ("features") in PF_ROX do (e.g.) not support updates, i.e., are far from becoming "main stream",
I guess the (only?) proper solution would (have been?) to also fork
PF_ROX
branches of the MonetDB_1-24 branch (of MonetDB) and the MonetDB_4-24
branch
(of MonetDB4) to accomodate Peter's yesterday's changes.
Note though, that this would / will exclude PF_ROX-specific changes in MonetDB & MonetDB$ from being tested --- this is already the case for the pathfinder PF_ROX branch and will be the case for the MonetDB_1-24 & MonetDB_4-24 branches, too, once we start prparing for the next release
no
later than next week.
Also, changes from the PF_ROX branches of MonetDB & MonetDB4 would/will than NOT be propagated to the respective development trunks; the same will be the case for (potential0 changes to the MonetDB_1-24 & MonetDB_4-24 branches as soon as we start prparing for the next release no later than next week.
In fact, I would even suggest to not propagate Peter's yesterday's changes to the development trunk since they are "PF_ROX-specific" (at least for now).
Stefan
See the results of "Stable" testing (once they are ready later today) to judged whether testing is "desired"/"required" or not ...
Stefan
On Tue, Oct 28, 2008 at 11:36:17PM +0100, Sjoerd Mullender wrote:
On 2008-10-28 22:37, Peter Boncz wrote:
Hi Sjoerd,
The one problem that could occur is people trying to check out
MonetDB_4-24
now, compiling it and trying to use with a precompiled MonetDB_1-24,
without
updating/recompiling that one. So, I agree with you that a problem
could
occur, but I would be surprised if any of our users would really hit
this.
Because we are going to abandon this branch I was willing to let the change pass. But in general I am opposed.
I agree that in this particular case, it is unlikely that people will encounter any problems. But again, that is because we are going to abandon this branch.
With excuses to pick nits, but contrary to you I believe that API
appends
are not "just as bad" as API modifications. An API modification in
MonetDB,
would force a release in MonetDB5, for instance. An API append does
not do
that (so it is "less bad").
It *is* bad. See the disaster involving the addition of True and False in Python in a minor release. They have sworn to never make that mistake again. People expect API stability in minor releases and
adding
something to the API is *not* stability.
PF_ROX is quite different from the HEAD in the pathfinder code due to
bot
branches making different changes to the indexing scheme; afaik this
has
prevented us from keeping it in sync with the HEAD. Various API
changes (not
appends) and even buildtool changes now force it to depend on the
stable
MonetDB/MonetDB4. For PF_ROX itself this is not a problem as it has
no
outside users. However, it might occur that this scenario will repeat (PF_ROX triggering a change/addition in MonetDB/MonetDB4 that would
trickle
towards the head via MonetDB_[14]-24).
I am not talking about the PF_ROX branch. It is not a "stable" branch and you can do with it what you want. But please do consider whether
it
is possible to use the development branches of MonetDB and MonetDB4
with
PF_ROX. If you can do that, you can make needed changes to the API
there.
Peter
-----Original Message----- From: Sjoerd Mullender [mailto:sjoerd@acm.org] Sent: Tuesday, October 28, 2008 9:50 PM To: monetdb-developers@lists.sourceforge.net Cc: monetdb-checkins@lists.sourceforge.net Subject: Re: [Monetdb-checkins] MonetDB4/src/modules/contrib
malalgebra.mx,
MonetDB_4-24, 1.8.2.1, 1.8.2.2
On 2008-10-28 20:20, Peter Boncz wrote:
Hi Sjoerd,
Thanks for keeping an eye on developer behavior and effort to
educate. My
change, however, only introduces new functions in GDK and commands
in MIL;
it
is only an API append.
API appends are just as bad as any other API change. If new code depends on the extra calls, it cannot run with the older version of
the
release. That is bad. It means that the branch is *not* "stable"
anymore.
It is checked into the stable branch because afaik PF_ROX only works
with
that. But I may as often be mistaken.
I noticed it was for PF_ROX, and that does pose a problem. It seems that the PF_ROX branch should depend on the current version of
MonetDB
and not on the stable version.
Peter
This type of change is not allowed on a stable branch. Changes of
API
are *not* allowed.
Since we're going to abandon this stable branch soon, I'll let it
pass.
On 2008-10-28 18:49, Peter Boncz wrote: > Update of /cvsroot/monetdb/MonetDB4/src/modules/contrib > In directory > 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv901/src/modules/contrib > > Modified Files: > Tag: MonetDB_4-24 > malalgebra.mx > Log Message: > changes to minimize sampling overhead (mostly in XML text index) > > - introduce bandmergejoin(), and leftmergejoin()/bandmergejoin()
with a
> cutoff > > > > U malalgebra.mx > Index: malalgebra.mx >
===================================================================
> RCS file:
/cvsroot/monetdb/MonetDB4/src/modules/contrib/malalgebra.mx,v
> retrieving revision 1.8.2.1 > retrieving revision 1.8.2.2 > diff -u -d -r1.8.2.1 -r1.8.2.2 > --- malalgebra.mx 10 Oct 2008 08:42:22 -0000 1.8.2.1 > +++ malalgebra.mx 28 Oct 2008 17:49:34 -0000 1.8.2.2 > @@ -43,7 +43,6 @@ > @:leftjoin(fetch,\nCAUTION: positional matches are assumed not to
be
> out-of-bounds!!)@ > @:leftjoin(merge)@ > @:leftjoin(hash)@ > - > .COMMAND leftthetajoin ( BAT[any::1,any::2] left,
BAT[any::2,any::3]
> right, int mode) : > BAT[any::1,any::3] = CMDleftthetajoin; > "Hook directly into the 'leftthetajoin' implementation of the
join.
> @@ -58,6 +57,22 @@ > Also, for each left tuple, all matching right tuples will
appear in
> their order > of appearrance in the right BAT. This property is handy for
XQuery
> processing." > > + .COMMAND bandmergejoin ( BAT[any::1,any::2] outer,
BAT[any::2,any::3]
> inner, > + any::2 minus, any::2 plus, bit
l_in, bit
> h_in) : > + BAT[any::1,any::3] =
CMDbandmergejoin;
> + "The bandjoin algorithm, but forced to use (left) mergejoin
rather
than
> nested loop" > + > + .COMMAND bandmergejoin ( BAT[any::1,any::2] outer,
BAT[any::2,any::3]
> inner, > + any::2 minus, any::2 plus, bit
l_in, bit
> h_in, lng limit) : > + BAT[lng,BAT] =
CMDbandmergejoin_limit;
> + "A bandjoin that uses the merge algorithm and limits the result
size.
> + Result generation is cut off after this point and the > + result [lng,BAT] holds
[estimated_full_resultsize,limited_result]"
> + > + .COMMAND leftmergejoin ( BAT[any::1,any::2] left,
BAT[any::2,any::3]
> right, lng limit) : > + BAT[lng,BAT] = CMDleftmergejoin_limit; > + "leftmergejoin, but limit the result (by cutting off result
generation)
> + return a BAT[lng,BAT] containing
[full_result_estimate,limited_result]"
> @= select > .COMMAND ord_@1select ( BAT[any::1,any::2] b, any::2 low, any::2
high)
:
> @@ -119,6 +134,7 @@ > #ifndef __MALALGEBRA_H__ > #define __MALALGEBRA_H__ > #include "gdk.h" > +#include "gdk_rangejoin.h" > > /* nothing much */ > > @@ -240,6 +256,40 @@ > return (*result = BATnlthetajoin(l, r, *mode, (size_t) *
estimate)) ?
> GDK_SUCCEED : GDK_FAIL; > } > > +int > +CMDbandmergejoin(BAT **result, BAT *left, BAT *right, ptr minus,
ptr
plus,
> bit *li, bit *hi) > +{ > + return (*result = BATbandmergejoin(left, right, minus,
plus,
*li,
> *hi)) ? GDK_SUCCEED : GDK_FAIL; > +} > + > +int > +CMDbandmergejoin_limit(BAT **result, BAT *left, BAT *right, ptr
minus,
ptr
> plus, bit *li, bit *hi, lng* limit) > +{ > + size_t cutoff = *limit; > + *result = BATbandmergejoin_limit(left, right, minus,
plus, *li,
> *hi, &cutoff); > + if (*result) { > + bat bid = (*result)->batCacheid; > + *result = BATnew(TYPE_lng, TYPE_bat, 1); > + if (*result) BUNins(*result, &cutoff, &bid,
FALSE);
> + BBPunfix(bid); > + } > + return (*result)?GDK_SUCCEED:GDK_FAIL; > +} > + > +int > +CMDleftmergejoin_limit(BAT **result, BAT *left, BAT *right, lng*
limit)
> +{ > + size_t cutoff = *limit; > + *result = BATleftmergejoin_limit(left, right, cutoff,
&cutoff);
> + if (*result) { > + bat bid = (*result)->batCacheid; > + *result = BATnew(TYPE_lng, TYPE_bat, 1); > + if (*result) BUNins(*result, &cutoff, &bid,
FALSE);
> + BBPunfix(bid); > + } > + return (*result)?GDK_SUCCEED:GDK_FAIL; > +} > + > @= selectcmd > int CMDord_@1select1(BAT **result, BAT* b, ptr value) { > return (*result = BAT_select_(b, value, 0, TRUE, TRUE, @2,
FALSE,
> TRUE))?GDK_SUCCEED:GDK_FAIL; > > >
> This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
> Build the coolest Linux based applications with Moblin SDK & win
great
> prizes > Grand prize is a trip for two to an Open Source event anywhere in
the
world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
Sjoerd Mullender
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win
great
prizes
Grand prize is a trip for two to an Open Source event anywhere in
the
world
http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win
great
prizes
Grand prize is a trip for two to an Open Source event anywhere in
the
world
http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
-- Sjoerd Mullender
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win
great
prizes Grand prize is a trip for two to an Open Source event anywhere in the
world
http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win
great prizes
Grand prize is a trip for two to an Open Source event anywhere in the
world
http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
-- Sjoerd Mullender
--
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win great
prizes
Grand prize is a trip for two to an Open Source event anywhere in the
world
http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
Hi Peter,
OK, porting your yesterday's changes to both MonetDB_1-24 and MonetDB_4-24 can still be propagated semi-automatically to the respective development trunks (MonetDB & MonetDB4) --- in fact the will be, anyway, unless we explicitly skip them ...
Changes to the PF_ROX branch (of pathfinder) have never been propagated to the pathfinder HEAD at all.
Just for info:
Porting PF_ROX to MonetDB & MonetDB4 head (or the upcoming MonetDB_1-26 & MonetDB_4-26 release branches?) migth require "a bit more" that "a little bit of effort". Semi-automatic propagation of pathfinder (HEAD) changes to the PF_ROX branch have been abandoned quite some time ago due to too many conflict. To port PF_ROX to the current MonetDB & MonetDB4 HEADs, all changes to the pathfinder HEAD since the XQuery_0-24 branch had been created in May (or June) have to be checked whether they are required to make PF_ROX compatible with the current MonetDB & MonetDB4 HEADs --- mainly all changs related to GDK changes like (internal) type systems (BUNs, hash_t, etc.) and BAT descriptors) --- and applied to the PF_ROX branch, resolving conflicts by hand ... ... unless we propagate all changes from the pathfinder HEAD to its PF_ROX branch (i.e., also those that provide conflichting implementations of similar functionality (e.g. indeces)), any future merge of the PF_ROX changes into the main trunk will also have to be done "by hand".
Alternatively, we could consider forking a new PF_ROX_2 branch of the current pathfinder HEAD (or the upcoming XQuery_0-26 release branch) and apply all changes from the ("old") PF_ROX branch (since its creation) to the new PF_ROX_2 branch --- again resolving all conflicts due to differing implementaions of similar features by hand ...
Stefan
On Wed, Oct 29, 2008 at 10:59:36AM +0100, Peter Boncz wrote:
Hi Stefan,
I engineered the changes to be MonetDB4 and even MonetDB5 friendly, and though PF_ROX is a proof of concept branch, the concepts will at a later stage re-appear in a main branch (certainly those on a GDK level).
However, the changes in MonetDB to support it are minimal; it is not a sacrifice to have them in a HEAD branch.
I actually think we should just make that little bit of effort to port PF_ROX to the MonetDB[4] HEAD, and take it from there. Part of that step is actually the propagation of my branch changes to the HEAD (as I was assuming would happen semi-automatically).
Peter
-----Original Message----- From: Stefan Manegold [mailto:Stefan.Manegold@cwi.nl] Sent: Wednesday, October 29, 2008 10:35 AM To: monetdb-developers@lists.sourceforge.net Cc: P.Boncz@cwi.nl Subject: Re: [Monetdb-checkins] MonetDB4/src/modules/contrib malalgebra.mx, MonetDB_4-24, 1.8.2.1, 1.8.2.2
On Wed, Oct 29, 2008 at 10:27:35AM +0100, Stefan Manegold wrote:
Gents,
given (1) the inherent incompatibility between the PF_ROX branch (forked
of
the XQuery_0-24 branch of pathfinder) and the pathfinder development
trunk
and (hence) the incompatibility between the PF_ROX branch and the development trunks of MonetDB & MonetDB4
and the facts that (2) PF_ROX (at least for now) is purely for internal research purposes and (3) cahnges ("features") in PF_ROX do (e.g.) not support updates, i.e., are far from becoming "main stream",
I guess the (only?) proper solution would (have been?) to also fork
PF_ROX
branches of the MonetDB_1-24 branch (of MonetDB) and the MonetDB_4-24
branch
(of MonetDB4) to accomodate Peter's yesterday's changes.
Note though, that this would / will exclude PF_ROX-specific changes in MonetDB & MonetDB$ from being tested --- this is already the case for the pathfinder PF_ROX branch and will be the case for the MonetDB_1-24 & MonetDB_4-24 branches, too, once we start prparing for the next release
no
later than next week.
Also, changes from the PF_ROX branches of MonetDB & MonetDB4 would/will than NOT be propagated to the respective development trunks; the same will be the case for (potential0 changes to the MonetDB_1-24 & MonetDB_4-24 branches as soon as we start prparing for the next release no later than next week.
In fact, I would even suggest to not propagate Peter's yesterday's changes to the development trunk since they are "PF_ROX-specific" (at least for now).
Stefan
See the results of "Stable" testing (once they are ready later today) to judged whether testing is "desired"/"required" or not ...
Stefan
On Tue, Oct 28, 2008 at 11:36:17PM +0100, Sjoerd Mullender wrote:
On 2008-10-28 22:37, Peter Boncz wrote:
Hi Sjoerd,
The one problem that could occur is people trying to check out
MonetDB_4-24
now, compiling it and trying to use with a precompiled MonetDB_1-24,
without
updating/recompiling that one. So, I agree with you that a problem
could
occur, but I would be surprised if any of our users would really hit
this.
Because we are going to abandon this branch I was willing to let the change pass. But in general I am opposed.
I agree that in this particular case, it is unlikely that people will encounter any problems. But again, that is because we are going to abandon this branch.
With excuses to pick nits, but contrary to you I believe that API
appends
are not "just as bad" as API modifications. An API modification in
MonetDB,
would force a release in MonetDB5, for instance. An API append does
not do
that (so it is "less bad").
It *is* bad. See the disaster involving the addition of True and False in Python in a minor release. They have sworn to never make that mistake again. People expect API stability in minor releases and
adding
something to the API is *not* stability.
PF_ROX is quite different from the HEAD in the pathfinder code due to
bot
branches making different changes to the indexing scheme; afaik this
has
prevented us from keeping it in sync with the HEAD. Various API
changes (not
appends) and even buildtool changes now force it to depend on the
stable
MonetDB/MonetDB4. For PF_ROX itself this is not a problem as it has
no
outside users. However, it might occur that this scenario will repeat (PF_ROX triggering a change/addition in MonetDB/MonetDB4 that would
trickle
towards the head via MonetDB_[14]-24).
I am not talking about the PF_ROX branch. It is not a "stable" branch and you can do with it what you want. But please do consider whether
it
is possible to use the development branches of MonetDB and MonetDB4
with
PF_ROX. If you can do that, you can make needed changes to the API
there.
Peter
-----Original Message----- From: Sjoerd Mullender [mailto:sjoerd@acm.org] Sent: Tuesday, October 28, 2008 9:50 PM To: monetdb-developers@lists.sourceforge.net Cc: monetdb-checkins@lists.sourceforge.net Subject: Re: [Monetdb-checkins] MonetDB4/src/modules/contrib
malalgebra.mx,
MonetDB_4-24, 1.8.2.1, 1.8.2.2
On 2008-10-28 20:20, Peter Boncz wrote:
Hi Sjoerd,
Thanks for keeping an eye on developer behavior and effort to
educate. My
change, however, only introduces new functions in GDK and commands
in MIL;
it
is only an API append.
API appends are just as bad as any other API change. If new code depends on the extra calls, it cannot run with the older version of
the
release. That is bad. It means that the branch is *not* "stable"
anymore.
It is checked into the stable branch because afaik PF_ROX only works
with
that. But I may as often be mistaken.
I noticed it was for PF_ROX, and that does pose a problem. It seems that the PF_ROX branch should depend on the current version of
MonetDB
and not on the stable version.
Peter
> This type of change is not allowed on a stable branch. Changes of
API
> are *not* allowed. > > Since we're going to abandon this stable branch soon, I'll let it
pass.
> > On 2008-10-28 18:49, Peter Boncz wrote: >> Update of /cvsroot/monetdb/MonetDB4/src/modules/contrib >> In directory >> 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv901/src/modules/contrib >> >> Modified Files: >> Tag: MonetDB_4-24 >> malalgebra.mx >> Log Message: >> changes to minimize sampling overhead (mostly in XML text index) >> >> - introduce bandmergejoin(), and leftmergejoin()/bandmergejoin()
with a
>> cutoff >> >> >> >> U malalgebra.mx >> Index: malalgebra.mx >>
===================================================================
>> RCS file:
/cvsroot/monetdb/MonetDB4/src/modules/contrib/malalgebra.mx,v
>> retrieving revision 1.8.2.1 >> retrieving revision 1.8.2.2 >> diff -u -d -r1.8.2.1 -r1.8.2.2 >> --- malalgebra.mx 10 Oct 2008 08:42:22 -0000 1.8.2.1 >> +++ malalgebra.mx 28 Oct 2008 17:49:34 -0000 1.8.2.2 >> @@ -43,7 +43,6 @@ >> @:leftjoin(fetch,\nCAUTION: positional matches are assumed not to
be
>> out-of-bounds!!)@ >> @:leftjoin(merge)@ >> @:leftjoin(hash)@ >> - >> .COMMAND leftthetajoin ( BAT[any::1,any::2] left,
BAT[any::2,any::3]
>> right, int mode) : >> BAT[any::1,any::3] = CMDleftthetajoin; >> "Hook directly into the 'leftthetajoin' implementation of the
join.
>> @@ -58,6 +57,22 @@ >> Also, for each left tuple, all matching right tuples will
appear in
>> their order >> of appearrance in the right BAT. This property is handy for
XQuery
>> processing." >> >> + .COMMAND bandmergejoin ( BAT[any::1,any::2] outer,
BAT[any::2,any::3]
>> inner, >> + any::2 minus, any::2 plus, bit
l_in, bit
>> h_in) : >> + BAT[any::1,any::3] =
CMDbandmergejoin;
>> + "The bandjoin algorithm, but forced to use (left) mergejoin
rather
than
>> nested loop" >> + >> + .COMMAND bandmergejoin ( BAT[any::1,any::2] outer,
BAT[any::2,any::3]
>> inner, >> + any::2 minus, any::2 plus, bit
l_in, bit
>> h_in, lng limit) : >> + BAT[lng,BAT] =
CMDbandmergejoin_limit;
>> + "A bandjoin that uses the merge algorithm and limits the result
size.
>> + Result generation is cut off after this point and the >> + result [lng,BAT] holds
[estimated_full_resultsize,limited_result]"
>> + >> + .COMMAND leftmergejoin ( BAT[any::1,any::2] left,
BAT[any::2,any::3]
>> right, lng limit) : >> + BAT[lng,BAT] = CMDleftmergejoin_limit; >> + "leftmergejoin, but limit the result (by cutting off result
generation)
>> + return a BAT[lng,BAT] containing
[full_result_estimate,limited_result]"
>> @= select >> .COMMAND ord_@1select ( BAT[any::1,any::2] b, any::2 low, any::2
high)
:
>> @@ -119,6 +134,7 @@ >> #ifndef __MALALGEBRA_H__ >> #define __MALALGEBRA_H__ >> #include "gdk.h" >> +#include "gdk_rangejoin.h" >> >> /* nothing much */ >> >> @@ -240,6 +256,40 @@ >> return (*result = BATnlthetajoin(l, r, *mode, (size_t) *
estimate)) ?
>> GDK_SUCCEED : GDK_FAIL; >> } >> >> +int >> +CMDbandmergejoin(BAT **result, BAT *left, BAT *right, ptr minus,
ptr
plus,
>> bit *li, bit *hi) >> +{ >> + return (*result = BATbandmergejoin(left, right, minus,
plus,
*li,
>> *hi)) ? GDK_SUCCEED : GDK_FAIL; >> +} >> + >> +int >> +CMDbandmergejoin_limit(BAT **result, BAT *left, BAT *right, ptr
minus,
ptr
>> plus, bit *li, bit *hi, lng* limit) >> +{ >> + size_t cutoff = *limit; >> + *result = BATbandmergejoin_limit(left, right, minus,
plus, *li,
>> *hi, &cutoff); >> + if (*result) { >> + bat bid = (*result)->batCacheid; >> + *result = BATnew(TYPE_lng, TYPE_bat, 1); >> + if (*result) BUNins(*result, &cutoff, &bid,
FALSE);
>> + BBPunfix(bid); >> + } >> + return (*result)?GDK_SUCCEED:GDK_FAIL; >> +} >> + >> +int >> +CMDleftmergejoin_limit(BAT **result, BAT *left, BAT *right, lng*
limit)
>> +{ >> + size_t cutoff = *limit; >> + *result = BATleftmergejoin_limit(left, right, cutoff,
&cutoff);
>> + if (*result) { >> + bat bid = (*result)->batCacheid; >> + *result = BATnew(TYPE_lng, TYPE_bat, 1); >> + if (*result) BUNins(*result, &cutoff, &bid,
FALSE);
>> + BBPunfix(bid); >> + } >> + return (*result)?GDK_SUCCEED:GDK_FAIL; >> +} >> + >> @= selectcmd >> int CMDord_@1select1(BAT **result, BAT* b, ptr value) { >> return (*result = BAT_select_(b, value, 0, TRUE, TRUE, @2,
FALSE,
>> TRUE))?GDK_SUCCEED:GDK_FAIL; >> >> >>
>> This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
>> Build the coolest Linux based applications with Moblin SDK & win
great
>> prizes >> Grand prize is a trip for two to an Open Source event anywhere in
the
world
>> http://moblin-contest.org/redirect.php?banner_id=100&url=/ >> _______________________________________________ >> Monetdb-checkins mailing list >> Monetdb-checkins@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/monetdb-checkins > -- > Sjoerd Mullender > > -------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
> Build the coolest Linux based applications with Moblin SDK & win
great
prizes
> Grand prize is a trip for two to an Open Source event anywhere in
the
world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins >
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win
great
prizes
Grand prize is a trip for two to an Open Source event anywhere in
the
world
http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
-- Sjoerd Mullender
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win
great
prizes Grand prize is a trip for two to an Open Source event anywhere in the
world
http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win
great prizes
Grand prize is a trip for two to an Open Source event anywhere in the
world
http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
-- Sjoerd Mullender
--
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win great
prizes
Grand prize is a trip for two to an Open Source event anywhere in the
world
http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
Hi Peter et al.
I just did a "quick" check, and propagating the PF_ROX changes from to the pathfinder main trunk (to replace the old PF_ROX by a new PF_ROX_2 branch and hence port PF_ROX to the HEAD versions of MonetDB & MonetDB4) --- i.e., running `cvs up -kk -jXQuery_0-24 -jPF_ROX` in a `cvs up -kk` checkout of the pathfinder trunk --- seems to work "fine" (i.e., without conflicts) for most of the changes in "compiler/" (minor conflicts in compiler/mil/milprint_summer.c, modules/pftijah/serialize_pftijah_options.mx, runtime5/xquery.mx can be resolved easily); in "runtime/", however, I get a total of 43 conflicts in runtime/ll_staircasejoin.mx, runtime/pathfinder.mx, runtime/pf_support.mx, runtime/serialize.mx, runtime/xrpc_client.mx that need to be checked and resolved by hand ...
(maybe another reason to also consider splitting the pathfinder repository in (stand-alone Pathfinder-) "compiler" and (MonetDB/XQuery-) "runtime" ...)
Stefan
On Wed, Oct 29, 2008 at 11:27:29AM +0100, Stefan Manegold wrote:
Hi Peter,
OK, porting your yesterday's changes to both MonetDB_1-24 and MonetDB_4-24 can still be propagated semi-automatically to the respective development trunks (MonetDB & MonetDB4) --- in fact the will be, anyway, unless we explicitly skip them ...
Changes to the PF_ROX branch (of pathfinder) have never been propagated to the pathfinder HEAD at all.
Just for info:
Porting PF_ROX to MonetDB & MonetDB4 head (or the upcoming MonetDB_1-26 & MonetDB_4-26 release branches?) migth require "a bit more" that "a little bit of effort". Semi-automatic propagation of pathfinder (HEAD) changes to the PF_ROX branch have been abandoned quite some time ago due to too many conflict. To port PF_ROX to the current MonetDB & MonetDB4 HEADs, all changes to the pathfinder HEAD since the XQuery_0-24 branch had been created in May (or June) have to be checked whether they are required to make PF_ROX compatible with the current MonetDB & MonetDB4 HEADs --- mainly all changs related to GDK changes like (internal) type systems (BUNs, hash_t, etc.) and BAT descriptors) --- and applied to the PF_ROX branch, resolving conflicts by hand ... ... unless we propagate all changes from the pathfinder HEAD to its PF_ROX branch (i.e., also those that provide conflichting implementations of similar functionality (e.g. indeces)), any future merge of the PF_ROX changes into the main trunk will also have to be done "by hand".
Alternatively, we could consider forking a new PF_ROX_2 branch of the current pathfinder HEAD (or the upcoming XQuery_0-26 release branch) and apply all changes from the ("old") PF_ROX branch (since its creation) to the new PF_ROX_2 branch --- again resolving all conflicts due to differing implementaions of similar features by hand ...
Stefan
On Wed, Oct 29, 2008 at 10:59:36AM +0100, Peter Boncz wrote:
Hi Stefan,
I engineered the changes to be MonetDB4 and even MonetDB5 friendly, and though PF_ROX is a proof of concept branch, the concepts will at a later stage re-appear in a main branch (certainly those on a GDK level).
However, the changes in MonetDB to support it are minimal; it is not a sacrifice to have them in a HEAD branch.
I actually think we should just make that little bit of effort to port PF_ROX to the MonetDB[4] HEAD, and take it from there. Part of that step is actually the propagation of my branch changes to the HEAD (as I was assuming would happen semi-automatically).
Peter
-----Original Message----- From: Stefan Manegold [mailto:Stefan.Manegold@cwi.nl] Sent: Wednesday, October 29, 2008 10:35 AM To: monetdb-developers@lists.sourceforge.net Cc: P.Boncz@cwi.nl Subject: Re: [Monetdb-checkins] MonetDB4/src/modules/contrib malalgebra.mx, MonetDB_4-24, 1.8.2.1, 1.8.2.2
On Wed, Oct 29, 2008 at 10:27:35AM +0100, Stefan Manegold wrote:
Gents,
given (1) the inherent incompatibility between the PF_ROX branch (forked
of
the XQuery_0-24 branch of pathfinder) and the pathfinder development
trunk
and (hence) the incompatibility between the PF_ROX branch and the development trunks of MonetDB & MonetDB4
and the facts that (2) PF_ROX (at least for now) is purely for internal research purposes and (3) cahnges ("features") in PF_ROX do (e.g.) not support updates, i.e., are far from becoming "main stream",
I guess the (only?) proper solution would (have been?) to also fork
PF_ROX
branches of the MonetDB_1-24 branch (of MonetDB) and the MonetDB_4-24
branch
(of MonetDB4) to accomodate Peter's yesterday's changes.
Note though, that this would / will exclude PF_ROX-specific changes in MonetDB & MonetDB$ from being tested --- this is already the case for the pathfinder PF_ROX branch and will be the case for the MonetDB_1-24 & MonetDB_4-24 branches, too, once we start prparing for the next release
no
later than next week.
Also, changes from the PF_ROX branches of MonetDB & MonetDB4 would/will than NOT be propagated to the respective development trunks; the same will be the case for (potential0 changes to the MonetDB_1-24 & MonetDB_4-24 branches as soon as we start prparing for the next release no later than next week.
In fact, I would even suggest to not propagate Peter's yesterday's changes to the development trunk since they are "PF_ROX-specific" (at least for now).
Stefan
See the results of "Stable" testing (once they are ready later today) to judged whether testing is "desired"/"required" or not ...
Stefan
On Tue, Oct 28, 2008 at 11:36:17PM +0100, Sjoerd Mullender wrote:
On 2008-10-28 22:37, Peter Boncz wrote:
Hi Sjoerd,
The one problem that could occur is people trying to check out
MonetDB_4-24
now, compiling it and trying to use with a precompiled MonetDB_1-24,
without
updating/recompiling that one. So, I agree with you that a problem
could
occur, but I would be surprised if any of our users would really hit
this.
Because we are going to abandon this branch I was willing to let the change pass. But in general I am opposed.
I agree that in this particular case, it is unlikely that people will encounter any problems. But again, that is because we are going to abandon this branch.
With excuses to pick nits, but contrary to you I believe that API
appends
are not "just as bad" as API modifications. An API modification in
MonetDB,
would force a release in MonetDB5, for instance. An API append does
not do
that (so it is "less bad").
It *is* bad. See the disaster involving the addition of True and False in Python in a minor release. They have sworn to never make that mistake again. People expect API stability in minor releases and
adding
something to the API is *not* stability.
PF_ROX is quite different from the HEAD in the pathfinder code due to
bot
branches making different changes to the indexing scheme; afaik this
has
prevented us from keeping it in sync with the HEAD. Various API
changes (not
appends) and even buildtool changes now force it to depend on the
stable
MonetDB/MonetDB4. For PF_ROX itself this is not a problem as it has
no
outside users. However, it might occur that this scenario will repeat (PF_ROX triggering a change/addition in MonetDB/MonetDB4 that would
trickle
towards the head via MonetDB_[14]-24).
I am not talking about the PF_ROX branch. It is not a "stable" branch and you can do with it what you want. But please do consider whether
it
is possible to use the development branches of MonetDB and MonetDB4
with
PF_ROX. If you can do that, you can make needed changes to the API
there.
Peter
-----Original Message----- From: Sjoerd Mullender [mailto:sjoerd@acm.org] Sent: Tuesday, October 28, 2008 9:50 PM To: monetdb-developers@lists.sourceforge.net Cc: monetdb-checkins@lists.sourceforge.net Subject: Re: [Monetdb-checkins] MonetDB4/src/modules/contrib
malalgebra.mx,
MonetDB_4-24, 1.8.2.1, 1.8.2.2
On 2008-10-28 20:20, Peter Boncz wrote: > Hi Sjoerd, > > Thanks for keeping an eye on developer behavior and effort to
educate. My
> change, however, only introduces new functions in GDK and commands
in MIL;
it > is only an API append.
API appends are just as bad as any other API change. If new code depends on the extra calls, it cannot run with the older version of
the
release. That is bad. It means that the branch is *not* "stable"
anymore.
> It is checked into the stable branch because afaik PF_ROX only works
with
> that. But I may as often be mistaken.
I noticed it was for PF_ROX, and that does pose a problem. It seems that the PF_ROX branch should depend on the current version of
MonetDB
and not on the stable version.
> Peter > >> This type of change is not allowed on a stable branch. Changes of
API
>> are *not* allowed. >> >> Since we're going to abandon this stable branch soon, I'll let it
pass.
>> >> On 2008-10-28 18:49, Peter Boncz wrote: >>> Update of /cvsroot/monetdb/MonetDB4/src/modules/contrib >>> In directory >>> 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv901/src/modules/contrib >>> >>> Modified Files: >>> Tag: MonetDB_4-24 >>> malalgebra.mx >>> Log Message: >>> changes to minimize sampling overhead (mostly in XML text index) >>> >>> - introduce bandmergejoin(), and leftmergejoin()/bandmergejoin()
with a
>>> cutoff >>> >>> >>> >>> U malalgebra.mx >>> Index: malalgebra.mx >>>
===================================================================
>>> RCS file:
/cvsroot/monetdb/MonetDB4/src/modules/contrib/malalgebra.mx,v
>>> retrieving revision 1.8.2.1 >>> retrieving revision 1.8.2.2 >>> diff -u -d -r1.8.2.1 -r1.8.2.2 >>> --- malalgebra.mx 10 Oct 2008 08:42:22 -0000 1.8.2.1 >>> +++ malalgebra.mx 28 Oct 2008 17:49:34 -0000 1.8.2.2 >>> @@ -43,7 +43,6 @@ >>> @:leftjoin(fetch,\nCAUTION: positional matches are assumed not to
be
>>> out-of-bounds!!)@ >>> @:leftjoin(merge)@ >>> @:leftjoin(hash)@ >>> - >>> .COMMAND leftthetajoin ( BAT[any::1,any::2] left,
BAT[any::2,any::3]
>>> right, int mode) : >>> BAT[any::1,any::3] = CMDleftthetajoin; >>> "Hook directly into the 'leftthetajoin' implementation of the
join.
>>> @@ -58,6 +57,22 @@ >>> Also, for each left tuple, all matching right tuples will
appear in
>>> their order >>> of appearrance in the right BAT. This property is handy for
XQuery
>>> processing." >>> >>> + .COMMAND bandmergejoin ( BAT[any::1,any::2] outer,
BAT[any::2,any::3]
>>> inner, >>> + any::2 minus, any::2 plus, bit
l_in, bit
>>> h_in) : >>> + BAT[any::1,any::3] =
CMDbandmergejoin;
>>> + "The bandjoin algorithm, but forced to use (left) mergejoin
rather
than >>> nested loop" >>> + >>> + .COMMAND bandmergejoin ( BAT[any::1,any::2] outer,
BAT[any::2,any::3]
>>> inner, >>> + any::2 minus, any::2 plus, bit
l_in, bit
>>> h_in, lng limit) : >>> + BAT[lng,BAT] =
CMDbandmergejoin_limit;
>>> + "A bandjoin that uses the merge algorithm and limits the result
size.
>>> + Result generation is cut off after this point and the >>> + result [lng,BAT] holds
[estimated_full_resultsize,limited_result]"
>>> + >>> + .COMMAND leftmergejoin ( BAT[any::1,any::2] left,
BAT[any::2,any::3]
>>> right, lng limit) : >>> + BAT[lng,BAT] = CMDleftmergejoin_limit; >>> + "leftmergejoin, but limit the result (by cutting off result generation) >>> + return a BAT[lng,BAT] containing [full_result_estimate,limited_result]" >>> @= select >>> .COMMAND ord_@1select ( BAT[any::1,any::2] b, any::2 low, any::2
high)
: >>> @@ -119,6 +134,7 @@ >>> #ifndef __MALALGEBRA_H__ >>> #define __MALALGEBRA_H__ >>> #include "gdk.h" >>> +#include "gdk_rangejoin.h" >>> >>> /* nothing much */ >>> >>> @@ -240,6 +256,40 @@ >>> return (*result = BATnlthetajoin(l, r, *mode, (size_t) *
estimate)) ?
>>> GDK_SUCCEED : GDK_FAIL; >>> } >>> >>> +int >>> +CMDbandmergejoin(BAT **result, BAT *left, BAT *right, ptr minus,
ptr
plus, >>> bit *li, bit *hi) >>> +{ >>> + return (*result = BATbandmergejoin(left, right, minus,
plus,
*li, >>> *hi)) ? GDK_SUCCEED : GDK_FAIL; >>> +} >>> + >>> +int >>> +CMDbandmergejoin_limit(BAT **result, BAT *left, BAT *right, ptr
minus,
ptr >>> plus, bit *li, bit *hi, lng* limit) >>> +{ >>> + size_t cutoff = *limit; >>> + *result = BATbandmergejoin_limit(left, right, minus,
plus, *li,
>>> *hi, &cutoff); >>> + if (*result) { >>> + bat bid = (*result)->batCacheid; >>> + *result = BATnew(TYPE_lng, TYPE_bat, 1); >>> + if (*result) BUNins(*result, &cutoff, &bid,
FALSE);
>>> + BBPunfix(bid); >>> + } >>> + return (*result)?GDK_SUCCEED:GDK_FAIL; >>> +} >>> + >>> +int >>> +CMDleftmergejoin_limit(BAT **result, BAT *left, BAT *right, lng*
limit)
>>> +{ >>> + size_t cutoff = *limit; >>> + *result = BATleftmergejoin_limit(left, right, cutoff,
&cutoff);
>>> + if (*result) { >>> + bat bid = (*result)->batCacheid; >>> + *result = BATnew(TYPE_lng, TYPE_bat, 1); >>> + if (*result) BUNins(*result, &cutoff, &bid,
FALSE);
>>> + BBPunfix(bid); >>> + } >>> + return (*result)?GDK_SUCCEED:GDK_FAIL; >>> +} >>> + >>> @= selectcmd >>> int CMDord_@1select1(BAT **result, BAT* b, ptr value) { >>> return (*result = BAT_select_(b, value, 0, TRUE, TRUE, @2,
FALSE,
>>> TRUE))?GDK_SUCCEED:GDK_FAIL; >>> >>>
>>>
>>> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge >>> Build the coolest Linux based applications with Moblin SDK & win
great
>>> prizes >>> Grand prize is a trip for two to an Open Source event anywhere in
the
world >>> http://moblin-contest.org/redirect.php?banner_id=100&url=/ >>> _______________________________________________ >>> Monetdb-checkins mailing list >>> Monetdb-checkins@lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/monetdb-checkins >> -- >> Sjoerd Mullender >> >> -------------------------------------------------------------------
>> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge >> Build the coolest Linux based applications with Moblin SDK & win
great
prizes >> Grand prize is a trip for two to an Open Source event anywhere in
the
world >> http://moblin-contest.org/redirect.php?banner_id=100&url=/ >> _______________________________________________ >> Monetdb-checkins mailing list >> Monetdb-checkins@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/monetdb-checkins >> > > --------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win
great
prizes > Grand prize is a trip for two to an Open Source event anywhere in
the
world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
-- Sjoerd Mullender
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win
great
prizes Grand prize is a trip for two to an Open Source event anywhere in the
world
http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win
great prizes
Grand prize is a trip for two to an Open Source event anywhere in the
world
http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
-- Sjoerd Mullender
--
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win great
prizes
Grand prize is a trip for two to an Open Source event anywhere in the
world
http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
Hi Stefan,
Thanks for the info. In my opinion, what needs to happen from a ROX point of view is re-applying the cut-off staircase joins that you made towards the HEAD only.
My analysis:
- for the short term (SIGMOD deadline), the ROX team just continues on PF_ROX and Monet*_24.
- for the mid term (post-SIGMOD, VLDB), we should port api.mil to the HEAD, which comes down to porting the staircase joins with cut-offs, and abandoning the old (modified) value indexing scheme in PF_ROX for the new scheme implemented by Lefteris in the HEAD. Apparently these cut-offs in staircase joins cause some conflicts now. It is unexpected, but these conflicts cannot be dramatic.
- for the longer term (XQ opt in M5) we will re-use the staircase joins with cut-offs (which by then are in the HEAD). We might consider the introduction of a parent NID in the document representation (as in PF_ROX; we should evaluate the trade-offs of this feature first). This latter one is thus related to the (re-) design of the XML document representation in M5.
the other changes/conflicts can simply be abandoned.
Peter
-----Original Message----- From: Stefan Manegold [mailto:Stefan.Manegold@cwi.nl] Sent: Wednesday, October 29, 2008 1:54 PM To: monetdb-developers@lists.sourceforge.net Cc: Peter Boncz Subject: Re: [Monetdb-developers] [Monetdb-checkins] MonetDB4/src/modules/contrib malalgebra.mx, MonetDB_4-24, 1.8.2.1, 1.8.2.2
Hi Peter et al.
I just did a "quick" check, and propagating the PF_ROX changes from to the pathfinder main trunk (to replace the old PF_ROX by a new PF_ROX_2 branch and hence port PF_ROX to the HEAD versions of MonetDB & MonetDB4) --- i.e., running `cvs up -kk -jXQuery_0-24 -jPF_ROX` in a `cvs up -kk` checkout of the pathfinder trunk --- seems to work "fine" (i.e., without conflicts) for most of the changes in "compiler/" (minor conflicts in compiler/mil/milprint_summer.c, modules/pftijah/serialize_pftijah_options.mx, runtime5/xquery.mx can be resolved easily); in "runtime/", however, I get a total of 43 conflicts in runtime/ll_staircasejoin.mx, runtime/pathfinder.mx, runtime/pf_support.mx, runtime/serialize.mx, runtime/xrpc_client.mx that need to be checked and resolved by hand ...
(maybe another reason to also consider splitting the pathfinder repository in (stand-alone Pathfinder-) "compiler" and (MonetDB/XQuery-) "runtime" ...)
Stefan
On Wed, Oct 29, 2008 at 11:27:29AM +0100, Stefan Manegold wrote:
Hi Peter,
OK, porting your yesterday's changes to both MonetDB_1-24 and MonetDB_4-24 can still be propagated semi-automatically to the respective development trunks (MonetDB & MonetDB4) --- in fact the will be, anyway, unless we explicitly skip them ...
Changes to the PF_ROX branch (of pathfinder) have never been propagated to the pathfinder HEAD at all.
Just for info:
Porting PF_ROX to MonetDB & MonetDB4 head (or the upcoming MonetDB_1-26 & MonetDB_4-26 release branches?) migth require "a bit more" that "a little bit of effort". Semi-automatic propagation of pathfinder (HEAD) changes to the PF_ROX branch have been abandoned quite some time ago due to too many conflict. To port PF_ROX to the current MonetDB & MonetDB4 HEADs, all changes to the pathfinder HEAD since the XQuery_0-24 branch had been
created
in May (or June) have to be checked whether they are required to make
PF_ROX
compatible with the current MonetDB & MonetDB4 HEADs --- mainly all changs related to GDK changes like (internal) type systems (BUNs, hash_t, etc.)
and
BAT descriptors) --- and applied to the PF_ROX branch, resolving conflicts by hand ... ... unless we propagate all changes from the pathfinder HEAD to its PF_ROX branch (i.e., also those that provide conflichting implementations of similar functionality (e.g. indeces)), any future merge of the PF_ROX changes into the main trunk will also have to be done "by hand".
Alternatively, we could consider forking a new PF_ROX_2 branch of the current pathfinder HEAD (or the upcoming XQuery_0-26 release branch) and apply all changes from the ("old") PF_ROX branch (since its creation) to
the
new PF_ROX_2 branch --- again resolving all conflicts due to differing implementaions of similar features by hand ...
Stefan
On Wed, Oct 29, 2008 at 10:59:36AM +0100, Peter Boncz wrote:
Hi Stefan,
I engineered the changes to be MonetDB4 and even MonetDB5 friendly, and
though
PF_ROX is a proof of concept branch, the concepts will at a later stage re-appear in a main branch (certainly those on a GDK level).
However, the changes in MonetDB to support it are minimal; it is not a
sacrifice
to have them in a HEAD branch.
I actually think we should just make that little bit of effort to port
PF_ROX to
the MonetDB[4] HEAD, and take it from there. Part of that step is
actually the
propagation of my branch changes to the HEAD (as I was assuming would
happen
semi-automatically).
Peter
-----Original Message----- From: Stefan Manegold [mailto:Stefan.Manegold@cwi.nl] Sent: Wednesday, October 29, 2008 10:35 AM To: monetdb-developers@lists.sourceforge.net Cc: P.Boncz@cwi.nl Subject: Re: [Monetdb-checkins] MonetDB4/src/modules/contrib
malalgebra.mx,
MonetDB_4-24, 1.8.2.1, 1.8.2.2
On Wed, Oct 29, 2008 at 10:27:35AM +0100, Stefan Manegold wrote:
Gents,
given (1) the inherent incompatibility between the PF_ROX branch
(forked
of
the XQuery_0-24 branch of pathfinder) and the pathfinder development
trunk
and (hence) the incompatibility between the PF_ROX branch and the development trunks of MonetDB & MonetDB4
and the facts that (2) PF_ROX (at least for now) is purely for
internal
research purposes and (3) cahnges ("features") in PF_ROX do (e.g.)
not
support updates, i.e., are far from becoming "main stream",
I guess the (only?) proper solution would (have been?) to also fork
PF_ROX
branches of the MonetDB_1-24 branch (of MonetDB) and the
MonetDB_4-24
branch
(of MonetDB4) to accomodate Peter's yesterday's changes.
Note though, that this would / will exclude PF_ROX-specific changes
in
MonetDB & MonetDB$ from being tested --- this is already the case
for the
pathfinder PF_ROX branch and will be the case for the MonetDB_1-24 & MonetDB_4-24 branches, too, once we start prparing for the next
release
no
later than next week.
Also, changes from the PF_ROX branches of MonetDB & MonetDB4
would/will
than NOT be propagated to the respective development trunks; the same will
be
the case for (potential0 changes to the MonetDB_1-24 & MonetDB_4-24
branches
as soon as we start prparing for the next release no later than next
week.
In fact, I would even suggest to not propagate Peter's yesterday's
changes
to the development trunk since they are "PF_ROX-specific" (at least
for
now).
Stefan
See the results of "Stable" testing (once they are ready later
today) to
judged whether testing is "desired"/"required" or not ...
Stefan
On Tue, Oct 28, 2008 at 11:36:17PM +0100, Sjoerd Mullender wrote:
On 2008-10-28 22:37, Peter Boncz wrote:
Hi Sjoerd,
The one problem that could occur is people trying to check out
MonetDB_4-24
now, compiling it and trying to use with a precompiled
MonetDB_1-24,
without
updating/recompiling that one. So, I agree with you that a
problem
could
occur, but I would be surprised if any of our users would really
hit
this.
Because we are going to abandon this branch I was willing to let
the
change pass. But in general I am opposed.
I agree that in this particular case, it is unlikely that people
will
encounter any problems. But again, that is because we are going
to
abandon this branch.
With excuses to pick nits, but contrary to you I believe that
API
appends
are not "just as bad" as API modifications. An API modification
in
MonetDB,
would force a release in MonetDB5, for instance. An API append
does
not do
that (so it is "less bad").
It *is* bad. See the disaster involving the addition of True and
False
in Python in a minor release. They have sworn to never make that mistake again. People expect API stability in minor releases and
adding
something to the API is *not* stability.
PF_ROX is quite different from the HEAD in the pathfinder code
due to
bot
branches making different changes to the indexing scheme; afaik
this
has
prevented us from keeping it in sync with the HEAD. Various API
changes (not
appends) and even buildtool changes now force it to depend on
the
stable
MonetDB/MonetDB4. For PF_ROX itself this is not a problem as it
has
no
outside users. However, it might occur that this scenario will
repeat
(PF_ROX triggering a change/addition in MonetDB/MonetDB4 that
would
trickle
towards the head via MonetDB_[14]-24).
I am not talking about the PF_ROX branch. It is not a "stable"
branch
and you can do with it what you want. But please do consider
whether
it
is possible to use the development branches of MonetDB and
MonetDB4
with
PF_ROX. If you can do that, you can make needed changes to the
API
there.
Peter
-----Original Message----- From: Sjoerd Mullender [mailto:sjoerd@acm.org] Sent: Tuesday, October 28, 2008 9:50 PM To: monetdb-developers@lists.sourceforge.net Cc: monetdb-checkins@lists.sourceforge.net Subject: Re: [Monetdb-checkins] MonetDB4/src/modules/contrib
malalgebra.mx,
MonetDB_4-24, 1.8.2.1, 1.8.2.2
On 2008-10-28 20:20, Peter Boncz wrote: > Hi Sjoerd, > > Thanks for keeping an eye on developer behavior and effort to
educate. My
> change, however, only introduces new functions in GDK and
commands
in MIL;
it > is only an API append.
API appends are just as bad as any other API change. If new
code
depends on the extra calls, it cannot run with the older version
of
the
release. That is bad. It means that the branch is *not*
"stable"
anymore.
> It is checked into the stable branch because afaik PF_ROX only
works
with
> that. But I may as often be mistaken.
I noticed it was for PF_ROX, and that does pose a problem. It
seems
that the PF_ROX branch should depend on the current version of
MonetDB
and not on the stable version.
> Peter > >> This type of change is not allowed on a stable branch.
Changes of
API
>> are *not* allowed. >> >> Since we're going to abandon this stable branch soon, I'll let
it
pass.
>> >> On 2008-10-28 18:49, Peter Boncz wrote: >>> Update of /cvsroot/monetdb/MonetDB4/src/modules/contrib >>> In directory >>>
23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv901/src/modules/contrib
>>> >>> Modified Files: >>> Tag: MonetDB_4-24 >>> malalgebra.mx >>> Log Message: >>> changes to minimize sampling overhead (mostly in XML text
index)
>>> >>> - introduce bandmergejoin(), and
leftmergejoin()/bandmergejoin()
with a
>>> cutoff >>> >>> >>> >>> U malalgebra.mx >>> Index: malalgebra.mx >>>
===================================================================
>>> RCS file:
/cvsroot/monetdb/MonetDB4/src/modules/contrib/malalgebra.mx,v
>>> retrieving revision 1.8.2.1 >>> retrieving revision 1.8.2.2 >>> diff -u -d -r1.8.2.1 -r1.8.2.2 >>> --- malalgebra.mx 10 Oct 2008 08:42:22 -0000 1.8.2.1 >>> +++ malalgebra.mx 28 Oct 2008 17:49:34 -0000 1.8.2.2 >>> @@ -43,7 +43,6 @@ >>> @:leftjoin(fetch,\nCAUTION: positional matches are assumed
not to
be
>>> out-of-bounds!!)@ >>> @:leftjoin(merge)@ >>> @:leftjoin(hash)@ >>> - >>> .COMMAND leftthetajoin ( BAT[any::1,any::2] left,
BAT[any::2,any::3]
>>> right, int mode) : >>> BAT[any::1,any::3] = CMDleftthetajoin; >>> "Hook directly into the 'leftthetajoin' implementation of
the
join.
>>> @@ -58,6 +57,22 @@ >>> Also, for each left tuple, all matching right tuples will
appear in
>>> their order >>> of appearrance in the right BAT. This property is handy
for
XQuery
>>> processing." >>> >>> + .COMMAND bandmergejoin ( BAT[any::1,any::2] outer,
BAT[any::2,any::3]
>>> inner, >>> + any::2 minus, any::2 plus,
bit
l_in, bit
>>> h_in) : >>> + BAT[any::1,any::3] =
CMDbandmergejoin;
>>> + "The bandjoin algorithm, but forced to use (left)
mergejoin
rather
than >>> nested loop" >>> + >>> + .COMMAND bandmergejoin ( BAT[any::1,any::2] outer,
BAT[any::2,any::3]
>>> inner, >>> + any::2 minus, any::2 plus,
bit
l_in, bit
>>> h_in, lng limit) : >>> + BAT[lng,BAT] =
CMDbandmergejoin_limit;
>>> + "A bandjoin that uses the merge algorithm and limits the
result
size.
>>> + Result generation is cut off after this point and the >>> + result [lng,BAT] holds
[estimated_full_resultsize,limited_result]"
>>> + >>> + .COMMAND leftmergejoin ( BAT[any::1,any::2] left,
BAT[any::2,any::3]
>>> right, lng limit) : >>> + BAT[lng,BAT] = CMDleftmergejoin_limit; >>> + "leftmergejoin, but limit the result (by cutting off
result
generation) >>> + return a BAT[lng,BAT] containing [full_result_estimate,limited_result]" >>> @= select >>> .COMMAND ord_@1select ( BAT[any::1,any::2] b, any::2 low,
any::2
high)
: >>> @@ -119,6 +134,7 @@ >>> #ifndef __MALALGEBRA_H__ >>> #define __MALALGEBRA_H__ >>> #include "gdk.h" >>> +#include "gdk_rangejoin.h" >>> >>> /* nothing much */ >>> >>> @@ -240,6 +256,40 @@ >>> return (*result = BATnlthetajoin(l, r, *mode, (size_t) *
estimate)) ?
>>> GDK_SUCCEED : GDK_FAIL; >>> } >>> >>> +int >>> +CMDbandmergejoin(BAT **result, BAT *left, BAT *right, ptr
minus,
ptr
plus, >>> bit *li, bit *hi) >>> +{ >>> + return (*result = BATbandmergejoin(left, right,
minus,
plus,
*li, >>> *hi)) ? GDK_SUCCEED : GDK_FAIL; >>> +} >>> + >>> +int >>> +CMDbandmergejoin_limit(BAT **result, BAT *left, BAT *right,
ptr
minus,
ptr >>> plus, bit *li, bit *hi, lng* limit) >>> +{ >>> + size_t cutoff = *limit; >>> + *result = BATbandmergejoin_limit(left, right, minus,
plus, *li,
>>> *hi, &cutoff); >>> + if (*result) { >>> + bat bid = (*result)->batCacheid; >>> + *result = BATnew(TYPE_lng, TYPE_bat, 1); >>> + if (*result) BUNins(*result, &cutoff, &bid,
FALSE);
>>> + BBPunfix(bid); >>> + } >>> + return (*result)?GDK_SUCCEED:GDK_FAIL; >>> +} >>> + >>> +int >>> +CMDleftmergejoin_limit(BAT **result, BAT *left, BAT *right,
lng*
limit)
>>> +{ >>> + size_t cutoff = *limit; >>> + *result = BATleftmergejoin_limit(left, right,
cutoff,
&cutoff);
>>> + if (*result) { >>> + bat bid = (*result)->batCacheid; >>> + *result = BATnew(TYPE_lng, TYPE_bat, 1); >>> + if (*result) BUNins(*result, &cutoff, &bid,
FALSE);
>>> + BBPunfix(bid); >>> + } >>> + return (*result)?GDK_SUCCEED:GDK_FAIL; >>> +} >>> + >>> @= selectcmd >>> int CMDord_@1select1(BAT **result, BAT* b, ptr value) { >>> return (*result = BAT_select_(b, value, 0, TRUE, TRUE, @2,
FALSE,
>>> TRUE))?GDK_SUCCEED:GDK_FAIL; >>> >>> >>>
---------------------------------------------------------------------
>>> This SF.Net email is sponsored by the Moblin Your Move
Developer's
challenge >>> Build the coolest Linux based applications with Moblin SDK &
win
great
>>> prizes >>> Grand prize is a trip for two to an Open Source event
anywhere in
the
world >>> http://moblin-contest.org/redirect.php?banner_id=100&url=/ >>> _______________________________________________ >>> Monetdb-checkins mailing list >>> Monetdb-checkins@lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/monetdb-checkins >> -- >> Sjoerd Mullender >> >>
-------------------------------------------------------------------
>> This SF.Net email is sponsored by the Moblin Your Move
Developer's
challenge >> Build the coolest Linux based applications with Moblin SDK &
win
great
prizes >> Grand prize is a trip for two to an Open Source event anywhere
in
the
world >> http://moblin-contest.org/redirect.php?banner_id=100&url=/ >> _______________________________________________ >> Monetdb-checkins mailing list >> Monetdb-checkins@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/monetdb-checkins >> > >
--------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move
Developer's
challenge > Build the coolest Linux based applications with Moblin SDK &
win
great
prizes > Grand prize is a trip for two to an Open Source event anywhere
in
the
world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
-- Sjoerd Mullender
---------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move
Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win
great
prizes Grand prize is a trip for two to an Open Source event anywhere
in the
world
http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
---------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move
Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win
great prizes
Grand prize is a trip for two to an Open Source event anywhere
in the
world
http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
-- Sjoerd Mullender
-----------------------------------------------------------------------
--
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win
great
prizes
Grand prize is a trip for two to an Open Source event anywhere in
the
world
http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win great
prizes
Grand prize is a trip for two to an Open Source event anywhere in the
world
http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win great
prizes
Grand prize is a trip for two to an Open Source event anywhere in the
world
http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
No virus found in this incoming message. Checked by AVG - http://www.avg.com Version: 8.0.175 / Virus Database: 270.8.4/1752 - Release Date: 10/28/2008 10:04 AM
participants (4)
-
Martin Kersten -
Peter Boncz -
Sjoerd Mullender -
Stefan Manegold