[Monetdb-developers] next release: mps or algebra?
Stefan suggested rightly to move this discussion to the mailing list. I also changed the language into english for that purpose.
Maurice' original questions were:
(a) whether the new release would be algebra-only (short answer: hopefully).
(b) whether our talk of a new release also refers to submodules, such as probxml and pftijah (short answer: yes)
(c) whether all issues on the wiki, see: http://www.pathfinder-xquery.org/wiki/index.php/Jupiter:_The_Algebra_Release needed for this action, have already been resolved (short answer: no)
On Tue, Oct 16, 2007 at 05:23:43PM +0200, Peter Boncz wrote:
Hi Maurice, et al,
The release of (hopefully) this week will still be mps based. I also anticipate at least one bug-fix release (ie 0.20.2), or even more. However, the next major release (0.22) should be algebra based, in my opinion. I am certainly reluctant to keep investing any more effort in mps. I would aim for an mps-free release, yes.
The time-goal of that release is "XMAS 2007", as far as we are concerned. For the work of built-in functions I have good hope that with the help of Lefteris we can come a long way.
I hope and think updates can also be ported by joerd in that time-frame. As for pftijah, if Jan Flokstra has time, it should be possible to port that as well in 2-3 months.
There are some hairy issues, one of them being free-form recursion (needed by you), another one being precompiled modules (needed by XRPC). The resolution of these needs assistance from Munich, and maybe related. It needs a method to compile certain functions into scripted procedures (here MIL) and calling them. One algorithm is to do this for all functions in modules, as well as for all recursive functions. The issue to resolve is the representation of the parameters.
Depending on how much time Jan Rittinger has available for this, we will have to find creative solutions for this. Indeed, a backup solution would be to keep handling certain queries for the time being with mps, but that surely is ugly.
Peter
-----Original Message----- From: Keulen, M. van (Maurice) [mailto:m.vankeulen@utwente.nl] Sent: dinsdag 16 oktober 2007 16:38 To: Peter Boncz [CWI]; Stefan Manegold [CWI] Cc: Djoerd Hiemstra; Jan Flokstra; Henning Rode; Abdel Kader, R. (Riham) Subject: Nieuwe releases en mps-backend
Hoi,
Sorry dat ik niet bij de Skype-sessie van vandaag kon zijn, maar Riham vertelde me dat het onderwerp van nieuwe releases en de mps-backend ter sprake is geweest. Het riep wat specifieke vragen bij me op die ze niet allemaal kon beantwoorden:
a.. Is het de bedoeling dat een nieuwe release de mps-backend helemaalniet meer bevat of alleen maar de algebra-backend als default heeft? b.. Als jullie praten over "nieuwe release", bedoel je dan alleen een package van alle modules van MonetDB/XQuery? Maw zegt het niets over eventuele `subreleases' van individuele modules?
c.. Op de wiki hebben we een pagina met open issues voor dealgebra-release. Wat is de status van die open issues? Ik zie namelijk op de pagina nergens dat bepaalde issues al opgelost/geïmplementeerd zijn. Ik Cc deze e-mail even aan de PF/Tijah-mensen, want hoewel er levendige discussies zijn hoe PF/Tijah naar de algebra-backend to porten, ik heb niet het idee dat er oplossingen zijn die op korte termijn realiseerbaar zijn. Evenzo het recursie-vraagstuk. Het lijkt me bijvoorbeeld voor de zichtbaarheid naar buiten toe onwenselijk als deze features ineens zouden wegvallen voor non-insiders. Of andere onwenselijke consequenties van het niet meer (sub)releasen van modules die gebaseerd zijn op de mps-backend.
Groetjes, Maurice.
--
Dr.Ir. M. van Keulen - Assistant Professor, Data Management Technology Univ. of Twente, Dept of EEMCS, POBox 217, 7500 AE Enschede, Netherlands Email: m.vankeulen@utwente.nl, Phone: +31 534893688, Fax: +31 534892927 Room: ZI 3039, WWW: http://www.cs.utwente.nl/~keulen
LS,
Good to see this discussion. Although the topic is primarily MonetDB/XQuery it is good to keep an eye on the global picture. Releases include more stuff and requires a lot of time to prepare. (the release itself easily consumes several months of full time work, many delivered by free (weekend) time!!!!) Good planning and long-term internal use are a prerequisite. Something we still have to learn. Once the release cycle starts, the train can not stop and full power is/should be in the hands of a single person. The release manager currently is Sjoerd, one of the project admins!
As soon as a RC is available, those with a non-standard platform (windows, sun, mac, vista) should immediately double check to assure it basically works. This holds for all addressed and developers.
In this respect, the time frame mentioned below is overly optimistic. Anyone close to the core group should work on the HEAD to help uncover problems from day 1. Code changes should *not* be collected by individual programmers until release time . This is a major source of frustration, because you simply may not expect your team members to spent day and night to fight the 'last' bugs. Let alone to quick-test your latest feature or performance improvement. PLEASE, check-in daily your changes and make sure you have at least ran mtest on your platform. And most of all, know that your changes are scrutinized for potential quality problems by others. It is the best way to avoid many bugs.
Furthermore, changes in the interfaces visible to users and the effect on their databases should be kept to a minimum. Backward compatibility is a must as long as it is manageable. Beware that a change in storage structure may help a few extreme users, but may affects hundreds or thousands in their daily use. E.g. one of our users would have to unload/reload a 0.5Tb business critical application.
MonetDB/Query and MonetDB/SQL are intertwined. This creates responsibility both ways.
Peter Boncz wrote:
Stefan suggested rightly to move this discussion to the mailing list. I also changed the language into english for that purpose.
Maurice' original questions were:
(a) whether the new release would be algebra-only (short answer: hopefully).
In my opinion this can only be answered if the algebra version, in its full glory (without mps) is used for several months internally, e.g. using students for stress testing.
(b) whether our talk of a new release also refers to submodules, such as probxml and pftijah (short answer: yes)
Submodules basically follow their own release cycle. Regardless the source, e.g. datacells geom, .... Keeping in line is the responsibility of their owners, and the core developers should be aware that their changes may break the work of others. Or at least drain a possibly substantial amount of time.
(c) whether all issues on the wiki, see: http://www.pathfinder-xquery.org/wiki/index.php/Jupiter:_The_Algebra_Release needed for this action, have already been resolved (short answer: no)
The pathfinder wiki is a good example of team documentation.
I am looking forward to the upcoming release. The next one might be this year, but not necessarily includes everything on the drawing board.
regards, Martin
On Tue, Oct 16, 2007 at 05:23:43PM +0200, Peter Boncz wrote:
Hi Maurice, et al,
The release of (hopefully) this week will still be mps based. I also anticipate at least one bug-fix release (ie 0.20.2), or even more. However, the next major release (0.22) should be algebra based, in my opinion. I am certainly reluctant to keep investing any more effort in mps. I would aim for an mps-free release, yes.
The time-goal of that release is "XMAS 2007", as far as we are concerned. For the work of built-in functions I have good hope that with the help of Lefteris we can come a long way.
I hope and think updates can also be ported by joerd in that time-frame. As for pftijah, if Jan Flokstra has time, it should be possible to port that as well in 2-3 months.
There are some hairy issues, one of them being free-form recursion (needed by you), another one being precompiled modules (needed by XRPC). The resolution of these needs assistance from Munich, and maybe related. It needs a method to compile certain functions into scripted procedures (here MIL) and calling them. One algorithm is to do this for all functions in modules, as well as for all recursive functions. The issue to resolve is the representation of the parameters.
Depending on how much time Jan Rittinger has available for this, we will have to find creative solutions for this. Indeed, a backup solution would be to keep handling certain queries for the time being with mps, but that surely is ugly.
Peter
-----Original Message----- From: Keulen, M. van (Maurice) [mailto:m.vankeulen@utwente.nl] Sent: dinsdag 16 oktober 2007 16:38 To: Peter Boncz [CWI]; Stefan Manegold [CWI] Cc: Djoerd Hiemstra; Jan Flokstra; Henning Rode; Abdel Kader, R. (Riham) Subject: Nieuwe releases en mps-backend
Hoi,
Sorry dat ik niet bij de Skype-sessie van vandaag kon zijn, maar Riham vertelde me dat het onderwerp van nieuwe releases en de mps-backend ter sprake is geweest. Het riep wat specifieke vragen bij me op die ze niet allemaal kon beantwoorden:
a.. Is het de bedoeling dat een nieuwe release de mps-backend helemaalniet meer bevat of alleen maar de algebra-backend als default heeft? b.. Als jullie praten over "nieuwe release", bedoel je dan alleen een package van alle modules van MonetDB/XQuery? Maw zegt het niets over eventuele `subreleases' van individuele modules?
c.. Op de wiki hebben we een pagina met open issues voor dealgebra-release. Wat is de status van die open issues? Ik zie namelijk op de pagina nergens dat bepaalde issues al opgelost/geïmplementeerd zijn. Ik Cc deze e-mail even aan de PF/Tijah-mensen, want hoewel er levendige discussies zijn hoe PF/Tijah naar de algebra-backend to porten, ik heb niet het idee dat er oplossingen zijn die op korte termijn realiseerbaar zijn. Evenzo het recursie-vraagstuk. Het lijkt me bijvoorbeeld voor de zichtbaarheid naar buiten toe onwenselijk als deze features ineens zouden wegvallen voor non-insiders. Of andere onwenselijke consequenties van het niet meer (sub)releasen van modules die gebaseerd zijn op de mps-backend.
Groetjes, Maurice.
--
Dr.Ir. M. van Keulen - Assistant Professor, Data Management Technology Univ. of Twente, Dept of EEMCS, POBox 217, 7500 AE Enschede, Netherlands Email: m.vankeulen@utwente.nl, Phone: +31 534893688, Fax: +31 534892927 Room: ZI 3039, WWW: http://www.cs.utwente.nl/~keulen
Dear all,
Very good: no more milprinting! In absence of Jan, please let me add the plans for PF/Tijah. To make things slightly more complicated: The PF/Tijah team would like to use the algebra release for porting our proprietary search extensions to XQuery Full-text, see: http://www.w3.org/TR/xquery-full-text/ Some initial design decisions that we like to follow have been made by Stefan Klinger at the University of Konstanz. The consequences of such a port would be:
1. XQuery syntax will be extended for text search queries (now only via user-defined functions) 2. All algebra operations must be score aware (i.e., if a score column exists, scores have to be propagated and aggregated)
We are aware that this affects everyone, although, only the extended queries will produce a score column, and algebra operations remain exactly the same if score columns are not there. The time goal of this release is XMAS 2008. In the mean time, we might not port the current PF/Tijah extensions to the algebra.
Best, Djoerd.
On Tue, 16 Oct 2007, Martin Kersten wrote:
LS,
Good to see this discussion. Although the topic is primarily MonetDB/XQuery it is good to keep an eye on the global picture. Releases include more stuff and requires a lot of time to prepare. (the release itself easily consumes several months of full time work, many delivered by free (weekend) time!!!!) Good planning and long-term internal use are a prerequisite. Something we still have to learn. Once the release cycle starts, the train can not stop and full power is/should be in the hands of a single person. The release manager currently is Sjoerd, one of the project admins!
As soon as a RC is available, those with a non-standard platform (windows, sun, mac, vista) should immediately double check to assure it basically works. This holds for all addressed and developers.
In this respect, the time frame mentioned below is overly optimistic. Anyone close to the core group should work on the HEAD to help uncover problems from day 1. Code changes should *not* be collected by individual programmers until release time . This is a major source of frustration, because you simply may not expect your team members to spent day and night to fight the 'last' bugs. Let alone to quick-test your latest feature or performance improvement. PLEASE, check-in daily your changes and make sure you have at least ran mtest on your platform. And most of all, know that your changes are scrutinized for potential quality problems by others. It is the best way to avoid many bugs.
Furthermore, changes in the interfaces visible to users and the effect on their databases should be kept to a minimum. Backward compatibility is a must as long as it is manageable. Beware that a change in storage structure may help a few extreme users, but may affects hundreds or thousands in their daily use. E.g. one of our users would have to unload/reload a 0.5Tb business critical application.
MonetDB/Query and MonetDB/SQL are intertwined. This creates responsibility both ways.
Peter Boncz wrote:
Stefan suggested rightly to move this discussion to the mailing list. I also changed the language into english for that purpose.
Maurice' original questions were:
(a) whether the new release would be algebra-only (short answer: hopefully).
In my opinion this can only be answered if the algebra version, in its full glory (without mps) is used for several months internally, e.g. using students for stress testing.
(b) whether our talk of a new release also refers to submodules, such as probxml and pftijah (short answer: yes)
Submodules basically follow their own release cycle. Regardless the source, e.g. datacells geom, .... Keeping in line is the responsibility of their owners, and the core developers should be aware that their changes may break the work of others. Or at least drain a possibly substantial amount of time.
(c) whether all issues on the wiki, see: http://www.pathfinder-xquery.org/wiki/index.php/Jupiter:_The_Algebra_Release needed for this action, have already been resolved (short answer: no)
The pathfinder wiki is a good example of team documentation.
I am looking forward to the upcoming release. The next one might be this year, but not necessarily includes everything on the drawing board.
regards, Martin
On Tue, Oct 16, 2007 at 05:23:43PM +0200, Peter Boncz wrote:
Hi Maurice, et al,
The release of (hopefully) this week will still be mps based. I also anticipate at least one bug-fix release (ie 0.20.2), or even more. However, the next major release (0.22) should be algebra based, in my opinion. I am certainly reluctant to keep investing any more effort in mps. I would aim for an mps-free release, yes.
The time-goal of that release is "XMAS 2007", as far as we are concerned. For the work of built-in functions I have good hope that with the help of Lefteris we can come a long way.
I hope and think updates can also be ported by joerd in that time-frame. As for pftijah, if Jan Flokstra has time, it should be possible to port that as well in 2-3 months.
There are some hairy issues, one of them being free-form recursion (needed by you), another one being precompiled modules (needed by XRPC). The resolution of these needs assistance from Munich, and maybe related. It needs a method to compile certain functions into scripted procedures (here MIL) and calling them. One algorithm is to do this for all functions in modules, as well as for all recursive functions. The issue to resolve is the representation of the parameters.
Depending on how much time Jan Rittinger has available for this, we will have to find creative solutions for this. Indeed, a backup solution would be to keep handling certain queries for the time being with mps, but that surely is ugly.
Peter
-----Original Message----- From: Keulen, M. van (Maurice) [mailto:m.vankeulen@utwente.nl] Sent: dinsdag 16 oktober 2007 16:38 To: Peter Boncz [CWI]; Stefan Manegold [CWI] Cc: Djoerd Hiemstra; Jan Flokstra; Henning Rode; Abdel Kader, R. (Riham) Subject: Nieuwe releases en mps-backend
Hoi,
Sorry dat ik niet bij de Skype-sessie van vandaag kon zijn, maar Riham vertelde me dat het onderwerp van nieuwe releases en de mps-backend ter sprake is geweest. Het riep wat specifieke vragen bij me op die ze niet allemaal kon beantwoorden:
a.. Is het de bedoeling dat een nieuwe release de mps-backend helemaalniet meer bevat of alleen maar de algebra-backend als default heeft? b.. Als jullie praten over "nieuwe release", bedoel je dan alleen een package van alle modules van MonetDB/XQuery? Maw zegt het niets over eventuele `subreleases' van individuele modules?
c.. Op de wiki hebben we een pagina met open issues voor dealgebra-release. Wat is de status van die open issues? Ik zie namelijk op de pagina nergens dat bepaalde issues al opgelost/geïmplementeerd zijn. Ik Cc deze e-mail even aan de PF/Tijah-mensen, want hoewel er levendige discussies zijn hoe PF/Tijah naar de algebra-backend to porten, ik heb niet het idee dat er oplossingen zijn die op korte termijn realiseerbaar zijn. Evenzo het recursie-vraagstuk. Het lijkt me bijvoorbeeld voor de zichtbaarheid naar buiten toe onwenselijk als deze features ineens zouden wegvallen voor non-insiders. Of andere onwenselijke consequenties van het niet meer (sub)releasen van modules die gebaseerd zijn op de mps-backend.
Groetjes, Maurice.
--
Dr.Ir. M. van Keulen - Assistant Professor, Data Management Technology Univ. of Twente, Dept of EEMCS, POBox 217, 7500 AE Enschede, Netherlands Email: m.vankeulen@utwente.nl, Phone: +31 534893688, Fax: +31 534892927 Room: ZI 3039, WWW: http://www.cs.utwente.nl/~keulen
This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
Hi all,
As a follow-up to Djoerd's response and my questions about releases, let me say a little bit more about certain activities and sketch an analogy which we might mimick.
We are trying to get many people enthusiastic about XML databases and XML-IR, as offered by MonetDB/XQuery in particular. To achieve some visibility for our work, I have for example started several activities in building production applications on top of MonetDB/XQuery and which use PF/Tijah. For example, I promised a demonstration in january to university-wide decision makers of the UT of a publication management application which uses XML-exports from our E-Prints system including the full-text of the papers. I hope it will be accepted as a production application used by all researchers of the UT next to E-Prints, which will over time demonstrate more advanced features and take over more and more functionality of E-Prints. As Djoerd pointed out, porting PF/Tijah to the algebra-backend in the proper way, is more a matter of a year than of a few months. However, it would be certain death to such production applications if the support for developing them would stop (since algebra-only means without PF/Tijah for a long time). Furthermore, many students following our courses and doing our projects are very interested in XML databases including XML-IR and probabilistic XML. A significant portion would be rather discouraged if they would not be able to grab a stable release to start working and get some bugs fixed.
Microsoft once had the problem that they had ideas for a fundamentally different operating system kernel (NT). It was infeasible to make it fully backward-compatible with Windows 95 (on top of MSDOS), but it could also not abandon a large part of their userbase if their applications would not work anymore. Therefore, they had two release streams: one NT-based and one MSDOS-based. Both knew their releases until in the end. When most of the applications were ported or at least working on the new technology, they stopped releasing in the MSDOS-based stream.
The algebra-backend is our `new technology', so can't we do something similar? Have an algebra-based release stream of MonetDB/XQuery (starting with 0.22) capable of running a significant portion of applications, and an mps-based release stream needed to serve the PF/Tijah- and ProbXML-applications. It would mean an extra package under "Download" on SF. Of course the algebra-based releases will have priority, but it would make it possible to have a new release for for example PF/Tijah when it is needed. Of course we will put in the required effort, but cannot do this alone.
What do you think of this?
Maurice.
Djoerd Hiemstra wrote:
Dear all,
Very good: no more milprinting! In absence of Jan, please let me add the plans for PF/Tijah. To make things slightly more complicated: The PF/Tijah team would like to use the algebra release for porting our proprietary search extensions to XQuery Full-text, see: http://www.w3.org/TR/xquery-full-text/ Some initial design decisions that we like to follow have been made by Stefan Klinger at the University of Konstanz. The consequences of such a port would be:
- XQuery syntax will be extended for text search queries (now only via user-defined functions)
- All algebra operations must be score aware (i.e., if a score column exists, scores have to be propagated and aggregated)
We are aware that this affects everyone, although, only the extended queries will produce a score column, and algebra operations remain exactly the same if score columns are not there. The time goal of this release is XMAS 2008. In the mean time, we might not port the current PF/Tijah extensions to the algebra.
Best, Djoerd.
On Tue, 16 Oct 2007, Martin Kersten wrote:
LS,
Good to see this discussion. Although the topic is primarily MonetDB/XQuery it is good to keep an eye on the global picture. Releases include more stuff and requires a lot of time to prepare. (the release itself easily consumes several months of full time work, many delivered by free (weekend) time!!!!) Good planning and long-term internal use are a prerequisite. Something we still have to learn. Once the release cycle starts, the train can not stop and full power is/should be in the hands of a single person. The release manager currently is Sjoerd, one of the project admins!
As soon as a RC is available, those with a non-standard platform (windows, sun, mac, vista) should immediately double check to assure it basically works. This holds for all addressed and developers.
In this respect, the time frame mentioned below is overly optimistic. Anyone close to the core group should work on the HEAD to help uncover problems from day 1. Code changes should *not* be collected by individual programmers until release time . This is a major source of frustration, because you simply may not expect your team members to spent day and night to fight the 'last' bugs. Let alone to quick-test your latest feature or performance improvement. PLEASE, check-in daily your changes and make sure you have at least ran mtest on your platform. And most of all, know that your changes are scrutinized for potential quality problems by others. It is the best way to avoid many bugs.
Furthermore, changes in the interfaces visible to users and the effect on their databases should be kept to a minimum. Backward compatibility is a must as long as it is manageable. Beware that a change in storage structure may help a few extreme users, but may affects hundreds or thousands in their daily use. E.g. one of our users would have to unload/reload a 0.5Tb business critical application.
MonetDB/Query and MonetDB/SQL are intertwined. This creates responsibility both ways.
Peter Boncz wrote:
Stefan suggested rightly to move this discussion to the mailing list. I also changed the language into english for that purpose.
Maurice' original questions were:
(a) whether the new release would be algebra-only (short answer: hopefully).
In my opinion this can only be answered if the algebra version, in its full glory (without mps) is used for several months internally, e.g. using students for stress testing.
(b) whether our talk of a new release also refers to submodules, such as probxml and pftijah (short answer: yes)
Submodules basically follow their own release cycle. Regardless the source, e.g. datacells geom, .... Keeping in line is the responsibility of their owners, and the core developers should be aware that their changes may break the work of others. Or at least drain a possibly substantial amount of time.
(c) whether all issues on the wiki, see: http://www.pathfinder-xquery.org/wiki/index.php/Jupiter:_The_Algebra_Release needed for this action, have already been resolved (short answer: no)
The pathfinder wiki is a good example of team documentation.
I am looking forward to the upcoming release. The next one might be this year, but not necessarily includes everything on the drawing board.
regards, Martin
On Tue, Oct 16, 2007 at 05:23:43PM +0200, Peter Boncz wrote:
Hi Maurice, et al,
The release of (hopefully) this week will still be mps based. I also anticipate at least one bug-fix release (ie 0.20.2), or even more. However, the next major release (0.22) should be algebra based, in my opinion. I am certainly reluctant to keep investing any more effort in mps. I would aim for an mps-free release, yes.
The time-goal of that release is "XMAS 2007", as far as we are concerned. For the work of built-in functions I have good hope that with the help of Lefteris we can come a long way.
I hope and think updates can also be ported by joerd in that time-frame. As for pftijah, if Jan Flokstra has time, it should be possible to port that as well in 2-3 months.
There are some hairy issues, one of them being free-form recursion (needed by you), another one being precompiled modules (needed by XRPC). The resolution of these needs assistance from Munich, and maybe related. It needs a method to compile certain functions into scripted procedures (here MIL) and calling them. One algorithm is to do this for all functions in modules, as well as for all recursive functions. The issue to resolve is the representation of the parameters.
Depending on how much time Jan Rittinger has available for this, we will have to find creative solutions for this. Indeed, a backup solution would be to keep handling certain queries for the time being with mps, but that surely is ugly.
Peter
-----Original Message----- From: Keulen, M. van (Maurice) [mailto:m.vankeulen@utwente.nl] Sent: dinsdag 16 oktober 2007 16:38 To: Peter Boncz [CWI]; Stefan Manegold [CWI] Cc: Djoerd Hiemstra; Jan Flokstra; Henning Rode; Abdel Kader, R. (Riham) Subject: Nieuwe releases en mps-backend
Hoi,
Sorry dat ik niet bij de Skype-sessie van vandaag kon zijn, maar Riham vertelde me dat het onderwerp van nieuwe releases en de mps-backend ter sprake is geweest. Het riep wat specifieke vragen bij me op die ze niet allemaal kon beantwoorden:
a.. Is het de bedoeling dat een nieuwe release de mps-backend helemaalniet meer bevat of alleen maar de algebra-backend als default heeft? b.. Als jullie praten over "nieuwe release", bedoel je dan alleen een package van alle modules van MonetDB/XQuery? Maw zegt het niets over eventuele `subreleases' van individuele modules?
c.. Op de wiki hebben we een pagina met open issues voor dealgebra-release. Wat is de status van die open issues? Ik zie namelijk op de pagina nergens dat bepaalde issues al opgelost/geïmplementeerd zijn. Ik Cc deze e-mail even aan de PF/Tijah-mensen, want hoewel er levendige discussies zijn hoe PF/Tijah naar de algebra-backend to porten, ik heb niet het idee dat er oplossingen zijn die op korte termijn realiseerbaar zijn. Evenzo het recursie-vraagstuk. Het lijkt me bijvoorbeeld voor de zichtbaarheid naar buiten toe onwenselijk als deze features ineens zouden wegvallen voor non-insiders. Of andere onwenselijke consequenties van het niet meer (sub)releasen van modules die gebaseerd zijn op de mps-backend.
Groetjes, Maurice.
--
Dr.Ir. M. van Keulen - Assistant Professor, Data Management Technology Univ. of Twente, Dept of EEMCS, POBox 217, 7500 AE Enschede, Netherlands Email: m.vankeulen@utwente.nl, Phone: +31 534893688, Fax: +31 534892927 Room: ZI 3039, WWW: http://www.cs.utwente.nl/~keulen
This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
Hi Maurice,
valid point.
Unfortunately, we (at least here at CWI) do not have the manpower an/or budget that Microsoft has (recall, our "clients" do not pay for our "product").
Hence, if a dual-path strategy is desired and/or required, please also consider who should be doing all the administration, maintenance, bug fixing, testing, releasing, etc...
Stefan
On Wed, Oct 17, 2007 at 10:15:17AM +0200, Keulen, M. van (Maurice) wrote:
Hi all,
As a follow-up to Djoerd's response and my questions about releases, let me say a little bit more about certain activities and sketch an analogy which we might mimick.
We are trying to get many people enthusiastic about XML databases and XML-IR, as offered by MonetDB/XQuery in particular. To achieve some visibility for our work, I have for example started several activities in building production applications on top of MonetDB/XQuery and which use PF/Tijah. For example, I promised a demonstration in january to university-wide decision makers of the UT of a publication management application which uses XML-exports from our E-Prints system including the full-text of the papers. I hope it will be accepted as a production application used by all researchers of the UT next to E-Prints, which will over time demonstrate more advanced features and take over more and more functionality of E-Prints. As Djoerd pointed out, porting PF/Tijah to the algebra-backend in the proper way, is more a matter of a year than of a few months. However, it would be certain death to such production applications if the support for developing them would stop (since algebra-only means without PF/Tijah for a long time). Furthermore, many students following our courses and doing our projects are very interested in XML databases including XML-IR and probabilistic XML. A significant portion would be rather discouraged if they would not be able to grab a stable release to start working and get some bugs fixed.
Microsoft once had the problem that they had ideas for a fundamentally different operating system kernel (NT). It was infeasible to make it fully backward-compatible with Windows 95 (on top of MSDOS), but it could also not abandon a large part of their userbase if their applications would not work anymore. Therefore, they had two release streams: one NT-based and one MSDOS-based. Both knew their releases until in the end. When most of the applications were ported or at least working on the new technology, they stopped releasing in the MSDOS-based stream.
The algebra-backend is our `new technology', so can't we do something similar? Have an algebra-based release stream of MonetDB/XQuery (starting with 0.22) capable of running a significant portion of applications, and an mps-based release stream needed to serve the PF/Tijah- and ProbXML-applications. It would mean an extra package under "Download" on SF. Of course the algebra-based releases will have priority, but it would make it possible to have a new release for for example PF/Tijah when it is needed. Of course we will put in the required effort, but cannot do this alone.
What do you think of this?
Maurice.
Djoerd Hiemstra wrote:
Dear all,
Very good: no more milprinting! In absence of Jan, please let me add the plans for PF/Tijah. To make things slightly more complicated: The PF/Tijah team would like to use the algebra release for porting our proprietary search extensions to XQuery Full-text, see: http://www.w3.org/TR/xquery-full-text/ Some initial design decisions that we like to follow have been made by Stefan Klinger at the University of Konstanz. The consequences of such a port would be:
- XQuery syntax will be extended for text search queries (now only via user-defined functions)
- All algebra operations must be score aware (i.e., if a score column exists, scores have to be propagated and aggregated)
We are aware that this affects everyone, although, only the extended queries will produce a score column, and algebra operations remain exactly the same if score columns are not there. The time goal of this release is XMAS 2008. In the mean time, we might not port the current PF/Tijah extensions to the algebra.
Best, Djoerd.
On Tue, 16 Oct 2007, Martin Kersten wrote:
LS,
Good to see this discussion. Although the topic is primarily MonetDB/XQuery it is good to keep an eye on the global picture. Releases include more stuff and requires a lot of time to prepare. (the release itself easily consumes several months of full time work, many delivered by free (weekend) time!!!!) Good planning and long-term internal use are a prerequisite. Something we still have to learn. Once the release cycle starts, the train can not stop and full power is/should be in the hands of a single person. The release manager currently is Sjoerd, one of the project admins!
As soon as a RC is available, those with a non-standard platform (windows, sun, mac, vista) should immediately double check to assure it basically works. This holds for all addressed and developers.
In this respect, the time frame mentioned below is overly optimistic. Anyone close to the core group should work on the HEAD to help uncover problems from day 1. Code changes should *not* be collected by individual programmers until release time . This is a major source of frustration, because you simply may not expect your team members to spent day and night to fight the 'last' bugs. Let alone to quick-test your latest feature or performance improvement. PLEASE, check-in daily your changes and make sure you have at least ran mtest on your platform. And most of all, know that your changes are scrutinized for potential quality problems by others. It is the best way to avoid many bugs.
Furthermore, changes in the interfaces visible to users and the effect on their databases should be kept to a minimum. Backward compatibility is a must as long as it is manageable. Beware that a change in storage structure may help a few extreme users, but may affects hundreds or thousands in their daily use. E.g. one of our users would have to unload/reload a 0.5Tb business critical application.
MonetDB/Query and MonetDB/SQL are intertwined. This creates responsibility both ways.
Peter Boncz wrote:
Stefan suggested rightly to move this discussion to the mailing list. I also changed the language into english for that purpose.
Maurice' original questions were:
(a) whether the new release would be algebra-only (short answer: hopefully).
In my opinion this can only be answered if the algebra version, in its full glory (without mps) is used for several months internally, e.g. using students for stress testing.
(b) whether our talk of a new release also refers to submodules, such as probxml and pftijah (short answer: yes)
Submodules basically follow their own release cycle. Regardless the source, e.g. datacells geom, .... Keeping in line is the responsibility of their owners, and the core developers should be aware that their changes may break the work of others. Or at least drain a possibly substantial amount of time.
(c) whether all issues on the wiki, see: http://www.pathfinder-xquery.org/wiki/index.php/Jupiter:_The_Algebra_Release needed for this action, have already been resolved (short answer: no)
The pathfinder wiki is a good example of team documentation.
I am looking forward to the upcoming release. The next one might be this year, but not necessarily includes everything on the drawing board.
regards, Martin
On Tue, Oct 16, 2007 at 05:23:43PM +0200, Peter Boncz wrote:
Hi Maurice, et al,
The release of (hopefully) this week will still be mps based. I also anticipate at least one bug-fix release (ie 0.20.2), or even more. However, the next major release (0.22) should be algebra based, in my opinion. I am certainly reluctant to keep investing any more effort in mps. I would aim for an mps-free release, yes.
The time-goal of that release is "XMAS 2007", as far as we are concerned. For the work of built-in functions I have good hope that with the help of Lefteris we can come a long way.
I hope and think updates can also be ported by joerd in that time-frame. As for pftijah, if Jan Flokstra has time, it should be possible to port that as well in 2-3 months.
There are some hairy issues, one of them being free-form recursion (needed by you), another one being precompiled modules (needed by XRPC). The resolution of these needs assistance from Munich, and maybe related. It needs a method to compile certain functions into scripted procedures (here MIL) and calling them. One algorithm is to do this for all functions in modules, as well as for all recursive functions. The issue to resolve is the representation of the parameters.
Depending on how much time Jan Rittinger has available for this, we will have to find creative solutions for this. Indeed, a backup solution would be to keep handling certain queries for the time being with mps, but that surely is ugly.
Peter
-----Original Message----- From: Keulen, M. van (Maurice) [mailto:m.vankeulen@utwente.nl] Sent: dinsdag 16 oktober 2007 16:38 To: Peter Boncz [CWI]; Stefan Manegold [CWI] Cc: Djoerd Hiemstra; Jan Flokstra; Henning Rode; Abdel Kader, R. (Riham) Subject: Nieuwe releases en mps-backend
Hoi,
Sorry dat ik niet bij de Skype-sessie van vandaag kon zijn, maar Riham vertelde me dat het onderwerp van nieuwe releases en de mps-backend ter sprake is geweest. Het riep wat specifieke vragen bij me op die ze niet allemaal kon beantwoorden:
a.. Is het de bedoeling dat een nieuwe release de mps-backend helemaalniet meer bevat of alleen maar de algebra-backend als default heeft? b.. Als jullie praten over "nieuwe release", bedoel je dan alleen een package van alle modules van MonetDB/XQuery? Maw zegt het niets over eventuele `subreleases' van individuele modules?
c.. Op de wiki hebben we een pagina met open issues voor dealgebra-release. Wat is de status van die open issues? Ik zie namelijk op de pagina nergens dat bepaalde issues al opgelost/geïmplementeerd zijn. Ik Cc deze e-mail even aan de PF/Tijah-mensen, want hoewel er levendige discussies zijn hoe PF/Tijah naar de algebra-backend to porten, ik heb niet het idee dat er oplossingen zijn die op korte termijn realiseerbaar zijn. Evenzo het recursie-vraagstuk. Het lijkt me bijvoorbeeld voor de zichtbaarheid naar buiten toe onwenselijk als deze features ineens zouden wegvallen voor non-insiders. Of andere onwenselijke consequenties van het niet meer (sub)releasen van modules die gebaseerd zijn op de mps-backend.
Groetjes, Maurice.
--
Dr.Ir. M. van Keulen - Assistant Professor, Data Management Technology Univ. of Twente, Dept of EEMCS, POBox 217, 7500 AE Enschede, Netherlands Email: m.vankeulen@utwente.nl, Phone: +31 534893688, Fax: +31 534892927 Room: ZI 3039, WWW: http://www.cs.utwente.nl/~keulen
This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
--
Dr.Ir. M. van Keulen - Assistant Professor, Data Management Technology Univ. of Twente, Dept of EEMCS, POBox 217, 7500 AE Enschede, Netherlands Email: m.vankeulen@utwente.nl, Phone: +31 534893688, Fax: +31 534892927 Room: ZI 3039, WWW: http://www.cs.utwente.nl/~keulen
This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
On Wed, Oct 17, 2007 at 10:27:41AM +0200, Stefan Manegold wrote: [...]
Unfortunately, we (at least here at CWI) do not have the manpower an/or budget that Microsoft has (recall, our "clients" do not pay for our "product").
... at least they don't pay us here at CWI (except from granting research projects) ...
Stefan
Hi Stefan,
Valid point yourself :-)
I wholeheartedly support a swift switch to the algebra-backend. What I'm basically looking for is a convenient way to keep on having others developing applications on top of MXQ/PFTijah whereby prohibitive bugs can still be solved for these people.
More thoughts: I presume the mps-code will remain in the CVS tree for as long as it takes, so bugs could be resolved, but others can only benefit if they would use a compilation of the CVS tree. Is it perhaps possible to easily create installation packages for the HEAD-version without actually releasing these packages as a formal release? We could 'release' these packages by putting them in the download-section of the PF/Tijah website. This would not make any claims concerning testing and stability, since it would be only a packaged new version of PF/Tijah with no formal status as release, but still allow new versions of PF/Tijah from being packaged and made available.
The downside of such an approach is that from the perspective of people in the outside world looking at MonetDB/XQuery as an open source product, MonetDb/XQuery will loose some of its distinguishing features (fulltext querying capability) for a long time. Is that what you want?
Any thoughts?
Maurice.
Stefan Manegold wrote:
Hi Maurice,
valid point.
Unfortunately, we (at least here at CWI) do not have the manpower an/or budget that Microsoft has (recall, our "clients" do not pay for our "product").
Hence, if a dual-path strategy is desired and/or required, please also consider who should be doing all the administration, maintenance, bug fixing, testing, releasing, etc...
Stefan
On Wed, Oct 17, 2007 at 10:15:17AM +0200, Keulen, M. van (Maurice) wrote:
Hi all,
As a follow-up to Djoerd's response and my questions about releases, let me say a little bit more about certain activities and sketch an analogy which we might mimick.
We are trying to get many people enthusiastic about XML databases and XML-IR, as offered by MonetDB/XQuery in particular. To achieve some visibility for our work, I have for example started several activities in building production applications on top of MonetDB/XQuery and which use PF/Tijah. For example, I promised a demonstration in january to university-wide decision makers of the UT of a publication management application which uses XML-exports from our E-Prints system including the full-text of the papers. I hope it will be accepted as a production application used by all researchers of the UT next to E-Prints, which will over time demonstrate more advanced features and take over more and more functionality of E-Prints. As Djoerd pointed out, porting PF/Tijah to the algebra-backend in the proper way, is more a matter of a year than of a few months. However, it would be certain death to such production applications if the support for developing them would stop (since algebra-only means without PF/Tijah for a long time). Furthermore, many students following our courses and doing our projects are very interested in XML databases including XML-IR and probabilistic XML. A significant portion would be rather discouraged if they would not be able to grab a stable release to start working and get some bugs fixed.
Microsoft once had the problem that they had ideas for a fundamentally different operating system kernel (NT). It was infeasible to make it fully backward-compatible with Windows 95 (on top of MSDOS), but it could also not abandon a large part of their userbase if their applications would not work anymore. Therefore, they had two release streams: one NT-based and one MSDOS-based. Both knew their releases until in the end. When most of the applications were ported or at least working on the new technology, they stopped releasing in the MSDOS-based stream.
The algebra-backend is our `new technology', so can't we do something similar? Have an algebra-based release stream of MonetDB/XQuery (starting with 0.22) capable of running a significant portion of applications, and an mps-based release stream needed to serve the PF/Tijah- and ProbXML-applications. It would mean an extra package under "Download" on SF. Of course the algebra-based releases will have priority, but it would make it possible to have a new release for for example PF/Tijah when it is needed. Of course we will put in the required effort, but cannot do this alone.
What do you think of this?
Maurice.
Djoerd Hiemstra wrote:
Dear all,
Very good: no more milprinting! In absence of Jan, please let me add the plans for PF/Tijah. To make things slightly more complicated: The PF/Tijah team would like to use the algebra release for porting our proprietary search extensions to XQuery Full-text, see: http://www.w3.org/TR/xquery-full-text/ Some initial design decisions that we like to follow have been made by Stefan Klinger at the University of Konstanz. The consequences of such a port would be:
- XQuery syntax will be extended for text search queries (now only via user-defined functions)
- All algebra operations must be score aware (i.e., if a score column exists, scores have to be propagated and aggregated)
We are aware that this affects everyone, although, only the extended queries will produce a score column, and algebra operations remain exactly the same if score columns are not there. The time goal of this release is XMAS 2008. In the mean time, we might not port the current PF/Tijah extensions to the algebra.
Best, Djoerd.
On Tue, 16 Oct 2007, Martin Kersten wrote:
LS,
Good to see this discussion. Although the topic is primarily MonetDB/XQuery it is good to keep an eye on the global picture. Releases include more stuff and requires a lot of time to prepare. (the release itself easily consumes several months of full time work, many delivered by free (weekend) time!!!!) Good planning and long-term internal use are a prerequisite. Something we still have to learn. Once the release cycle starts, the train can not stop and full power is/should be in the hands of a single person. The release manager currently is Sjoerd, one of the project admins!
As soon as a RC is available, those with a non-standard platform (windows, sun, mac, vista) should immediately double check to assure it basically works. This holds for all addressed and developers.
In this respect, the time frame mentioned below is overly optimistic. Anyone close to the core group should work on the HEAD to help uncover problems from day 1. Code changes should *not* be collected by individual programmers until release time . This is a major source of frustration, because you simply may not expect your team members to spent day and night to fight the 'last' bugs. Let alone to quick-test your latest feature or performance improvement. PLEASE, check-in daily your changes and make sure you have at least ran mtest on your platform. And most of all, know that your changes are scrutinized for potential quality problems by others. It is the best way to avoid many bugs.
Furthermore, changes in the interfaces visible to users and the effect on their databases should be kept to a minimum. Backward compatibility is a must as long as it is manageable. Beware that a change in storage structure may help a few extreme users, but may affects hundreds or thousands in their daily use. E.g. one of our users would have to unload/reload a 0.5Tb business critical application.
MonetDB/Query and MonetDB/SQL are intertwined. This creates responsibility both ways.
Peter Boncz wrote:
Stefan suggested rightly to move this discussion to the mailing list. I also changed the language into english for that purpose.
Maurice' original questions were:
(a) whether the new release would be algebra-only (short answer: hopefully).
In my opinion this can only be answered if the algebra version, in its full glory (without mps) is used for several months internally, e.g. using students for stress testing.
(b) whether our talk of a new release also refers to submodules, such as probxml and pftijah (short answer: yes)
Submodules basically follow their own release cycle. Regardless the source, e.g. datacells geom, .... Keeping in line is the responsibility of their owners, and the core developers should be aware that their changes may break the work of others. Or at least drain a possibly substantial amount of time.
(c) whether all issues on the wiki, see: http://www.pathfinder-xquery.org/wiki/index.php/Jupiter:_The_Algebra_Release needed for this action, have already been resolved (short answer: no)
The pathfinder wiki is a good example of team documentation.
I am looking forward to the upcoming release. The next one might be this year, but not necessarily includes everything on the drawing board.
regards, Martin
On Tue, Oct 16, 2007 at 05:23:43PM +0200, Peter Boncz wrote:
Hi Maurice, et al,
The release of (hopefully) this week will still be mps based. I also anticipate at least one bug-fix release (ie 0.20.2), or even more. However, the next major release (0.22) should be algebra based, in my opinion. I am certainly reluctant to keep investing any more effort in mps. I would aim for an mps-free release, yes.
The time-goal of that release is "XMAS 2007", as far as we are concerned. For the work of built-in functions I have good hope that with the help of Lefteris we can come a long way.
I hope and think updates can also be ported by joerd in that time-frame. As for pftijah, if Jan Flokstra has time, it should be possible to port that as well in 2-3 months.
There are some hairy issues, one of them being free-form recursion (needed by you), another one being precompiled modules (needed by XRPC). The resolution of these needs assistance from Munich, and maybe related. It needs a method to compile certain functions into scripted procedures (here MIL) and calling them. One algorithm is to do this for all functions in modules, as well as for all recursive functions. The issue to resolve is the representation of the parameters.
Depending on how much time Jan Rittinger has available for this, we will have to find creative solutions for this. Indeed, a backup solution would be to keep handling certain queries for the time being with mps, but that surely is ugly.
Peter
-----Original Message----- From: Keulen, M. van (Maurice) [mailto:m.vankeulen@utwente.nl] Sent: dinsdag 16 oktober 2007 16:38 To: Peter Boncz [CWI]; Stefan Manegold [CWI] Cc: Djoerd Hiemstra; Jan Flokstra; Henning Rode; Abdel Kader, R. (Riham) Subject: Nieuwe releases en mps-backend
Hoi,
Sorry dat ik niet bij de Skype-sessie van vandaag kon zijn, maar Riham vertelde me dat het onderwerp van nieuwe releases en de mps-backend ter sprake is geweest. Het riep wat specifieke vragen bij me op die ze niet allemaal kon beantwoorden:
a.. Is het de bedoeling dat een nieuwe release de mps-backend helemaalniet meer bevat of alleen maar de algebra-backend als default heeft? b.. Als jullie praten over "nieuwe release", bedoel je dan alleen een package van alle modules van MonetDB/XQuery? Maw zegt het niets over eventuele `subreleases' van individuele modules?
c.. Op de wiki hebben we een pagina met open issues voor dealgebra-release. Wat is de status van die open issues? Ik zie namelijk op de pagina nergens dat bepaalde issues al opgelost/geïmplementeerd zijn. Ik Cc deze e-mail even aan de PF/Tijah-mensen, want hoewel er levendige discussies zijn hoe PF/Tijah naar de algebra-backend to porten, ik heb niet het idee dat er oplossingen zijn die op korte termijn realiseerbaar zijn. Evenzo het recursie-vraagstuk. Het lijkt me bijvoorbeeld voor de zichtbaarheid naar buiten toe onwenselijk als deze features ineens zouden wegvallen voor non-insiders. Of andere onwenselijke consequenties van het niet meer (sub)releasen van modules die gebaseerd zijn op de mps-backend.
Groetjes, Maurice.
--
Dr.Ir. M. van Keulen - Assistant Professor, Data Management Technology Univ. of Twente, Dept of EEMCS, POBox 217, 7500 AE Enschede, Netherlands Email: m.vankeulen@utwente.nl, Phone: +31 534893688, Fax: +31 534892927 Room: ZI 3039, WWW: http://www.cs.utwente.nl/~keulen
This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
--
Dr.Ir. M. van Keulen - Assistant Professor, Data Management Technology Univ. of Twente, Dept of EEMCS, POBox 217, 7500 AE Enschede, Netherlands Email: m.vankeulen@utwente.nl, Phone: +31 534893688, Fax: +31 534892927 Room: ZI 3039, WWW: http://www.cs.utwente.nl/~keulen
This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
Maurice and other,
Code that has once been released will never "disappear" form that (packaged) release or the respective release branches in CVS. For MPS, this means that it will always be available in the to-be-release MonetDB/XQuery 0.20 release. Hence, users are free to use it --- we have users that happily used MonetDB 4.6 until recently and even some that (though "unconciously") still use some anchient MonetDB 4.0 and/or 4.1 version --- and developers are wellcome to fix bugs on these release branches. They'll always be available in and from CVS. To why point in time and/or extend those bugfixes will be automa[tng]ically propagated to the head needs to be decided. For sure, bug fixes on the branch for release x.y will be automa[tng]ically propagated to the head until the branch for the subsequent release x.y+2 will be created.
For the next x.22 release we need to decide at some point whether it will (as major feature) contain only GDK-2 or also the move from MPS to ALG as default or even only PF backend. To be honest, since GDK-2 is "almost ready", but ALG sill needs quite some work (even without PFtijah & probxml: updates, modules/XRPC, built-in functions, stand-off support), this needs to be evaluated carefully to avoid too many dependencies and hence delays for either of them.
Hence, for pure usage & bugfixing, the MPS version as it will be released "these days" with 0.20 is and remains available. Further, there is a chance that the next release will still contain MPS (possibly even as default), though without major/any(?) changes, development, and/or new features as these should indeed focus *as of now* on ALG --- preferably also for PFtijah & propxml.
Thus, if your main point is to keep MPS, and hence PFtijah & propxml (the latter is not enabled in the binary release packages!) available for *users* and development on top of it that's obviously no problem, even incl. bugfixes in CVS and possibly even incl. bugfix releases.
In case you plan to do major development of PFtijah and/or propxml inside the PF code base before porting these to ALG, we indeed need to find a solution, possibly be extracting these two from the PF source tree and moving them in to one (or two) repositories of there own that can then be release separately (on top of the latest PF release that still comes with MPS).
We don't want to discontinue any of the MonetDB related functionality, but we also don't want to encurage further development of code that has been or will be replaced soon, nor of code that depends on such code.
Stefan
On Wed, Oct 17, 2007 at 04:11:53PM +0200, Keulen, M. van (Maurice) wrote:
Hi Stefan,
Valid point yourself :-)
I wholeheartedly support a swift switch to the algebra-backend. What I'm basically looking for is a convenient way to keep on having others developing applications on top of MXQ/PFTijah whereby prohibitive bugs can still be solved for these people.
More thoughts: I presume the mps-code will remain in the CVS tree for as long as it takes, so bugs could be resolved, but others can only benefit if they would use a compilation of the CVS tree. Is it perhaps possible to easily create installation packages for the HEAD-version without actually releasing these packages as a formal release? We could 'release' these packages by putting them in the download-section of the PF/Tijah website. This would not make any claims concerning testing and stability, since it would be only a packaged new version of PF/Tijah with no formal status as release, but still allow new versions of PF/Tijah from being packaged and made available.
The downside of such an approach is that from the perspective of people in the outside world looking at MonetDB/XQuery as an open source product, MonetDb/XQuery will loose some of its distinguishing features (fulltext querying capability) for a long time. Is that what you want?
Any thoughts?
Maurice.
Stefan Manegold wrote:
Hi Maurice,
valid point.
Unfortunately, we (at least here at CWI) do not have the manpower an/or budget that Microsoft has (recall, our "clients" do not pay for our "product").
Hence, if a dual-path strategy is desired and/or required, please also consider who should be doing all the administration, maintenance, bug fixing, testing, releasing, etc...
Stefan
On Wed, Oct 17, 2007 at 10:15:17AM +0200, Keulen, M. van (Maurice) wrote:
Hi all,
As a follow-up to Djoerd's response and my questions about releases, let me say a little bit more about certain activities and sketch an analogy which we might mimick.
We are trying to get many people enthusiastic about XML databases and XML-IR, as offered by MonetDB/XQuery in particular. To achieve some visibility for our work, I have for example started several activities in building production applications on top of MonetDB/XQuery and which use PF/Tijah. For example, I promised a demonstration in january to university-wide decision makers of the UT of a publication management application which uses XML-exports from our E-Prints system including the full-text of the papers. I hope it will be accepted as a production application used by all researchers of the UT next to E-Prints, which will over time demonstrate more advanced features and take over more and more functionality of E-Prints. As Djoerd pointed out, porting PF/Tijah to the algebra-backend in the proper way, is more a matter of a year than of a few months. However, it would be certain death to such production applications if the support for developing them would stop (since algebra-only means without PF/Tijah for a long time). Furthermore, many students following our courses and doing our projects are very interested in XML databases including XML-IR and probabilistic XML. A significant portion would be rather discouraged if they would not be able to grab a stable release to start working and get some bugs fixed.
Microsoft once had the problem that they had ideas for a fundamentally different operating system kernel (NT). It was infeasible to make it fully backward-compatible with Windows 95 (on top of MSDOS), but it could also not abandon a large part of their userbase if their applications would not work anymore. Therefore, they had two release streams: one NT-based and one MSDOS-based. Both knew their releases until in the end. When most of the applications were ported or at least working on the new technology, they stopped releasing in the MSDOS-based stream.
The algebra-backend is our `new technology', so can't we do something similar? Have an algebra-based release stream of MonetDB/XQuery (starting with 0.22) capable of running a significant portion of applications, and an mps-based release stream needed to serve the PF/Tijah- and ProbXML-applications. It would mean an extra package under "Download" on SF. Of course the algebra-based releases will have priority, but it would make it possible to have a new release for for example PF/Tijah when it is needed. Of course we will put in the required effort, but cannot do this alone.
What do you think of this?
Maurice.
Djoerd Hiemstra wrote:
Dear all,
Very good: no more milprinting! In absence of Jan, please let me add the plans for PF/Tijah. To make things slightly more complicated: The PF/Tijah team would like to use the algebra release for porting our proprietary search extensions to XQuery Full-text, see: http://www.w3.org/TR/xquery-full-text/ Some initial design decisions that we like to follow have been made by Stefan Klinger at the University of Konstanz. The consequences of such a port would be:
- XQuery syntax will be extended for text search queries (now only via user-defined functions)
- All algebra operations must be score aware (i.e., if a score column exists, scores have to be propagated and aggregated)
We are aware that this affects everyone, although, only the extended queries will produce a score column, and algebra operations remain exactly the same if score columns are not there. The time goal of this release is XMAS 2008. In the mean time, we might not port the current PF/Tijah extensions to the algebra.
Best, Djoerd.
On Tue, 16 Oct 2007, Martin Kersten wrote:
LS,
Good to see this discussion. Although the topic is primarily MonetDB/XQuery it is good to keep an eye on the global picture. Releases include more stuff and requires a lot of time to prepare. (the release itself easily consumes several months of full time work, many delivered by free (weekend) time!!!!) Good planning and long-term internal use are a prerequisite. Something we still have to learn. Once the release cycle starts, the train can not stop and full power is/should be in the hands of a single person. The release manager currently is Sjoerd, one of the project admins!
As soon as a RC is available, those with a non-standard platform (windows, sun, mac, vista) should immediately double check to assure it basically works. This holds for all addressed and developers.
In this respect, the time frame mentioned below is overly optimistic. Anyone close to the core group should work on the HEAD to help uncover problems from day 1. Code changes should *not* be collected by individual programmers until release time . This is a major source of frustration, because you simply may not expect your team members to spent day and night to fight the 'last' bugs. Let alone to quick-test your latest feature or performance improvement. PLEASE, check-in daily your changes and make sure you have at least ran mtest on your platform. And most of all, know that your changes are scrutinized for potential quality problems by others. It is the best way to avoid many bugs.
Furthermore, changes in the interfaces visible to users and the effect on their databases should be kept to a minimum. Backward compatibility is a must as long as it is manageable. Beware that a change in storage structure may help a few extreme users, but may affects hundreds or thousands in their daily use. E.g. one of our users would have to unload/reload a 0.5Tb business critical application.
MonetDB/Query and MonetDB/SQL are intertwined. This creates responsibility both ways.
Peter Boncz wrote:
Stefan suggested rightly to move this discussion to the mailing list. I also changed the language into english for that purpose.
Maurice' original questions were:
(a) whether the new release would be algebra-only (short answer: hopefully).
In my opinion this can only be answered if the algebra version, in its full glory (without mps) is used for several months internally, e.g. using students for stress testing.
(b) whether our talk of a new release also refers to submodules, such as probxml and pftijah (short answer: yes)
Submodules basically follow their own release cycle. Regardless the source, e.g. datacells geom, .... Keeping in line is the responsibility of their owners, and the core developers should be aware that their changes may break the work of others. Or at least drain a possibly substantial amount of time.
(c) whether all issues on the wiki, see: http://www.pathfinder-xquery.org/wiki/index.php/Jupiter:_The_Algebra_Release needed for this action, have already been resolved (short answer: no)
The pathfinder wiki is a good example of team documentation.
I am looking forward to the upcoming release. The next one might be this year, but not necessarily includes everything on the drawing board.
regards, Martin
On Tue, Oct 16, 2007 at 05:23:43PM +0200, Peter Boncz wrote:
> Hi Maurice, et al, > > The release of (hopefully) this week will still be mps based. I also > anticipate at least one bug-fix release (ie 0.20.2), or even more. However, > the next major release (0.22) should be algebra based, in my opinion. I am > certainly reluctant to keep investing any more effort in mps. I would aim > for an mps-free release, yes. > > The time-goal of that release is "XMAS 2007", as far as we are concerned. > For the work of built-in functions I have good hope that with the help of > Lefteris we can come a long way. > > I hope and think updates can also be ported by joerd in that time-frame. As > for pftijah, if Jan Flokstra has time, it should be possible to port that as > well in 2-3 months. > > There are some hairy issues, one of them being free-form recursion (needed > by you), another one being precompiled modules (needed by XRPC). The > resolution of these needs assistance from Munich, and maybe related. It needs > a method to compile certain functions into scripted procedures (here MIL) > and calling them. One algorithm is to do this for all functions in modules, > as well as for all recursive functions. The issue to resolve is the > representation of the parameters. > > Depending on how much time Jan Rittinger has available for this, we will > have to find creative solutions for this. Indeed, a backup solution would be > to keep handling certain queries for the time being with mps, but that > surely is ugly. > > Peter > > -----Original Message----- > From: Keulen, M. van (Maurice) [mailto:m.vankeulen@utwente.nl] > Sent: dinsdag 16 oktober 2007 16:38 > To: Peter Boncz [CWI]; Stefan Manegold [CWI] > Cc: Djoerd Hiemstra; Jan Flokstra; Henning Rode; Abdel Kader, R. (Riham) > Subject: Nieuwe releases en mps-backend > > > Hoi, > > Sorry dat ik niet bij de Skype-sessie van vandaag kon zijn, maar Riham > vertelde me dat het onderwerp van nieuwe releases en de mps-backend ter > sprake is geweest. Het riep wat specifieke vragen bij me op die ze niet > allemaal kon beantwoorden: > > a.. Is het de bedoeling dat een nieuwe release de mps-backend helemaal > niet meer bevat of alleen maar de algebra-backend als default heeft? > b.. Als jullie praten over "nieuwe release", bedoel je dan alleen een > package van alle modules van MonetDB/XQuery? Maw zegt het niets over > eventuele `subreleases' van individuele modules? > > c.. Op de wiki hebben we een pagina met open issues voor de > algebra-release. Wat is de status van die open issues? Ik zie namelijk op de > pagina nergens dat bepaalde issues al opgelost/geïmplementeerd zijn. > Ik Cc deze e-mail even aan de PF/Tijah-mensen, want hoewel er levendige > discussies zijn hoe PF/Tijah naar de algebra-backend to porten, ik heb niet > het idee dat er oplossingen zijn die op korte termijn realiseerbaar zijn. > Evenzo het recursie-vraagstuk. Het lijkt me bijvoorbeeld voor de > zichtbaarheid naar buiten toe onwenselijk als deze features ineens zouden > wegvallen voor non-insiders. Of andere onwenselijke consequenties van het > niet meer (sub)releasen van modules die gebaseerd zijn op de mps-backend. > > Groetjes, > Maurice. > > -- > ---------------------------------------------------------------------- > Dr.Ir. M. van Keulen - Assistant Professor, Data Management Technology > Univ. of Twente, Dept of EEMCS, POBox 217, 7500 AE Enschede, Netherlands > Email: m.vankeulen@utwente.nl, Phone: +31 534893688, Fax: +31 534892927 > Room: ZI 3039, WWW: http://www.cs.utwente.nl/~keulen > > >
This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
--
Dr.Ir. M. van Keulen - Assistant Professor, Data Management Technology Univ. of Twente, Dept of EEMCS, POBox 217, 7500 AE Enschede, Netherlands Email: m.vankeulen@utwente.nl, Phone: +31 534893688, Fax: +31 534892927 Room: ZI 3039, WWW: http://www.cs.utwente.nl/~keulen
This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
--
Dr.Ir. M. van Keulen - Assistant Professor, Data Management Technology Univ. of Twente, Dept of EEMCS, POBox 217, 7500 AE Enschede, Netherlands Email: m.vankeulen@utwente.nl, Phone: +31 534893688, Fax: +31 534892927 Room: ZI 3039, WWW: http://www.cs.utwente.nl/~keulen
This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
Hi,
Don't underestimate the cost of rolling forward to new development strands or even willing to maintain multiple software stacks. The necessary expert manpower is scarce. The easiest solution is to focus effort on the algebra strand and 'ignore' any further request to improve the old line. This however, only works if the two lines are in 'sink' for some time.
Experience with M4/SQL and M5/SQL is that it takes few dedicated people to push a line, and it took almost a year to run them in a lab setting in parallel before releasing it to a larger audience. This move went smoothly, but only by taking careful steps of those involved. No last minute large checkins and changes.
So, I advice to make a clear decision to work on the current head only. Bugs in the stable MonetDB/XQuery still has to be solved, but avoid any feature addition.
There seems to be at least two steps considered Algebra version on M4 and M5. You better be aware of this and make a conscious decision on spending time. The M4 server is deprecated! http://monetdb.cwi.nl/projects/monetdb/MonetDB/index.html
Keulen, M. van (Maurice) wrote:
Hi all,
As a follow-up to Djoerd's response and my questions about releases, let me say a little bit more about certain activities and sketch an analogy which we might mimick.
We are trying to get many people enthusiastic about XML databases and XML-IR, as offered by MonetDB/XQuery in particular. To achieve some visibility for our work, I have for example started several activities in building production applications on top of MonetDB/XQuery and which use PF/Tijah. For example, I promised a demonstration in january to university-wide decision makers of the UT of a publication management application which uses XML-exports from our E-Prints system including the full-text of the papers. I hope it will be accepted as a production application used by all researchers of the UT next to E-Prints, which will over time demonstrate more advanced features and take over more and more functionality of E-Prints. As Djoerd pointed out, porting PF/Tijah to the algebra-backend in the proper way, is more a matter of a year than of a few months. However, it would be certain death to such production applications if the support for developing them would stop (since algebra-only means without PF/Tijah for a long time). Furthermore, many students following our courses and doing our projects are very interested in XML databases including XML-IR and probabilistic XML. A significant portion would be rather discouraged if they would not be able to grab a stable release to start working and get some bugs fixed.
Microsoft once had the problem that they had ideas for a fundamentally different operating system kernel (NT). It was infeasible to make it fully backward-compatible with Windows 95 (on top of MSDOS), but it could also not abandon a large part of their userbase if their applications would not work anymore. Therefore, they had two release streams: one NT-based and one MSDOS-based. Both knew their releases until in the end. When most of the applications were ported or at least working on the new technology, they stopped releasing in the MSDOS-based stream.
The algebra-backend is our `new technology', so can't we do something similar? Have an algebra-based release stream of MonetDB/XQuery (starting with 0.22) capable of running a significant portion of applications, and an mps-based release stream needed to serve the PF/Tijah- and ProbXML-applications. It would mean an extra package under "Download" on SF. Of course the algebra-based releases will have priority, but it would make it possible to have a new release for for example PF/Tijah when it is needed. Of course we will put in the required effort, but cannot do this alone.
What do you think of this?
Maurice.
Djoerd Hiemstra wrote:
Dear all,
Very good: no more milprinting! In absence of Jan, please let me add the plans for PF/Tijah. To make things slightly more complicated: The PF/Tijah team would like to use the algebra release for porting our proprietary search extensions to XQuery Full-text, see: http://www.w3.org/TR/xquery-full-text/ Some initial design decisions that we like to follow have been made by Stefan Klinger at the University of Konstanz. The consequences of such a port would be:
- XQuery syntax will be extended for text search queries (now only via user-defined functions)
- All algebra operations must be score aware (i.e., if a score column exists, scores have to be propagated and aggregated)
We are aware that this affects everyone, although, only the extended queries will produce a score column, and algebra operations remain exactly the same if score columns are not there. The time goal of this release is XMAS 2008. In the mean time, we might not port the current PF/Tijah extensions to the algebra.
Best, Djoerd.
On Tue, 16 Oct 2007, Martin Kersten wrote:
LS,
Good to see this discussion. Although the topic is primarily MonetDB/XQuery it is good to keep an eye on the global picture. Releases include more stuff and requires a lot of time to prepare. (the release itself easily consumes several months of full time work, many delivered by free (weekend) time!!!!) Good planning and long-term internal use are a prerequisite. Something we still have to learn. Once the release cycle starts, the train can not stop and full power is/should be in the hands of a single person. The release manager currently is Sjoerd, one of the project admins!
As soon as a RC is available, those with a non-standard platform (windows, sun, mac, vista) should immediately double check to assure it basically works. This holds for all addressed and developers.
In this respect, the time frame mentioned below is overly optimistic. Anyone close to the core group should work on the HEAD to help uncover problems from day 1. Code changes should *not* be collected by individual programmers until release time . This is a major source of frustration, because you simply may not expect your team members to spent day and night to fight the 'last' bugs. Let alone to quick-test your latest feature or performance improvement. PLEASE, check-in daily your changes and make sure you have at least ran mtest on your platform. And most of all, know that your changes are scrutinized for potential quality problems by others. It is the best way to avoid many bugs.
Furthermore, changes in the interfaces visible to users and the effect on their databases should be kept to a minimum. Backward compatibility is a must as long as it is manageable. Beware that a change in storage structure may help a few extreme users, but may affects hundreds or thousands in their daily use. E.g. one of our users would have to unload/reload a 0.5Tb business critical application.
MonetDB/Query and MonetDB/SQL are intertwined. This creates responsibility both ways.
Peter Boncz wrote:
Stefan suggested rightly to move this discussion to the mailing list. I also changed the language into english for that purpose.
Maurice' original questions were:
(a) whether the new release would be algebra-only (short answer: hopefully).
In my opinion this can only be answered if the algebra version, in its full glory (without mps) is used for several months internally, e.g. using students for stress testing.
(b) whether our talk of a new release also refers to submodules, such as probxml and pftijah (short answer: yes)
Submodules basically follow their own release cycle. Regardless the source, e.g. datacells geom, .... Keeping in line is the responsibility of their owners, and the core developers should be aware that their changes may break the work of others. Or at least drain a possibly substantial amount of time.
(c) whether all issues on the wiki, see: http://www.pathfinder-xquery.org/wiki/index.php/Jupiter:_The_Algebra_Release needed for this action, have already been resolved (short answer: no)
The pathfinder wiki is a good example of team documentation.
I am looking forward to the upcoming release. The next one might be this year, but not necessarily includes everything on the drawing board.
regards, Martin
On Tue, Oct 16, 2007 at 05:23:43PM +0200, Peter Boncz wrote:
Hi Maurice, et al,
The release of (hopefully) this week will still be mps based. I also anticipate at least one bug-fix release (ie 0.20.2), or even more. However, the next major release (0.22) should be algebra based, in my opinion. I am certainly reluctant to keep investing any more effort in mps. I would aim for an mps-free release, yes.
The time-goal of that release is "XMAS 2007", as far as we are concerned. For the work of built-in functions I have good hope that with the help of Lefteris we can come a long way.
I hope and think updates can also be ported by joerd in that time-frame. As for pftijah, if Jan Flokstra has time, it should be possible to port that as well in 2-3 months.
There are some hairy issues, one of them being free-form recursion (needed by you), another one being precompiled modules (needed by XRPC). The resolution of these needs assistance from Munich, and maybe related. It needs a method to compile certain functions into scripted procedures (here MIL) and calling them. One algorithm is to do this for all functions in modules, as well as for all recursive functions. The issue to resolve is the representation of the parameters.
Depending on how much time Jan Rittinger has available for this, we will have to find creative solutions for this. Indeed, a backup solution would be to keep handling certain queries for the time being with mps, but that surely is ugly.
Peter
-----Original Message----- From: Keulen, M. van (Maurice) [mailto:m.vankeulen@utwente.nl] Sent: dinsdag 16 oktober 2007 16:38 To: Peter Boncz [CWI]; Stefan Manegold [CWI] Cc: Djoerd Hiemstra; Jan Flokstra; Henning Rode; Abdel Kader, R. (Riham) Subject: Nieuwe releases en mps-backend
Hoi,
Sorry dat ik niet bij de Skype-sessie van vandaag kon zijn, maar Riham vertelde me dat het onderwerp van nieuwe releases en de mps-backend ter sprake is geweest. Het riep wat specifieke vragen bij me op die ze niet allemaal kon beantwoorden:
a.. Is het de bedoeling dat een nieuwe release de mps-backend helemaalniet meer bevat of alleen maar de algebra-backend als default heeft? b.. Als jullie praten over "nieuwe release", bedoel je dan alleen een package van alle modules van MonetDB/XQuery? Maw zegt het niets over eventuele `subreleases' van individuele modules?
c.. Op de wiki hebben we een pagina met open issues voor dealgebra-release. Wat is de status van die open issues? Ik zie namelijk op de pagina nergens dat bepaalde issues al opgelost/geïmplementeerd zijn. Ik Cc deze e-mail even aan de PF/Tijah-mensen, want hoewel er levendige discussies zijn hoe PF/Tijah naar de algebra-backend to porten, ik heb niet het idee dat er oplossingen zijn die op korte termijn realiseerbaar zijn. Evenzo het recursie-vraagstuk. Het lijkt me bijvoorbeeld voor de zichtbaarheid naar buiten toe onwenselijk als deze features ineens zouden wegvallen voor non-insiders. Of andere onwenselijke consequenties van het niet meer (sub)releasen van modules die gebaseerd zijn op de mps-backend.
Groetjes, Maurice.
--
Dr.Ir. M. van Keulen - Assistant Professor, Data Management Technology Univ. of Twente, Dept of EEMCS, POBox 217, 7500 AE Enschede, Netherlands Email: m.vankeulen@utwente.nl, Phone: +31 534893688, Fax: +31 534892927 Room: ZI 3039, WWW: http://www.cs.utwente.nl/~keulen
This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
participants (5)
-
Djoerd Hiemstra -
Keulen, M. van (Maurice) -
Martin Kersten -
Peter Boncz -
Stefan Manegold