[Monetdb-developers] next release: mps or algebra?

Stefan Manegold Stefan.Manegold at cwi.nl
Wed Oct 17 10:27:41 CEST 2007


Hi Maurice,

valid point. 

Unfortunately, we (at least here at CWI) do not have the manpower an/or
budget that Microsoft has (recall, our "clients" do not pay for our
"product").

Hence, if a dual-path strategy is desired and/or required, please also
consider who should be doing all the administration, maintenance, bug
fixing, testing, releasing, etc...

Stefan

On Wed, Oct 17, 2007 at 10:15:17AM +0200, Keulen, M. van (Maurice) wrote:
> Hi all,
> 
> As a follow-up to Djoerd's response and my questions about releases, let 
> me say a little bit more about certain activities and sketch an analogy 
> which we might mimick.
> 
> We are trying to get many people enthusiastic about XML databases and 
> XML-IR, as offered by MonetDB/XQuery in particular. To achieve some 
> visibility for our work, I have for example started several activities 
> in building production applications on top of MonetDB/XQuery and which 
> use PF/Tijah.  For example, I promised a demonstration in january to 
> university-wide decision makers of the UT of a publication management 
> application which uses XML-exports from our E-Prints system including 
> the full-text of the papers. I hope it will be accepted as a production 
> application used by all researchers of the UT next to E-Prints, which 
> will over time demonstrate more advanced features and take over more and 
> more functionality of E-Prints. As Djoerd pointed out, porting PF/Tijah 
> to the algebra-backend in the proper way, is more a matter of a year 
> than of a few months. However, it would be certain death to such 
> production applications if the support for developing them would stop 
> (since algebra-only means without PF/Tijah for a long time). 
> Furthermore, many students following our courses and doing our projects 
> are very interested in XML databases including XML-IR and probabilistic 
> XML. A significant portion would be rather discouraged if they would not 
> be able to grab a stable release to start working and get some bugs fixed.
> 
> Microsoft once had the problem that they had ideas for a fundamentally 
> different operating system kernel (NT). It was infeasible to make it 
> fully backward-compatible with Windows 95 (on top of MSDOS), but it 
> could also not abandon a large part of their userbase if their 
> applications would not work anymore. Therefore, they had two release 
> streams: one NT-based and one MSDOS-based. Both knew their releases 
> until in the end. When most of the applications were ported or at least 
> working on the new technology, they stopped releasing in the MSDOS-based 
> stream.
> 
> The algebra-backend is our `new technology', so can't we do something 
> similar? Have an algebra-based release stream of MonetDB/XQuery 
> (starting with 0.22) capable of running a significant portion of 
> applications, and an mps-based release stream needed to serve the 
> PF/Tijah- and ProbXML-applications. It would mean an extra package under 
> "Download" on SF. Of course the algebra-based releases will have 
> priority, but it would make it possible to have a new release for for 
> example PF/Tijah when it is needed. Of course we will put in the 
> required effort, but cannot do this alone.
> 
> What do you think of this?
> 
> Maurice.
> 
> Djoerd Hiemstra wrote:
> > Dear all,
> >
> > Very good: no more milprinting! In absence of Jan, please let me add the 
> > plans for PF/Tijah. To make things slightly more complicated: The PF/Tijah 
> > team would like to use the algebra release for porting our proprietary 
> > search extensions to XQuery Full-text, see: 
> > http://www.w3.org/TR/xquery-full-text/ Some initial design decisions that 
> > we like to follow have been made by Stefan Klinger at the University of 
> > Konstanz. The consequences of such a port would be:
> >
> >  1. XQuery syntax will be extended for text search queries (now only via
> >     user-defined functions)
> >  2. All algebra operations must be score aware (i.e., if a score column
> >     exists, scores have to be propagated and aggregated)
> >
> > We are aware that this affects everyone, although, only the extended 
> > queries will produce a score column, and algebra operations remain exactly 
> > the same if score columns are not there. The time goal of this release is 
> > XMAS 2008. In the mean time, we might not port the current PF/Tijah 
> > extensions to the algebra.
> >
> > Best,  Djoerd.
> >
> > On Tue, 16 Oct 2007, Martin Kersten wrote:
> >
> >   
> >> LS,
> >>
> >> Good to see this discussion. Although the topic is primarily 
> >> MonetDB/XQuery it is good
> >> to keep an eye on the global picture. Releases include more stuff and 
> >> requires a lot of
> >> time to prepare. (the release itself easily consumes several months of 
> >> full time work,
> >> many delivered by free (weekend) time!!!!)
> >> Good planning and long-term internal use are a prerequisite.
> >> Something we still have to learn.
> >> Once the release cycle starts, the train can not stop and full power 
> >> is/should
> >> be in the hands of a single person. The release manager currently is Sjoerd,
> >> one of the project admins!
> >>
> >> As soon as a RC is available, those with a non-standard platform 
> >> (windows, sun,
> >> mac, vista) should immediately double check to assure it basically works.
> >> This holds for all addressed and developers.
> >>
> >> In this respect, the time frame mentioned below is overly optimistic. 
> >> Anyone close to
> >> the core group should work on the HEAD to help uncover problems from day 1.
> >> Code changes should *not* be collected by individual programmers until
> >> release time .
> >> This is a major source of frustration, because you simply may not expect
> >> your team members to spent day and night to fight the 'last' bugs. Let 
> >> alone to
> >> quick-test your latest feature or performance improvement.
> >> PLEASE, check-in daily your changes and make sure you have at least ran
> >> mtest on your platform. And most of all, know that your changes are
> >> scrutinized for potential quality problems by others. It is the best way
> >> to avoid many bugs.
> >>
> >> Furthermore, changes in the interfaces visible to users and the effect 
> >> on their databases
> >> should be kept to a minimum. Backward compatibility is a must as long as 
> >> it is manageable.
> >> Beware that a change in storage structure may help a few extreme users, 
> >> but may affects
> >> hundreds or thousands in their daily use. E.g. one of our users would 
> >> have to unload/reload
> >> a 0.5Tb business critical application.
> >>
> >> MonetDB/Query and MonetDB/SQL are intertwined. This creates responsibility
> >> both ways.
> >>
> >> Peter Boncz wrote:
> >>     
> >>> Stefan suggested rightly to move this discussion to the mailing list.
> >>> I also changed the language into english for that purpose.
> >>>
> >>> Maurice' original questions were:
> >>>
> >>> (a) whether the new release would be algebra-only (short answer: hopefully).
> >>>       
> >> In my opinion this can only be answered if the algebra version, in its 
> >> full glory (without mps)
> >> is used for several months internally, e.g. using students for stress 
> >> testing.
> >>     
> >>> (b) whether our talk of a new release also refers to submodules, such as
> >>> probxml and pftijah (short answer: yes)
> >>>   
> >>>       
> >> Submodules basically follow their own release cycle. Regardless the 
> >> source, e.g. datacells
> >> geom, ....  Keeping in line is the responsibility of their owners, and 
> >> the core developers
> >> should be aware that their changes may break the work of others. Or at 
> >> least drain a
> >> possibly substantial amount of time.
> >>     
> >>> (c) whether all issues on the wiki, see:
> >>> http://www.pathfinder-xquery.org/wiki/index.php/Jupiter:_The_Algebra_Release
> >>>     needed for this action, have already been resolved (short answer: no)
> >>>   
> >>>       
> >> The pathfinder wiki is a good example of team documentation.
> >>
> >> I am looking forward to the upcoming release. The next one might be this 
> >> year,
> >> but not necessarily includes everything on the drawing board.
> >>
> >> regards, Martin
> >>     
> >>> On Tue, Oct 16, 2007 at 05:23:43PM +0200, Peter Boncz wrote:
> >>>   
> >>>       
> >>>> Hi Maurice, et al,
> >>>>
> >>>> The release of (hopefully) this week will still be mps based. I also
> >>>> anticipate at least one bug-fix release (ie 0.20.2), or even more.  However,
> >>>> the next major release (0.22) should be algebra based, in my opinion. I am
> >>>> certainly reluctant to keep investing any more effort in mps. I would aim
> >>>> for an mps-free release, yes.
> >>>>
> >>>> The time-goal of that release is "XMAS 2007", as far as we are concerned.
> >>>> For the work of built-in functions I have good hope that with the help of
> >>>> Lefteris we can come a long way.
> >>>>
> >>>> I hope and think updates can also be ported by joerd in that time-frame. As
> >>>> for pftijah, if Jan Flokstra has time, it should be possible to port that as
> >>>> well in 2-3 months.
> >>>>
> >>>> There are some hairy issues, one of them being free-form recursion (needed
> >>>> by you), another one being precompiled modules (needed by XRPC). The
> >>>> resolution of these needs assistance from Munich, and maybe related. It needs
> >>>> a method to compile certain functions into scripted procedures (here MIL)
> >>>> and calling them. One algorithm is to do this for all functions in modules,
> >>>> as well as for all recursive functions. The issue to resolve is the
> >>>> representation of the parameters.
> >>>>
> >>>> Depending on how much time Jan Rittinger has available for this, we will
> >>>> have to find creative solutions for this. Indeed, a backup solution would be
> >>>> to keep handling certain queries for the time being with mps, but that
> >>>> surely is ugly.
> >>>>
> >>>> Peter
> >>>>
> >>>>   -----Original Message-----
> >>>>   From: Keulen, M. van (Maurice) [mailto:m.vankeulen at utwente.nl]
> >>>>   Sent: dinsdag 16 oktober 2007 16:38
> >>>>   To: Peter Boncz [CWI]; Stefan Manegold [CWI]
> >>>>   Cc: Djoerd Hiemstra; Jan Flokstra; Henning Rode; Abdel Kader, R. (Riham)
> >>>>   Subject: Nieuwe releases en mps-backend
> >>>>
> >>>>
> >>>>   Hoi,
> >>>>
> >>>>   Sorry dat ik niet bij de Skype-sessie van vandaag kon zijn, maar Riham
> >>>> vertelde me dat het onderwerp van nieuwe releases en de mps-backend ter
> >>>> sprake is geweest. Het riep wat specifieke vragen bij me op die ze niet
> >>>> allemaal kon beantwoorden:
> >>>>
> >>>>     a.. Is het de bedoeling dat een nieuwe release de mps-backend helemaal
> >>>> niet meer bevat of alleen maar de algebra-backend als default heeft?
> >>>>     b.. Als jullie praten over "nieuwe release", bedoel je dan alleen een
> >>>> package van alle modules van MonetDB/XQuery? Maw zegt het niets over
> >>>> eventuele `subreleases' van individuele modules?
> >>>>
> >>>>     c.. Op de wiki hebben we een pagina met open issues voor de
> >>>> algebra-release. Wat is de status van die open issues? Ik zie namelijk op de
> >>>> pagina nergens dat bepaalde issues al opgelost/geïmplementeerd zijn.
> >>>>   Ik Cc deze e-mail even aan de PF/Tijah-mensen, want  hoewel er levendige
> >>>> discussies zijn hoe PF/Tijah naar de algebra-backend to porten, ik heb niet
> >>>> het idee dat er oplossingen zijn die op korte termijn realiseerbaar zijn.
> >>>> Evenzo het recursie-vraagstuk. Het lijkt me bijvoorbeeld voor de
> >>>> zichtbaarheid naar buiten toe onwenselijk als deze features ineens zouden
> >>>> wegvallen voor non-insiders. Of andere onwenselijke consequenties van het
> >>>> niet meer (sub)releasen van modules die gebaseerd zijn op de mps-backend.
> >>>>
> >>>>   Groetjes,
> >>>>   Maurice.
> >>>>
> >>>> --
> >>>> ----------------------------------------------------------------------
> >>>> Dr.Ir. M. van Keulen - Assistant Professor, Data Management Technology
> >>>> Univ. of Twente, Dept of EEMCS, POBox 217, 7500 AE Enschede, Netherlands
> >>>> Email: m.vankeulen at utwente.nl, Phone: +31 534893688, Fax: +31 534892927
> >>>> Room: ZI 3039, WWW: http://www.cs.utwente.nl/~keulen
> >>>>     
> >>>>         
> >>>   
> >>>       
> >> -------------------------------------------------------------------------
> >> This SF.net email is sponsored by: Splunk Inc.
> >> Still grepping through log files to find problems?  Stop.
> >> Now Search log events and configuration files using AJAX and a browser.
> >> Download your FREE copy of Splunk now >> http://get.splunk.com/
> >> _______________________________________________
> >> Monetdb-developers mailing list
> >> Monetdb-developers at lists.sourceforge.net
> >> https://lists.sourceforge.net/lists/listinfo/monetdb-developers
> >>     
> >> ------------------------------------------------------------------------
> >>
> >> -------------------------------------------------------------------------
> >> This SF.net email is sponsored by: Splunk Inc.
> >> Still grepping through log files to find problems?  Stop.
> >> Now Search log events and configuration files using AJAX and a browser.
> >> Download your FREE copy of Splunk now >> http://get.splunk.com/
> >> ------------------------------------------------------------------------
> >>
> >> _______________________________________________
> >> Monetdb-developers mailing list
> >> Monetdb-developers at lists.sourceforge.net
> >> https://lists.sourceforge.net/lists/listinfo/monetdb-developers
> >>     
> 
> -- 
> ----------------------------------------------------------------------
> Dr.Ir. M. van Keulen - Assistant Professor, Data Management Technology
> Univ. of Twente, Dept of EEMCS, POBox 217, 7500 AE Enschede, Netherlands
> Email: m.vankeulen at utwente.nl, Phone: +31 534893688, Fax: +31 534892927
> Room: ZI 3039, WWW: http://www.cs.utwente.nl/~keulen
> 
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems?  Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >> http://get.splunk.com/
> _______________________________________________
> Monetdb-developers mailing list
> Monetdb-developers at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/monetdb-developers

-- 
| Dr. Stefan Manegold | mailto:Stefan.Manegold at cwi.nl |
| CWI,  P.O.Box 94079 | http://www.cwi.nl/~manegold/  |
| 1090 GB Amsterdam   | Tel.: +31 (20) 592-4212       |
| The Netherlands     | Fax : +31 (20) 592-4312       |




More information about the developers-list mailing list