[Monetdb-developers] next release: mps or algebra?

Keulen, M. van (Maurice) m.vankeulen at utwente.nl
Wed Oct 17 10:15:17 CEST 2007


Hi all,

As a follow-up to Djoerd's response and my questions about releases, let 
me say a little bit more about certain activities and sketch an analogy 
which we might mimick.

We are trying to get many people enthusiastic about XML databases and 
XML-IR, as offered by MonetDB/XQuery in particular. To achieve some 
visibility for our work, I have for example started several activities 
in building production applications on top of MonetDB/XQuery and which 
use PF/Tijah.  For example, I promised a demonstration in january to 
university-wide decision makers of the UT of a publication management 
application which uses XML-exports from our E-Prints system including 
the full-text of the papers. I hope it will be accepted as a production 
application used by all researchers of the UT next to E-Prints, which 
will over time demonstrate more advanced features and take over more and 
more functionality of E-Prints. As Djoerd pointed out, porting PF/Tijah 
to the algebra-backend in the proper way, is more a matter of a year 
than of a few months. However, it would be certain death to such 
production applications if the support for developing them would stop 
(since algebra-only means without PF/Tijah for a long time). 
Furthermore, many students following our courses and doing our projects 
are very interested in XML databases including XML-IR and probabilistic 
XML. A significant portion would be rather discouraged if they would not 
be able to grab a stable release to start working and get some bugs fixed.

Microsoft once had the problem that they had ideas for a fundamentally 
different operating system kernel (NT). It was infeasible to make it 
fully backward-compatible with Windows 95 (on top of MSDOS), but it 
could also not abandon a large part of their userbase if their 
applications would not work anymore. Therefore, they had two release 
streams: one NT-based and one MSDOS-based. Both knew their releases 
until in the end. When most of the applications were ported or at least 
working on the new technology, they stopped releasing in the MSDOS-based 
stream.

The algebra-backend is our `new technology', so can't we do something 
similar? Have an algebra-based release stream of MonetDB/XQuery 
(starting with 0.22) capable of running a significant portion of 
applications, and an mps-based release stream needed to serve the 
PF/Tijah- and ProbXML-applications. It would mean an extra package under 
"Download" on SF. Of course the algebra-based releases will have 
priority, but it would make it possible to have a new release for for 
example PF/Tijah when it is needed. Of course we will put in the 
required effort, but cannot do this alone.

What do you think of this?

Maurice.

Djoerd Hiemstra wrote:
> Dear all,
>
> Very good: no more milprinting! In absence of Jan, please let me add the 
> plans for PF/Tijah. To make things slightly more complicated: The PF/Tijah 
> team would like to use the algebra release for porting our proprietary 
> search extensions to XQuery Full-text, see: 
> http://www.w3.org/TR/xquery-full-text/ Some initial design decisions that 
> we like to follow have been made by Stefan Klinger at the University of 
> Konstanz. The consequences of such a port would be:
>
>  1. XQuery syntax will be extended for text search queries (now only via
>     user-defined functions)
>  2. All algebra operations must be score aware (i.e., if a score column
>     exists, scores have to be propagated and aggregated)
>
> We are aware that this affects everyone, although, only the extended 
> queries will produce a score column, and algebra operations remain exactly 
> the same if score columns are not there. The time goal of this release is 
> XMAS 2008. In the mean time, we might not port the current PF/Tijah 
> extensions to the algebra.
>
> Best,  Djoerd.
>
> On Tue, 16 Oct 2007, Martin Kersten wrote:
>
>   
>> LS,
>>
>> Good to see this discussion. Although the topic is primarily 
>> MonetDB/XQuery it is good
>> to keep an eye on the global picture. Releases include more stuff and 
>> requires a lot of
>> time to prepare. (the release itself easily consumes several months of 
>> full time work,
>> many delivered by free (weekend) time!!!!)
>> Good planning and long-term internal use are a prerequisite.
>> Something we still have to learn.
>> Once the release cycle starts, the train can not stop and full power 
>> is/should
>> be in the hands of a single person. The release manager currently is Sjoerd,
>> one of the project admins!
>>
>> As soon as a RC is available, those with a non-standard platform 
>> (windows, sun,
>> mac, vista) should immediately double check to assure it basically works.
>> This holds for all addressed and developers.
>>
>> In this respect, the time frame mentioned below is overly optimistic. 
>> Anyone close to
>> the core group should work on the HEAD to help uncover problems from day 1.
>> Code changes should *not* be collected by individual programmers until
>> release time .
>> This is a major source of frustration, because you simply may not expect
>> your team members to spent day and night to fight the 'last' bugs. Let 
>> alone to
>> quick-test your latest feature or performance improvement.
>> PLEASE, check-in daily your changes and make sure you have at least ran
>> mtest on your platform. And most of all, know that your changes are
>> scrutinized for potential quality problems by others. It is the best way
>> to avoid many bugs.
>>
>> Furthermore, changes in the interfaces visible to users and the effect 
>> on their databases
>> should be kept to a minimum. Backward compatibility is a must as long as 
>> it is manageable.
>> Beware that a change in storage structure may help a few extreme users, 
>> but may affects
>> hundreds or thousands in their daily use. E.g. one of our users would 
>> have to unload/reload
>> a 0.5Tb business critical application.
>>
>> MonetDB/Query and MonetDB/SQL are intertwined. This creates responsibility
>> both ways.
>>
>> Peter Boncz wrote:
>>     
>>> Stefan suggested rightly to move this discussion to the mailing list.
>>> I also changed the language into english for that purpose.
>>>
>>> Maurice' original questions were:
>>>
>>> (a) whether the new release would be algebra-only (short answer: hopefully).
>>>       
>> In my opinion this can only be answered if the algebra version, in its 
>> full glory (without mps)
>> is used for several months internally, e.g. using students for stress 
>> testing.
>>     
>>> (b) whether our talk of a new release also refers to submodules, such as
>>> probxml and pftijah (short answer: yes)
>>>   
>>>       
>> Submodules basically follow their own release cycle. Regardless the 
>> source, e.g. datacells
>> geom, ....  Keeping in line is the responsibility of their owners, and 
>> the core developers
>> should be aware that their changes may break the work of others. Or at 
>> least drain a
>> possibly substantial amount of time.
>>     
>>> (c) whether all issues on the wiki, see:
>>> http://www.pathfinder-xquery.org/wiki/index.php/Jupiter:_The_Algebra_Release
>>>     needed for this action, have already been resolved (short answer: no)
>>>   
>>>       
>> The pathfinder wiki is a good example of team documentation.
>>
>> I am looking forward to the upcoming release. The next one might be this 
>> year,
>> but not necessarily includes everything on the drawing board.
>>
>> regards, Martin
>>     
>>> On Tue, Oct 16, 2007 at 05:23:43PM +0200, Peter Boncz wrote:
>>>   
>>>       
>>>> Hi Maurice, et al,
>>>>
>>>> The release of (hopefully) this week will still be mps based. I also
>>>> anticipate at least one bug-fix release (ie 0.20.2), or even more.  However,
>>>> the next major release (0.22) should be algebra based, in my opinion. I am
>>>> certainly reluctant to keep investing any more effort in mps. I would aim
>>>> for an mps-free release, yes.
>>>>
>>>> The time-goal of that release is "XMAS 2007", as far as we are concerned.
>>>> For the work of built-in functions I have good hope that with the help of
>>>> Lefteris we can come a long way.
>>>>
>>>> I hope and think updates can also be ported by joerd in that time-frame. As
>>>> for pftijah, if Jan Flokstra has time, it should be possible to port that as
>>>> well in 2-3 months.
>>>>
>>>> There are some hairy issues, one of them being free-form recursion (needed
>>>> by you), another one being precompiled modules (needed by XRPC). The
>>>> resolution of these needs assistance from Munich, and maybe related. It needs
>>>> a method to compile certain functions into scripted procedures (here MIL)
>>>> and calling them. One algorithm is to do this for all functions in modules,
>>>> as well as for all recursive functions. The issue to resolve is the
>>>> representation of the parameters.
>>>>
>>>> Depending on how much time Jan Rittinger has available for this, we will
>>>> have to find creative solutions for this. Indeed, a backup solution would be
>>>> to keep handling certain queries for the time being with mps, but that
>>>> surely is ugly.
>>>>
>>>> Peter
>>>>
>>>>   -----Original Message-----
>>>>   From: Keulen, M. van (Maurice) [mailto:m.vankeulen at utwente.nl]
>>>>   Sent: dinsdag 16 oktober 2007 16:38
>>>>   To: Peter Boncz [CWI]; Stefan Manegold [CWI]
>>>>   Cc: Djoerd Hiemstra; Jan Flokstra; Henning Rode; Abdel Kader, R. (Riham)
>>>>   Subject: Nieuwe releases en mps-backend
>>>>
>>>>
>>>>   Hoi,
>>>>
>>>>   Sorry dat ik niet bij de Skype-sessie van vandaag kon zijn, maar Riham
>>>> vertelde me dat het onderwerp van nieuwe releases en de mps-backend ter
>>>> sprake is geweest. Het riep wat specifieke vragen bij me op die ze niet
>>>> allemaal kon beantwoorden:
>>>>
>>>>     a.. Is het de bedoeling dat een nieuwe release de mps-backend helemaal
>>>> niet meer bevat of alleen maar de algebra-backend als default heeft?
>>>>     b.. Als jullie praten over "nieuwe release", bedoel je dan alleen een
>>>> package van alle modules van MonetDB/XQuery? Maw zegt het niets over
>>>> eventuele `subreleases' van individuele modules?
>>>>
>>>>     c.. Op de wiki hebben we een pagina met open issues voor de
>>>> algebra-release. Wat is de status van die open issues? Ik zie namelijk op de
>>>> pagina nergens dat bepaalde issues al opgelost/geïmplementeerd zijn.
>>>>   Ik Cc deze e-mail even aan de PF/Tijah-mensen, want  hoewel er levendige
>>>> discussies zijn hoe PF/Tijah naar de algebra-backend to porten, ik heb niet
>>>> het idee dat er oplossingen zijn die op korte termijn realiseerbaar zijn.
>>>> Evenzo het recursie-vraagstuk. Het lijkt me bijvoorbeeld voor de
>>>> zichtbaarheid naar buiten toe onwenselijk als deze features ineens zouden
>>>> wegvallen voor non-insiders. Of andere onwenselijke consequenties van het
>>>> niet meer (sub)releasen van modules die gebaseerd zijn op de mps-backend.
>>>>
>>>>   Groetjes,
>>>>   Maurice.
>>>>
>>>> --
>>>> ----------------------------------------------------------------------
>>>> Dr.Ir. M. van Keulen - Assistant Professor, Data Management Technology
>>>> Univ. of Twente, Dept of EEMCS, POBox 217, 7500 AE Enschede, Netherlands
>>>> Email: m.vankeulen at utwente.nl, Phone: +31 534893688, Fax: +31 534892927
>>>> Room: ZI 3039, WWW: http://www.cs.utwente.nl/~keulen
>>>>     
>>>>         
>>>   
>>>       
>> -------------------------------------------------------------------------
>> This SF.net email is sponsored by: Splunk Inc.
>> Still grepping through log files to find problems?  Stop.
>> Now Search log events and configuration files using AJAX and a browser.
>> Download your FREE copy of Splunk now >> http://get.splunk.com/
>> _______________________________________________
>> Monetdb-developers mailing list
>> Monetdb-developers at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/monetdb-developers
>>     
>> ------------------------------------------------------------------------
>>
>> -------------------------------------------------------------------------
>> This SF.net email is sponsored by: Splunk Inc.
>> Still grepping through log files to find problems?  Stop.
>> Now Search log events and configuration files using AJAX and a browser.
>> Download your FREE copy of Splunk now >> http://get.splunk.com/
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Monetdb-developers mailing list
>> Monetdb-developers at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/monetdb-developers
>>     

-- 
----------------------------------------------------------------------
Dr.Ir. M. van Keulen - Assistant Professor, Data Management Technology
Univ. of Twente, Dept of EEMCS, POBox 217, 7500 AE Enschede, Netherlands
Email: m.vankeulen at utwente.nl, Phone: +31 534893688, Fax: +31 534892927
Room: ZI 3039, WWW: http://www.cs.utwente.nl/~keulen





More information about the developers-list mailing list