[Monetdb-developers] next release: mps or algebra?

Keulen, M. van (Maurice) m.vankeulen at utwente.nl
Wed Oct 17 16:11:53 CEST 2007


Hi Stefan,

Valid point yourself :-)

I wholeheartedly support a swift switch to the algebra-backend. What I'm 
basically looking for is a convenient way to keep on having others 
developing applications on top of MXQ/PFTijah whereby prohibitive bugs 
can still be solved for these people.

More thoughts:
I presume the mps-code will remain in the CVS tree for as long as it 
takes, so bugs could be resolved, but others can only benefit if they 
would use a compilation of the CVS tree. Is it perhaps possible to 
easily create installation packages for the HEAD-version without 
actually releasing these packages as a formal release? We could 
'release' these packages by putting them in the download-section of the 
PF/Tijah website. This would not make any claims concerning testing and 
stability, since it would be only a packaged new version of PF/Tijah 
with no formal status as release, but still allow new versions of 
PF/Tijah from being packaged and made available.

The downside of such an approach is that from the perspective of people 
in the outside world looking at MonetDB/XQuery as an open source 
product, MonetDb/XQuery will loose some of its distinguishing features 
(fulltext querying capability) for a long time. Is that what you want?

Any thoughts?

Maurice.


Stefan Manegold wrote:
> Hi Maurice,
>
> valid point. 
>
> Unfortunately, we (at least here at CWI) do not have the manpower an/or
> budget that Microsoft has (recall, our "clients" do not pay for our
> "product").
>
> Hence, if a dual-path strategy is desired and/or required, please also
> consider who should be doing all the administration, maintenance, bug
> fixing, testing, releasing, etc...
>
> Stefan
>
> On Wed, Oct 17, 2007 at 10:15:17AM +0200, Keulen, M. van (Maurice) wrote:
>   
>> Hi all,
>>
>> As a follow-up to Djoerd's response and my questions about releases, let 
>> me say a little bit more about certain activities and sketch an analogy 
>> which we might mimick.
>>
>> We are trying to get many people enthusiastic about XML databases and 
>> XML-IR, as offered by MonetDB/XQuery in particular. To achieve some 
>> visibility for our work, I have for example started several activities 
>> in building production applications on top of MonetDB/XQuery and which 
>> use PF/Tijah.  For example, I promised a demonstration in january to 
>> university-wide decision makers of the UT of a publication management 
>> application which uses XML-exports from our E-Prints system including 
>> the full-text of the papers. I hope it will be accepted as a production 
>> application used by all researchers of the UT next to E-Prints, which 
>> will over time demonstrate more advanced features and take over more and 
>> more functionality of E-Prints. As Djoerd pointed out, porting PF/Tijah 
>> to the algebra-backend in the proper way, is more a matter of a year 
>> than of a few months. However, it would be certain death to such 
>> production applications if the support for developing them would stop 
>> (since algebra-only means without PF/Tijah for a long time). 
>> Furthermore, many students following our courses and doing our projects 
>> are very interested in XML databases including XML-IR and probabilistic 
>> XML. A significant portion would be rather discouraged if they would not 
>> be able to grab a stable release to start working and get some bugs fixed.
>>
>> Microsoft once had the problem that they had ideas for a fundamentally 
>> different operating system kernel (NT). It was infeasible to make it 
>> fully backward-compatible with Windows 95 (on top of MSDOS), but it 
>> could also not abandon a large part of their userbase if their 
>> applications would not work anymore. Therefore, they had two release 
>> streams: one NT-based and one MSDOS-based. Both knew their releases 
>> until in the end. When most of the applications were ported or at least 
>> working on the new technology, they stopped releasing in the MSDOS-based 
>> stream.
>>
>> The algebra-backend is our `new technology', so can't we do something 
>> similar? Have an algebra-based release stream of MonetDB/XQuery 
>> (starting with 0.22) capable of running a significant portion of 
>> applications, and an mps-based release stream needed to serve the 
>> PF/Tijah- and ProbXML-applications. It would mean an extra package under 
>> "Download" on SF. Of course the algebra-based releases will have 
>> priority, but it would make it possible to have a new release for for 
>> example PF/Tijah when it is needed. Of course we will put in the 
>> required effort, but cannot do this alone.
>>
>> What do you think of this?
>>
>> Maurice.
>>
>> Djoerd Hiemstra wrote:
>>     
>>> Dear all,
>>>
>>> Very good: no more milprinting! In absence of Jan, please let me add the 
>>> plans for PF/Tijah. To make things slightly more complicated: The PF/Tijah 
>>> team would like to use the algebra release for porting our proprietary 
>>> search extensions to XQuery Full-text, see: 
>>> http://www.w3.org/TR/xquery-full-text/ Some initial design decisions that 
>>> we like to follow have been made by Stefan Klinger at the University of 
>>> Konstanz. The consequences of such a port would be:
>>>
>>>  1. XQuery syntax will be extended for text search queries (now only via
>>>     user-defined functions)
>>>  2. All algebra operations must be score aware (i.e., if a score column
>>>     exists, scores have to be propagated and aggregated)
>>>
>>> We are aware that this affects everyone, although, only the extended 
>>> queries will produce a score column, and algebra operations remain exactly 
>>> the same if score columns are not there. The time goal of this release is 
>>> XMAS 2008. In the mean time, we might not port the current PF/Tijah 
>>> extensions to the algebra.
>>>
>>> Best,  Djoerd.
>>>
>>> On Tue, 16 Oct 2007, Martin Kersten wrote:
>>>
>>>   
>>>       
>>>> LS,
>>>>
>>>> Good to see this discussion. Although the topic is primarily 
>>>> MonetDB/XQuery it is good
>>>> to keep an eye on the global picture. Releases include more stuff and 
>>>> requires a lot of
>>>> time to prepare. (the release itself easily consumes several months of 
>>>> full time work,
>>>> many delivered by free (weekend) time!!!!)
>>>> Good planning and long-term internal use are a prerequisite.
>>>> Something we still have to learn.
>>>> Once the release cycle starts, the train can not stop and full power 
>>>> is/should
>>>> be in the hands of a single person. The release manager currently is Sjoerd,
>>>> one of the project admins!
>>>>
>>>> As soon as a RC is available, those with a non-standard platform 
>>>> (windows, sun,
>>>> mac, vista) should immediately double check to assure it basically works.
>>>> This holds for all addressed and developers.
>>>>
>>>> In this respect, the time frame mentioned below is overly optimistic. 
>>>> Anyone close to
>>>> the core group should work on the HEAD to help uncover problems from day 1.
>>>> Code changes should *not* be collected by individual programmers until
>>>> release time .
>>>> This is a major source of frustration, because you simply may not expect
>>>> your team members to spent day and night to fight the 'last' bugs. Let 
>>>> alone to
>>>> quick-test your latest feature or performance improvement.
>>>> PLEASE, check-in daily your changes and make sure you have at least ran
>>>> mtest on your platform. And most of all, know that your changes are
>>>> scrutinized for potential quality problems by others. It is the best way
>>>> to avoid many bugs.
>>>>
>>>> Furthermore, changes in the interfaces visible to users and the effect 
>>>> on their databases
>>>> should be kept to a minimum. Backward compatibility is a must as long as 
>>>> it is manageable.
>>>> Beware that a change in storage structure may help a few extreme users, 
>>>> but may affects
>>>> hundreds or thousands in their daily use. E.g. one of our users would 
>>>> have to unload/reload
>>>> a 0.5Tb business critical application.
>>>>
>>>> MonetDB/Query and MonetDB/SQL are intertwined. This creates responsibility
>>>> both ways.
>>>>
>>>> Peter Boncz wrote:
>>>>     
>>>>         
>>>>> Stefan suggested rightly to move this discussion to the mailing list.
>>>>> I also changed the language into english for that purpose.
>>>>>
>>>>> Maurice' original questions were:
>>>>>
>>>>> (a) whether the new release would be algebra-only (short answer: hopefully).
>>>>>       
>>>>>           
>>>> In my opinion this can only be answered if the algebra version, in its 
>>>> full glory (without mps)
>>>> is used for several months internally, e.g. using students for stress 
>>>> testing.
>>>>     
>>>>         
>>>>> (b) whether our talk of a new release also refers to submodules, such as
>>>>> probxml and pftijah (short answer: yes)
>>>>>   
>>>>>       
>>>>>           
>>>> Submodules basically follow their own release cycle. Regardless the 
>>>> source, e.g. datacells
>>>> geom, ....  Keeping in line is the responsibility of their owners, and 
>>>> the core developers
>>>> should be aware that their changes may break the work of others. Or at 
>>>> least drain a
>>>> possibly substantial amount of time.
>>>>     
>>>>         
>>>>> (c) whether all issues on the wiki, see:
>>>>> http://www.pathfinder-xquery.org/wiki/index.php/Jupiter:_The_Algebra_Release
>>>>>     needed for this action, have already been resolved (short answer: no)
>>>>>   
>>>>>       
>>>>>           
>>>> The pathfinder wiki is a good example of team documentation.
>>>>
>>>> I am looking forward to the upcoming release. The next one might be this 
>>>> year,
>>>> but not necessarily includes everything on the drawing board.
>>>>
>>>> regards, Martin
>>>>     
>>>>         
>>>>> On Tue, Oct 16, 2007 at 05:23:43PM +0200, Peter Boncz wrote:
>>>>>   
>>>>>       
>>>>>           
>>>>>> Hi Maurice, et al,
>>>>>>
>>>>>> The release of (hopefully) this week will still be mps based. I also
>>>>>> anticipate at least one bug-fix release (ie 0.20.2), or even more.  However,
>>>>>> the next major release (0.22) should be algebra based, in my opinion. I am
>>>>>> certainly reluctant to keep investing any more effort in mps. I would aim
>>>>>> for an mps-free release, yes.
>>>>>>
>>>>>> The time-goal of that release is "XMAS 2007", as far as we are concerned.
>>>>>> For the work of built-in functions I have good hope that with the help of
>>>>>> Lefteris we can come a long way.
>>>>>>
>>>>>> I hope and think updates can also be ported by joerd in that time-frame. As
>>>>>> for pftijah, if Jan Flokstra has time, it should be possible to port that as
>>>>>> well in 2-3 months.
>>>>>>
>>>>>> There are some hairy issues, one of them being free-form recursion (needed
>>>>>> by you), another one being precompiled modules (needed by XRPC). The
>>>>>> resolution of these needs assistance from Munich, and maybe related. It needs
>>>>>> a method to compile certain functions into scripted procedures (here MIL)
>>>>>> and calling them. One algorithm is to do this for all functions in modules,
>>>>>> as well as for all recursive functions. The issue to resolve is the
>>>>>> representation of the parameters.
>>>>>>
>>>>>> Depending on how much time Jan Rittinger has available for this, we will
>>>>>> have to find creative solutions for this. Indeed, a backup solution would be
>>>>>> to keep handling certain queries for the time being with mps, but that
>>>>>> surely is ugly.
>>>>>>
>>>>>> Peter
>>>>>>
>>>>>>   -----Original Message-----
>>>>>>   From: Keulen, M. van (Maurice) [mailto:m.vankeulen at utwente.nl]
>>>>>>   Sent: dinsdag 16 oktober 2007 16:38
>>>>>>   To: Peter Boncz [CWI]; Stefan Manegold [CWI]
>>>>>>   Cc: Djoerd Hiemstra; Jan Flokstra; Henning Rode; Abdel Kader, R. (Riham)
>>>>>>   Subject: Nieuwe releases en mps-backend
>>>>>>
>>>>>>
>>>>>>   Hoi,
>>>>>>
>>>>>>   Sorry dat ik niet bij de Skype-sessie van vandaag kon zijn, maar Riham
>>>>>> vertelde me dat het onderwerp van nieuwe releases en de mps-backend ter
>>>>>> sprake is geweest. Het riep wat specifieke vragen bij me op die ze niet
>>>>>> allemaal kon beantwoorden:
>>>>>>
>>>>>>     a.. Is het de bedoeling dat een nieuwe release de mps-backend helemaal
>>>>>> niet meer bevat of alleen maar de algebra-backend als default heeft?
>>>>>>     b.. Als jullie praten over "nieuwe release", bedoel je dan alleen een
>>>>>> package van alle modules van MonetDB/XQuery? Maw zegt het niets over
>>>>>> eventuele `subreleases' van individuele modules?
>>>>>>
>>>>>>     c.. Op de wiki hebben we een pagina met open issues voor de
>>>>>> algebra-release. Wat is de status van die open issues? Ik zie namelijk op de
>>>>>> pagina nergens dat bepaalde issues al opgelost/geïmplementeerd zijn.
>>>>>>   Ik Cc deze e-mail even aan de PF/Tijah-mensen, want  hoewel er levendige
>>>>>> discussies zijn hoe PF/Tijah naar de algebra-backend to porten, ik heb niet
>>>>>> het idee dat er oplossingen zijn die op korte termijn realiseerbaar zijn.
>>>>>> Evenzo het recursie-vraagstuk. Het lijkt me bijvoorbeeld voor de
>>>>>> zichtbaarheid naar buiten toe onwenselijk als deze features ineens zouden
>>>>>> wegvallen voor non-insiders. Of andere onwenselijke consequenties van het
>>>>>> niet meer (sub)releasen van modules die gebaseerd zijn op de mps-backend.
>>>>>>
>>>>>>   Groetjes,
>>>>>>   Maurice.
>>>>>>
>>>>>> --
>>>>>> ----------------------------------------------------------------------
>>>>>> Dr.Ir. M. van Keulen - Assistant Professor, Data Management Technology
>>>>>> Univ. of Twente, Dept of EEMCS, POBox 217, 7500 AE Enschede, Netherlands
>>>>>> Email: m.vankeulen at utwente.nl, Phone: +31 534893688, Fax: +31 534892927
>>>>>> Room: ZI 3039, WWW: http://www.cs.utwente.nl/~keulen
>>>>>>     
>>>>>>         
>>>>>>             
>>>>>   
>>>>>       
>>>>>           
>>>> -------------------------------------------------------------------------
>>>> This SF.net email is sponsored by: Splunk Inc.
>>>> Still grepping through log files to find problems?  Stop.
>>>> Now Search log events and configuration files using AJAX and a browser.
>>>> Download your FREE copy of Splunk now >> http://get.splunk.com/
>>>> _______________________________________________
>>>> Monetdb-developers mailing list
>>>> Monetdb-developers at lists.sourceforge.net
>>>> https://lists.sourceforge.net/lists/listinfo/monetdb-developers
>>>>     
>>>> ------------------------------------------------------------------------
>>>>
>>>> -------------------------------------------------------------------------
>>>> This SF.net email is sponsored by: Splunk Inc.
>>>> Still grepping through log files to find problems?  Stop.
>>>> Now Search log events and configuration files using AJAX and a browser.
>>>> Download your FREE copy of Splunk now >> http://get.splunk.com/
>>>> ------------------------------------------------------------------------
>>>>
>>>> _______________________________________________
>>>> Monetdb-developers mailing list
>>>> Monetdb-developers at lists.sourceforge.net
>>>> https://lists.sourceforge.net/lists/listinfo/monetdb-developers
>>>>     
>>>>         
>> -- 
>> ----------------------------------------------------------------------
>> Dr.Ir. M. van Keulen - Assistant Professor, Data Management Technology
>> Univ. of Twente, Dept of EEMCS, POBox 217, 7500 AE Enschede, Netherlands
>> Email: m.vankeulen at utwente.nl, Phone: +31 534893688, Fax: +31 534892927
>> Room: ZI 3039, WWW: http://www.cs.utwente.nl/~keulen
>>
>>
>> -------------------------------------------------------------------------
>> This SF.net email is sponsored by: Splunk Inc.
>> Still grepping through log files to find problems?  Stop.
>> Now Search log events and configuration files using AJAX and a browser.
>> Download your FREE copy of Splunk now >> http://get.splunk.com/
>> _______________________________________________
>> Monetdb-developers mailing list
>> Monetdb-developers at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/monetdb-developers
>>     
>
>   

-- 
----------------------------------------------------------------------
Dr.Ir. M. van Keulen - Assistant Professor, Data Management Technology
Univ. of Twente, Dept of EEMCS, POBox 217, 7500 AE Enschede, Netherlands
Email: m.vankeulen at utwente.nl, Phone: +31 534893688, Fax: +31 534892927
Room: ZI 3039, WWW: http://www.cs.utwente.nl/~keulen





More information about the developers-list mailing list