[Monetdb-developers] RFC: Upcoming release of MonetDB 4.10
Dear Developers,
Enclosed you find the draft cover letter for the upcoming MonetDB release. Please provide any comments that from your perspective are relevant to mention.
In preparing the distribution content we have finally decided not to include anything of the Xupdate functionality, SOAP, or other XML modules. The primary reason is that implementation and validation of these components have not reached a point were we are confident of their stability. These issues will be moved forward to the next release planned for March 2006 when the system will be presented at EDBT.
Nevertheless, we can be proud of the progress made. Especially interaction with the server using a large number of small SQL update queries has been boosted to the point where we are in the same arena as MySQL and PostgreSQL using JDBC connectivity.
The code freeze date is Friday, Jan 13 2006. We plan to release MonetDB v4.10 on Tuesday, Jan 24 2006.
Regards, The MonetDB Development Team
------------------- Cover release 4.10 ----------
The MonetDB release 4.10 marks the end of a period where much attention was given to improving the performance of simple SQL applications. These improvements came from careful analysis of student programs and participating in the "c't" Magazine's database contest featuring the DVDstore benchmark.
It led to re-consideration of MAPI -- the communication protocol -- aiming for less overhead and complex structures to be parsed. To facilitate this, JDBC has been largely be rewritten at core level to streamline the communication flow even more. Not only JDBC has benefited from the changes made, all MAPI driven communication has been sped up.
A second Perl library has been prepared, which more directly uses the MAPI library. It has been registered at the official Perl archive for distribution (ICAN). ODBC has been upgraded to support server side prepared statements like JDBC. More meta-data is now available for columns in JDBC.
MonetDB/SQL has been extended with SQL:03's sequences support. This allows for identity columns, serials (PostgreSQL) or auto_increments (MySQL) in MonetDB. A complete rewrite of the catalog and transaction handling removed the overhead encountered in small interactions. The SQL transaction scheme is based on three BATs (stable,inserts,deletes) which are merged at the end of a transaction.
MonetDB/XQuery performance has been boosted using query/result caching and a rewrite of the code to manage the internal representation of MIL programs. Further, MonetDB/XQuery no also supports XQuery module definitions. XQuery is currently actively used in large scale applications and is stable.
The MonetDB software base has been split to ease source distribution. The tools to prepare the source compilation (Mx), source code generators (MEL, Burg) and makefile generators (autogen) are assembled in the buildtools package. They are not needed to recompile source code distributed. ! TODO: ! Actually make the latter work. ! For now also the source distributions (still) require the buildtools package.
In the kernel area we have added the 'constant' module. It provides functions to work with BATs whose tail consist of a constant value.
During this period we have solved numerous bugs reported by our users. In addition we organized 3 bug testing days to encapsulate the problems reported into scripts that became part of our nightly testing scheme.
Several smaller performance improvements have been made to the inner core of the server. One semantic error has been the cause of confusion. Bats with signature [void,void] are now allowed. A final clean-up of the respective semantic is currently being implemented; a detailed description of the changes will follow soon.
Lack of manpower, expiring access to external machines, ceasing system maintenance and decrepit hardware forced us to stop nightly testing on IBM AIX, SGI IRIX and Sun Solaris. Thus, all active support for these (aging) platforms had to be suspended, too, but could easily be revived on request, once we (re-)gain access to such systems. Higher on our wishlist are more modern systems, especially 64-bit Windows and 64-bit Mac OS X. We welcome any donation in this area: access to respective systems, hardware, and/or (especially in case of the pending 64-bit Windows port of MonetDB) development-expertise and -time.
The next release of MonetDB version 4 is scheduled for March 2006. By that time we also plan to release the alpha version of the next generation of MonetDB, version 5.
Regards, The MonetDB Development Team
Dear developers,
thank you very much for all your comments so far.
I just recall some more TODOs for the release, which we should not forget:
- make source distributions (tarball, source RPMs, ebuild) independent of buildtools (and possibly also other code generators like flex/bison)
- make configure use the local C compilers standard to decide whether to compile for 32- or 64-bits (for now, the standard of our configure script is to use 32-bits, "even" on 64-bit systems)
- location of 64-bit libraries:
On systems that support both 32- and 64-bits, only 32-bit libraries are usually located to [/usr[/local]]/lib/ while 64-bit libraries are located in (e.g.) [/usr[/local]]/lib64/ .
With MonetDB, however, we install all libs in either case to <prefix>/lib/ which (a.o.) causes `make rpm` to fail on 64-bit SuSE Linux on x86_64 systems: ======== + /usr/lib/rpm/brp-lib64-linux sf@suse.de: if you find problems with this script, drop me a note /var/tmp/MonetDB-root/usr/lib/libmutils.so.0.0.0: file format elf64-x86-64 /var/tmp/MonetDB-root/usr/lib/libmutils.so.0.0.0: should be in */lib64 error: Bad exit status from /tmp/rpm-tmp.55606 (%install)
RPM build errors: Bad exit status from /tmp/rpm-tmp.55606 (%install) make: *** [rpm] Error 1 ========
Hence, we should consider installing 64-bit libraries to <prefix>/lib/ on Linux (ia64) <prefix>/lib64/ on Linux (x86_64), IRIX64 <prefix>/lib/64/ on SunOS/Solaris
Open: What about AIX, Windows, Mac OS X ?
Stefan
On 12-01-2006 11:54:59 +0100, Stefan Manegold wrote:
Dear developers,
thank you very much for all your comments so far.
I just recall some more TODOs for the release, which we should not forget:
make source distributions (tarball, source RPMs, ebuild) independent of buildtools (and possibly also other code generators like flex/bison)
make configure use the local C compilers standard to decide whether to compile for 32- or 64-bits (for now, the standard of our configure script is to use 32-bits, "even" on 64-bit systems)
location of 64-bit libraries:
On systems that support both 32- and 64-bits, only 32-bit libraries are usually located to [/usr[/local]]/lib/ while 64-bit libraries are located in (e.g.) [/usr[/local]]/lib64/ .
With MonetDB, however, we install all libs in either case to <prefix>/lib/ which (a.o.) causes `make rpm` to fail on 64-bit SuSE Linux on x86_64 systems: ========
- /usr/lib/rpm/brp-lib64-linux
sf@suse.de: if you find problems with this script, drop me a note /var/tmp/MonetDB-root/usr/lib/libmutils.so.0.0.0: file format elf64-x86-64 /var/tmp/MonetDB-root/usr/lib/libmutils.so.0.0.0: should be in */lib64 error: Bad exit status from /tmp/rpm-tmp.55606 (%install)
RPM build errors: Bad exit status from /tmp/rpm-tmp.55606 (%install) make: *** [rpm] Error 1 ========
Hence, we should consider installing 64-bit libraries to <prefix>/lib/ on Linux (ia64) <prefix>/lib64/ on Linux (x86_64), IRIX64 <prefix>/lib/64/ on SunOS/Solaris
Open: What about AIX, Windows, Mac OS X ?
Darwin is different, as it has universal binaries. Nett effect is that you can always install in ${PREFIX}/lib, because building fat binaries (using "-march=ppc -march=ppc64 -march=i386" or something) is what we don't support. In fact... I'm against building 64-bits on Darwin by default because: - Tiger is the only 64-bits OS, but 64-bits support was dead on the day Tiger was released. Only the kernel + -libs are 64-bits, the whole userland is 32-bits. - Having ppc64 arch is not sufficient (need darwin8+) - Since userland is 32-bits, if pcre is there (through Fink, Gentoo, DP or whatever), it won't be used, because it can't, which sucks. - x86-macos is lurking around the corner, to make things complicated.
Dear developers,
I have one more issue (well, one to be discussed here ...) concening the upcoming release:
Shall we leave Wouters 'Burkowski Axis Steps' in and release right now, or shall we leave them for the next release with XQuery updates and SOAP?
Well, I guess, the first questions are to Wouter: - What the current state of the implementaion? Is it finished? Is it tested? Are there tests? How does it work? Is there a documentation?
In case we leave it in, shall we "advertize" it, or just release it "silently"?
Since the database schema has been changed to support the Burkowski Axis Steps, it requires re-shredding of documents, which we must document with the release. However, the schema has IMHO changed anyway since the last release (indices, etc.), hence, re-shredding is required, anyway. Moreover, the schema will change again with the update release ... Thus, this is not an issue here.
Opinions?
Thanks in advance,
Stefan
On Thu, Jan 12, 2006 at 11:54:59AM +0100, Stefan Manegold wrote:
Dear developers,
thank you very much for all your comments so far.
I just recall some more TODOs for the release, which we should not forget:
make source distributions (tarball, source RPMs, ebuild) independent of buildtools (and possibly also other code generators like flex/bison)
make configure use the local C compilers standard to decide whether to compile for 32- or 64-bits (for now, the standard of our configure script is to use 32-bits, "even" on 64-bit systems)
location of 64-bit libraries:
On systems that support both 32- and 64-bits, only 32-bit libraries are usually located to [/usr[/local]]/lib/ while 64-bit libraries are located in (e.g.) [/usr[/local]]/lib64/ .
With MonetDB, however, we install all libs in either case to <prefix>/lib/ which (a.o.) causes `make rpm` to fail on 64-bit SuSE Linux on x86_64 systems: ========
- /usr/lib/rpm/brp-lib64-linux
sf@suse.de: if you find problems with this script, drop me a note /var/tmp/MonetDB-root/usr/lib/libmutils.so.0.0.0: file format elf64-x86-64 /var/tmp/MonetDB-root/usr/lib/libmutils.so.0.0.0: should be in */lib64 error: Bad exit status from /tmp/rpm-tmp.55606 (%install)
RPM build errors: Bad exit status from /tmp/rpm-tmp.55606 (%install) make: *** [rpm] Error 1 ========
Hence, we should consider installing 64-bit libraries to <prefix>/lib/ on Linux (ia64) <prefix>/lib64/ on Linux (x86_64), IRIX64 <prefix>/lib/64/ on SunOS/Solaris
Open: What about AIX, Windows, Mac OS X ?
Stefan
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
- We decided that we would not do these kinds of extras. - Since Wouter just checked it in, there is no testweb coverage (yet). - Changing plans to now advertise Burkowski steps, means you also have to reconsider SOAP to be fair to others.
A big "no-go" from me. We should freeze the sources, not add all kinds of things at the very last moment. Why are the sources not frozen yet? (Does this list also know the reason?) Many people are waiting for an updated snapshot of MonetDB. Why delay and delay and delay. If you want to have these options, just release more often. IMHO the releases take far too much human resources, and should be much more relaxed. If you haven't got it in front of you on the table, tested and all, then it's simply not in. Not every release must be an eye-opener for the world, people are just waiting on a new version which fixes all of their bugs. They are waiting since August 1, 2005.
On 16-01-2006 23:44:31 +0100, Stefan Manegold wrote:
Dear developers,
I have one more issue (well, one to be discussed here ...) concening the upcoming release:
Shall we leave Wouters 'Burkowski Axis Steps' in and release right now, or shall we leave them for the next release with XQuery updates and SOAP?
Well, I guess, the first questions are to Wouter:
- What the current state of the implementaion? Is it finished? Is it tested? Are there tests? How does it work? Is there a documentation?
In case we leave it in, shall we "advertize" it, or just release it "silently"?
Since the database schema has been changed to support the Burkowski Axis Steps, it requires re-shredding of documents, which we must document with the release. However, the schema has IMHO changed anyway since the last release (indices, etc.), hence, re-shredding is required, anyway. Moreover, the schema will change again with the update release ... Thus, this is not an issue here.
Opinions?
Thanks in advance,
Stefan
On Tue, Jan 17, 2006 at 09:07:29AM +0100, Fabian Groffen wrote:
- We decided that we would not do these kinds of extras.
It was in before we decided that (and hence before the original freeze date). as far as I recall, we hadn't decided, yet, to actively "throw it out". (Hence my question...)
- Since Wouter just checked it in, there is no testweb coverage (yet).
- Changing plans to now advertise Burkowski steps, means you also have to reconsider SOAP to be fair to others.
SOAP is partly in, but AFAIK far from finished; we will not announce it, but removing code traces is impossible (as opposed to the Burkowski stuff, which is nicely tagged in CVS (I tagged part one, Wouter tagged part two). SOAP development currently proceeds on a branch and will be merged in and released with the next release.
A big "no-go" from me. We should freeze the sources, not add all kinds of things at the very last moment. Why are the sources not frozen yet?
well, kind of my decision/fault, since the void/oid changes/fixes are still pending. to reduce (my) work, a feature freeze for me coinsides with making a new stable branch; doing this before the void/oid changes/fixes are in would require extra propagation...
(btw, a freeze on Friday Jan 13 would only have kept your latest JDBC changes out of the release ... ;-))
(Does this list also know the reason?)
well, now yes ;-) (sorry for the delay...)
Many people are waiting for an updated snapshot of MonetDB. Why delay and delay and delay. If you want to have these options, just release more often. IMHO the releases take far too much human resources, and should be much more relaxed. If you haven't got it in front of you on the table, tested and all, then it's simply not in. Not every release must be an eye-opener for the world, people are just waiting on a new version which fixes all of their bugs. They are waiting since August 1, 2005.
Agree (almost) completely.
The point/problem is, that somehow all of us want (in one way or an other) "perfect" releases, both from the users' and from the developers' point of view (some major contributions to "sell" is, all bugs fixed, better packaging, better/more documentation, better/simpler compilation, more/better testing, ...), and all only start thinking (and acting) about this, once release "plans" become more concrete, and that happens only shortly before a (planned) release....
maybe too many (contradicting?) optimization targets? maybe too many "cooks"???
Stefan
On 17-01-2006 10:04:04 +0100, Stefan Manegold wrote:
(btw, a freeze on Friday Jan 13 would only have kept your latest JDBC changes out of the release ... ;-))
I already made my own assessment to that I just was too late. However, when I heard yesterday that nothing was frozen yet, and that it is most likely not to be frozen for another week or so, I took the freedom to lock redirection support into JDBC, as I can now use a stable JDBC driver to work with for instance a DBPoolv2 or M5. "Halcyon & on & on"
Agree (almost) completely.
The point/problem is, that somehow all of us want (in one way or an other) "perfect" releases, both from the users' and from the developers' point of view (some major contributions to "sell" is, all bugs fixed, better packaging, better/more documentation, better/simpler compilation, more/better testing, ...), and all only start thinking (and acting) about this, once release "plans" become more concrete, and that happens only shortly before a (planned) release....
maybe too many (contradicting?) optimization targets? maybe too many "cooks"???
<monetdb:rant type="release">
probably. no planning, no milestones, maximal postponing to deadlines... "Ominous & on & on"
</monetdb:rant>
On Tue, Jan 17, 2006 at 10:34:25AM +0100, Fabian Groffen wrote:
On 17-01-2006 10:04:04 +0100, Stefan Manegold wrote:
(btw, a freeze on Friday Jan 13 would only have kept your latest JDBC changes out of the release ... ;-))
I already made my own assessment to that I just was too late. However, when I heard yesterday that nothing was frozen yet, and that it is most likely not to be frozen for another week or so, I took the freedom to
what made you think that ? I hope to checkin today....
Niels
lock redirection support into JDBC, as I can now use a stable JDBC driver to work with for instance a DBPoolv2 or M5. "Halcyon & on & on"
Agree (almost) completely.
The point/problem is, that somehow all of us want (in one way or an other) "perfect" releases, both from the users' and from the developers' point of view (some major contributions to "sell" is, all bugs fixed, better packaging, better/more documentation, better/simpler compilation, more/better testing, ...), and all only start thinking (and acting) about this, once release "plans" become more concrete, and that happens only shortly before a (planned) release....
maybe too many (contradicting?) optimization targets? maybe too many "cooks"???
<monetdb:rant type="release">
probably. no planning, no milestones, maximal postponing to deadlines... "Ominous & on & on"
</monetdb:rant>
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&da... _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
On Tue, Jan 17, 2006 at 10:34:25AM +0100, Fabian Groffen wrote:
I already made my own assessment to that I just was too late. However, when I heard yesterday that nothing was frozen yet, and that it is most likely not to be frozen for another week or so, I took the freedom to lock redirection support into JDBC, as I can now use a stable JDBC driver to work with for instance a DBPoolv2 or M5. "Halcyon & on & on"
Perfectly fine with me. No objections at all. (I assume, everything is well tested and documented ;-))
I just wanted to be "more verbose" and let this list know about what happened ... ;-)
Stefan
My opinion, for what it is worth...
The code I checked in did not generate more errors on the testweb than before (so I guess it does no harm to anyone using the code WITHOUT the burkowski-compile-time-switch). The code also generates a 'invalid character' error at compile time when trying to use the burkowski steps when they are disabled.
When the Burkowski steps are enabled (I currently only use this mode), the new steps seem to behave fine. I think there still is a memory leak, but I haven't located it yet. I should have checked in some tests. Still busy with that.
I would like to ask to leave the code in, because it (probably) would not harm anybody, and I would be happy.
Wouter
-----Original Message----- From: monetdb-developers-admin@lists.sourceforge.net [mailto:monetdb-developers-admin@lists.sourceforge.net] On Behalf Of Stefan Manegold Sent: dinsdag 17 januari 2006 11:12 To: Stefan Manegold; monetdb-developers@lists.sourceforge.net Subject: Re: * Re: * Re: [Monetdb-developers] RFC: Upcoming release of MonetDB 4.10
On Tue, Jan 17, 2006 at 10:34:25AM +0100, Fabian Groffen wrote:
I already made my own assessment to that I just was too late. However, when I heard yesterday that nothing was frozen yet, and that it is most likely not to be frozen for another week or so, I took the freedom to lock redirection support into JDBC, as I can now use a stable JDBC driver to work with for instance a DBPoolv2 or M5. "Halcyon & on & on"
Perfectly fine with me. No objections at all. (I assume, everything is well tested and documented ;-))
I just wanted to be "more verbose" and let this list know about what happened ... ;-)
Stefan
On Wed, Jan 11, 2006 at 04:37:29PM +0100, Stefan Manegold wrote:
Dear Developers,
Enclosed you find the draft cover letter for the upcoming MonetDB release. Please provide any comments that from your perspective are relevant to mention.
[..]
Hi all,
I took Stefan's latest email to look into his release notes draft. I guess we in fact made sufficient progress to justify a major release.
I have one comment on Stefan's draft, though:
MonetDB/XQuery performance has been boosted using query/result caching and a rewrite of the code to manage the internal representation of MIL programs. Further, MonetDB/XQuery no also supports XQuery module definitions. XQuery is currently actively used in large scale applications and is stable.
Pathfinder actually solely supports our own interpretation of XQuery modules. We never meant to implement full XQuery module support. The code that Peter and I had added was rather meant as some tailor-made support for the optimizations Peter did.
That still means we provide some support for XQuery modules. But I would not advocate this that loudly. How about “Further, MonetDB/XQuery now provides (limited) support for XQuery modules.”?
Just my 2¢
Jens
One thing we might want to make a note about, is the renewed Windows(tm) Installer(r). Or has that experiment failed, Niels?
On 11-01-2006 16:37:29 +0100, Stefan Manegold wrote:
Dear Developers,
Enclosed you find the draft cover letter for the upcoming MonetDB release. Please provide any comments that from your perspective are relevant to mention.
On Fri, Jan 13, 2006 at 09:11:32AM +0100, Fabian Groffen wrote:
One thing we might want to make a note about, is the renewed Windows(tm) Installer(r). Or has that experiment failed, Niels?
This experiment didn't fail, and yes we should add it. Problem is it needs some more work/testing.
Niels
On 11-01-2006 16:37:29 +0100, Stefan Manegold wrote:
Dear Developers,
Enclosed you find the draft cover letter for the upcoming MonetDB release. Please provide any comments that from your perspective are relevant to mention.
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
participants (5)
-
Fabian Groffen -
Jens Teubner -
Niels Nes -
Stefan Manegold -
Wouter Alink