February 2009

MonetDB Feb2009-SP2 release

Release "Feb2009-SP2" consists of the following updated components

IMPORTANT NOTE:
Before upgrading to the latest "Feb2009-SP2" release from a release of MonetDB older than the Nov2008 release, please make a dump (SQL, XQuery) of your database, and rename/move/backup the database directories (dbfarm, sql_logs, xquery_logs). After the upgrade, restore (SQL, XQuery) your database, again. Once you have checked and confirmed the correctness of the restore, you can delete the above renamed/moved/backed-up old database directories.

Contents

• Feb 2009 SP2 bug-fix release
• Feb 2009 SP1 bug-fix release
• Feb 2009 feature release
  • Packaging
  • MonetDB Server 5
  • SQL
  • XQuery
  • Mtest
  • Bug fixes
  • Known Issues in this Release

Feb 2009 SP2 bug-fix release

One set of fixes that affects all components is that stack overflow checking has been improved. It is now much less likely that the server crashes when queries need too much stack space. Instead, the server will return an error.

MonetDB/XQuery (tijah): fixes to UTF-8 handling; fixes towards being able to index more than 2**31 words on a 64 bit architecture.

MonetDB5: fixed a crash on exit of the server (this crash was not harmful, just annoying).

MonetDB: avoid integer overflow on very large tables.

mclient/msqldump: fixes to the order in which tables, views, and functions are dumped; also dump triggers; fixed the Ubuntu and Debian installers to include mclient and msqldump.

MonetDB/XQuery: fixed a bug where documents were not available when another document in the collection was loaded.

merovingian/monetdb: fixes to the way the server is stopped.

MonetDB/SQL: fixed a bad concurrency bug where updates were lost or data was incorrectly inserted.

The following bug tracker items were fixed since the Feb2009 SP1 release:

• [ 1811229 ] [ADT] Adding large document, with update support
• [ 1836980 ] XQ: Error with declared variable in module
• [ 1894477 ] PF: "fatal error: aborted too deep recursion" on Windows
• [ 2255453 ] mclient terminates connection unexpectedly
• [ 2418722 ] pf:docname() problem
• [ 2549582 ] M4: leftthetajoin estimate way to small
• [ 2550429 ] nlthetajoin asks for 128 bytes too much memory
• [ 2567316 ] Segfault after long period of use
• [ 2636285 ] ancestor-or-self: illegal data in size column or context lis
• [ 2637051 ] Wrong error message for invalid identifier in group by
• [ 2664405 ] fn:collection("DOES-NOT-EXIST") does not fail
• [ 2668437 ] PF runtime: parent step produces not/wrongly sorted result
• [ 2683759 ] PF:memory leak when fireing multiple simultaneous clients
• [ 2687705 ] Performance degradation due to memory management
• [ 2693776 ] xquery fails: memory allocation
• [ 2698327 ] XQ/GDK: strPut: Assertion failed
• [ 2704016 ] 'null' from copy into gets wrong
• [ 2710426 ] PF/TIJAH: dbat_extend fails
• [ 2711258 ] Mtest.py fails with Python <=2.3.?
• [ 2716585 ] Java fails to run make dist with latest autotools
• [ 2722117 ] PF/TIJAH: dbat_finalize fails
• [ 2722174 ] XQ: shredding 70GB XML fails.
• [ 2726337 ] M4: Mserver startup errors after changes on 2009/04/01
• [ 2747661 ] Merovingian: wait for stopping MonetDB
• [ 2752158 ] M4: test output differs due to reduced error buffer size
• [ 2756553 ] stethoscope -p parameter doesn't work
• [ 2757113 ] crash in la_cse
• [ 2757642 ] Inaccurate AVG of SQRT with 51M records
• [ 2761480 ] compilation error
• [ 2776100 ] SQL: "CREATE VIEW AS SELECT 1;" fails
• [ 2777473 ] PF: incremental shredding broken

Feb 2009 SP1 bug-fix release

The main fix that was done in this release that affects all parts of the code, but most especially MonetDB/SQL is that we have plugged a large number of memory leaks. MonetDB/SQL can now handle larger databases and more demanding queries. This is especially noticeable on 32 bit Windows systems where the memory management system (the system malloc library) is not as sophisticated as the one in Linux.

mclient/msqldump: Fix to the order in which tables and views are dumped: tables must be dumped before views, since views may reference tables and the view creation will only work if the referenced tables already exist.

mclient: Fixed potential deadlock situation when there is a large input file and several queries early on produce large results.

MonetDB/XQuery: Fixed a bug where a replace operation could corrupt the document (bug #2642003).

The following bug tracker items were fixed since the Feb2009 release:

• [ 2581675 ] mal variable type is persistent over sessions
• [ 2627138 ] SQL: test fails with BATpropcheck error
• [ 2642003 ] XQUF replace insert cycle corrupts database
• [ 2651397 ] M5: manual.summary() produces empty result
• [ 2654133 ] simple join fails on latest stable
• [ 2654300 ] problem with views reusing aliases ?
• [ 2654603 ] assertion failed in store.mx
• [ 2656762 ] Views with select on top of join - attribute lookup problem
• [ 2658243 ] SQL: INSERT INTO + ORDER BY
• [ 2671445 ] Mapi read error when rows exceed maximum
• [ 2678502 ] Scary: (not)equal evaluated wrong
• [ 2686045 ] M5 fails to compile with optimization enabled (gcc & icc)
• [ 2686121 ] XQ: oids in DocmgmTape
• [ 2687040 ] M4: test src/modules/plain/Tests/bat.py fails
• [ 2705666 ] Merovingian does not remove pid-file on clean shutdown

The following things were fixed or changed in merovingian and monetdb:

• merovingian.mx: Remove the pid file upon shutdown, fixes bug #2705666
• monetdb.mx: Small cosmetic fix not to overflow the uptime in monetdb status display
• merovingian.mx: Fix hang on Linux when shutting down. select() calls are not interrupted on Linux, so make sure they time out. Fixes bug #2623838
• merovingian.mx: Fixed crash of merovingian when a to be started database terminates immediately after startup (such as when it crashes)
• merovingian.mx: Protect shared linked list structures from being accessed and/or modified by two or more threads concurrently. This resolves random crashes. Fixes bug #2605706
• monetdb.mx: Fixed crash when using monetdb stop on a non-started database. Bug #2605105
• merovingian.mx: Create logfiles when they do not exist. Bug #2556973

The following was changed in the JDBC interface:

• Connection.setReadOnly() now always succeeds but sets a warning when called with true. This allows Mondrian to be used with MonetDB/SQL.
• Holdability invocation methods were implemented, because the JDBC specs state that they need to be implemented and just be warned upon when unavailable (15.1.3.1).
• Implemented retrieved keys, works only for the first auto-generated column, which is always of int-type.

Feb 2009 feature release

Packaging

We now provide Debian (Lenny, i.e. 5.0) and Ubuntu (Intrepid, i.e. 8.10) packages. These packages have only been very lightly tested (the packaged software is tested extensively, though). For now we provide 32 and 64 bit versions for Debian and only a 64 bit version for Ubuntu. This of course in addition to the Fedora packages we already provided.

MonetDB Server 5

The code base has been tested for memory leaks reducing the change of running out of memory too quickly.

The Merovingian has undergone more thorough stress testing making it much more stable.

SQL

The optimizer of SQL has been overhauled. Rather than working on the binary plans, it performs optimization on the relational algebra tree before it produces the binary plans.

XQuery

Like the previous "Nov 2008 SP2" release, the default Algebra-based Pathfinder compiler ("ALG") still does not support a few XQuery features and extensions. A detailed list and temporary work-arounds by resorting to the deprecated "milprint_summer" ("MPS") version of the Pathfinder compiler can be found here.

Time and date related types and the comparison and extraction functions on top of them have been implemented in MonetDB/XQuery.

Mtest

Mainly for MonetDB developers:

The MonetDB testing tools (Mtest.py & friends) from MonetDB/src/testing/ have been moved into their own "testing" CVS module / MonetDB component.

IMPORTANT NOTE:

In case you plan to upgrade your existing CVS-based "MonetDB" installation, please run `make uninstall` in your "MonetDB" build directory, first, i.e., before running `cvs update` in your "MonetDB" source directory.

Once you update your "MonetDB" to the latest CVS development trunk version, and plan to continue (or start) using Mtest.py, you also need to check out the new "testing" CVS module (from SF) and compile and install it any time after you compiled and installed "MonetDB", and before you plan to use Mtest.py.

The new "testing" CVS module depends only on "MonetDB" (and hence requires that to be installed), but on no other MonetDB component (except from the "buildtools", in case your are compiling from CVS sources). No other MonetDB component depends on the new "testing" CVS module other than that you obviously need to have "testing" installed once you want/need to run tests using Mtest.py.

Details:

The MonetDB testing tools (Mtest.py & friends) are not part of the actual MonetDB functionality, but mainly for MonetDB development purposes (and hence only distributed in the source tarballs, but not in any (official) binary packages).

To make them less dependent on release cycles (and vice versa), as well as to prepare making them usable without MonetDB --- for now, they actually still depend on MonetDB; see below) --- we decided to separate the MonetDB testing tools from the "MonetDB" CVS module into their own "testing" CVS module.

For the time being (i.e., until further notice), "testing" depends on "MonetDB" --- more precisely only on some utils from MonetDB/src/common/, but not on the GDK kernel from MonetDB/src/gdk/, as

1. Mtest.py (mainly for "convenience") still uses monet_options.py; and
2. (on Windows only), Mdiff.c uses "our" getopt code.

Python modules "subprocess26.py" & "trace.py" have been moved from <PYTHON_LIBDIR>/MonetDB/ to <PYTHON_LIBDIR>/MonetDBtesting/ .

Bug fixes

A detailed list of the 65 bug reports that have been closed since the previous "Nov 2008 SP2" release can be found in the MonetDB Bug Tracker at SourceForge: page 1/2, page 2/2.
The detailed list of changes can be found in the source code CVS logs.

Known Issues in this Release

• Parallel loading is known to be buggy. Actual workaround is to start the server using gdk_nr_threads=1 set in your monetdb5.conf file (or given via --set gdk_nr_threads=1 on your mserver5's command line) during the bulkload using COPY INTO FROM FILE with mclient.
  Details: (no known bug tracker reference)
• ...