Skip to main content

MonetDB Jul2015 released

The MonetDB team at CWI & MonetDB Solutions is pleased to announce the Jul2015 feature release of the MonetDB suite of programs. The plethora of platforms to be supported and OS upgrades called for a slight delay in our release cycle. A number of issues have been dealt with and a couple of highly interesting features have been included for the community to try out. Here are highlights of some of the changes in this release.

SQL

We have added 128-bit integers (called HUGEINT) on all platforms that can support this. We improved a less often used transaction feature called SAVE POINTS in SQL. Access to FITS files, the default for managing images in many scientific domains, has become more mature.

Performance

The overall system performance profile is slightly shifting, some queries may run significant faster and some somewhat slower due to improvements in the optimizer pipelines and key algorithms, such as range joins. Inspired by a project to handle point-cloud queries over 640 billion points [1][2][3], we have included a feature to make auto-generated index structures persistent on disk. We will keep monitoring the performance ourselves and we look forward towards feedback from the community.

Data loading

The bulk data loading functionality now uses a double IO buffering scheme in addition to the multi-core parsing and loading scheme. Depending on the data being loaded this can safe up to 40%. In the same venue we have included an experimental BEST EFFORT option, which continues parsing the load file upon detecting errors and leaves the set of corrupted rows in the table sys.rejects() [4].

Distributed database support

Significant progress has been made in distributed processing. MonetDB now supports a master-slave distribution function. For the time being, the slave nodes use the data from the master in read-only mode. Tables can be defined as MERGE and REMOTE tables, supporting a flexible range partitioning scheme for distributed query processing. The query optimize will use the partitioning information to restrict the number of workers involved in handling a request. It is used in the point cloud project [1][2][3].

Query profilers

The performance profilers have been extended with a progress-bar tool, the Tachograph, and the Tomograph underwent a major rewrite to become easier to use.

Java

Java application programming has been simplified using improvement for PREPARED statements. The JDBC driver along with a number of improvements, We now compile the Java classes using the latest Java 8 version, and we tell it to compile for Java 7.

OS X

For all Mac users, installing MonetDB has become a few simple clicks using the .pkg installer for OS X.

Removed modules

Some experimental modules has been removed from the distribution. The code remains accessible from Mercurial. This involves the RDF branch, the DataCell branch, and the MiniSEED Data Vaults (now found in DataVaults and DVframework branches).

License

Last, but not least, we upgraded our licensing model to Mozilla Public License (MPL) 2.0. Equally permissive for deploying, changing, and extending the code base. We love to hear where the system is being deployed, as it provides us the ingredients to keep on supporting the project as part of the ongoing research at CWI.

For a full list of fixes and more on this release, please see the release notes. We'll be upgrading the documentation of the new features. Head over to the downloads section or use your package manager to upgrade.

Outlook

We continue the development of features requested and addressing issues raised. Given the success of the MonetDB/R integration, we are planning to release MonetDB/Python and Embedded MonetDB soon. Most likely the next feature release. Those living at the bleeding edge of technology are invited to help harden out the code.

[1] Oscar Martinez-Rubi, Peter van Oosterom, Romulo Goncalves, Theo Tijssen, Milena Ivanova, Martin L. Kersten, Foteini Alvanaki: Benchmarking and improving point cloud data management in MonetDB. SIGSPATIAL Special 6(2): 11-18 (2014).

[2] Foteini Alvanaki, Romulo Goncalves, Milena Ivanova, Martin L. Kersten, Kostis Kyzirakos: GIS Navigation Boosted by Column Stores. PVLDB 8(12): 1956-1967 (2015).

[3] R. A. Goncalves, M. Ivanova, F. Alvanaki, J. Maassen, K. Kyzirakos, O. Martinez-Rubi, Hannes Mühleisen. A round table for multi-disciplinary research on Geospatial and Climate Data. IEEE e-Science, 2015.

 

[4] An issue calls for using the latest repository source code until Jul2015-SP1 is released.