Skip to main content

MonetDB Dec2016 Release

Dec 2016 feature release

There are lots of changes in this (Dec2016) release and the previous (Jun2016).  Here is an outline.

  • The client interfaces that are not written in C, that is to say, the Java (JDBC driver), Perl, PHP, Python, and Ruby interfaces all now have their own repository and are no longer part of the main MonetDB repository.  (Actually, the Ruby interface was already separate in the Jun2016 release.)  These client interfaces are still supported but now have an independent release schedule.  Their first releases are synchronized with this release. Debian and Ubuntu users, please note that the package names for the client interfaces have changed.  If you have the old versions installed, they will not get updated automatically.  Use: sudo apt-get install libdbd-monetdb-perl php-monetdb python-pymonetdb python3-pymonetdb removing the ones from the list above you don't want.
  • For years we have been working towards a version that we've internally called "headless".  This is now complete.  This has had consequences for quite a few internal interfaces, also ones that could potentially be used by extensions written by third parties. The trickiest one is probable BATkey() which has the same calling interface but does something different.  The compiler should complain about other functions since their arguments changed or the functions were completely removed.
  • In the Jun2016 release it was possible to have user-defined functions (UDF) callable from SQL that were written in the R language.  In this release, the same is now possible for UDFs written in Python.  Also see this blog.
  • The possibility of having 32 bit OID values in a system that was compiled for a 64 bit architecture has been removed.  The size of the OID type is now always the same as the "natural" size of the architecture.  This was a compile-time feature and we haven't made any binary releases in a long time in which this feature was used. The automatic conversion from a database in which the size of the OID type was 32 to one in which the size is 64 is still in place. This will be removed in a future version.
  • The internal type "wrd" has been removed.  The "wrd" type was an architecture-dependent integer type, 32 bits wide on a 32-bit system and 64 bits wide on a 64-bit system.  Databases where this type is used will automatically have the type converted to either "int" or "lng", the internal types for 32-bit and 64-bit integers.
  • Some old and unused code and interfaces have been removed.
  • By making some adjustments to internal parameters, we have been able to get a significant improvements on query speed for many types of queries (in informal tests we have seen a 20% improvement compared to the previous release when running the queries from the TPC-H benchmark).
  • A new type of experimental column index has been introduced: the ordered index.  Since creating this index is expensive, it is only used when explicitly created.  To create an index in SQL use CALL createorderindex(schema,table,column) where the procedure arguments are all strings.
  • We have fixed quite a few memory leaks and other bugs.

 

MonetDB5 Server
  • Removed the zorder module with functions zorder.encode, zorder.decode_x and zorder.decode_y.

  • Removed command bbp.getHeat().

  • Removed bat.setColumn with two arguments and bat.setRole. Use bat.setColumn with one argument instead.

  • Removed function BKCappend_reverse_val_wrap: it was unused.

  • The "wrd" type has been removed from GDK and MAL. The type was defined to be a 32 bit integer on 32 bit architectures and a 64 bit integer on 64 bit architectures. We now generally use "lng" (always 64 bits) where "wrd" was used.

  • Keep a collection of full traces. Each time the SQL user applies the TRACE option, the full json trace is retained within the <dbpath>/<dbname>/sql_traces

Build Environment
  • New packages MonetDB-python2 (Fedora) and monetdb-python2 (Debian/Ubuntu) have been created for Python 2 integration into MonetDB.

  • With OID size equal to ABI/word size, mserver5 does not need to print the OID size, anymore.

  • Removed obsolete code associated with long gone static linking option.

  • Removed configure option --enable-oid32 to compile with 32 bit OIDs on a 64 bit architecture.

.
MonetDB Common
  • The tnokey values must now be 0 if it is not known whether all values in a column are distinct.

  • The 2-bit tkey field in the bat descriptor has been split into two single bit fields: tkey and tunique. The old tkey&BOUND2BTRUE value is now stored in tunique.

  • Implemented conversion to str from any type (not just the internal types).

  • VALcopy and VALinit both return their first argument on success or (and that's new) NULL on (allocation) failure.

  • BATattach now can also create a str BAT from a file consisting of null-terminated strings. The input file must be encoded using UTF-8.

  • BATattach now copies the input file instead of "stealing" it.

  • Removed the lastused "timestamp" from the BBP.

  • Removed batStamp field from BAT descriptor, and removed the BBPcurstamp function.

  • Removed unused functions BBPhot and BBPcold.

  • Removed BATderiveTailProps and BATderiveProps. Just set the properties you know about, or use BATsettrivprop.

  • Removed the macro BUNfirst. It can be replaced by 0.

  • Changed BATroles by removing the argument to set the name of the head column.

  • The head column is now completely gone. MonetDB is completely "headless".

  • The format of the BBP.dir file was simplified. Since the head column is VOID, the only value that needs to be stored is the head seqbase.

  • BATs now only have a single (logical) name.

  • The function BATmirror is gone. The HEAD column is always VOID (with a non-nil seqbase) and the TAIL column carries the data. All functions that deal with data work on the TAIL column.

  • BATkey now works on the TAIL column instead of the HEAD column.

  • Replaced BATseqbase with BAThseqbase and BATtseqbase, the former for setting the seqbase on the HEAD, the latter for setting the seqbase on the TAIL.

  • Replaced function BATnew with COLnew with slightly different arguments: the first argument of COLnew is the SEQBASE of the head column (which is always VOID).

  • The "wrd" type has been removed from GDK and MAL. The type was defined to be a 32 bit integer on 32 bit architectures and a 64 bit integer on 64 bit architectures. We now generally use "lng" (always 64 bits) where "wrd" was used.

SQL
  • Removed functions sys.zorder_encode, sys.zorder_decode_x, and sys.zorder_decode_y.

  • The experimental recycler code is moved to the attic.

  • The syntax of bat.new(:oid,:any) has been changed by dropping the superflous :oid. All BATs are now binary associations between a void column and a materialized value column. (except for the internal :bat[:void,:void] representation of simple oid ranged tails.)

Bug Fixes
  • 3357: Implement setQueryTimeout()
  • 3445: Add support for database name to dotmonetdb file
  • 3597: SQL to MAL listing looses types
  • 3973: JDBC hangs
  • 3976: Performance enhancement to LIKE without wildcards
  • 4005: Correlated update causes incorrect null constraint violation
  • 4016: merge table only optimises for point query
  • 4040: sys.storage call can take a long time
  • 4047: Segfault when updating a dropped table
  • 4050: Database corruption when running low on inode
  • 4057: missing bulk operations between constant and bat
  • 4061: SIGSEGV in candscan_lng
  • 4066: Deadlocked monetdbd
  • 6068: Error message about incompatible BBP version should be clearer
  • 6069: query with union all silently crashes
  • 6070: setting negative session query timeout should not be possible/allowed
  • 6071: where clause with cast and floor fails to sigsegv
  • 6072: Bind to UPD delta column does not get/show type information in EXPLAIN
  • 6073: Missing type information for constants in MAL explain
  • 6074: SET ROLE command does not work
  • 6075: gdk_calc.c:13113: BATcalcifthenelse_intern: Assertion `col2 != NULL' failed.
  • 6076: rel_optimizer.c:5426: rel_push_project_up: Assertion `e' failed.
  • 6077: mserver5: rel_optimizer.c:5444: rel_push_project_up: Assertion `e' failed.
  • 6078: rel_bin.c:2402: rel2bin_project: Assertion `0' failed.
  • 6084: Merge table point to wrong columns if columns in partition tables are deleted
  • 6108: monetdb5-sql sysv init script does not wait for shutdown
  • 6114: segfault raised in the query rewriter due to a null pointer
  • 6115: Assertion hit in the codegen
  • 6116: Codegen does not support certain kind of selects on scalar subqueries
  • 6117: Assertion hit in the query rewriting stage during the push down phase
  • 6118: SIGSEGV in strPut due to shared heap
  • 6119: Assertion hit in the MAL optimiser on a complex query
  • 6120: QUANTILE() treats NULL as if it is zero
  • 6121: SELECT a.col IN ( b.col FROM b ) FROM a statements with no error but no result
  • 6123: Assertion hit in the codegen #2
  • 6124: CASE <column> WHEN NULL THEN 0 ELSE 1 END returns wrong result
  • 6125: Stack overflow in the query rewriter with a query having an OR condition and a nested SELECT subexpression
  • 6126: batcalc.== can't handle void BATs
  • 6139: Debian libmonetdb13 conflicts with libmonetdb5-server-geom