December 2016

Dec 2016-SP5 bugfix release

Bug Fixes

• 6290: Crash (and assertion failure) with a correlated subquery with NOT IN in the select-list
• 6291: crash if executes a function with sample operator
• 6294: Sqlitelogictest crash
• 6296: Another sqllitelogictest crash :(
• 6297: 7th sqllitelogictest crash :(
• 6300: Protect against missing BATs (sqlsmith)
• 6314: Lateral crash report (sqlsmith)
• 6315: Exist operator on type bigint missing (sqlsmith)
• 6316: Coalesc and limit error (sqlsmith)
• 6317: Two-column integer aggregation extremely slow
• 6318: Daemon crashes if .merovignian_properties for a database contains more than 42 entries
• 6320: The daemon ignores all but the last entry in the .merovignian_properties file when spawning mserver5

Dec 2016-SP4 bugfix release

MonetDB5 Server

• Fixed a bug causing a crash during cleanup when mserver5 is stopped with monetdb stop database.
• Some memory leaks were plugged.

MonetDB Common

• A potential deadlock was fixed in order index creation.
• 98ad79c555cc for details.

SQL

• Some memory leaks were plugged.
• Upgrade code was added for an old change in the sys.settimeout function.
• A bug was fixed with the automatic “vacuum” operation on system tables.

Bug Fixes

• 6110: cast of a SQL boolean value to a string or clob or (var)char is wrong
• 6254: Crash (and assertion failure) after querying a view which uses a correlated subquery in the select-list
• 6256: Assertion Trigger on FULL OUTER JOIN with more than two BETWEEN clauses
• 6257: wrong count values (1 instead of 0) for correlated aggregation queries
• 6258: Vulnerability in FITS and NETCDF data vaults
• 6259: crash on select query from sqlitelogictests
• 6260: Sqlitelogictest crash
• 6288: Function cannot find column in merge table
• 6295: msqldump writes unescaped timestamp values when using inserts

Dec 2016-SP3 bugfix release

MonetDB Common

• Fixed a bug when appending string bats that are fully duplicate eliminated. It could happend that the to-be-appended bat had an empty string at an offset and at that same offset in the to-be-appended-to bat there happened to be a (sequence of) NULL(s). Then this offset would be used, even though it might nog be the right offset for the empty string in the to-be-appended-to bat. This would result in multiple offsets for the empty string, breaking the promise of being duplicate eliminated.

Bug Fixes

• 6138: Weak duplicate elimination in string heaps > 64KB
• 6183: ResultSet returns double quoted column name if name contains space characters
• 6216: Assertion raised (sqlsmith)
• 6219: Crash in rel_optimizer (sqlsmith)
• 6227: Monetdb fails on remote tables
• 6228: mclient crashes if real column is multiplied by it itself
• 6229: ANALYZE, unexpected end of input
• 6230: ANALYZE, syntax error
• 6237: semijoin with empty right bat does not return immediately
• 6242: Crash on rel_reduce_groupby_exps (sqlsmith)
• 6243: Static optimization gives wrong result (1 + NULL = -127)
• 6245: Nested query crashes all versions of MonetDB or gives wrong result starting from Dec2016-SP2
• 6246: update statements: references to a table do not bind to its alias
• 6247: Type analysis issue (sqlsmith)
• 6248: update statements: the semantic stage does not resolve the relation in the from clause
• 6250: Assertion failure when querying a Blob column with order by DESC
• 6251: Crash after adding an ordered index on sys.statistics column and querying sys.statistics
• 6253: FITS Data Vaults does not work when using user/pw and other than sys schema name

Dec 2016-SP2 bugfix release

Bug Fixes

• 4034: argnames array in rapi.c has fixed length (that was too short)
• 6080: mserver5: rel_bin.c:2391: rel2bin_project: Assertion `0’ failed.
• 6081: Segmentation fault (core dumped)
• 6082: group.subgroup is undefined if group by is used on an expression involving only constants
• 6111: Maximum number of digits for hge decimal is listed as 39 in sys.types. Should be 38 as DECIMAL(39) is not supported.
• 6112: Crash upgrading Jul2015->Jun2016
• 6130: Query rewriter crashes on a NULL pointer when having a correlated subquery
• 6133: A crash occurs when preparing an INSERT on joined tables during the query semantic phase
• 6141: Getting an error message regarding a non-GROUP-BY column rather than an unknown identifier
• 6177: Server crashes
• 6186: Null casting causes no results (silent server crash?)
• 6189: Removing a NOT NULL constraint from a PKey column should NOT be allowed
• 6190: CASE query crashes database
• 6191: MT_msync failed with “Cannot allocate memory”
• 6192: Numeric column stores wrong values after adding large numbers
• 6193: converting to a smaller precision (fewer or no decimals after decimal point) should round/truncate consistently
• 6194: splitpart returns truncated last part if it contains non ascii caracters
• 6195: Cast from huge decimal type to smaller returns wrong results
• 6196: Database crashes after generate_series query
• 6198: COALESCE could be more optimized
• 6201: MonetDB completely giving up on certain queries - no error and no result
• 6202: querying a table with an ordered index on string/varchar column crashes server and makes server unrestartable!
• 6203: copy into: Failed to import table Leftover data ‘False’
• 6205: Integer addition overflow
• 6206: casting strings with more than one trailing zero (‘0’) to decimal goes wrong
• 6209: Aggregation over complex OR expressions produce wrong results
• 6210: Upgrading a database from Jun2015 or older crashes the server
• 6213: SQLsmith causes server to crash
• 6217: Segfault in rel_optimizer (sqlsmith)
• 6218: grouped quantiles with all null group causes following groups to return null
• 6224: mal_parser: cannot refer to types containing an underscore

Dec 2016-SP1 bugfix release

Bug Fixes

• 4039: Slow mserver5 start after drop of tables (> 1 hour)
• 4048: Segfault on vacuum with parallel updates
• 6079: pushselect optimizer bug on MAL snippet
• 6140: INNER JOIN gives the results of a CROSS JOIN
• 6150: Query giving wrong results, extra records are appearing
• 6175: The program can’t start because python27.dll is missing from your computer.
• 6178: AVG + GROUP BY returns NULL for some records that should have results
• 6179: mergetable optimizer messes up sample
• 6182: sys.shutdown triggers assertion in clients.c
• 6184: Incorrect result set - Extra records in result set

Dec 2016 feature release

There are lots of changes in this (Dec2016) release and the previous (Jun2016). Here is an outline.

• Ruby interfaces all now have their own repository and are no longer part of the main MonetDB repository. (Actually, the Ruby interface was already separate in the Jun2016 release.) These client interfaces are still supported but now have an independent release schedule. Their first releases are synchronized with this release. Debian and Ubuntu users, please note that the package names for the client interfaces have changed. If you have the old versions installed, they will not get updated automatically. Use: sudo apt-get install libdbd-monetdb-perl php-monetdb python-pymonetdb python3-pymonetdb removing the ones from the list above you don’t want.
• For years we have been working towards a version that we’ve internally called “headless”. This is now complete. This has had consequences for quite a few internal interfaces, also ones that could potentially be used by extensions written by third parties. The trickiest one is probable BATkey() which has the same calling interface but does something different. The compiler should complain about other functions since their arguments changed or the functions were completely removed.
• this blog.
• The possibility of having 32 bit OID values in a system that was compiled for a 64 bit architecture has been removed. The size of the OID type is now always the same as the “natural” size of the architecture. This was a compile-time feature and we haven’t made any binary releases in a long time in which this feature was used. The automatic conversion from a database in which the size of the OID type was 32 to one in which the size is 64 is still in place. This will be removed in a future version.
• The internal type “wrd” has been removed. The “wrd” type was an architecture-dependent integer type, 32 bits wide on a 32-bit system and 64 bits wide on a 64-bit system. Databases where this type is used will automatically have the type converted to either “int” or “lng”, the internal types for 32-bit and 64-bit integers.
• Some old and unused code and interfaces have been removed.
• By making some adjustments to internal parameters, we have been able to get a significant improvements on query speed for many types of queries (in informal tests we have seen a 20% improvement compared to the previous release when running the queries from the TPC-H benchmark).
• A new type of experimental column index has been introduced: the ordered index. Since creating this index is expensive, it is only used when explicitly created. To create an index in SQL use CALL createorderindex(schema,table,column) where the procedure arguments are all strings.
• We have fixed quite a few memory leaks and other bugs.

MonetDB5 Server

• Removed the zorder module with functions zorder.encode, zorder.decode_x and zorder.decode_y.
• Removed command bbp.getHeat().
• Removed bat.setColumn with two arguments and bat.setRole. Use bat.setColumn with one argument instead.
• Removed function BKCappend_reverse_val_wrap: it was unused.
• The “wrd” type has been removed from GDK and MAL. The type was defined to be a 32 bit integer on 32 bit architectures and a 64 bit integer on 64 bit architectures. We now generally use “lng” (always 64 bits) where “wrd” was used.
• Keep a collection of full traces. Each time the SQL user applies the TRACE option, the full json trace is retained within the <dbpath>/<dbname>/sql_traces

Build Environment

• New packages MonetDB-python2 (Fedora) and monetdb-python2 (Debian/Ubuntu) have been created for Python 2 integration into MonetDB.
• With OID size equal to ABI/word size, mserver5 does not need to print the OID size, anymore.
• Removed obsolete code associated with long gone static linking option.
• Removed configure option –enable-oid32 to compile with 32 bit OIDs on a 64 bit architecture.
• The Perl, PHP, and Python clients, and the JDBC driver each now have their own repositories and release cycles. The Python client is maintained by Gijs Molenaar on Github, the other clients are maintained by CWI/MonetDB on our own server.

MonetDB Common

• The tnokey values must now be 0 if it is not known whether all values in a column are distinct.
• The 2-bit tkey field in the bat descriptor has been split into two single bit fields: tkey and tunique. The old tkey&BOUND2BTRUE value is now stored in tunique.
• Implemented conversion to str from any type (not just the internal types).
• VALcopy and VALinit both return their first argument on success or (and that’s new) NULL on (allocation) failure.
• BATattach now can also create a str BAT from a file consisting of null-terminated strings. The input file must be encoded using UTF-8.
• BATattach now copies the input file instead of “stealing” it.
• Removed the lastused “timestamp” from the BBP.
• Removed batStamp field from BAT descriptor, and removed the BBPcurstamp function.
• Removed unused functions BBPhot and BBPcold.
• Removed BATderiveTailProps and BATderiveProps. Just set the properties you know about, or use BATsettrivprop.
• Removed the macro BUNfirst. It can be replaced by 0.
• Changed BATroles by removing the argument to set the name of the head column.
• The head column is now completely gone. MonetDB is completely “headless”.
• The format of the BBP.dir file was simplified. Since the head column is VOID, the only value that needs to be stored is the head seqbase.
• BATs now only have a single (logical) name.
• The function BATmirror is gone. The HEAD column is always VOID (with a non-nil seqbase) and the TAIL column carries the data. All functions that deal with data work on the TAIL column.
• BATkey now works on the TAIL column instead of the HEAD column.
• Replaced BATseqbase with BAThseqbase and BATtseqbase, the former for setting the seqbase on the HEAD, the latter for setting the seqbase on the TAIL.
• Replaced function BATnew with COLnew with slightly different arguments: the first argument of COLnew is the SEQBASE of the head column (which is always VOID).
• The “wrd” type has been removed from GDK and MAL. The type was defined to be a 32 bit integer on 32 bit architectures and a 64 bit integer on 64 bit architectures. We now generally use “lng” (always 64 bits) where “wrd” was used.

SQL

• Removed functions sys.zorder_encode, sys.zorder_decode_x, and sys.zorder_decode_y.
• The experimental recycler code is moved to the attic.
• The syntax of bat.new(:oid,:any) has been changed by dropping the superflous :oid. All BATs are now binary associations between a void column and a materialized value column. (except for the internal :bat[:void,:void] representation of simple oid ranged tails.)

Bug Fixes

• 3357: Implement setQueryTimeout()
• 3445: Add support for database name to dotmonetdb file
• 3597: SQL to MAL listing looses types
• 3973: JDBC hangs
• 3976: Performance enhancement to LIKE without wildcards
• 4005: Correlated update causes incorrect null constraint violation
• 4016: merge table only optimises for point query
• 4040: sys.storage call can take a long time
• 4047: Segfault when updating a dropped table
• 4050: Database corruption when running low on inode
• 4057: missing bulk operations between constant and bat
• 4061: SIGSEGV in candscan_lng
• 4066: Deadlocked monetdbd
• 6068: Error message about incompatible BBP version should be clearer
• 6069: query with union all silently crashes
• 6070: setting negative session query timeout should not be possible/allowed
• 6071: where clause with cast and floor fails to sigsegv
• 6072: Bind to UPD delta column does not get/show type information in EPLAIN
• 6073: Missing type information for constants in MAL explain
• 6074: SET ROLE command does not work
• 6075: gdk_calc.c:13113: BATcalcifthenelse_intern: Assertion `col2 != NULL’ failed.
• 6076: rel_optimizer.c:5426: rel_push_project_up: Assertion `e’ failed.
• 6077: mserver5: rel_optimizer.c:5444: rel_push_project_up: Assertion `e’ failed.
• 6078: rel_bin.c:2402: rel2bin_project: Assertion `0’ failed.
• 6084: Merge table point to wrong columns if columns in partition tables are deleted
• 6108: monetdb5-sql sysv init script does not wait for shutdown
• 6114: segfault raised in the query rewriter due to a null pointer
• 6115: Assertion hit in the codegen
• 6116: Codegen does not support certain kind of selects on scalar subqueries
• 6117: Assertion hit in the query rewriting stage during the push down phase
• 6118: SIGSEGV in strPut due to shared heap
• 6119: Assertion hit in the MAL optimiser on a complex query
• 6120: QUANTILE() treats NULL as if it is zero
• 6121: SELECT a.col IN ( b.col FROM b ) FROM a statements with no error but no result
• 6123: Assertion hit in the codegen #2
• 6124: CASE <column> WHEN NULL THEN 0 ELSE 1 END returns wrong result
• 6125: Stack overflow in the query rewriter with a query having an OR condition and a nested SELECT subexpression
• 6126: batcalc.== can’t handle void BATs
• 6139: Debian libmonetdb13 conflicts with libmonetdb5-server-geom