Jun 2020-SP1 bugfix release (11.37.11)

MonetDB Common

• Hash buckets come in variable widths. But if a BAT grows long enough so that the BAT indexes that are stored in the buckets don't fit anymore, the buckets need to be widened. This is now fixed.

Bug Fixes

• 6844: sys.getUser('https://me:pw@www.monetdb.org/Doc') does not return the user: me
• 6845: the url sys.get...(url) functions do not allow null as a parameter
• 6858: json.keyarray(json '{ "":0 }') fails with error: Could not allocate space
• 6859: only first character of the separator string in json.text(js json, sep string) is used
• 6873: sys.hot_snapshot() creates incomplete snapshots if the write-ahead log is very large
• 6876: tar files created by sys.hot_snapshot() produce warnings on some implementations of tar
• 6877: MonetDB produces malformed LZ4 files
• 6880: Left fuzzy queries are much slower than other fuzzy queries.
• 6882: cgroups limits no longer respected?
• 6883: SQLancer crash on delete query
• 6884: SQLancer generates query with unclear error message
• 6885: SQLancer causes assertion error on UTF8_strlen
• 6886: SQLancer alter table add unique gives strange error message
• 6887: SQLancer crash on complex query
• 6888: SQLancer crash on cross join on view
• 6889: SQLancer crash on long query
• 6892: SQLancer crash on query with HAVING
• 6893: SQLancer inner join reporting GDK error
• 6894: SQLancer crash on rtrim function
• 6895: SQLancer causing 'algebra.select' undefined error
• 6896: SQLancer algebra.select' undefined 2
• 6897: SQLancer distinct aggregate with error on group by constant
• 6898: SQLancer crash on join query
• 6899: SQLancer TLP query with wrong results
• 6900: SQLancer generated SIGFPE
• 6901: SQLancer TLP query with wrong results 2
• 6902: SQLancer query: batcalc.between undefined
• 6903: SQLancer calc.abs undefined
• 6904: SQLancer aggr.subavg undefined
• 6905: SQLancer TLP query with wrong results 3
• 6906: SQLancer crash on complex join
• 6907: SQLancer algebra.select undefined
• 6908: SQLancer inputs not the same size
• 6909: SQLancer query with wrong results
• 6911: SQLancer query: 'calc.bit' undefined
• 6917: Decimal parsing fails
• 6918: SQLancer query compilation error
• 6919: SQLancer insert function doesn't handle utf-8 strings
• 6920: SQLancer project_bte: does not match always
• 6922: Timestamp columns not migrated to new format
• 6923: Imprints data files for timestamp BAT not migrated to the new format
• 6924: SQLancer query copy on unique pair of columns fails and complex query with GDK error
• 6925: Count string rows in union of string tables leaks (RSS) memory
• 6926: SQLancer query with wrong results
• 6927: SQLancer inputs not the same size
• 6928: SQLancer crash on coalesce
• 6929: SQLancer calc.date undefined
• 6932: Syntax error while parsing JSON numbers with exponent
• 6934: sys.isauuid() returns wrong answer for some invalid uuid strings

Jun 2020 feature release (11.37.7)

MonetDB5 Server

• The mserver5 option --verbose (-v) was removed. A similar effect can be had by issuing the query CALL logging.setcomplevel('SQL_TRANS', 'INFO'); as the monetdb user.
• The functions algebra.intersect, algebra.semijoin, and algebra.crossproduct have an extra argument, bool max_one, which indicates that there must be no more than one match in the join.
• The example module opt_sql_append is not installed in the binary packages anymore.
• Added mserver5 option (--dbtrace=<path>) in order to be able to specify the output file any produced traces.
• Removed function bat.setKey().
• algebra.rangejoin now has two extra arguments: anti:bit and symmetric:bit.
• Added session identifier, number of workers and memory claim to the sysmon queue.
• The worker (number of threads), memory (in MB) and optimizer pipeline limits can now be set per user session basis. The query and session timeouts are now set in seconds.
• With required privileges an user can set resource limits for a session.
• There are now versions of group.(sub)group(done) that produce a single output containing just the groups.
• algebra.join and algebra.leftjoin now have forms which return a single column. The column that is returned is the left column of the two column version.
• The MAL profiler now assigns the SQL TRACE output to the client record thereby avoiding the interaction with other queries, but loosing insight of competing queries. The stethoscope should be used for that.

SELinux Support

• There was a problem with the MonetDB SELinux support on Fedora 32. That is fixed in this release. In order to do a proper upgrade of the package if you have already installed MonetDB-selinux on Fedora 32, you may need to uninstall (dnf remove) the old package and then install the new.

Whole System

• A new system to deal with debug output has been implemented. There is now an option --dbtrace to mserver5 that takes a file argument to which debug output is written. The default value is the file mdbtrace.log inside the database directory. This option can also be set through the monetdb program.
• The home directory of the automatically created monetdb user was changed from /var/MonetDB to /var/lib/monetdb (RPM based systems only). This home directory is (currently) not used for anything, though.
• Python 2 support has been removed. There is now only support for using Python 3.
• Removed support for bam and sam files.
• Added mserver5 option (--set raw_strings=true|false) and monetdb database property (raw_strings=yes|no) to control interpretation of strings.

Merovingian

• On Fedora and RHEL systems (not RHEL 6), if monetdbd runs under systemd, when the package is updated, monetdbd (and hence any mserver5 process it runs) is restarted.
• Added dbtrace mserver5 option to the daemon in order to set mserver5's output directory for the produced traces.

Client Package

• The monetdb-client-tools (Debian/Ubuntu) and MonetDB-client-tools (Fedora/RH) containing the stethoscope, tachograph, and tomograph has been removed. A completely new version of stethoscope will be released to replace the old version.
• Removed the possibility of using the MD5 checksum for authentication purposes. It was never actively used but was there as an option. Now the option has been removed.
• Add port and host as fields in the .monetdb file.
• Added 'sessionid' column to system function sys.queue(), so each query gets tagged with the current session identifier
• Allow monetdb user to control session and query time out and selectively stopping a client sessions with a soft termination request.

ODBC Driver

• The NUMERIC and FLOAT types are now handled fully. Before only DECIMAL, FLOAT, and DOUBLE were handled fully.
• Some bugs were fixed in the passing back and forth between application and server of values of type GUID (UUID).

MonetDB Common

• The functions BATintersect, BATsemijoin, and BATsubcross have an extra argument, bool max_one, which indicates that there must be no more than one match in the join.
• The "unique" property on BATs was removed. The property indicated that all values in a BAT *had* to be distinct, but this was not actually used.
• A new type of candidate list has been introduced. Candidate lists are used internally to specify which rows of a column participate in an operation. Before, candidate lists always contained a list of candidate row IDs. The new candidate list type specifies a list of row IDs that should NOT be considered (negative candidates).
• The maximum number of BATs in the system has been increased for 64 bit architectures.
• The hash tables used internally by the system now uses a technique based on Linear Hashing which allows them to grow gracefully. This means that hash tables aren't removed and recreated nearly as often anymore. This also meant that the hash table had to be split into two files, which means that after an upgrade the hash tables have to be recreated.
• Implemented a version of BATproject, called BATproject2, with two "right" arguments which conceptually follow each other.
• Removed MT_mmap and MT_munmap from the list of exported functions. Use GDKmmap and GDKmunmap with the same parameters instead.
• Changed the interface of the atom "fix" and "unfix" functions. They now return a value of type gdk_return to indicate success/failure.
• Removed the tunique property. The tunique property indicated that all values in the column had to be distinct. It was removed because it wasn't used.
• BATrangeselect now has two extra arguments: anti and symmetric (both bool).

SQL Frontend

• Updating the value of a sequence now requires privilege on its own schema.
• The sys.querylog_enable(threshold integer) now actually enables the querylog and uses a threshold in milliseconds.
• Removed UNION JOIN statements. They were dropped by the SQL:2003 standard, plus MonetDB implementation was not fully compliant.
• The OFFSET value in the COPY INTO query now counts uninterpreted newlines. Before it counted "unquoted record separators" which meant that if you had a single quote on a line that you want to skip, you could not use the feature.
• Added support for FROM RANGE MINVALUE TO RANGE MAXVALUE and FROM RANGE MINVALUE TO RANGE MAXVALUE WITH NULL VALUES cases in partitioned tables by range (before they weren't).
• Removed support for Python 2. Python 2 itself is no longer supported. Use Python 3 instead. Functions that were declared as LANGUAGE PYTHON2 or LANGUAGE PYTHON2_MAP are changed to LANGUAGE PYTHON and LANGUAGE PYTHON_MAP respectively (without changing the Python code).
• Implemented 'covar_pop' and 'covar_samp' aggregate functions, as well as their window function counterparts. Implemented 'stddev_samp', 'stddev_pop', 'var_samp', 'var_pop', 'corr' and 'group_concat' window function correspondents.
• Extended SQL catalog with CREATE WINDOW syntax for user-defined SQL:2003 window functions. At the moment, window functions must be defined on the backend engine, i.e. on this case MAL. In the current implementation, the backend code generation creates two additional columns of type lng with the start and end offsets for each row.
• Added prepared_statements_args view, which details the arguments for the prepared statements created in the current session.
• Added sys.prepared_statements view, which lists the available prepared statements in the current session.
• Added deallocate statements with the syntax 'DEALLOCATE [PREPARE] { number | ALL }', to close an existing prepared statement or all, through the SQL layer. Previously this feature was available via MAPI exclusively with the "release" command.
• Added support for raw strings using the syntax r'' or R''. This means that C-like escapes will remain uninterpreted within those strings. For instance SELECT r'\"' returns a string of length two. The user needs to escape single quotes by doubling them: SELECT r''''.
• Implemented ROLLUP, CUBE and GROUPING SETS from SQL:1999. They define grouping subsets used with the GROUP BY clause in order to compute partial groupings. Also, the GROUPING aggregate was added. This aggregate is a bitmask identifying the grouping columns not present in the generated grouping row when used with the operators described above.
• Updated user session procedures by adding the possibility to set properties based on a session identifier. Optimizer pipeline: sys.setoptimizer(int, string) Number of worker threads: sys.setworkerlimit(int, int) Memory limits (in MB): sys.setmemorylimit(int, int) Query timeout (in ms): sys.setquerytimeout(int, int) Session timeout (in ms): sys.setsessiontimeout(int, int) The first argument corresponds to the id of the session to modify, and these procedures are bound to the monetdb user exclusively. The versions of the mentioned procedures with just the second argument were added as well, where the changes are reflected in the current user session, and therefore every user can call them.
• The procedures sys.settimeout(bigint), sys.settimeout(bigint,bigint) and sys.session(bigint) are now deprecated. Instead use sys.setquerytimeout and sys.setsessiontimeout mentioned above.
• Added SQL procedure sys.hot_snapshot() which can be used to write a snapshot of the database to a tar file. For example, sys.hot_snapshot('/tmp/snapshot.tar'). If compression support is compiled in, snapshots can also be compressed ('/tmp/snapshot.tar.gz'). The tar file expands to a single directory with the same name as the database that was snapshotted. This directory can be passed directly as the --dbpath argument of mserver5 or it can be copied into an existing dbfarm and started from monetdbd.

Bug Fixes

• 6298: unexpectedly slow execution of SELECT length(fieldname) FROM tablename LIMIT 1 queries
• 6401: Suspected memory leak in mserver5 when creating/dropping tables
• 6687: Count distinct very slow and use too much the hard drive
• 6723: columns aliases duplicates should not be allowed. automatic aliasing required.
• 6731: Add system view to allow querying of available prepared statements and their parameters
• 6732: Add SQL command to close a specific prepared statement
• 6750: Executing a query on a non-existing column on a remote table crashes the remote server
• 6785: function sys.isaURL(url) should have been declared as sys.isaURL(string)
• 6808: reveal the alarm.sleep procedure in SQL
• 6813: function not_uniques(bigint) returns error when called
• 6818: usage of multiple column expressions in where-clause (f(a), f(b)) in (select a, b) causes assertion failure on mserver5
• 6821: Failed to start monetdb with embedded python
• 6828: Server crashes when executing a window query with ordering by EXTRACT date
• 6846: Global temporary table not accessible in other connections / sessions
• 6847: A simple way of speeding up impscheck for dense canditers
• 6850: Idle timestamp not set
• 6851: json parser doesn't parse integers correctly
• 6863: thash files not released upon drop table
• 6864: (I)LIKE with multiple % doen't find matches

Nov 2019-SP3 bugfix release (11.35.19)

The main fix in this release is the problem with running monetdbd under the control of systemd.  There was also a related problem with the SELinux configuration that was fixed in this release.

Bug Fixes

• 6827: CUME_DIST window function returns incorrect results
• 6829: NTILE window function returns incorrect results

Nov 2019-SP2 bugfix release (11.35.15)

There is a known problem with running monetdbd under the control of systemd. The monetdbd service won't start.  A new release to fix this problem is coming soon.

Bug Fixes

• 6805: Using the cascade operator in a drop table statement ends in an exit from the Monetdb shell.
• 6807: Median_avg and quantile_avg ignore NULL values
• 6815: query with ifthenelse() crashes mserver5
• 6816: Monetdb Crashes on INSERT statement after ALTER statement in another connection
• 6817: running analyze on a schema which contains a stream table stops with an error
• 6819: functions do not persist

Nov 2019-SP1 bugfix release (11.35.9)

Bug Fixes

• 6724: Prepare confuses types when more than one argument is used
• 6726: Python aggregation does not create aggr_group when aggregating over all rows
• 6765: GRANT SELECT privilege on a subset of table columns results in access denied error when selecting the same columns from the table
• 6790: Count distinct giving wrong results
• 6791: str_to_time('11:40', '%H:%M') creates wrong time value
• 6792: JSON path compiler accepts invalid input
• 6793: cast(interval second value to int or decimal) is wrong (by a factor of 1000), cast(interval month value to decimal or floating point) fails
• 6794: external name fits.listdir not bound (sys.listdir) Fatal error during initialization:
• 6796: Incorrect crash time reported by monetdb tool after crash of mserver5
• 6798: json.text off by one error
• 6804: DNS resolution of 0.0.0.0 fails on recent Ubuntus

Nov 2019 feature release (11.35.3)

MonetDB5 Server

• The server "console" has been removed, as has the --daemon option. The server now doesn't read from standard input anymore. The way to stop a server is by sending it a TERM signal (on Linux/Unix) or by sending it an interrupt signal (usually control-C -- on all systems).
• Implemented a function bat.diffcand to calculate difference of two candidate lists.
• The mtime module was completely rewritten, the atom types date, daytime, and timestamp were changed. Upgrade code for BATs containing these types has been implemented. The old daytime type used a 32 bit integer to record milliseconds since the start of the day. The new daytime type uses a 64 bit integer to record microseconds since the start of the day. The old date type recorded days since or before the year 1. The new daytime type records the day of the month and the number of months since the year -4712 separately in a single 32 bit integer of which only 26 bits are used. Dates now use the proleptic Gregorian calendar, meaning the normal Gregorian callendar has been extended backward, and before the year 1, we have the year 0. Both the old and new timestamp types are a combination of a daytime and a date value, but since those types have changed, the timestamp type has also changed. The new date type has a smaller range than the old. The new date range is from the year -4712 to the year 170049. During conversion of date and timestamp columns, the dates are clamped to this range.
• The tzone and rule atom types have been removed. They were not used by any code, and they were defined in a non-portable way.
• Added "mapi_ipv6" property to monet_options to force ipv6 address binding only. This property is inherited while forking from monetdbd if it is also set there.
• Removed (bat)calc.between_symmetric and changed (bat)calc.between by adding a number of extra arguments, all of type :bit: symmetric, low inclusive, high inclusive, nils false.

Merovingian

• Added "vmmaxsize" and "memmaxsize" mserver5 options to the daemon in order to set mserver5's maximum virtual and committed memory respectively.
• Added ipv6 property to monetdbd properties to force IPv6 addresses binding only. By default this property is false to allow IPv4 addresses as well.

MonetDB Common

• BATcalcbetween and all its variants now have an extra bool parameter "anti" to invert the test.
• All forms of BATcalcbetween and VARcalcbetween have extra bool arguments for low inclusive, high inclusive and nils false. The latter causes the result to be false instead of nil if the value being checked is nil.

SQL Frontend

• Removed functions json.text(string) returns string and json.text(int) returns string. Their MAL implementation didn't exist, so they were meaningless.
• There are new aggregate functions sys.median_avg and sys.quantile_avg that return the interpolated value if the median/quantile doesn't fall exactly on a particular row. These functions always return a value of type DOUBLE and only work for numeric types (various width integers and floating point).
• Added sys.deltas ("schema" string, "table" string, "column" string) returns table ("values" bigint) system function which returns a single column with 6 values: a flag indicating if the column's upper table is cleared or not, the count of the RDONLY, RD_INS and RD_UPD_ID deltas of the column itself, the number of deleted values of the column's table, as well as the level of the current transaction in the transaction level tree. It should be used for debugging purposes only.
• Added "VALUES row_list" statement as a top SQL projection statement.
• The implementation of in-expression now follows a join-based approach instead of using iterative union/selects. This greatly improves performance for large in-value-lists. Furthermore the old approach has large in-value-lists generate large MAL plans. This is now no longer the case.
• Strings are now limited to 1GB, double-quoted tokens are limited to 2kB. These sizes are bytes of (UTF-8 encoded) input data.

Bug Fixes

• 3533: SQL aggregate functions avg(), sum() and median() return an error when used on a column with datatype interval second
• 6134: Query produces error: HEAPalloc: Insufficient space for HEAP of 1168033427456 bytes.
• 6613: LATERAL crash /.../rel_bin.c:1473: rel2bin_table: Assertion `0' failed.
• 6683: Bug in subselect
• 6686: Bug in subselect (count function)
• 6688: Bug in subselect (or condition)
• 6689: Trying to improve the performance of SQL queries with a large list of members in IN clause.
• 6695: timestamp transformation
• 6700: Monetdb Bugs in Subselect statements:
• 6722: window functions issues
• 6740: while upgrading the database from verison (MonetDB-11.27.13) to (MonetDB-11.33.3) we are unable to bring up the database
• 6754: in mclient a strang msg is reported after issueing command: set schema sys;
• 6755: Assertion failure in rel_bin.c
• 6756: Error in optimizer garbageCollector on merge tables select
• 6757: Double free or corruption (out)
• 6758: SIGSEGV in __strcmp_sse2_unaligned()
• 6759: Python JSON loader creates invalid data type for strings
• 6761: Error: Program contains errors.:(NONE).multiplex
• 6762: mserver5 crashes on (re-)start
• 6764: mserver5 crashes with corruption, double free, invalid size or invalid pointer
• 6766: Missing bulk implementation for get_value and next_value calls
• 6769: ProfilerStart is not threadsafe
• 6771: R-devel
• 6772: TRACE includes information from all active sessions
• 6773: json.filter returns corrupted string when selecting JSON null value
• 6774: PROD aggregation gives wrong result
• 6775: NOT IN with an AND containing an OR gives wrong result
• 6776: Creating a table with a full outer join query gives type with wrong digits on the joined key.
• 6779: Using Windows Messages translation for errno error codes.
• 6780: Wrong value of the rank function
• 6781: Insert after index creation crash
• 6783: AVG changes scale of its results
• 6784: function sys.isauuid(string) should return false if string value cannot be converted to a UUID

Apr 2019-SP1 bugfix release (11.33.11)

Build Environment

• Removed restriction on using combinations of --enable-assert, --enable-debug, and --enable-optimize. --enable-debug adds a -g (or -even -g3) option, --enable-debug=gdb adds a -ggdb3 flag (for GCC); --enable-optimize adds a bunch of optimization flags; --disable-debug (or --enable-debug=no) removes any -g flags; --disable-optimize removes any -O flags; --enable-optimize=auto leaves the optimization flags untouched.

MonetDB Common

• We now look at the limits imposed by cgroups and the setrlimit system call to initialize some internal values related to how much (virtual) memory we think is available.

SQL Frontend

• Many fixes for using the server in a highly concurrent setting with many clients doing read-only queries mixed with clients doing updating queries.
• Changed the internal representation of RANGE MINVALUE and RANGE MAXVALUE in MERGE TABLE partitions. Before, the limits were represented by the smallest and largest value of the domain of the column, now these are represented by a NULL value. This has the added benefit that if you use TO RANGE MAXVALUE, the largest value of the domain is included in the partition (before it wasn't). Also, these new limits now work for any ordered type, so including VARCHAR.

Bug Fixes

• 6576: Sqlitelogictest aritmetic expressions with negative numbers handling
• 6697: Duplicate expressions not eliminated with long CASE statement
• 6701: When changing the schema name of a table, referencing rows from sys.columns, sys.keys and more tables are not removed
• 6703: SQL optimizer enters loop and goes into stack overflow
• 6706: prepare doesn't recognize merge statement
• 6712: Where clause with cast ignores sub-select
• 6713: COPY INTO FROM 'file.xz' does not work
• 6714: Assertion failure in rel_select.c for correlated subquery in aggregation query (with group by and having and order by and limit)
• 6715: Assertion failure in rel_bin.c for MERGE INTO command
• 6716: COPY INTO does not load UTF8 encoded text
• 6718: Can't copy table into another table with constraints (mkey.bulk_rotate_xor_hash)
• 6720: Compilation issues on gcc 9.1.1 [-Werror=stringop-truncation]
• 6721: Possibly incorrect call to pcre_exec
• 6725: Prepare statement on non-existing function crashes
• 6728: SELECT DISTINCT not removing duplicates
• 6729: Creating a table with duplicate column name should not be possible
• 6730: sys.stop or sys.pause a INSERT query triggers "BATproject: does not match always"
• 6736: Unexpected query result with merge tables and view after upgrade
• 6738: issue with window functions and hugeint type coercion
• 6741: report no such table when execute “alter table tb1 rename to tb2” according to using mapi
• 6744: CREATE SEQUENCE seq; fails
• 6748: MonetDB "forgets" columns after ALTER TABLE statements
• 6749: mserver5 restart aborts/segfaults after dropping column with constraint
• 6751: ODBC driver(11.33.3) Seg Faults when "fn ucase" is used in SQL

Apr 2019 feature release (11.33.3)

MonetDB5 Server

• Changed the way blobs are ordered. Before, shorter blobs came before longer, now the contents is compared first.
• Removed function blob.tostring() since it cannot guarantee that the resulting string is properly encoded in UTF-8.
• Removed the type "sqlblob", changed the external format of the type "blob" to be like what "sqlblob" used to be. In other words, the "blob" type is now SQL-compatible.
• The MAL functions algebra.sort and algebra.firstn now have a new, extra second-to-last argument, nilslast:bit, which indicates where NIL values are to be sorted: at the beginning, or at the end. The old behavior is when for algebra.sort, the value of nilslast is equal to the value or the reverse:bit parameter, for algebra.firstn when the value of nilslast is opposite to the value of the asc:bit parameter.

Build Environment

• We now use the C11 atomic operations if available, although libatomic_ops is still used if present and not disabled on the configure command line.
• We now also build the Python 3 integration packages for Fedora, Debian, Ubuntu, and Windows (not for Debian 8, Ubuntu 14.04, or EPEL 6 and 7 which are too old).

SELinux Support

• There is one tunable parameter, mserver5_can_read_home, which can be set using "setsebool -P mserver5_can_read_home=true" to allow an mserver5 process started by monetdbd under the control of systemd to read files in users' home directories.

Merovingian

• Added a new database property named `profilerbeatfreq`. Its value, <freq>, should be an integer and, if it is set the command `profilerstart` will call stethoscope with this value as the argument to -b. This will instruct stethoscope to collect heartbeat events from the MonetDB server every <freq> milliseconds. Please note that no type checking is done, and if <freq> is not an integer, stethoscope will silently ignore it.

Client Package

• The mclient program can now be quit by typing quit or exit on the command-line. This makes it easier for novice users to quit the program if they do not know that the standard command to quit is \q.

Mapi Library

• Removed function mapi_cache_shuffle.
• Removed function mapi_stream_query.

Stream Library

• The functions mnstr_write{Sht,Int,Lng,Hge} and their Array variants now swap bytes of the written values if the stream is set to swap bytes (i.e. big-endian on a little-endian machine and v.v.).
• Removed defines ST_ASCII, ST_BIN, ST_READ, ST_WRITE from stream.h.
• Changed function mnstr_set_byteorder(stream, char) to mnstr_set_bigendian(stream, bool) where the second argument should be `true' for specifying that the stream is bigendian. This sets an internal flag whether or not to swap bytes which can be retrieved with mnstr_get_swapbytes(stream).
• Changed function `int mnstr_byteorder(stream)' to `bool mnstr_get_swapbytes(stream)'; it now returns whether bytes are swapped.
• Changed function `int mnstr_type(stream)' to `bool mnstr_isbinary(stream)'.

MonetDB Common

• During processing, we now try to maintain the smallest and largest values that occur in a BAT. This information is not saved across server restarts.
• Added an extra argument of type `bool' to the atom to string and string to atom methods to indicate whether the string is for internal use (casting the value to or from a string) of external use (to be printed).
• Implemented a nilslast option for BATfirstn. If set, NILs come last in the ordering that BATfirstn simulates, so non-NIL values are preferentially returned. The old behavior can be obtained by setting nilslast to !asc(ending).
• Implemented a nilslast option for BATsort. This option should be equal to the reverse option for stable sort (it is not implemented for stable sort), but can be different from reverse for non-stable sort. The functions BATsort and GDKqsort have extra parameters, the function GDKqsort_rev has been removed (superseded by GDKqsort with the new `reverse' parameter).
• The BUNtail, BUNtvar, BUNtloc, and BUNtpos macros (and Tloc and Tpos) now return a `void *' instead of a `char *'.
• Function PROPdestroy now takes a BAT* parameter.
• A whole bunch of functions now take a bool argument instead of an int argument. Some functions now return a bool instead of an int. In all these cases, the int was used as a Boolean value.

SQL Frontend

• Implemented two-argument least/greatest functions. If one argument is NULL, the functions return the other value, otherwise they return the least/greatest of the two arguments.
• Implemented SQL2003 natural logarithm function: ln(num_expr).
• Function octet_length now also accepts a BLOB argument and returns the length of the BLOB (i.e. the same as length when called with a BLOB argument).
• Implemented length(blob) and comparison between blobs.
• Implemented a two argument function log which calculates the logarithm of the first argument using the second argument as base.
• Next to the functions log (natural logarithm) and log10 (base 10 logarithm), there is now also log2 (base 2 logarithm).
• Removed unused global SQL variable "history".
• Allow usage of custom expressions in GROUP BY and PARTITION BY clauses. The same expressions can be used in the projection phase as long as it is exactly the same used in the grouping clause (textual lookup is performed). As an example the query: SELECT col1*2 FROM t1 GROUP BY col1*2; is correct, while SELECT sin(col1+5) FROM t1 GROUP BY col1*2; is wrong.
• Allow usage of WITH clauses in insert, update, delete and merge statements.
• Implemented merge statements from SQL:2003 standard. Using a source relation R, a target table T is updating depending on the result of the merge. In case of a match, the table T's row is either updated or deleted with R's row. In a non-match case, R's row is inserted into T. The grammar is the follows: > MERGE INTO target [ [AS] ident ] USING source ON search_condition { WHEN MATCHED [ AND search_condition ] THEN { UPDATE SET assignment_list | DELETE } } | { WHEN NOT MATCHED [ AND search_condition ] THEN INSERT [ column_list ] [ { DEFAULT VALUES | VALUES row_values } ] }
• Added possibility to change the schema of a table with the syntax: > ALTER TABLE [ IF EXISTS ] qname SET SCHEMA ident
• Added optional alias option for target table in update and delete statements: (e.g. UPDATE target AS alias SET col1=1 WHERE alias.col2 < 5)
• Improved and extended storagemodel functionality. Changes include:
• Added views sys."tablestorage" and sys."schemastorage" for easy and quick assesment of aggregated storage usage per table or per schema.
• Excluded listing system tables in views sys."storage", sys."tablestorage" and sys."schemastorage". You can still use sys."storage"() to query storage of system tables and columns.
• In procedure sys.storagemodelinit() when populating table sys.storagemodelinput it now excludes system tables as system tables are not useful to be modeled for storagesize by application users. It now also computes and populates the atomwidth column more correctly for variable size data types (strings, blobs) when the table has representative data in those columns.
• System function sys.columnsize() has been corrected for types: tinyint, real, date, time, timetz, sec_interval, month_interval, decimal, uuid, mbr, char, clob, json, url, blob, geometry and geometrya. For variable size data types (varchar, char, clob, json, url, blob, geometry, geometrya) it now returns the columnsize excluding the variable heapsize. The heapsize is retrievable via sys.heapsize().
• System function sys.heapsize() has been corrected for all variable size data types: varchar, char, clob, json, url, blob, geometry and geometrya.
• System function sys.imprintsize() has been corrected. The imprintsize depends on the width (1 or 2 or 4 or 8 or 16 bytes) of the data type. Also instead of 12% it now uses 20%, which is a better estimate.
• System function sys.storagemodel() has been removed as it outputs the same data as view sys.storagemodel. Use view sys.storagemodel instead.
• Corrected views sys.storagemodel and sys.tablestoragemodel by returning a computed orderidxsize when the count has been changed in the sys.storagemodelinput table. Also the views now return data ordered by schema, table and column names.
• Extended view sys.tablestoragemodel with column: "storages". Besides columns also keys (primary, foreign and unique keys) and indexes (ordered, imprints) use storage, so the "storages" count can be higher than the number of columns per table.
• Corrected the data type of columns "schema", "table", "column", "type", "mode" and location in table sys.storagemodelinput and functions sys."storage"(), sys."storage"(sname), sys."storage"(sname, tname) and sys."storage"(sname, tname, cname) from string into varchar(1024). Consequently also the views based on the table or functions will be reporting varchar(1024) as column meta data instead of clob for those columns. This allows faster querying and reporting by generic SQL programs which treat clob querying different from varchar columns.
• Extended windowing functions catalog with SQL standard semantics. Standard aggregation functions can now be used in windowing functions: (avg,max,min,sum,prod,count). Other windowing specific functions were also implemented: (percent_rank,cume_dist,ntile, lag,lead,first_value,last_value,nth_value).
• The standard frame specification was implemented for aggregation functions as well as first_value, last_value and nth_value functions. The available frames are rows, range and groups. Bounds can be unbounded (partition limit), the current row, a fixed number of rows (constant), or variable (column as input). (e.g SELECT COUNT(col1) OVER (PARTITION BY col2 ORDER BY col3 RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) FROM t1;).
• Added WINDOW keyword which is optionally provided after the FROM clause, with window specifications used in the projection (e.g SELECT SUM(col1) OVER w1, LAST_VALUE(col2) OVER w2 FROM t1 WINDOW w1 AS (ROWS BETWEEN 5 PRECEDING AND 0 FOLLOWING), w2 AS (w1);).
• Our previous partitioning implementation didn’t impose order in the input. With this reexamination, partitioning now imposes ascending order by default, thus pairing with the industry standard implementation.
• Implemented X'...' style binary string literals.
• Implemented U&'...' Unicode character string literals and U&"..." Unicode delimited identifiers, including UESCAPE. For the string literals, you can have U&'...' '...' '...' UESCAPE '...' where the escape must be as single character and the other '...' strings are also Unicode character string literals. For now, these latter strings also undergo C-style backslash interpretation.
• Implemented PostgreSQL-like E'...' strings. The strings can contain C-style backslash escapes. The old format strings '...' currently still also accept C-style escapes, but that feature will be removed in a future release.
• Implemented the NULLS FIRST and NULLS LAST option to ORDER BY. The default is NULLS FIRST for ASC(ending) and NULLS LAST for DESC(ending).
• Added possibility to rename SQL schemas, tables and columns with ALTER statements: > ALTER SCHEMA [ IF EXISTS ] ident RENAME TO ident > ALTER TABLE [ IF EXISTS ] qname RENAME TO ident > ALTER TABLE [ IF EXISTS ] qname RENAME [ COLUMN ] ident TO ident Also added optional IF EXISTS clause to other existing ALTER TABLE statements.
• The sql SAMPLE syntax is extended to include an optional integral SEED parameter that sets the seed for the internal random number generator in the sample algorithm. When the seed is set, the user can obtain a reproducible sample set from a static dataset with the same sample query. The new syntax is SELECT ... FROM ... WHERE ... SAMPLE <expr> [SEED <integer>]
• Implemented a version of COPY INTO that reads/writes files from/to the client instead of doing it in the server. This has the advantage that COPY INTO is then no longer restricted to only the "super user" monetdb, nor only to absolute file names. The syntax to have the server communicate with the client for file content is COPY INTO table FROM file ON CLIENT ...; and COPY query INTO file ON CLIENT ...;. This also works for COPY BINARY INTO. There is also the possibility to specify that files are to be read/written by the server by using ON SERVER. This is also the default when ON CLIENT or ON SERVER is not specified.
• The sys.functions table now has an extra column called "system" of type BOOLEAN whose value is TRUE for system functions (i.e. functions that should not be dumped). The table sys.systemfunctions has been changed to a view and is now officially deprecated.
• Extended merge tables with partitioning using a predicate: > CREATE MERGE TABLE [ IF NOT EXISTS ] table_name (... columns ...) [ PARTITION BY { RANGE | VALUES } { ON '(' column_name ')' | USING '(' expression ')' } ] The partitioning can occur by range or list of values using one of the table's columns or an expression. The domain of the partitioning scheme will be covered by each child table added with alter statements depending if the table is partitioned by range or list of values: > ALTER TABLE merge_table_name { ADD | SET } TABLE child_table_name AS PARTITION IN '(' expression [ ',' ... ] ')' [ WITH NULL ] > ALTER TABLE merge_table_name { ADD | SET } TABLE child_table_name AS PARTITION BETWEEN { RANGE MINVALUE | expression } AND { RANGE MAXVALUE | expression } [ WITH NULL ] > ALTER TABLE merge_table_name { ADD | SET } TABLE child_table_name AS PARTITION WITH NULL When updating a single partition range/list of values, the SET keyword should be used instead of ADD. INSERT, UPDATE and DELETE statements are possible on partitioned tables with corresponding validation on each partition domain. The system tables sys.table_partitions, sys.range_partitions and sys.value_partitions were added to store meta-information about each existing partition.

Bug Fixes

• 2403: stream: http read support in streams
• 2416: GDK: file/dir creation mask
• 2478: SQL: rename columns
• 2496: SQL: implement greatest / least
• 3384: Auxiliary window functions
• 3416: log() SQL function only supports natural logarithm
• 3448: Make lbatomic_ops optional if installed
• 3520: Add support for scalar function LN(num)
• 3530: sigabort on window aggr function
• 3567: Add support for: ALTER TABLE [schema1.]oldtablename RENAME [TO] [schema2.]newtablename
• 3743: DELETE FROM merge table not supported.
• 3832: Cannot use expressions in GROUP BY clause
• 3946: Expose C log2 function to SQL
• 4056: mapi_timeout is not implemented
• 6181: comments cause error with return in function
• 6326: Eliminate unused UNION tables
• 6347: Mserver returns with "memory exhausted in" error if query size exceeds 2.5MB
• 6475: Remove unused SQL global variable "history"
• 6588: Table aliasing are not supported in SQL update queries
• 6591: Name mangling in the SQL catalog results in ambiguity
• 6608: Sqlitelogictest error message in group by with coalesce
• 6609: Rename Schema in MonetDB
• 6636: sys.queue() extension: allow DBA to see all running queries
• 6647: Add suport to Python 3 on Windows
• 6669: COPY [xxx RECORDS] INTO foo FROM STDIN ... doesn't work without specifying nr of to be copied records
• 6671: Error when running user function on merge table with remote part
• 6674: R UDF with Date type fails to convert RAPI
• 6676: Max data length of 2048 for column sys._tables.query is too small for the actual data (2811 for view sys.ids)
• 6678: Binding NULL parameter to parametrized query results in syntax error on execution
• 6680: Copy cannot open CSV file if systemctl enabled
• 6684: Inserting into a replica table crashes the server
• 6685: adding a view to a merge table gives unexpected behaviour
• 6690: Unable to fetch empty string with ODBC driver
• 6696: Re-use of MAL explain plans with complex type coercions

Aug 2018-SP2 bugfix release (11.31.13)

SQL Frontend

• Disabled function sys.getcontent(url).

Bug Fixes

• 6643: schema name qualifier in create global temporary table json.table6643 ... is not honered
• 6645: optimizer treats all the functions with no or constant parameters as constant
• 6650: PREPARE SQL statement fails to compile user defined functions with parameter/s
• 6651: Multi-column IN clause for subquery produces wrong results
• 6653: CREATE TABLE accepts empty table/column name
• 6654: Incorrect handling of 'TRUE' in compound select
• 6657: Restart sequence with a non atomic sub-query cardinality gives wrong error message
• 6660: GRANTing a ROLE is not idempotent
• 6662: Concurrency Conflicts Exception string "!transaction is aborted because of concurrency conflicts, will ROLLBACK instead" not shown on prompt.
• 6664: mserver5 crash: infinite recursive happens at rel_bin.c:489
• 6665: Creation of serial types does not accept negative numbers
• 6666: COPY INTO from .. LOCKED doubles input data
• 6668: The SAMPLE key word doesn't work in a subquery.
• 6669: COPY [xxx RECORDS] INTO foo FROM STDIN ... doesn't work without specifying nr of to be copied records
• 6672: SQLGetData with SQL_C_WCHAR string truncation and invalid StrLen_or_Ind value

Aug 2018-SP1 bugfix release (11.31.11)

Build Environment

• On Ubuntu 18.10 (Cosmic Cuttlefish), the libmonetdb5-server-bam package cannot be built because of an incompatibility in the libbam library (it cannot be used in a shared object).

Whole System

• Some subtle dependencies between RPMs have been fixed.

Bug Fixes

• 6640: timestamp_to_str returning incorrectly adjusted results
• 6641: race condition in SQL UDF with update
• 6642: Hanging query
• 6648: key property potentially wrong after type conversion
• 6649: Projection inside within transaction gives wrong results

Aug 2018 feature release (11.31.7)

MonetDB5 Server

• The lsst module was moved to a separate repository (https://dev.monetdb.org/hg/MonetDB-lsst/).

Build Environment

• Build the MonetDB-cfitsio RPM and libmonetdb5-server-cfitsio Debian/Ubuntu package.
• On Windows, the separate MonetDB5-Geom installer has been incorporated into the main MonetDB5-SQL installer and is therefore no longer available as a separate download.

Merovingian

• Added a "logrotate" configuration file. See /etc/logrotate.d/monetdbd.
• Changed the monetdb profilerstart command to be more robust. If the server or stethoscope crashed before, the pid file is still there, so the next time we try to start stethoscope, it will fail. Now the profilerstart command will check if a stethoscope process with the recorded pid is running. If not, we start stethoscope, assuming that something went wrong before.
• Changed the monetdb stop command to try to stop stethoscope before stoping the server. The error conditions that can arrise from attempting to stop stethoscope are:
  • The database is not running.
  • The profilerlogpath is not set.
  • The profiler.pid file does not exist or is inaccessible.
  • The contents of the profiler.pid are not valid.
  • Shutdown of stethoscope did not succeed.
  • Removing the profiler.pid file failed.
  In all the cases, the attempt to stop the server can continue normally, so we actually ignore any errors that rise from the attempt to stop stethoscope.

Client Package

• ODBC: Implemented SQL_ATTR_QUERY_TIMEOUT parameter in SQLSetStmtAttr.
• ODBC SQLGetInfo now returns a positive numeric value for InfoTypes: SQL_MAX_COLUMN_NAME_LEN, SQL_MAX_DRIVER_CONNECTIONS, SQL_MAX_IDENTIFIER_LEN, SQL_MAX_PROCEDURE_NAME_LEN, SQL_MAX_SCHEMA_NAME_LEN, SQL_MAX_TABLE_NAME_LEN and SQL_MAX_USER_NAME_LEN.
• Added a '-f rowcount' option in mclient to repress printing the actual data of a resultset, but only print the number of returned tuples

Stream Library

• Added support for lz4 compressed files in the stream library

MonetDB Common

• Hash indexes are now persistent across server restarts.
• The macros bunfastapp and tfastins and variants no longer set the dirty flag of the heap they write to. This now needs to be done separately (and preferably outside of the inner loop).
• Removed batDirty flag from BAT record. Its function is completely superseded by batDirtydesc and the dirty flags on the various heaps.
• Removed "masksize" argument of function BAThash.
• A whole bunch of functions that took an int argument that was used as a Boolean (true/false) value now take a value of type bool. The functions BATkeyed, BATordered and BATordered_rev now return a bool instead of an int.
• Removed the tdense property: it's function is completely replaced by whether or not tseqbase is equal to oid_nil.

Testing Environment

• Removed helper programs Mtimeout and MkillUsers: they have long been superseded by timeout handling by Mtest.py itself.

SQL Frontend

• Removed deprecated table producing system functions: sys.dependencies_columns_on_functions() sys.dependencies_columns_on_indexes() sys.dependencies_columns_on_keys() sys.dependencies_columns_on_triggers() sys.dependencies_columns_on_views() sys.dependencies_functions_on_functions() sys.dependencies_functions_on_triggers() sys.dependencies_keys_on_foreignkeys() sys.dependencies_owners_on_schemas() sys.dependencies_schemas_on_users() sys.dependencies_tables_on_foreignkeys() sys.dependencies_tables_on_functions() sys.dependencies_tables_on_indexes() sys.dependencies_tables_on_triggers() sys.dependencies_tables_on_views() sys.dependencies_views_on_functions() sys.dependencies_views_on_triggers() They are replaced by new system dependency_* views: sys.dependency_args_on_types sys.dependency_columns_on_functions sys.dependency_columns_on_indexes sys.dependency_columns_on_keys sys.dependency_columns_on_procedures sys.dependency_columns_on_triggers sys.dependency_columns_on_types sys.dependency_columns_on_views sys.dependency_functions_on_functions sys.dependency_functions_on_procedures sys.dependency_functions_on_triggers sys.dependency_functions_on_types sys.dependency_functions_on_views sys.dependency_keys_on_foreignkeys sys.dependency_owners_on_schemas sys.dependency_schemas_on_users sys.dependency_tables_on_foreignkeys sys.dependency_tables_on_functions sys.dependency_tables_on_indexes sys.dependency_tables_on_procedures sys.dependency_tables_on_triggers sys.dependency_tables_on_views sys.dependency_views_on_functions sys.dependency_views_on_procedures sys.dependency_views_on_views
• Implemented group_concat(X,Y) aggregate function which also concatenates a column of strings X, but using a supplied string Y as the separator. This function is also a SQL extension.
• Implemented group_concat(X) aggregate function which concatenates a column of strings using a comma as a separator. This function is not featured in the SQL standard.

Bug Fixes

• 4020: Importing timestamp with zone from copy into
• 6506: Improper performance counters
• 6556: Sqlitelogictest division by zero on COALESCE call
• 6564: Changes to the Remote Table definition
• 6575: Sqlitelogictest crash on groupby query with coalesce call
• 6579: Sqlitelogic test infinite loop while compiling SQL query
• 6586: Sqlitelogictest crash on complex aggregation query
• 6593: Poor performance with like operator and escape clause
• 6596: Multicolumn aggregation very slow after ANALYZE when persistent hashes are enabled
• 6605: Sqlitelogictest set queries with wrong results
• 6606: Misleading parameter name in generate_series function
• 6610: Sqlitelogictest algebra.rangejoin undefined
• 6611: Cannot compile with GCC 8.1 and --enable-debug=no
• 6612: Implement BLOB handling in python UDFs
• 6618: dependency column on sequence violated by DROP SEQUENCE
• 6621: SELECT FROM REMOTE TABLE WHERE <> returns wrong results
• 6624: "Cannot use non GROUP BY column in query results without an aggregate function" when using aggregate function in both HAVING and ORDER BY clauses.
• 6625: OR in subselect causes the server to crash with segmentation fault
• 6627: stddev_pop inconsistent behaviour
• 6628: User cannot insert into own local temporary table
• 6629: CREATE TABLE IF NOT EXISTS returns 42000!
• 6630: Sqlitelogictest cast NULL to integer failing
• 6632: Dataflow causes crash when THRnew fails
• 6633: ILIKE clauses don't work on certain characters
• 6635: monetdbd exits due to "Too many open files" error
• 6637: Within a transaction, \d after an error causes mclient to exit
• 6638: (sequences of) mkey.bulk_rotate_xor_hash() can generate NIL from non-NIL making multi-col joins return wrong results
• 6639: COMMENT ON TABLE abc IS NULL invalidly sets the remark column to null where remark column is defined as NOT NULLable

   

Mar 2018-SP1 bugfix release (11.29.7)

Build Environment

• On Linux and Unix, try not to link to libraries multiple times: that only causes the code (and worse, the variables in those libraries) to be included multiple times. On Windows, we do need to link to libraries multiple times (once for each DLL we create if the DLL references anything from the library) but that doesn't cause the linked library to be loaded multiple times.

SQL Frontend

• Corrected the definition of view: sys.ids.

Bug Fixes

• 6562: Sqlitelogictest crash on group by query with not in operator
• 6565: Sqlitelogictest crash on complex select query with coalesce call
• 6566: Sqlitelogictest unavailable calc.- MAL operations
• 6568: Sqlitelogictest crash on complex case query
• 6569: Sqlitelogictest select query with not between cause with wrong results
• 6570: Sqlitelogictest select coalesce undefined calc
• 6572: ordered index Error in optimizer garbageCollector
• 6573: Sqlitelogictest crash on complex select query with case statement
• 6574: server crashed could not find ordered index while creating a table
• 6576: Sqlitelogictest aritmetic expressions with negative numbers handling
• 6577: creating temp table kills performance of the original query
• 6578: One two-tuple insert gives different results than two single inserts
• 6581: Join condition errors.
• 6583: Fixed size string concatenation with integer results in fixed size string of size 0
• 6584: SELECT FROM REMOTE TABLE WHERE IS NOT NULL produces wrong result
• 6585: Nested Merge tables cause an infinite loop in rel_optimizer
• 6587: Sqlitelogictest crash on complex case statement
• 6589: Sqlitelogictest crash on complex on complex expression
• 6594: Sqlitelogictest crash on complex case statement
• 6595: Remote decimal division triggers assertion / returns wrong answer
• 6598: Python 3.4 not supported (due to usage of Py_DecodeLocale)
• 6600: Sqlitelogictest queries fail to execute
• 6601: "where is null" clause on remote table causes problem with next query
• 6602: Sqlitelogictest wrong results in IN query
• 6603: Sqlitelogictest: Aggregation query with distinct clause produces duplicated rows

Mar 2018 feature release (11.29.3)

MonetDB5 Server

• Implemented function pcre.replace_first which is like pcre.replace, except it only replaces the first match.
• The EXPLAIN command now shows all the MAL type resolutions, because in general users may not be aware of their signatures. It also simplifies programs to analyze such plans.
• Implemented versions of group.(sub)group(done) that don't return a histogram.
• Removed MAL functions streams.socketRead, streams.socketReadBytes, streams.socketWrite, and streams.socketWriteBytes.
• Removed MAL functions streams.openRead(s:streams):streams and streams.openWrite(s:streams):streams.
• Lots of changes to streamline the internal error handling. In principle, all errors should now include a SQLSTATE error code (see the SQL standard).

Build Environment

• Added the .pdb files needed for debug symbols to the Windows installer for MonetDB/SQL.

Whole System

• mclient's execution time profiling options and output format have been changed. Instead of implicitly via the "--interactive"/"-i" option (or when using an interactive mclient console), execution time profiling is now controlled via an explicit "--timer=timermode"/"-t timermode" command-line option, or a "\t timermode" command in the interactive mclient console. The default (also in the interactive mclient console) is now timermode "none", i.e., no timing information is given. Timermode "clock" activates client-side wall-clock timing ("clk") in "human-friendly" format much like the interactive mode did before. Timermode "performance" also provides detailed server-side timings: "sql" is the time to parse the SQL query, optimize the logical relational plan and create the initial physical (MAL) plan; "opt" is the time to optimize the physical (MAL) plan; "run" is the time to execute the physical (MAL) plan. With timermode "performance" all server-side timings and the client-side wall-clock time are given in milliseconds (ms). Note that the client-measured wall-clock time "clk" is reported per query only when options "--interactive" or "--echo" are used, because only then does mclient send individual lines (statements) of the SQL script to the server. Otherwise, mclient sends the SQL script in large(r) batch(es) to the server, and, thus, only the total wall-clock time per batch is measured and reported. The server-measured detailed performance timings "sql", "opt", "run" are always measured and reported per query. Also, all timing information is now given on a separate line and sent to stderr rather than stdout.
• Some types and constants were moved from configure (and hence monetdb_config.h) to gdk.h. In particular, the types "lng" and "ulng" have been moved and can therefore no longer be used by code that doesn't (ultimately) include gdk.h. Just use int64_t instead. A bunch of format defines have been removed: SZFMT, SSZFMT, PTRFMT, PDFMT. Just use the C standard codes for those (%zu, %zd, %p, %td). The define for printing a lng (LLFMT) was also moved. Use PRId64 for printing int64_t in code not using gdk.h. Removed all references to __int64 and long long (use int64_t instead).

Merovingian

• Add daemon commands for starting/stopping collection of profiler (stethoscope) logs.

Client Package

• ODBC: The driver function SQLProcedureColumns was implemented.
• ODBC: Changed table types as used by SQLTables from "LOCAL TEMPORARY" and "GLOBAL TEMPORARY" to "LOCAL TEMPORARY TABLE" and "GLOBAL TEMPORARY TABLE" respectively.
• ODBC: Initial support for the HUGEINT type in SQL was added. Any value with type HUGEINT can be retrieved in a C variable with type SQL_C_CHAR or SQL_C_WCHAR. A value of type HUGEINT can be retrieved in other C types as long as they fit, the largest C type supported being a 64 bit integer (equivalent to BIGINT).
• The functions in the mapi library that require 64 bit integers now use the standard type int64_t instead of the non-standard mapi_int64. This requires a compilation environment that has the stdint.h include file (standardized in C99). Compilation of the library also requires the inttypes.h include file (also standardized in C99).
• Add a new pretty printing option to stethoscope Running stethoscope with the flag -j will produce not pretty printed output (one json object per line). Running with the -y flag will produce pretty printed output. Running with neither will produce the legacy, line oriented format

mapilib

• The three mapi_explain* functions and mapi_trace don't return any useful information, so they now return void.
• The functions mapi_error_str and mapi_result_error now return const char * instead of plain char * to indicate that the returned data belongs to the library and should not be changed or freed by the application.
• New function const char *mapi_result_errorcode(MapiHdl) which returns the SQLSTATE code if available when an error has occurred.

stream

• The interface of mnstr_fgetpos and mnstr_fsetpos was changed to look more like the standard C functions fsetpos and fgetpos: they both have a second argument "pointer to fpos_t".
• Removed function wbstream.
• Removed functions udp_rastream and udp_wastream.
• Removed functions socket_rstream and socket_wstream.
• Removed functions append_wstream and append_wastream.
• Removed functions mnstr_rstream and mnstr_wstream.

utils

• The function mcrypt_getHashAlgorithms now returns a static, constant string, so the result should not be modified or freed.

MonetDB Common

• Changed return type of function void_replace_bat from BUN to gdk_return: it now only returns whether the operation succeeded or not.
• Changed the return type of BATroles from void to gdk_return: it can fail due to malloc failure.
• Removed functions ALIGNsetH, ALIGNsetT, and CREATEview_ (mind the underscore). The first can easily be replace by using BAThseqbase (that's all it did), the second was only used once, and the third can be replace by VIEWcreate.
• Removed unused functions BATmemsize and BATvmsize.
• Removed the tnodense property: it was maintained but never actually used, not even stored.
• The NIL representation of the internal flt and dbl types was changed from the smallest representable finite value to NaN (not-a-number).
• Changed the interface of ATOMformat and VALformat: they now return a pointer to the allocated string.
• The length "method" for atoms now returns a size_t, the "len" field of a ValRecord is now a size_t, the "size" field of the atomDesc structure is now unsigned short.
• Removed the "align" field from the ATOM descriptor (atomDesc) structure.
• The atomtostr and atomfromstr "methods" for atoms now return ssize_t and require a pointer to size_t for the size of the buffer.
• The atom tostr and fromstr "methods" now always return -1 on error. A return value greater than 0 is normal, a return value of 0 is not normal, but technically not an error.

Testing Environment

• Added a --data_path option to Mtest.py that defines an external data repository. See the commit message of c484932c7fd8 for more info.

SQL

• Extended support to use CREATE ORDERED INDEX on columns of type: char, varchar, clob, blob, url, json, inet and uuid.
• Added new system view: sys.ids which contains all database objects ids which can be used in sys.dependencies table.
• Added new system view: sys.dependencies_vw which shows all data of sys.dependencies including names on objects, object types and dependency types.
• Added 25 new system views for finding out dependencies between database objects. These new dependency views improve, extend and replace the 17 sys.dependencies_X_on_Y() functions as previously defined in 21_dependency_functions.sql. Those sys.dependencies_X_on_Y() functions are now marked as deprecated.
• Added new system view: sys.roles which contains all defined roles.
• Added new system view: sys.var_values which shows the values for system variables.
• Added support for COMMENT ON statements using SQL syntax: COMMENT ON { SCHEMA | TABLE | VIEW | COLUMN | INDEX | SEQUENCE | FUNCTION | PROCEDURE | AGGREGATE | FILTER FUNCTION | LOADER } qname IS { 'my description text' | NULL | '' } ; For COLUMN the qname can be "table_name"."column_name" or fully qualified as in: "schema_name"."table_name"."column_name". For FUNCTION, PROCEDURE, AGGREGATE, FILTER FUNCTION and LOADER the qname may need to include the signature (argument types) to be able to differentiate between multiple overloaded functions which have the same name and schema. By specifying IS NULL or IS '' you remove the comment for the database object. If a database object is dropped, the associated comment is also removed. Note: it is not allowed or possible to add comments for temporary tables or objects in schema "tmp". The sql catalog has been extended with system table: sys.comments. The keyword 'COMMENT' has now become a reserved keyword.
• Removed system function sys.environment(). It was a duplicate of system function sys.env().
• Implemented behavior for DROP SCHEMA my_schema RESTRICT command. Previously the RESTRICT keyword was accepted but not obeyed. It would always do a CASCADE operation. Also the default behavior of DROP SCHEMA my_schema command is now changed into RESTRICT behavior (was CASCADE).
• The internal NULL representation of the REAL (FLOAT) and DOUBLE types was changed from the smallest representable finite value to NaN (not-a-number).
• A column default value can be used in a UPDATE statement: UPDATE tname SET cname = DEFAULT, and INSERT statements: INSERT INTO tname VALUES (..., DEFAULT, ...)
• Added support for TRUNCATE statements conforming to the SQL:2008 standard: TRUNCATE [ TABLE ] qname [ CONTINUE IDENTITY | RESTART IDENTITY ] [ RESTRICT | CASCADE ] In a TRUNCATE statement a 'CONTINUE IDENTITY' or 'RESTART IDENTITY' clause can be passed to restart or not, being the former the default one. The 'CASCADE' option instructs to truncate referencing table(s) also if the referencing table(s) have foreign key references to this table. The default behavior is 'RESTRICT'. Note: it is possible to use TRUNCATE statements in a transaction and thus to rollback the effects of a truncate. The keywords 'TRUNCATE' and 'CONTINUE' have now become reserved keywords.
• Added possibility to GRANT or REVOKE a TRUNCATE privilege to a user or role.
• Added possibility to define a TRIGGER on a TRUNCATE event.
• Added possibility to specify 'OR REPLACE' in following CREATE commands: CREATE [ OR REPLACE ] VIEW qname ... CREATE [ OR REPLACE ] TRIGGER qname ...
• Added possibility to specify 'IF EXIST' in following DROP commands: DROP AGGREGATE [ IF EXISTS ] qname ... DROP FUNCTION [ IF EXISTS ] qname ... DROP FILTER FUNCTION [ IF EXISTS ] qname ... DROP LOADER [ IF EXISTS ] qname ... DROP PROCEDURE [ IF EXISTS ] qname ... DROP TRIGGER [ IF EXISTS ] qname ...
• Lots of changes to streamline the internal error handling. In principle, all errors should now include a SQLSTATE error code (see the SQL standard).
• Added support for extracting the quarter (number between 1 and 4) of a date or a timestamp or a timestamp with timezone in SQL: EXTRACT ( QUARTER FROM my_date_expr ). Added support for extracting the week (number between 1 and 53) of a date or a timestamp or a timestamp with timezone in SQL: EXTRACT ( WEEK FROM my_date_expr ). Added support for scalar functions: quarter(date_expr), quarter(timestamp_expr) and quarter(timestamptz_expr).

Bug Fixes

• 3574: Add support for: create OR REPLACE view ...
• 3824: Created table not visible from ODBC
• 3831: Extend date part extraction and date formating functions to support more formats like "quarter", "iso year/week"
• 6244: Add support for: TRUNCATE TABLE
• 6249: DEFAULT in row-values missing (sqlsmith)
• 6346: BATsort returns GDK_SUCCEED when **sorted bat is null
• 6438: Implement functionality to enforce the restrict option in: DROP SCHEMA xyz RESTRICT;
• 6507: Column Header coming with trailing spaces in compiled version of MonetDB 11.27.9/11
• 6513: Sqlitelogictest: Wrong MAL plan generation for column product
• 6526: Crash using aggregate function inside a case statement in having clause
• 6529: Sqlitelogictest crash in select query with IN operator and cast
• 6530: Sqlitelogictest: select query with NOT IN giving wrong results
• 6532: copy into ignore null as directive if first column doesn't come from file
• 6534: [Mar2018]: mclient -f tab / --format=tab complains about "unsupported formatter"
• 6535: [Mar2018]: mclient -t / --timer does not work as documented
• 6536: [Mar2018]: timing output of mclient -t / --timer= should go to stderr rather than stdout
• 6537: [Mar2018]: mclient's default timing mode should (again) be "none" rather than "clock"
• 6541: [Mar2018]: mclient reports incorrect wall-clock time
• 6542: assertion failure when querying: select count(*) from sys.commented_function_signatures;
• 6543: Mar2018: truncate on SQL system tables should NOT be allowed
• 6545: Sqlitelogictest crash in IN query
• 6546: Sqlitelogictest crash in IN query with division
• 6547: OS-dependent behaviour for ilike
• 6548: Select from remote table leaves session open
• 6549: Add log messages when listen fails
• 6550: Sqlitelogictest crash on complex CASE statement
• 6551: Sqlitelogictest wrong NULL value cast
• 6552: Sqlitelogictest crash on complex case statement
• 6553: Sqlitelogictest crash on aggregation with having statement
• 6554: Sqlitelogictest crash on complex case statement
• 6555: Sqlitelogictest wrong result set generated from complex case statement
• 6557: Sqlitelogictest crash on aggregation query with not in
• 6559: rows in sys.statistics are not removed when a temporary table is dropped.
• 6560: Sqlitelogictest crash on group by query with having in
• 6561: Sqlitelogictest crash on group by query with having not in clause

 

 

Jul 2017-SP4 bugfix release (11.27.13)

Build Environment

• Added the .lib and .h files needed for building extensions to the Windows installer.

Bug Fixes

• 3470: Support setClob without length restrictions
• 6468: JDBC 2.27 fails with year < 1000
• 6482: Query failures on order by on union
• 6483: Monetdb crashes, on query
• 6487: UNION of NULLs with several tables fails
• 6488: Semijoin returns duplicate values from a column with unique values
• 6489: Sqlitelogictest - Wrong result set of complex conditional query
• 6490: Sqlitelogictest - Select query with an IN clause parse error
• 6491: SELECT IN returns NULL instead of false when not found
• 6492: Persistent hashes stored and then ignored. Storage info not in sync with actual indices.
• 6493: Sqlitelogictest - Aggregation query on empty table with DISTINCT clause
• 6494: Sqlitelogictest - Algebra operators priority in select query
• 6495: Sqlitelogictest - Omitting AS in a result set column alias name
• 6496: Sqlitelogictest - Select interval comparisons between floating-points and NULL
• 6497: Sqlitelogictest - Select not between query producing wrong results
• 6498: large virtual memory spike on BLOB column COUNT
• 6499: Crash when trying to replace a function defined in sys from a different schema
• 6502: Query with multiple limit clauses does not return anything
• 6508: Segmentation fault in mserver5 on Python2 UDF with TIMESTAMP column input that has NULL values (conversion.c:438, PyNullMask_FromBAT)
• 6510: Sqlitelogictest: Wrong output in aggregation query
• 6512: Monetdb crashes on query with limit after sort with case
• 6514: Sqlitelogictest: Range query between NULL values not possible
• 6515: Insert null second interval value results in 0
• 6516: Sqlitelogictest unknown bat append operation
• 6517: Sqlitelogictest overflow in conversion during MAL plan execution
• 6518: Sqlitelogictest: count aggregation with not in operator
• 6519: Sqlitelogictest: algebra join between lng and int BATs undefined
• 6520: UPDATE with correlated subquery causes assertion (or segfault)
• 6522: Sqlitelogictest: IN operator return a single column
• 6523: Sqlitelogictest: Case statement subquery missing
• 6524: Sqlitelogictest: Crash in aggregation query with IN operator
• 6527: Crash using order by alias in subquery

Jul 2017-SP3 bugfix release (11.27.11)

MonetDB Common

• Reimplemented summing of a column of floating point (flt and dbl) values. The old code could give wildly inaccurate results when adding up lots and lots of values due to lack of precision. Try SELECT sum(c) FROM t; where t is 100,000,000 rows, c is of type REAL and all values are equal to 1.1. (The old code returned 33554432 instead of 1.1e8.)

Bug Fixes

• 3898: Deadlock on insertion
• 6429: ROUND produces wrong data type
• 6436: Query sequence with 2x ifthenelse() and next nullif() causes mserver5 Segmentation fault
• 6437: System schemas "profiler" and "json" shouldn't be allowed to be dropped.
• 6439: Invalid references to sys.columns.id from sys.statistics.column_id
• 6442: SEGFAULT with COPY INTO BEST EFFORT and skipping input columns
• 6443: complex(?) query forgets(?) column name
• 6444: Using 'with' keyword with table returning function crashes monetdb
• 6445: Sqlitelogictest crash in MySQL query
• 6446: sql_parser.y bug?
• 6448: 'insert into' with multiple rows containing subqueries crashes
• 6449: Assertion error in rel_dce_refs (sqlsmith)
• 6450: Assertion error in exp_bin (sqlsmith)
• 6451: Assertion error in sql_ref_dec (sqlsmith)
• 6453: Assertion error in rel_rename_exps (sqlsmith)
• 6454: SQL lexical error
• 6455: Assertion error in rel_apply_rewrite (sqlsmith)
• 6456: NULL becomes 0 in outer join
• 6459: Assertion error in exp_bin (sqlsmith)
• 6462: large virtual memory spike on BLOB column select
• 6465: appending to variables sized atom bats other than str bats with force flag may result in corrupted heap
• 6467: date_to_str formatter is wrong
• 6470: mitosis gets in the way of simple select
• 6471: calls to sys.generate_series should auto-convert arguments
• 6472: Assertion failure in rel_rename (Sqlsmith)
• 6477: assertion eror rel_push_project_up (sqlsmith)
• 6478: Crash with nested order by/ limit offset
• 6479: Mserver receives an assertion error on a procedure call
• 6480: Segfault in mvc_find_subexp (sqlsmith)

Jul 2017-SP2 bugfix release (11.27.9)

MonetDB Common

• A serious bug, possibly resulting in database corruption, having to do with appending data to a string BAT was fixed.

Bug Fixes

• 4017: server crashes when executing particular loopback query in embedded python
• 6207: identifier ambiguous when grouping and selecting the same column twice
• 6239: Incorrect profiling
• 6261: New handling of delta tables hurts badly reusage of bats
• 6287: should the CORR function return some numeric type that allows fractions?
• 6321: Two-column aggregation on join result extremely slow.
• 6335: Sqlitelogictest crash in complex SQL query
• 6343: MERGE TABLE issue: unable to find
• 6348: Interference of procedure/table name
• 6350: Carriage return and form feed in TEXT fields are returned as 'r' and 'f' in jdbcclient and ResultSets
• 6352: Scope resolution problem (sqlsmith)
• 6353: implicit NULL value not propagated in distributed/remote query
• 6374: Wrong answer from merge table after content changes
• 6379: Table UDF: SEGV raised when invoking a non existing function
• 6380: unable to create new databases from clean installation
• 6381: Parser misses error messages in conditional
• 6382: Can't set JSON fields via PreparedStatement
• 6384: crash when setting a wrong listenaddr
• 6385: AGGREGATE UDFs with more than 2 parameters incorrectly processed
• 6386: Unexpected error from server for query with long floats
• 6387: Performance degradation on multi column sort
• 6388: JDBC Connection via user voc produces errors when fetching certain meta data information
• 6392: SELECT EXISTS (empty table) returns 'true'
• 6395: BAT leak of scalar result sets
• 6397: Isolation of generating functions not correct
• 6398: Null Matches in outer join are not supported
• 6399: UDF crashes when subquery and scalar values are passed as pameters
• 6400: getCharacterStream() currently not supported
• 6404: COPY INTO crashes if table has primary key or foreign key constraint
• 6405: Compilation error if DISABLE_PARENT_HASH not defined
• 6409: sqllogictest crash on aggregation query with NOT IN clause in HAVING clause
• 6410: Sqlitelogictest crash on aggregation query with IN clause
• 6411: Sqlitelogictest crash in aggregation query
• 6412: Overflow in sys.epoch
• 6413: Wrong answer for TPC-H Q17
• 6414: Using RPAD returns: could not allocate space
• 6416: Sqlitelogictest crash in aggregation query with a NOT IN clause
• 6417: Segfault encountered (sqlsmith)
• 6418: Segfault in renaming (sqlsmith)
• 6419: segfault in rel_optimizer (sqlsmith)
• 6420: Assertion error in mergetable task (sqlsmith)
• 6422: Another assertion error in rel_or (sqlsmith)
• 6423: Dereference null pointer (sqlsmith)
• 6424: Assertion error in rel_rename_expr (sqlsmith)
• 6425: Assertion error in exp_bin (sqlsmith)
• 6426: Assertion error in rel_find_exp_ (sqlsmith)
• 6427: Assertion error in eq_typeswitchloop (sqlsmith)
• 6428: Sqlitelogictest crash in aggregation query
• 6430: Assertion raised in another eq_typeswitch error (sqlsmith)
• 6431: Sqlitelogictest crash in aggregation query with a long having clause
• 6432: Assertion error in exp_bin (sqlsmith)
• 6433: Sqlitelogictest crash in complex SELECT query with IN operator
• 6435: Sqlitelogictest crash in simple select query
• 6437: System schemas "profiler" and "json" shouldn't be allowed to be dropped.
• 6438: Implement functionality to enforce the restrict option in: DROP SCHEMA xyz RESTRICT;
• 6440: Faulty plan generated. Query returns more rows than expected or existing in the view sys.tables.

Important: we found a bug in the Jul2017 and Jul2017-SP1 releases (only) that can corrupt your database.  We recommend to not upgrade to these two releases, but to skip directly to the Jul2017-SP2 release. See this email message.

Jul 2017-SP1 bugfix release (11.27.5)

Debian and Ubuntu uses may want to reimport the MonetDB GPG public key as per the instructions on the download page.  See bug 6383 for more information.

Build Environment

• The Debian and Ubuntu installers have been fixed: there was a file missing in the Jul2017 release.
• Added a new RPM called MonetDB-selinux which provides the SELinux policy required to run MonetDB under systemd, especially on Fedora 26.
• The Windows installers (*.msi files) are now created using the WiX Toolset.
• The Windows binaries are now built using Visual Studio 2015. Because of this, you may need to install the Visual C++ Redistributable for Visual Studio 2015 before being able to run MonetDB.

Merovingian

• monetdbd was leaking open file descriptors to the mserver5 process it started. This has been fixed.

MonetDB Common

• Many functions in GDK are now annotated with the GCC attribute __warn_unused_result__ meaning that the compiler will issue a warning if the result of the function (usually an indication of an error) is not used.

Bug Fixes

• 6325: Merge table unusable in other connections
• 6328: Transactional/multi-connection issues with merge tables
• 6336: VALUES multiple inserts error
• 6339: Mserver5 crashes on nested SELECT
• 6340: sample operator takes effect after the execution of the query, expected before
• 6341: MERGE TABLE issue: Cannot register
• 6342: MERGE TABLE issue: hang
• 6344: Spurious errors and assertions (SQLsmith)
• 6375: MAL profiler truncates JSON objects larger than 8192 characters

Jul 2017 feature release (11.27.1)

On Fedora 26 there is a known issue with this release.  When you use MonetDB under systemd (if you have enabled the system service with systemctl enable monetdbd.service), and you have SELinux (Security-Enhanced Linux) enabled, the database server processes will not start when you try to connect.

MonetDB5 Server

• The "sub" prefix of many functions, both at the MAL and the C level, has been removed.
• Changed the interfaces of the AUTH* functions: pass values, not pointers to values.
• Removed calc.setoid().
• group.subgroup is now called group.group if it is not refining a group. Both group.group and group.subgroup now also have variants with a candidate list.
• The allocation schemes for MAL blocks and Variables has been turned into block-based. This reduces the number of malloc()/free() calls.

.

• Added a new server-side protocol implementation. The new protocol is backwards compatible with the old protocol. Clients can choose whether they want to use the old or the new protocol during the initial handshake with the server. The new protocol is a binary column-based protocol that is significantly faster than the old protocol when transferring large result sets. In addition, the new protocol supports compression using Snappy or LZ4.
• Moved the sphinx extension module to its own repository. See https://dev.monetdb.org/hg/MonetDB-sphinx/.
• Removed GSL module: it's now a separate (extension) package. See https://dev.monetdb.org/hg/MonetDB-gsl/.
• The PCRE library is now optional for systems that support POSIX regular expressions.
• Added 5 new sys schema tables: function_languages, function_types, key_types, index_types and privilege_codes. They are pre-loaded with static content and contain descriptive names for the various integer type and code values. See also sql/scripts/51_sys_schema_extension.sql

Merovingian

• Added handling of a dbextra property per database at the daemon level. The user can set the dbextra property for a database using the command: $ monetdb set dbextra=<path> <database> and the daemon will make sure to start the new server using the correct --dbextra parameter.

Client Package

• The mclient and msqldump programs lost compatibility with old mserver5 versions (pre 2014) which didn't have a "system" column in the sys.schemas table.
• The mclient and msqldump programs lost compatibility with ancient mserver5 versions (pre 2011) which didn't have the sys.systemfunctions table.
• Removed the "array" and "quick" functions from the mapi library. To be precise, the removed functions are: mapi_execute_array, mapi_fetch_field_array, mapi_prepare_array, mapi_query_array, mapi_quick_query, mapi_quick_query_array, and mapi_quick_response.
• Added a more elaborate \help command for SQL expressions.

MonetDB Common

• Improved error checking in the logger code (dealing with the write-ahead log); changed return types a several functions from int to gdk_return (i.e., they now return GDK_SUCCEED or GDK_FAIL). The logger no longer calls GDKfatal on error. Instead the caller is responsible for dealing with errors.
• BATsort may now create an order index as a by product.
• Quantile calculations now use the order index if available (and use BATsort otherwise, producing an order index).
• Quantiles calculate a position in the sorted column. If this position is not an integer, we now choose the nearest position, favoring the lower if the distance to the two adjacent positions is equal (round down to nearest integer).
• Removed function BATprintf. Use BATprint or BATprintcolumns instead.
• Removed BATsave from the list of exported functions.
• Replaced BBPincref/BBPdecref with BBPfix/BBPunfix for physical reference count and BBPretain/BBPrelease for logical reference count maintenance.
• Removed automatic conversion of 32-bit OIDs to 64 bits on 64-bit architectures.
• Removed functions OIDbase() and OIDnew().
• Removed talign field from BAT descriptor.
• BATappend now takes an optional (NULL if not used) candidate list for the to-be-appended BAT.
• New function BATkeyed(BAT *b) that determines (possibly using a hash table) whether all values in b are distinct.

SQL

• Made the operator precedence of % equal to those of * and /. All three are evaluated from left to right.
• Removed table sys.connections. It was a remnant of an experimental change that had already been removed in 2012.
• Protect against runaway profiler events If you hit a barrier block during profiling, the JSON event log may quickly become unwieldy. Event production is protected using a high water mark, which ensures that never within the single execution of MAL block the instruction causes excessive event records.

Bug Fixes

• 3465: Request: add support for CREATE VIEW with ORDER BY clause
• 3545: monetdb commands don't work with -h -P -p options (locally and remotely)
• 3996: select * from sys.connections always returns 0 rows. Expected to see at least one row for the active connection.
• 6187: Nested WITH queries not supported
• 6225: Order of evaluation of the modulo operator
• 6289: Crashes and hangs with remote tables
• 6292: Runaway SQL optimizer in too many nested operators
• 6310: Name resolution error (sqlsmith)
• 6312: Object not found in LIMIT clause (sqlsmith)
• 6313: Null type resolution in disjunction fails (sqlsmith)
• 6319: Server crash on LATERAL (sqlsmith)
• 6322: Crash on disjunction with LIMIT (sqlsmith)
• 6323: Deadlock calling sys.bbp()
• 6324: Sqlitelogictest crash in a IN query (8th)
• 6327: The daemon does not respect the actual name of the mserver5 executable
• 6330: Sqlitelogictest crash on a complex SELECT query
• 6331: sys.statistics column "nils" always contains 0. Expected a positive value for columns that have one or more nils/NULLs
• 6332: Sqlitelogictest crash related to an undefined MAL function

 

 

 

Dec 2016-SP5 bugfix release

Bug Fixes

6290: Crash (and assertion failure) with a correlated subquery with NOT IN in the select-list 6291: crash if executes a function with sample operator 6294: Sqlitelogictest crash 6296: Another sqllitelogictest crash :( 6297: 7th sqllitelogictest crash :( 6300: Protect against missing BATs (sqlsmith) 6314: Lateral crash report (sqlsmith) 6315: Exist operator on type bigint missing (sqlsmith) 6316: Coalesc and limit error (sqlsmith) 6317: Two-column integer aggregation extremely slow 6318: Daemon crashes if .merovignian_properties for a database contains more than 42 entries 6320: The daemon ignores all but the last entry in the .merovignian_properties file when spawning mserver5

Dec 2016-SP4 bugfix release

MonetDB5 Server	Fixed a bug causing a crash during cleanup when mserver5 is stopped with monetdb stop database. Some memory leaks were plugged.
MonetDB Common	A potential deadlock was fixed in order index creation. A bug that could happen during recovery of the write-ahead log (WAL) was fixed. See changeset 98ad79c555cc for details.
SQL	Some memory leaks were plugged. Upgrade code was added for an old change in the sys.settimeout function. A bug was fixed with the automatic "vacuum" operation on system tables.
Bug Fixes	6110: cast of a SQL boolean value to a string or clob or (var)char is wrong 6254: Crash (and assertion failure) after querying a view which uses a correlated subquery in the select-list 6256: Assertion Trigger on FULL OUTER JOIN with more than two BETWEEN clauses 6257: wrong count values (1 instead of 0) for correlated aggregation queries 6258: Vulnerability in FITS and NETCDF data vaults 6259: crash on select query from sqlitelogictests 6260: Sqlitelogictest crash 6288: Function cannot find column in merge table 6295: msqldump writes unescaped timestamp values when using inserts

Dec 2016-SP3 bugfix release

MonetDB Common

Fixed a bug when appending string bats that are fully duplicate eliminated. It could happend that the to-be-appended bat had an empty string at an offset and at that same offset in the to-be-appended-to bat there happened to be a (sequence of) NULL(s). Then this offset would be used, even though it might nog be the right offset for the empty string in the to-be-appended-to bat. This would result in multiple offsets for the empty string, breaking the promise of being duplicate eliminated.

Bug Fixes

6138: Weak duplicate elimination in string heaps > 64KB 6183: ResultSet returns double quoted column name if name contains space characters 6216: Assertion raised (sqlsmith) 6219: Crash in rel_optimizer (sqlsmith) 6227: Monetdb fails on remote tables 6228: mclient crashes if real column is multiplied by it itself 6229: ANALYZE, unexpected end of input 6230: ANALYZE, syntax error 6237: semijoin with empty right bat does not return immediately 6242: Crash on rel_reduce_groupby_exps (sqlsmith) 6243: Static optimization gives wrong result (1 + NULL = -127) 6245: Nested query crashes all versions of MonetDB or gives wrong result starting from Dec2016-SP2 6246: update statements: references to a table do not bind to its alias 6247: Type analysis issue (sqlsmith) 6248: update statements: the semantic stage does not resolve the relation in the from clause 6250: Assertion failure when querying a Blob column with order by DESC 6251: Crash after adding an ordered index on sys.statistics column and querying sys.statistics 6253: FITS Data Vaults does not work when using user/pw and other than sys schema name

Dec 2016-SP2 bugfix release

Bug Fixes

4034: argnames array in rapi.c has fixed length (that was too short) 6080: mserver5: rel_bin.c:2391: rel2bin_project: Assertion `0' failed. 6081: Segmentation fault (core dumped) 6082: group.subgroup is undefined if group by is used on an expression involving only constants 6111: Maximum number of digits for hge decimal is listed as 39 in sys.types. Should be 38 as DECIMAL(39) is not supported. 6112: Crash upgrading Jul2015->Jun2016 6130: Query rewriter crashes on a NULL pointer when having a correlated subquery 6133: A crash occurs when preparing an INSERT on joined tables during the query semantic phase 6141: Getting an error message regarding a non-GROUP-BY column rather than an unknown identifier 6177: Server crashes 6186: Null casting causes no results (silent server crash?) 6189: Removing a NOT NULL constraint from a PKey column should NOT be allowed 6190: CASE query crashes database 6191: MT_msync failed with "Cannot allocate memory" 6192: Numeric column stores wrong values after adding large numbers 6193: converting to a smaller precision (fewer or no decimals after decimal point) should round/truncate consistently 6194: splitpart returns truncated last part if it contains non ascii caracters 6195: Cast from huge decimal type to smaller returns wrong results 6196: Database crashes after generate_series query 6198: COALESCE could be more optimized 6201: MonetDB completely giving up on certain queries - no error and no result 6202: querying a table with an ordered index on string/varchar column crashes server and makes server unrestartable! 6203: copy into: Failed to import table Leftover data 'False' 6205: Integer addition overflow 6206: casting strings with more than one trailing zero ('0') to decimal goes wrong 6209: Aggregation over complex OR expressions produce wrong results 6210: Upgrading a database from Jun2015 or older crashes the server 6213: SQLsmith causes server to crash 6217: Segfault in rel_optimizer (sqlsmith) 6218: grouped quantiles with all null group causes following groups to return null 6224: mal_parser: cannot refer to types containing an underscore

Dec 2016-SP1 bugfix release

Bug Fixes

4039: Slow mserver5 start after drop of tables (> 1 hour) 4048: Segfault on vacuum with parallel updates 6079: pushselect optimizer bug on MAL snippet 6140: INNER JOIN gives the results of a CROSS JOIN 6150: Query giving wrong results, extra records are appearing 6175: The program can't start because python27.dll is missing from your computer. 6178: AVG + GROUP BY returns NULL for some records that should have results 6179: mergetable optimizer messes up sample 6182: sys.shutdown triggers assertion in clients.c 6184: Incorrect result set - Extra records in result set

Dec 2016 feature release

There are lots of changes in this (Dec2016) release and the previous (Jun2016).  Here is an outline.

• The client interfaces that are not written in C, that is to say, the Java (JDBC driver), Perl, PHP, Python, and Ruby interfaces all now have their own repository and are no longer part of the main MonetDB repository.  (Actually, the Ruby interface was already separate in the Jun2016 release.)  These client interfaces are still supported but now have an independent release schedule.  Their first releases are synchronized with this release. Debian and Ubuntu users, please note that the package names for the client interfaces have changed.  If you have the old versions installed, they will not get updated automatically.  Use: sudo apt-get install libdbd-monetdb-perl php-monetdb python-pymonetdb python3-pymonetdb removing the ones from the list above you don't want.
• For years we have been working towards a version that we've internally called "headless".  This is now complete.  This has had consequences for quite a few internal interfaces, also ones that could potentially be used by extensions written by third parties. The trickiest one is probable BATkey() which has the same calling interface but does something different.  The compiler should complain about other functions since their arguments changed or the functions were completely removed.
• In the Jun2016 release it was possible to have user-defined functions (UDF) callable from SQL that were written in the R language.  In this release, the same is now possible for UDFs written in Python.  Also see this blog.
• The possibility of having 32 bit OID values in a system that was compiled for a 64 bit architecture has been removed.  The size of the OID type is now always the same as the "natural" size of the architecture.  This was a compile-time feature and we haven't made any binary releases in a long time in which this feature was used. The automatic conversion from a database in which the size of the OID type was 32 to one in which the size is 64 is still in place. This will be removed in a future version.
• The internal type "wrd" has been removed.  The "wrd" type was an architecture-dependent integer type, 32 bits wide on a 32-bit system and 64 bits wide on a 64-bit system.  Databases where this type is used will automatically have the type converted to either "int" or "lng", the internal types for 32-bit and 64-bit integers.
• Some old and unused code and interfaces have been removed.
• By making some adjustments to internal parameters, we have been able to get a significant improvements on query speed for many types of queries (in informal tests we have seen a 20% improvement compared to the previous release when running the queries from the TPC-H benchmark).
• A new type of experimental column index has been introduced: the ordered index.  Since creating this index is expensive, it is only used when explicitly created.  To create an index in SQL use CALL createorderindex(schema,table,column) where the procedure arguments are all strings.
• We have fixed quite a few memory leaks and other bugs.

 

MonetDB5 Server	Removed the zorder module with functions zorder.encode, zorder.decode_x and zorder.decode_y. Removed command bbp.getHeat(). Removed bat.setColumn with two arguments and bat.setRole. Use bat.setColumn with one argument instead. Removed function BKCappend_reverse_val_wrap: it was unused. The "wrd" type has been removed from GDK and MAL. The type was defined to be a 32 bit integer on 32 bit architectures and a 64 bit integer on 64 bit architectures. We now generally use "lng" (always 64 bits) where "wrd" was used. Keep a collection of full traces. Each time the SQL user applies the TRACE option, the full json trace is retained within the <dbpath>/<dbname>/sql_traces
Build Environment	New packages MonetDB-python2 (Fedora) and monetdb-python2 (Debian/Ubuntu) have been created for Python 2 integration into MonetDB. With OID size equal to ABI/word size, mserver5 does not need to print the OID size, anymore. Removed obsolete code associated with long gone static linking option. Removed configure option --enable-oid32 to compile with 32 bit OIDs on a 64 bit architecture.
.	The Perl, PHP, and Python clients, and the JDBC driver each now have their own repositories and release cycles. The Python client is maintained by Gijs Molenaar on Github (https://github.com/gijzelaerr/pymonetdb), the other clients are maintained by CWI/MonetDB on our own server (https://dev.monetdb.org/hg/monetdb-java, https://dev.monetdb.org/hg/monetdb-perl, https://dev.monetdb.org/hg/monetdb-php).
MonetDB Common	The tnokey values must now be 0 if it is not known whether all values in a column are distinct. The 2-bit tkey field in the bat descriptor has been split into two single bit fields: tkey and tunique. The old tkey&BOUND2BTRUE value is now stored in tunique. Implemented conversion to str from any type (not just the internal types). VALcopy and VALinit both return their first argument on success or (and that's new) NULL on (allocation) failure. BATattach now can also create a str BAT from a file consisting of null-terminated strings. The input file must be encoded using UTF-8. BATattach now copies the input file instead of "stealing" it. Removed the lastused "timestamp" from the BBP. Removed batStamp field from BAT descriptor, and removed the BBPcurstamp function. Removed unused functions BBPhot and BBPcold. Removed BATderiveTailProps and BATderiveProps. Just set the properties you know about, or use BATsettrivprop. Removed the macro BUNfirst. It can be replaced by 0. Changed BATroles by removing the argument to set the name of the head column. The head column is now completely gone. MonetDB is completely "headless". The format of the BBP.dir file was simplified. Since the head column is VOID, the only value that needs to be stored is the head seqbase. BATs now only have a single (logical) name. The function BATmirror is gone. The HEAD column is always VOID (with a non-nil seqbase) and the TAIL column carries the data. All functions that deal with data work on the TAIL column. BATkey now works on the TAIL column instead of the HEAD column. Replaced BATseqbase with BAThseqbase and BATtseqbase, the former for setting the seqbase on the HEAD, the latter for setting the seqbase on the TAIL. Replaced function BATnew with COLnew with slightly different arguments: the first argument of COLnew is the SEQBASE of the head column (which is always VOID). The "wrd" type has been removed from GDK and MAL. The type was defined to be a 32 bit integer on 32 bit architectures and a 64 bit integer on 64 bit architectures. We now generally use "lng" (always 64 bits) where "wrd" was used.
SQL	Removed functions sys.zorder_encode, sys.zorder_decode_x, and sys.zorder_decode_y. The experimental recycler code is moved to the attic. The syntax of bat.new(:oid,:any) has been changed by dropping the superflous :oid. All BATs are now binary associations between a void column and a materialized value column. (except for the internal :bat[:void,:void] representation of simple oid ranged tails.)
Bug Fixes	3357: Implement setQueryTimeout() 3445: Add support for database name to dotmonetdb file 3597: SQL to MAL listing looses types 3973: JDBC hangs 3976: Performance enhancement to LIKE without wildcards 4005: Correlated update causes incorrect null constraint violation 4016: merge table only optimises for point query 4040: sys.storage call can take a long time 4047: Segfault when updating a dropped table 4050: Database corruption when running low on inode 4057: missing bulk operations between constant and bat 4061: SIGSEGV in candscan_lng 4066: Deadlocked monetdbd 6068: Error message about incompatible BBP version should be clearer 6069: query with union all silently crashes 6070: setting negative session query timeout should not be possible/allowed 6071: where clause with cast and floor fails to sigsegv 6072: Bind to UPD delta column does not get/show type information in EXPLAIN 6073: Missing type information for constants in MAL explain 6074: SET ROLE command does not work 6075: gdk_calc.c:13113: BATcalcifthenelse_intern: Assertion `col2 != NULL' failed. 6076: rel_optimizer.c:5426: rel_push_project_up: Assertion `e' failed. 6077: mserver5: rel_optimizer.c:5444: rel_push_project_up: Assertion `e' failed. 6078: rel_bin.c:2402: rel2bin_project: Assertion `0' failed. 6084: Merge table point to wrong columns if columns in partition tables are deleted 6108: monetdb5-sql sysv init script does not wait for shutdown 6114: segfault raised in the query rewriter due to a null pointer 6115: Assertion hit in the codegen 6116: Codegen does not support certain kind of selects on scalar subqueries 6117: Assertion hit in the query rewriting stage during the push down phase 6118: SIGSEGV in strPut due to shared heap 6119: Assertion hit in the MAL optimiser on a complex query 6120: QUANTILE() treats NULL as if it is zero 6121: SELECT a.col IN ( b.col FROM b ) FROM a statements with no error but no result 6123: Assertion hit in the codegen #2 6124: CASE <column> WHEN NULL THEN 0 ELSE 1 END returns wrong result 6125: Stack overflow in the query rewriter with a query having an OR condition and a nested SELECT subexpression 6126: batcalc.== can't handle void BATs 6139: Debian libmonetdb13 conflicts with libmonetdb5-server-geom

Jun 2016-SP2 bugfix release

Build Environment

We now use the CommonCrypto library instead of the OpenSSL library on Darwin.

Bug Fixes

3939: Assert failure on concurrent queries when querying sys.queue 4019: Casting a timestamp from a string results in NULL 4025: expressions in the WHERE clause that evaluates incorrectly 4038: After upgrade from 11.21.19, jdbc couldn't list tables for non sys users 4044: Server crash when trying to delete a table has been added to a merge table with "cascade" at the end 4049: Wrong results for queries with "OR" and "LEFT JOIN" 4052: Infinite loop in rel_select 4054: copy into file wrongly exports functions 4058: Server crashes with a particular conditional query 4059: Geom functions only visible by user monetdb 4060: BAT leak in some aggregate queries 4062: Error: SELECT: no such binary operator 1800 790 9186'like(varchar,varchar)' when used in query running in other schema than sys 4064: Assertion: column not found 4067: Relevant column name not printed when a CSV parsing error occurs 4070: Extra condition in join predicate of explicit join produces wrong MAL code 4074: Cannot use prepared statements when caching disabled 6065: CTE with row number and union fails within MAL

Jun 2016-SP1 bugfix release

.	Lots of memory leaks have been plugged across the whole system.
Java Module	Corrected PROCEDURE_TYPE output value of method DatabaseMetaData.getProcedures(). It used to return procedureReturnsResult. Now it returns procedureNoResult. Corrected ORDINAL_POSITION output value of method DatabaseMetaData.getProcedureColumns(). It used to start with 0, but as procedures do not return a result value it now starts with 1 for all the procedure arguments, as defined by the JDBC API. Improved output of method DatabaseMetaData.getProcedures(). The REMARKS column now contains the procedure definition as stored in sys.functions.func. The SPECIFIC_NAME column now contains the procedure unique identifier as stored in sys.functions.id. This allows the caller to retrieve the specific overloaded procedure which has the same name, but different arguments. Also improved output of method DatabaseMetaData.getProcedureColumns(). The SPECIFIC_NAME column now contains the procedure unique identifier as stored in sys.functions.id. This allows the caller to retrieve the proper arguments of the specific overloaded procedure by matching the SPECIFIC_NAME value. Improved output of method DatabaseMetaData.getFunctions(). The REMARKS column now contains the function definition as stored in sys.functions.func. The SPECIFIC_NAME column now contains the function unique identifier as stored in sys.functions.id. This allows the caller to retrieve the specific overloaded function which has the same name, but different arguments. Also improved output of method DatabaseMetaData.getFunctionColumns(). The SPECIFIC_NAME column now contains the function unique identifier as stored in sys.functions.id. This allows the caller to retrieve the proper arguments of the specific overloaded function by matching the SPECIFIC_NAME value.
Bug Fixes	4014: KILL signal 4021: Analyze query does not escape input [security] 4026: JDBC driver incorrectly converts TINYINT fields to String instead of an integer type. 4028: inputs not the same size 4031: mclient doesn't accept - argument to refer to stdin 4032: no decimal places after update. ODBC driver 4035: SQL Function call bug 4036: Possible sql_catalog corruption due to unclean backuped tail

Jun 2016 feature release

MonetDB5 Server	Extended the storage() table producing function to also accept storage([schemaname [, tablename [, columnname]]]) The :bat[:oid,:any] type descriptor has been turned into its columnar version :bat[:any] Converted the *.mal scripts into *.malC versions to prepare for removal of the mserver console. Removed functions mat.hasMoreElements, mat.info, mat.mergepack, mat. newIterator, mat.project, mat.pack2, mat.sortReverse, mat.sort, and mat.slice. Removed grouped aggregate functions from the aggr module in which the groups were indicated by the head column of the bat to be aggregated. Use the interface with a separate group bat instead. The server now stops executing a query when the client disconnects. Removed algebra.join. Use algebra.subjoin instead. Removed algebra.antijoin. Use algebra.subantijoin or algebra.subthetajoin instead. The MAL function 'leftfetchjoin' is renamed to its relational algebra version 'projection' The generic property handling scheme has been removed. It was used in just a few places, for widely different purposes and contained cruft. Removed functions str.iconv and str.codeset. Internally, strings are always in UTF-8 encoding, so we can't allow code set conversions. Removed algebra.like with a BAT argument. Use algebra.likesubselect instead. Removed algebra.leftjoin. Use algebra.subleftjoin or algebra.leftfetchjoin instead. Removed algebra.tdiff and algebra.tinter. Removed algebra.sample. Use sample.subuniform instead. Removed algebra.select. Use algebra.subselect instead. Removed algebra.revert. Removed bat.order and bat.orderReverse functions. Changed client.getUsers to return two bats, one with the user id and one with the user name. Implemented algebra.subdiff and algebra.subinter. Removed algebra.tdifference and algebra.tintersect. Removed algebra.tunion.
Build Environment	A new package MonetDB-lidar (Fedora) or libmonetdb5-server-lidar (Debian/Ubuntu) has been created to work with LiDAR data. Implemented a systemd configuration file for a monetdbd.service on systems that support it (Fedora, newer Ubuntu).
Client Package	The Ruby client is now in a separate repository (http://dev.monetdb.org/hg/monetdb-ruby) and released independently.
Geom Module	The geom module has been completely overhauled. Types are now specified as GEOMETRY(POINT) instead of POINT, old functions have been removed, new ones introduced. However, even with all the changes, a database upgrade should still be possible (as always, make a backup first).
MonetDB Common	Removed BATconst. Use BATconstant instead. Changed BATconstant. It now has a new first argument with the seqbase for the head column. Removed BATmmap. It was no longer used. BUNdelete and BATdel don't accept a foce argument and only allow deleting values that have not yet been committed. BUNdelete exchanges the deleted value with the last value (if the deleted value isn't the last value). BATdel compacts the BAT by shifting values after the deleted values up. The list of to-be-deleted values in BATdel must be sorted and unique. Removed BUNreplace from list of exported functions. It wasn't used, and there is still BUNinplace and void_inplace that do more-or-less the same thing. Changed BATderiveHeadProps to BATderiveTailProps (and it now works on the tail column). Removed unused functions BATalpha, BATdelta, and BATprev. Removed function VIEWcombine. Use BATdense instead. Removed "left" parameter from BUNinplace. It wasn't used since the BAT it works on is dense headed. Created function BATdense to easily create a [void,void] BAT with specified seqbases for head and tail and a count. Removed function BATmark. When all head columns are dense, BATmark basically only created a new [void,void] BAT. Renamed BATsubsort to BATsort. Removed "sub" from the names of the function BATsubselect, BATthetasubselect, BATsubcross, BATsubleftjoin, BATsubouterjoin, BATsubthetajoin, BATsubsemijoin, BATsubdiff, BATsubjoin, BATsubbandjoin, BATsubrangejoin, and BATsubunique. Removed BATsubleftfetchjoin: it was not used. Removed BATcross1. Use BATsubcross instead. Removed all versions of the SORTloop macro. Removed Batkdiff. Use BATsubdiff instead. Removed BATselect. Use BATsubselect instead. Removed BATsemijoin. Use BATsubsemijoin instead. Removed BATjoin. Use BATsubjoin instead. Removed BATleftjoin. Use BATsubleftjoin or BATproject instead. Removed BATleftfetchjoin. Use BATproject instead. Removed BUNins from the list of exported functions. Removed legacy functions BATuselect and BATuselect_. Removed legacy functions BATantijoin, BATbandjoin, BATouterjoin, BATrangejoin, and BATthetajoin. Removed function BATrevert. BATordered now works on the TAIL column. Removed obsolete functions BATorder() and BATorder_rev(). Implemented BATsubdiff which returns a list of OIDs (sorted, i.e. usable as candidate list) of tuples in the left input whose value does not occur in the right input. Removed function BATkintersect. It wasn't used anymore. It's functionality can be obtained by using BATsubsemijoin. Removed the function BATkunion.
Java Module	Fixed problem in DatabaseMetaData.getUDTs() when it was called with types parameter filled. It used to throw SQException with message: SELECT: identifier 'DATA_TYPE' unknown. Now it returns the UDTs which match the provided array of data types. Implemented MonetDatabaseMetaData.supportsConvert() and MonetDatabaseMetaData.supportsConvert(int fromType, int toType) methods. It used to always return false. Now it returns true for the supported conversions. This fixes Bug 3460. ResultSet.setFetchSize(): added a dummy implementation to get rid of the SQLFeatureNotSupportedException. In MonetDB, it does not make sense to set the fetch size of a result set. If one really wants to set the fetch size, one should use Statement.setFetchSize() instead. Fixed resource leak in ResultSetMetaData. It created and cached a ResultSet object for each column but never closed the ResultSet objects. Corrected DatabaseMetaData methods which accept a catalog filter argument. Those methods will now filter the results on the specified catalog name, whereas previously the catalog filter argument was ignored. Corrected output of column KEY_SEQ of DatabaseMetaData methods: getPrimaryKeys(), getImportedKeys(), getExportedKeys() and getCrossReference(). It now starts at 1 instead of 0 previously. Corrected DatabaseMetaData.getSchemas() by returning 2 instead of 3 columns. Improved DatabaseMetaData.getColumns() by returning two additional columns: IS_AUTOINCREMENT and IS_GENERATEDCOLUMN. Improved DatabaseMetaData.getTypeInfo(). It now returns better information on LITERAL_PREFIX, LITERAL_SUFFIX, CREATE_PARAMS, CASE_SENSITIVE, FIXED_PREC_SCALE and MAXIMUM_SCALE for some data types. Also the returned rows are now ordered by DATA_TYPE, TYPE_NAME, PRECISION as required by the specs. Also corrected output column names "searchable" into "SEARCHABLE" and "MAXIMUM SCALE" into "MAXIMUM_SCALE" to match the JDBC specification. Corrected DatabaseMetaData.getPseudoColumns(). It used to return 12 empty rows. Now it returns no rows as MonetDB does not have pseudo columns. Implemented method DatabaseMetaData.getClientProperties(). It used to always return a resultset with 4 completely empty rows. It now returns a resultset with the possible connection properties. Implemented method DatabaseMetaData.getUDTs(). It used to return an empty resultset. Now it returns the User Defined Types such as inet, json, url and uuid. Corrected the returned table types in DatabaseMetaData.getTableTypes(). It now returns all 10 table types (as stored in sys.table_types) instead of the previously 8 hardcoded table types. For old MonetDB servers which do not have the sys.table_types table, the old behavior is retained. Implemented methods DatabaseMetadata.getProcedures() and DatabaseMetadata.getProcedureColumns(). They used to return an empty resultset. Now they return the expected Procedures and ProcedureColumns. Also getProcedureColumns() now returns a resultset with all 20 columns instead of 13 columns previously. Method getFunctionColumns() in DatabaseMetadata used to throw an SQLException: getFunctionColumns(String, String, String, String) is not implemented. This method is now implemented and returns a resultset. Method getFunctions() in DatabaseMetadata used to throw an SQLException: SELECT: no such column 'functions.sql' This has been corrected. It now returns a resultset as requested. The resultsets of DatabaseMetadata methods now no longer return a value for the *_CAT columns as MonetDB does not support Catalogs. Fixed a memory leak in MonetDatabaseMetaData.java for a static cache which kept references to closed Connection objects. Improved JdbcClient program when presenting query data to console. It used to send an SQL catalog query for each query result column which slowed down the interactive response considerably. These additional SQL catalog queries have been eliminated. Corrected MonetResultSet.getObject(String columnName). It no longer throws a NullPointerException in cases where internally a MonetVirtualResultSet is used. Fixed Connection.isValid(): this method should never attempt to close the connection, even if an error has occurred.
SQL	Disallow GRANT <some-user> TO <role-or-use>. Only explicitly created roles can be granted. Extended grantable schema privileges: when a user is granted the "sysadmin" role, the user now hcan not only create schemas, but also drop schemas. Introduced COPY INTO/ COPY FROM global privileges. These privileges allows other users than 'monetdb' to be granted the privileges to use the MonetDB bulk data loading/exporting features, i.e., COPY FROM <file> and COPY INTO <file>.
Bug Fixes	2407: Merovingian: allow binds to given ip/interface 2815: No SRID support 3361: constants as MAL function parameters prevent intermediate reuse 3374: UNIQUE constraint does not set tkey property on the corresponding BAT 3460: incomplete implementation of JDBC driver supportsConvert(), supportsConvert(int fromType, int toType) methods in MonetDatabaseMetaData.java 3711: JDBC connection using jdbcclient.jar is very slow 3877: MonetDBLite should allow close then re-open databases? without restarting R 3911: Invalid connect() call in 'redirect' mode 3920: query with DISTINCT and correlated scalar subquery in SELECT list causes Assertion failure and crash of mserver5 3927: COUNT( distinct my_clob ) , QUANTILE( my_double , number ) fails in dev build 3956: MonetDBLite unable to execute LIMIT 1 statement 3972: Drastic Memory leak of 600GBs while generating plan for Query with 25 joins 3974: Prepared statement rel_bin.c:2378: rel2bin_project: Assertion `0' failed. 3975: Suspicious code in store_manager() on exit path 3978: SQL returns TypeException 'aggr.subcorr' undefined for sys.corr function 3980: JOIN with references on both sides crashes mserver 3981: Incorrect LEFT JOIN when FROM clause contains nested subqueries 3983: Creation of a Foreign Key which partially maps to a primary key is accepted without a warning 3984: Multiple paths in the .profile 3985: ruby-monetdb-sql gem fails for negative timezone offset (USA, etc.) 3987: Segfault on malformed csv import 3991: MonetDBLite feature request: default monetdb.sequential to FALSE on windows 3994: MonetDBLite dbDisconnect with shutdown=TRUE freezes the console on windows 3995: NullPointerException when calling getObject() 3997: calling scalar functions sys.isaUUID(str) or sys.isaUUID(uuid) fail 3999: length() returns wrong length for strings which have spaces at the end of the string 4010: RELEASE SAVEPOINT after ALTER TABLE crashes mserver5 4011: sys.sessions.user column always shows 'monetdb' 4013: GDKextendf does not free up memory when it fails due to insufficient resources 4015: Daemon crashes on database release command

Jul 2015-SP4 bugfix release

MonetDB Common	Fixed a bug that caused various instances where old data returned or where crashes occurred. The problem was that internally data wasn't always marked dirty when it was being changed, causing later processing to not deal with the changed data correctly.
Bug Fixes	3905: MonetDB doesn't handle ANY/SOME/ALL operator correctly 3929: R aggregate not recognized when using 3 or more parameters 3965: Not possible to quote/escape single quote character in the name of the file to load. 3968: Missing double use of column names

Jul 2015-SP3 bugfix release

MonetDB5 Server

Fixed potential crash in MAL debugger when accessing BATs by index. Functionality dropped as it is also a security leak.

Bug Fixes

2972: SQL URL functionality contains errors 3549: bulk string operations very slow 3881: Server crashes on bulk load 3890: Window function + group by in subselect, rel2bin_project: Assertion `0' failed 3891: MonetDB crashes when executing SQL with window function 3900: null handling in some sql statements is incorrect 3906: Multi-column 1-N table-function with mitosis produces different column counts 3908: LEFT JOIN with OR conditions triggers assertion 3909: Incorrect column name in OR condition of LEFT JOIN crashes mserver 3910: COPY INTO table (column1, column2) got wrong result 3912: When table/column names conflicts, data ends in multiple tables! 3917: Date difference returns month_interval instead of day_interval 3918: MonetDB.R version 1.0.1 incorrectly constructs the batfile script 3919: Table conflict when the table name and fields are identical 3921: Creating a table from a complex query crashes mserver or triggers assertion 3922: AVG( column ) returns NaN rather than Inf when column contains Inf 3928: When killing a virtual machine, sql_logs/sql/log is empty 3930: Wrong typecast on character columns in prepared statements when using Umlaute 3932: CASE expressions with constants are not evaluated correctly 3933: replace "exit" by "throw new Exception" 3937: bad BAT properties with binary copy into and NULL values 3938: Wrong error message on violating foreign key constraint 3940: Date calculation and comparison produce wrong result 3941: Wrong coercion priority 3948: SQL: select * from sys.sys.table_name; is accepted but should return an error 3951: extern table_funcs not visible from Windows DLL for extensions like vaults (crashes) 3952: Stream table gives segfault 3953: MIN/MAX of a UUID column produces wrong results 3954: Consolidate table assertion error 3955: (incorrect) MAL loop instead of manifold triggered by simple change in target list

Jul 2015-SP2 bugfix release

Bug Fixes

2014: 'null' from copy into gets wrong 3817: opt_pushselect stuck with multi-table UDF 3835: windows does not release ram after operations 3836: rand() only gets evaluated once when used as an expression 3838: Update column with or without parenthesis produce different results 3840: savepoints may crash the database 3841: mclient fails with response "Challenge string is not valid" 3842: SQL execution fails to finish and reports bogus error messages 3845: Too many VALUES in INSERT freeze mserver5 3847: Wrong SQL results for a certain combination of GROUP BY / ORDER BY / LIMIT 3848: mserver segfault during bulk loading/updating 3849: HUGEINT incorrect value 3850: DEL character not escaped 3851: expression that should evaluate to FALSE evaluates to TRUE in SELECT query 3852: CASE statement produces GDK error on multithreaded database: BATproject does not match always 3854: Complex expression with comparison evaluates incorrectly in WHERE clause 3855: Foreign key referencing table in a different schema - not allowed. 3857: Large LIMIT in SELECT may abort the query 3861: Using window functions cause a crash 3864: Error in bulk import for chinese character 3871: NOT x LIKE triggers "too many nested operators" 3872: mserver crashes under specific combination of JOIN and WHERE conditions 3873: mserver5: gdk_bat.c:1015: setcolprops: Assertion `x != ((void *)0) || col->type == 0' failed. 3879: Database crashes when querying with several UNION ALLs. 3887: Querying "sys"."tracelog" causes assertion violation and crash of mserver5 process 3889: read only does not protect empty tables 3895: read only does not protect this table

Jul 2015-SP1 bugfix release

Client Package

In the SQL formatter of mclient (the default) we now properly align East Asian wide characters.

Bug Fixes

3789: Query on large string table fails on HEAPextend 3794: table function sys.rejects() and view sys.rejects() are listed are metadata objects but give an (incorrect) error when they are queried 3797: COPY INTO with incorrect number columns 3798: SELECT query with INTERSECT causes assertion failure 3800: LIKE is broken for many patterns 3802: Disk space never freed: a logical ref is keeped on a deleted BATs 3803: SQL query parser fails to parse valid SELECT query with a CASE .. END in it. It fails with parser error: identifier 'x' ambiguous 3804: `monetdb status` command crashes under certain conditions 3809: Inefficient plan is generated for queries with many (>= 24) joined tables which take a long time or an HEAPalloc error. I get Error: GDK reported error. HEAPalloc: Insufficie 3810: Fix statistics gathering 3811: NOT LIKE not working if the operand doesn't contains wildcards. 3813: COPY INTO fails on perfectly clean CSV file 3814: Server crash when using bitwise NOT operation in SQL query 3816: Server crashes when trying to convert timestamp to str with incorrect format 3818: Crash when performing UNION/GROUP BY over tables with different columns 3819: order of tables in FROM-clause has negative impact on generated plan (using crossproducts instead of joins) 3820: mclient accepts table with repeated constraint which causes crash on insert 3821: Unexpected error when using a number instead of a boolean 3822: Yet another LIKE operator issue 3823: JDBC Connection to a schema - setSchema() error 3825: MonetDB not cleaning intermediate results which leads to filling up disk space and ultimately server crash 3827: Certains comparisons between UUID produce a MAL error 3828: Schema corruption after several ALTER TABLE statements and server restart 3829: Certains simple WHERE clause cause MonetDB to segfault without explanation 3830: Coalesce typing inconsistencies 3833: NULL literals refused at many places 3834: Date comparison returns incorrect results 3839: msqldump generates incorrect syntax ON UPDATE (null)

Jul 2015 feature release

Do not skip upgrades! If you are running a release older than the Oct2014 series (11.19.X), do not skip upgrading to the Oct2014 release before upgrading to the Jul2015 release.  In fact, you should first upgrade to the Oct2014-SP2 (11.19.9).  Don't forget to start the server and to connect a client to the server for each of your databases.

One of the new features in this release is the possibility to use "best effort" when importing CSV files into the server.  However, we have found a bug that causes this feature to misbehave when the CSV file cannot be read correctly (exactly when one might want to use best effort).  Hence we recommend to not use the feature.  See bug 3797.

MonetDB5 Server	Implemented batcalc.min and batcalc.max. Made calc.min and calc.max generic so that no other implementations are needed. Removed function batcalc.ifthen.
Build Environment	Upgraded the license to the Mozilla Public License Version 2.0.
Client Package	Added a new output format to mclient: --format=expanded (or -fx). In this format, column values are shown in full and below each other.
MonetDB Common	BUNtvar and BUNhvar macros no longer work for TYPE_void columns. Changed interfaces of a lot of GDK-level functions. When they modify a BAT, don't return the same BAT or NULL, but instead return GDK_SUCCEED or GDK_FAIL. Changed a bunch of hash-related functions to work on the tail column. The functions that have been changed to work on the tail column are: BAThash, BATprepareHash, HASHgonebad, HASHins, and HASHremove.
Java Module	Improved JDBC driver to not throw NullPointerException anymore when calling isNullable() or getPrecision() or getScale() or getColumnDisplaySize() or getSchemaName() or getTableName() or getColumnClassName() on a ResultSetMetaData object. We now compile the Java classes using the latest Java 1.8 version, and we tell it to compile for Java 1.7.
SQL	Differences between time, timestamp, and date values now return properly typed interval types (second or month intervals) instead of integers. Added support for 128-bit integers (called HUGEINT) on platforms that support this. Removed support for the mseed library. Removed support for RDF. Removed DataCell. It was experimental code that was never enabled.
Bug Fixes	2618: Implement master slave scheme 3361: constants as MAL function parameters prevent intermediate reuse 3364: Cannot set role back to a user's default role 3365: Unable to grant object privileges while having a non-default current_role 3440: Sequence type errors 3449: mserver crash on start - Freebsd 10 amd64 3476: Cannot revoke object access 3496: autocompletion table names does not work correctly 3556: when 2 multiplexed functions in MAL plan, only one is mapped correctly to bat<mod>.function primitive 3564: Request: add support for postgresql specific scalar function: split_part(string text, delimiter text, field int) 3625: SIGSEGV because mat array can overrun in opt_mergetable.c 3627: SQRT in CASE does not work as of Oct2014 3654: configure --enable-fits requires extra commands after creating a database instance 3673: mclient 'expanded' row formatter 3674: Obfuscate event tracing 3679: No error is given when incorrect timezone value is specified for a timetz column 3686: Wrong associativity of multiply/divide 3702: Filter function not found if created in a user schema 3708: wrong scoping for cross-schema view references 3716: alter table my_merge_table drop table t1; crashes mserver5 with Segmentation fault 3724: Wrong size calculation in BATsubjoin 3730: SAMPLE function not sampling randomly 3732: memory leak (of InstrRecord) in opt_mergetable 3733: "(TRUE OR <Exp>) AND <Exp>" is evaluated incorrectly 3735: python connection with unix_socket 3736: crash if mclient disconnects abruptly during a query 3738: Database inconsistency when using savepoint 3739: CASE statements do not handle NULLs in the IN () operator properly 3740: select epoch(now()); types timestamptz(7,0) and bigint(64,0) are not equal 3742: Division By Zero 3744: cast to int gives different results for decimal than double 3747: joins fail in the presence of nulls 3748: Missing META-INF/services/java.sql.Driver in JDBC package 3753: Hang on json field parsing 3754: select from a REMOTE TABLE referring local table crashes mserver5 3756: column type conversion sticks to subsequent queries 3758: "COPY INTO ..." doesn't work, if executing from 2 processes concurrently. 3759: select data from "sys"."rejects" returns unexpected error and when next select data from "sys"."sessions" causes an assertion failure in mal_interpreter.c:646. 3760: SQL parser has problem with (position of) a scalar subquery in a SELECT-list 3761: SQL executor has problem with (position of) a subquery in a SELECT-list. Inconsistent behavior. 3763: JDBC PreparedStatement for a table with 14 Foreign Keys crashing the Database 3764: DROPping multiple users causes a crash 3765: Re-granting a revoked privilege does not work 3766: VIEW not visible if created under a different schema 3767: CREATE TEMP TABLE using "LIKE" incorrectly handled 3769: SIGSEGV when combining a cast/column alias with a UNION ALL view 3770: combined conditions on declared table in User Defined Function definition crashes monetdb 3771: Owner of the schema loses rights if assumes the monetdb role. 3772: Any user can grant a role. 3773: quantile(col, 0) and quantile(col, 1) fail 3774: mclient is unaware of merge tables and remote tables 3775: COPY INTO: Backslash preceding field separator kills import 3778: Crash on remote table schema mismatch 3779: server crashes on MAX() on SELECT DISTINCT something combo 3783: Behavioural change in Jul2015 for 'timestamp minus timestamp' 3784: Assertion failed: (bn->batCapacity >= cnt), function BAT_scanselect, file gdk_select.c, line 1008. 3785: sum(interval) causes overflow in conversion to bte 3786: ResultSet.close() never sends Xclose to free resources 3787: "b and g must be aligned" from complex group/union query 3791: HEAPextend: failed to extend to 2420077101056

Important consideration when upgrading to this release. In this release we have fixed bug 3474. This bug made it possible that there were duplicates despite there being a unique or primary key constraint present. This could only happen for multi-column constraints. MonetDB maintains an internal column containing a hash for multi-column constraints, and there was a bug in the maintenance code of this column. During an upgrade, this column needs to be recreated. This is done by dropping and adding back the constraints. If there are no duplicates, adding back the constrains should work without problem, but if there are duplicates, adding the constraints will fail. There is unfortunately no easy way to tell the user, so after having made the first connection to the server (when the upgrade happens), check the merovingian log (if you use merovingian -- also known as monetdbd) or the server output window (if you don't use merovingian).

Jaql support is removed. The experimental code for Jaql support has been removed. If you have RPM or DEB packages for Jaql installed from an earlier version (MonetDB-jaql and monetdb5_jaql respectively), you will need to uninstall them before the upgrade.

Upgrade code fixed. The code to upgrade from previous releases still had some bugs. These have now been fixed. Upgrades directly from the Feb2013 release should work again. Also, upgrading to this version should fix some problems that were caused by the upgrade to the Oct2014 release.

Oct 2014-SP4 bugfix release

Due to a code conflict it is not possible to upgrade to this bug fix release from a release older than the Oct2014 release.  If you are running an older release, first upgrade to the Oct2014 release, and then to Oct2014-SP4.

Bug Fixes

3705: Assertion failure in rel_bin.c:2274: rel2bin_project: Assertion `0' failed. 3707: var() possibly not working in debug builds 3712: Concurrency issue on querying the SQL catalog 3713: Long startup cost for simple session 3715: Crash with two ALTER TABLE statements in a transaction 3718: Adding and dropping a non existing tablename to/from a merge table is accepted without an error 3719: Assertion failure in /MonetDB-11.19.11/gdk/gdk_bat.c:2841: BATassertHeadProps: Assertion `!b->H->key || cmp != 0' failed. 3720: Incorrect results on joining with same table 3723: Assertion failure in rel_bin.c:2548: rel2bin_groupby: Assertion `0' failed. 3725: LEFT JOIN bug with CONST value 3731: left shift for IP addresses not available to non-system users

Oct 2014-SP3 bugfix release

Build Environment	We now also create debug packages for Debian and Ubuntu.
MonetDB Common	Replaced the rangejoin implementation with one that uses imprints if it can.
Bug Fixes	3466: UPDATE statements fails with "GDKerror: MT_mremap() failed" 3602: Surprising overload resolution of generate_series 3613: SQL data dictionary contains columns names which are also special keywords. This causes unexpected/unneeded SQL query errors 3645: Network address operators such as << and <<= do not work 3647: missing BAT for a column leads to crash in gtr_update_delta 3648: memory corruption on unclean connection shutdown with local temporary tables 3650: Naming of persistent BATs is fragile 3653: PREPARE crashes mserver if unbound variable is function parameter 3655: SQL WHERE -1 in (-1) issue? 3656: error message after calling fitsload() 3660: Incorrect Results for Comparison Operators on inet Datatype 3661: Ship debug symbols for pre-built binaries 3662: UPDATE row with row value constructor crashes monetdb server 3663: Incorrect result ROW_NUMBER in subquery 3664: SQLstatementIntern missing parameter when using jsonstore 3665: inter-session starvation issue, particularly affects sys.queue 3666: casting text column to inet truncating text column and resulting inet for first occurrence only 3667: insert of negative value for oid column aborts mserver5 process with assertion failure 3669: ALTER TABLE <tbl_nm> ADD CONSTRAINT <tbl_uc1> UNIQUE (col1, col2, col3) causes Assertion failure and abort 3671: ODBC-Access on Windows 2012 does not work - E_FAIL 3672: libbat_la-gdk_utils.o: relocation R_X86_64_PC32 against `MT_global_exit' can not be used when making a shared object 3676: merovingian hangs trying to exit 3677: Crash in BATgroup_internal (caused by 87379087770d?) 3678: Ruby driver installation ignores prefix 3680: Prepared statements fail on execution with message 'Symbol type not found' 3684: Wrong query result set WHERE "IS NULL" or "NOT IN" clauses uses in combination with ORDER, LIMIT and OFFSET 3687: 'bat.insert' undefined 3688: Crash at exit (overrun THRerrorcount?) 3689: No more connections accepted if a single client misbehaves 3690: find_fk: Assertion `t && i' failed. 3691: conversion of whitespaces string to double or float is accepted without an error during insert 3693: algebra.join undefined (caused by non-existing variables in the plan) 3696: Inconsistent behavior between dbl (SQL double) and flt (SQL real) data types and across platforms 3697: mserver5[26946]: segfault at 0 ip 00007f3d0e1ab808 sp 00007f3cefbfcad0 error 4 in lib_sql.so[7f3d0e180000+16c000] 3699: segfault again! (during last week I found 3 segfault bugs already) 3703: INSERT INTO a MERGE TABLE crashes mserver5 3704: Unknown identifier from subquery 3705: Assertion failure in rel_bin.c:2274: rel2bin_project: Assertion `0' failed. 3706: Assertion failure in gdk_bat.c: BATassertHeadProps: Assertion `!b->H->sorted || cmp <= 0' failed. 3709: "BATproject: does not match always" on abusive use of ALTER TABLE SET READ ONLY

Oct 2014-SP2 bugfix release

Client Package	Changes to the Perl interface, thanks to Stefan O'Rear: removes "use sigtrap", because this has global effects and should not be used by modules, only by the application. allows Perl 5.8.1+ Unicode strings to be passed to quote() and included in statements (UTF-8 encoded, as expected by Monet's str module) quote and unquote now use the same quoting rules as the MonetDB server, allowing for all characters except NUL to be round-tripped several character loops have been reimplemented in regex for much greater performance micro-optimizations to the result fetch loop block boundaries are preserved in piggyback data so that Xclose is not appended or prepended to a SQL command diagnostic messages #foo before a result header are ignored, this is necessary to use recycler_pipe fail quickly and loudly if we receive a continuation prompt (or any other response that starts with a non-ASCII character) header lines must start with %, not merely contain %, fixing a bug when querying a table where string values contain % after closing a large resultset, account for the fact that a reply will come and do not lose sync allow a MAPI_TRACE environment variable to dump wire protocol frames to standard output fixes maximum MAPI block size to match the server limit of 16k. previously would crash on blocks larger than 16k
SQL	Fixed a typo in a column name of the sys.tablestoragemodel view (auxillary changed to auxiliary).
Bug Fixes	3467: Field aliases with '#' character excise field names in result set. 3605: relational query without result 3619: Missing dll on MonetDB Start 3622: Type resolution error 3624: insert of incomplete or invalid ip address values in inet column is silently accepted but the values are not stored (they become/show nil) 3626: casting a type without alias results in program contains errors 3628: mclient and ODBC driver report 'type mismatch' when stddev_pop used in a select which returns 0 rows 3629: IF THEN ELSEIF always evaluates the first test as true 3630: segv on rel_order_by_column_exp 3632: running make clean twice gives an error in clients/ruby/adapter 3633: Wrong result for HAVING with floating-point constant 3640: Missing implementation of scalar function: sql_sub(<date>, <month interval>) 3641: SQL lexer fails to detect string end if it the last character is U+FEFF ZERO WIDTH NO-BREAK SPACE 3642: Combined WHERE conditions less-than plus equals-to produce incorrect results 3643: Missing implementations of scalar function: sql_sub(<timetz>, arg2) 3644: COPY INTO fails to import "inet" data type when value has prefix length in CIDR notation 3646: ORDER BY clause does not produce proper results on 'inet' datatype 3649: recycler crashes with concurrent transactions

Oct 2014 SP1 bugfix release

MonetDB5 Server	Fixed adding of 0 intervals to dates.
Build Environment	The libraries included in the Windows installers have been upgraded. We now use libxml2-2.9.2, openssl-1.0.1j, pcre-8.36, and zlib-1.2.8.
MonetDB Common	Implemented a change to the way in which string bats are appended. We now try harder to limit the growth of the string heap. Fixed some problems with BATsample. It was possible for BATsample to return a value that was just beyond the end of the sampled BAT. Also, on some systems the range of the rand() function is rather limited (0..32767) and trying to get a sample larger than this range would result in an infinite loop.
SQL	Fixed sys.queue() implementation to report on other queries being executed. A number of bugs were fixed in the code to upgrade a database from previous releases. This version should fix the upgrade of a database that had been upgraded to the Oct2014 release, but also properly upgrade directly from Jan2014 and Feb2013 releases.
Bug Fixes	3580: cosmetic change (append newline) 3609: Incorrect use of generate_series 3611: quantile() and median() commands crash when used 2x on the same variable on a null table 3612: assertion failure when deleting rows from table to which a FK constraint is defined 3620: ORDER BY broken when using UNION ALL 3621: Hexadecimal literal vs decimal literal

Known issue when upgrading old versions. We have noticed that upgrading directly from a version older than the Jan2014 release gave an error. Upgrading to Jan2014 first and then on to Oct2014 should work.

Backup. As always, make a backup of your databases before upgrading.

Oct 2014 feature release

MonetDB5 Server	Removed algebra.materialize. Removed algebra.kunique and algebra.tunique. They were subsumed by algebra.subunique. Remove algebra.antiuselect and algebra.thetauselect. They were subsumed by algebra.subselect. Removed algebra.topN and its imlementation BATtopN. The function was not used. Removed aggr.histogram and its implementation BAThistogram. The function was not used, and did not produce output in the "headless" (i.e. dense-headed) format. Histograms can be created as a by-product of group.subgroup.
Build Environment	Jacqueline, the MonetDB/JAQL frontend, has been removed. The frontend never grew beyond being experimental, and there is no interest anymore to maintain the code.
Merovingian	monetdb create: add -p flag to set monetdb user password on creation, and therefore allow creating the database in unlocked state
mapilib	Changed mapi_timeout argument from seconds to milliseconds.
stream	Changed mnstr_settimeout function so that the specified timeout is now in milliseconds (used to be seconds), and that it also needs an extra argument specifying a callback function (no arguments, int result) that should return TRUE if the timeout should cause the function to abort or continue what it was doing.
MonetDB Common	Added "multifarm" capability. It is now possible to separate persistent and transient BATs into different directories (presumably on different disks). This can be done by using the --dbextra option of mserver5 (see the man page).
SQL	Added PostgreSQL compatible string TRIM, LTRIM, RTRIM, LPAD and RPAD functions Stop support for upgrading directly from a database created with a server from the Oct2012 release or older. You can upgrade via the Feb2013 or Jan2014 release.
Bug Fixes	2945: evaluation of SQL "between SYMMETRIC" requires MAL iterator because there is no (bulk) MIN/MAX(a,b) 3204: monetdb create: allow setting admin password during creation 3390: Missing definition for pushSht in monetdb5/mal/mal_builder.h 3402: We should have a C implementation of mal.multiplex. 3422: Segmentation fault after large table insert 3459: incomplete implementation of JDBC driver getNumericFunctions(), getStringFunctions(), getSystemFunctions(), getTimeDateFunctions(), getSQLKeywords() methods in MonetDatabaseM 3471: JDBC driver: incorrect output result of SQL query: SELECT 1 ; 3474: bulk and scalar versions of mkey.rotate_xor_hash differ 3484: COPY INTO on a file works fine on Linux/OSX, but not on Windows 3488: Slow SQL execution for correlated scalar subquery 3489: SQL query with ORDER BY does not order its result as requested 3490: SQL query kills the mserver5 (Segmentation fault) 3491: SQL query kills the mserver5 (Segmentation fault) 3493: Test monetdb5/modules/mal/Tests/pqueue.mal fails since recent checkins 3494: Tests monetdb5/modules/mal/Tests/pqueue[23].mal lack correct/expected/intended output 3495: Test sql/test/centipede/Tests/olap.sql lacks correct/expected/intended output 3497: monetdb start reports crash in merovingian.log 3498: SQL throws TypeException if aggregations and limit statements are both present 3502: Database was killed by signal SIGBUS 3504: COPY INTO does not allow OFFSET without specifying amount of records 3505: expression with <boolean> = NOT <boolean> returns a syntax error but NOT <boolean> = <boolean> not 3506: conversion to varchar terminates mserver 3508: conversion of string '0 ' to type smallint or integer fails 3510: timestamp + month interval generates bogus MAL? 3511: When having multiple selections combined with aliases not all of them seem to be evalauted. 3512: auto-conversion of string to `sht` type no longer works 3513: COPY BINARY INTO fails on 6gb file; works fine on 3gb 3516: inserting '0' into a column of datatype numeric fails 3518: UNION with subqueries 3521: large results of function exp() are not automatically returned as double 3522: SQL catalog table sys.columns lists columns for table ids which do not exist in sys.tables 3523: Window function over union gives no result 3524: wrong error on missing aggregation column 3527: select distinct - order by - limit 2 results in one single result 3528: segfault at mal_session.c:521 3532: several geom tests crash after manifold changes 3534: missing table name with invalid column in join using (and problems after resolving it) 3536: program contains error with join using integer and smallint 3542: gdk/gdk_bat.c:2904: BATassertHeadProps: Assertion `!b->H->revsorted || cmp >= 0' failed. 3543: invalid behavior and incorrect data results for SQL data type: numeric(4,4) 3544: sys.reuse() corrupts data 3546: Division by zero in CASE statement that should avoid it 3547: Empty query when selecting a field from a view made of UNION ALL 3551: Wrong ticks in TRACE 3552: incorrect data results for "WHERE int_col <> 0" 3554: Issue with subselect and ORDER BY 3555: Order of evaluation inside CASE WHEN 3558: numeric values (as strings) are incorrectly parsed/converted and invalid strings are accepted without error 3560: Error "BATproject: does not match always" with subselect/groupby/having 3562: mserver5: gdk_bat.c:2855: BATassertHeadProps: Assertion `!b->H->revsorted || cmp >= 0' failed. 3563: incorrect results for scalar function locate(in_str, search_str, occurrence) 3565: Wrong/confusing error message when trying to add a FK to a TEMP TABLE 3572: Table names with escaped double quotes are rejected 3573: alter table alter_not_null_test alter test set NOT NULL; is accepted when test contains null. This used to be restricted but isn't anymore 3575: segmentation fault in mserver5 process 3576: Dropping default value definitions from a table does not work as expected 3577: SIGSEGV in BATins_kdiff 3579: segmentation fault in mserver5 process 3581: mserver5: rel_bin.c:2504: rel2bin_groupby: Assertion `0' failed. 3582: mserver5: sql_mem.c:48: sql_ref_dec: Assertion `r->refcnt > 0' failed. 3583: Possible buffer overflow in max(varchar) 3585: Incorrect query terminates connection 3586: mserver5: sql/storage/store.c:3610: sys_drop_func: Assertion `rid_func != oid_nil' failed. 3592: SIGSEGV in MANIFOLDjob 3593: delta_append_val: Assertion `!c || ((c)->S->count) == bat->ibase' failed. 3594: gdk/gdk_bat.c:2855: BATassertHeadProps: Assertion `!b->H->revsorted || cmp >= 0' failed. 3595: Race/heap corruption on thread exit 3596: gdk_bat.c:2861: BATassertHeadProps: Assertion `!b->H->nonil || cmp != 0' failed. 3597: SQL to MAL listing looses types 3598: SQL bulk load should ignore leading/trailing spaces also with type decimal (as with integers & real/double) 3599: Double-free of imprints 3601: Trivial typo in debian/monetdb5-sql.init.d 3603: "monetdb create -p" hangs monetdbd 3604: Sys.queue ignored upon errors

Feb 2013 feature release
Testing Environment	enabled "top-level" Mtest.py So far, while Mtest.py could be called in any subdirectory of the MonetDB source tree (and could then run all tests in the entire sub-tree), it was not possible to call Mtest.py in the top-level MonetDB source directory to run all tests. Instead, to run all tests, Mtest.py had to be called at least 4 times, once in each of these directories: "clients", "monetdb5", "sql", "geom". Now, it is possible to call Mtest.py once in the top-level MonetDB source directory to run all tests in one go. The behaviour of calling Mtest.py in any subdirectory, including the four mentioned above, did not changed, other than that now obsolete command line options "-p / --package <package>" and "-5 / --monetdb5" have been removed.
Java Module	merocontrol was changed to return server URIs, and lastStop time. Connections and dbpath were removed. Mapi protocol v8 support was removed from MapiSocket. Protocol v8 has not been used by the servers any more since Apr2012 release
Client Package	Mapi protocol v8 support was removed from all client drivers. Protocol v8 has not been used by the servers any more since Apr2012 release The tool mnc was removed from installations msqldump: Implmented an option (--table/-t) to dump a single table. Changed msqdump's trace option to be in line with mclient. In both cases, the long option is --Xdebug and the short option is -X.
MonetDB5 Server	mserver5: The --dbname and --dbfarm options have been replaced by the single --dbpath option. The scheduler of mserver5 was changed to use a fixed set of workers to perform the work for all connected clients. Previously, each client connection had its own set of workers, easily causing resource problems upon multiple connections to the server.
Merovingian	Upgrade support for dbfarms from Mar2011 and Aug2011 was dropped monetdb status now uses a more condensed output, to cater for the uris being shown, and prints how long a database is stopped, or how long ago it crashed monetdb status now prints the connection uri for each database, when available. The connections and database path properties have been dropped. monetdb status now prints last crash date only if the database has not been started since.
Bug Fixes	2291: small doubles end up as NULL after arithmetic 3215: Calculation Date function using interval and year 3033: stethoscope needs better documentation 3084: Timestamp arithmetic very slow (especially on Windows) 3125: Python tests fail after recent Python API changes 3172: assertion fails on table function with subselects as parameters 3178: one scan is enough to implement ALGstdev_@1 in monetdb5/modules/kernel/algebra.mx 3179: LIKE: batstr.like+algebra.uselect called instead of pcre.like_filter 3193: Expressions not supported in the GROUP BY or ORDER BY clause. 3216: "unknown property" error setting format and width in .monetdb file 3217: gdk_posix fails to compile under NetBSD 3221: can no execute large statements 3222:  Monet DB giving wrong value from a table when a value is queried after insertion/ Data Loss 3225: mserver segfault on select 3227: MT_set_lock() call on an non-initialized lock

 

Feb 2013-SP1 bugfix release

Bug Fixes

3228: Python client - mapi.Connection.__getbyes should throw exception when 0 bytes received from socket 3229: GRANT does not work for UPDATE 3230: GRANT SELECT priveleges not set in SQL FUNCTIONs 3233: getting a crash of server with segmentation fault on SLES 11 64 SP2 3235: MapiPP.pm unquote '0' as 'undef' 3236: mclient \d produces no output 3237: Assertion on BATgroup_internal 3241: SEGFAULT on string processing when using optimizer.reduce()

In addition to these changes, we also fixed the database upgrade.  Due to internal changes in the Feb2013 release, some changes had to be made to the SQL catalog.  These changes were done incompletely in the Feb2013 release.  This has now been fixed in the Feb2013-SP1 release.  Both a direct upgrade from Oct2012 and from Feb2013 should work.

Feb 2013-SP2 bugfix release
MonetDB5 Server	Fixed argument parsing of mapi.reconnect() with 5 arguments.
MonetDB Common	Fixed a bug in case the candidate list is dense and completely outside the range of the bat being worked upon.
Java Module	The pre-compiled .jar files are now created using Java 7 instead of Java 6.
SQL	Added TEMPORARY to the non-reserved keywords, ie no need for double quotes when used as identifier.
Bug Fixes	3245: SELECT from a view of a view 3250: SELECT fails with subquery result missing when adding condition 3251: Name of coercions optimizer wrong in opt_wrapper.c 3257: stddev_pop and stddev_samp break during a GROUP BY 3259: BETWEEN clause not producing rangejoin 3261: Incorrect empty string values in query results 3264: select + group by expression alias from a view kills database by signal SIGSEGV 3276: Problem with starting monetdb on debian sqeeze

Feb 2013-SP3 bugfix release
MonetDB Common	System calls to flush files to disks were added. This may cause some slowdown, but it should provide better durability, especially in the face of power failures.
Java Module	Further improved setBigDecimal() method, based on patch by Ben Reilly in bug #3290 Fixed bug where PreparedStatement.setBigDecimal() wouldn't format its input well enough for the server causing odd errors. Allow PreparedStatement.setXXX() methods to be called with null arguments, bug #3288
Bug Fixes	3265: import decimal with copy into fails 3269: copy into results into gdk_utils.c:728: GDKfree_: Assertion `(size & 2) == 0' failed. 3277: count( case when COL1 in ( NUM1 , NUM2 ) then COL2 end ) cause problems in functions 3278: SQL order by <column number> only sorts on first call 3279: column alias combined with ORDER BY returns wrong data 3280: median function breaks when the column contains NULL values 3281: pqueue.enqueue_max type mismatch error 3282: bulk loaded data corrupted when restart monetdb 3286: TypeException when creating table with data by selecting from sys.storagemodel() 3287: lower(name) clause in query causing "BATsubselect: invalid argument.." error 3288: PreparedStatements may throw NullPointerException when setting parameter 3289: SELECT: no such binary operator 'sql_add(boolean,boolean)' 3290: Faulty type handling/casting in JDBC driver 3295: mserver5 fails to start properly: "address JSONrenderobject not found" 3296: Type checking in table functions 3297: Data type changes when arithmetic in prepared statements. 3300: explain select from storagemodel causes mserver5 to segfault 3302: test exists-select.Bug-2933.sql fails after recent changes 3303: Assertion on concurrent write

Feb 2013-SP4 bugfix release
Merovingian	monetdb now no longer compresses output when not connected to a terminal, bug #3292
Bug Fixes	2781: SQL localtime() returns GMT time 2962: client timezone information needs to be handled also for modifications to timestamp values 3232: The function weekofyear returns results not according to the iso 8601 standard 3255: M5 tests inlineFunction & ifthencst fail since changeset 207ed7d7e03a 3292: monetdb status should print full line 3307: distinct multicolumn performance and implementation suggestion for count distinct 3310: Cast value to integer gives wrong result 3311: strLength assertion on join and concat 3313: Assertion in rel2bin_project 3315: Incorrect behavior with subqueries in DELETE 3317: correlated subselect does not work 3318: UPDATE does not work with DECLEAREd TABLE in FUNCTION 3319: UNIQUE column in DECLAREd TABLE in FUNCTION results in segfault 3329: Function AVG fails during runtime in a procedure 3330: Corrupt mitosis plan with group-by query 3332: MEDIAN: no such operator 'median(decimal)' 3336: DB files not removed if all rows are deleted, even after restart 3337: Monet ODBC gives wrong schema information 3339: Segvault in complex function call 3340: crash on seeming incorrect SQL query 3341: segfault caused by select rank() on a newly-loaded table 3342: incorrect automatic cast from decimal to tinyint 3345: copy binary into fails with the wrong error message 3346: Deadlock in mal_dataflow's q_dequeue() (SciQL-2 branch) 3348: SQL: LIKE incorrect with _ 3349: When selecting everything from a joined result, column contents are incorrect in one of my use cases. 3351: gzip'ed files cannot be bulkloaded into DB 3352: Median function with decimal type doesn't work 3353: Readonly tables call for UNSET 3358: Remote client connection stays in CLOSE_WAIT state 3360: uninitialized memory causing bad output in query plans 3362: ALTER TABLE operations silently remove previously set READ ONLY property 3363: a join causes SIGSEGV after creation of Unique Key 3367: Fully qualified order by column gives "relational query without result" 3369: \0 Byte appears in MAPI

Feb 2013-SP5 bugfix release
Client Package	ODBC: Implemented retrieving variable-length data in parts with SQLGetData. See bug 3381.
MonetDB Common	Removed the heap cache. Since the fix for bug 3323 which made that the cache was actually getting used, bug 3376 made clear that it didn't work very well. In addition, on Linux at least, the heap cache slows things down.
Java Module	Rearranged order of returned columns of certain metadata functions to comply with the JDBC documentation. See bug 3356.
Bug Fixes	3323: Heapcache bugs/performance issues 3331: SAMPLE will return same result every time. 3356: DatabaseMetaData.getColumns() doesn't work correctly when using index-based getters 3367: Fully qualified order by column gives "relational query without result" 3368: BAT sortedness info ignored on ORDER BY and TOPN 3370: SQL environment settings are updated even when the set statement fails 3373: Setting table to read-only (Bug 3362) fails on big-endian 3375: LIKE join: BATfetchjoin does not hit always 3376: COPY INTO fails with HEAPextend: failed to extend: MT_mremap() failed 3377: Query interfering with next query in same transaction, after SP4 3380: Python DBAPI driver throws exception when fetching EXPLAIN results 3381: Windows ODBC driver hangs or crashes on simple queries

Feb 2013-SP6 bugfix release
MonetDB5 Server	Fixed a possible buffer overflow in the COPY INTO code. Fixed a problem that when the server is using all available threads for a query, it was not possible to attach another client and have it execute even the smallest query. This is fixed by creating extra threads for each client entering the fray at the cost of having more threads that execute queries. But at least there is guaranteed progress for all clients (modulo the operating system scheduler). See bug #3258.
Merovingian	monetdbd(1) now refuses to startup if it cannot read the properties from the dbfarm, bug #3398
Client Package	mclient: Fixed a bug where the -H option only worked if the readline history file already existed. Now we properly create and use the history file. ODBC: Fixed interpretation SQL_C_SLONG/SQL_C_ULONG/SQL_C_LONG to refer to a 32 bit integer always (i.e. "int" on 64 bit architectures despite the name and the Microsoft documentation). This seems to be the consensus. ODBC: Fixed transaction level: MonetDB only supports the highest level (SQL_TXN_SERIALIZABLE), so setting the transaction level can be accepted and ignored.
MonetDB Common	Stopped using the deprecated sbrk() system call. Fixed a problem when reverse sorting a sorted column. Fixed bugs that deal with problems that could possibly occur when transactions are aborted and the server is restarted. See bug #3243. A bug was fixed in the handling of grouped aggregates when all values in a group are null. See bug #3388.
SQL	Fixed a bug where the server at some point stopped saving compiled queries in the SQL query cache.
Bug Fixes	3243: Segmentation fault (possible data corruption) after clean shutdown 3258: Scheduling issues 3368: BAT sortedness info ignored on ORDER BY and TOPN 3374: UNIQUE constraint does not set tkey property on the corresponding BAT 3382: Response to PREPARE emtpy if query contains a LIMIT 3385: Simple query fails with 'identifier not found' 3387: mclient does not properly double quote schema names when using autofill tab 3388: case statement in "order by" clause doesn't work when used together with "group by" 3389: median function with "group by" - SIGSEGV 3392: ODBC datatype conversion for INTEGER not working properly 3394: "Cannot find column type" error in temporary tables in functions 3395: error occurred during a query: "'CASE WHEN" sentence 3396: Improper UDF expansion 3397: Error in ODBC-Driver when using Prepared Statements 3398: Cannot stop monetdbd after erroneously starting an uninitialized dbfarm

 

 

Oct 2012 feature release
Build Environment	Removed --enable-noexpand configure option.
Java Module	Fixed problem with PreparedStatements and setXXX() methods using column numbers instead of names, bug #3158
Client Package	mclient now accepts URIs as database to connect to. all strings returned by python2 are unicode, removed use_unicode option python2 and 3 type convertion speed improvements python2 uses new style objects now (bug #3104) split python2 and python3
MonetDB5 Server	Removed module attach since it wasn't used or even tested. The MAL-to-C Compiler (mcc) was removed. The code wasn't tested and most likely non-functional.
Merovingian	Fixed problem where monetdbd would refuse to startup when discovery was set to false, bug #3155
MonetDB Common	Removed the gdk_embedded (and embedded) option. The code wasn't tested and most likely non-functional. BAT-of-BATs is no longer allowed. It was already not allowed to make these types of BATs persistent, but now they can't be created at all anymore.
Bug Fixes	3084: Timestamp arithmetic very slow (especially on Windows) 3125: Python tests fail after recent Python API changes 3155: Provide an option for setting discovery to false

 

No Debian and Ubuntu packages were built for this release (i.e. Oct2012) due to a packaging failure.  We expect to have resolved that problem by Oct2012-SP1.  We apologise for the inconvenience.

 

Oct 2012-SP1 bugfix release
Bug Fixes	3155: Provide an option for setting discovery to false 3161: invalid tuple received from server, got 138 columns, expected 476, ignoring 3182: When using aggregate SQL functions, precision & decimal metadata for numeric/decimal type returned is 0,0

For this release (i.e. Oct2012-SP1) there are once again Debian and Ubuntu packages.  From this release onward, Ubuntu Lucid Lynx (10.04) is no longer supported.

 

Oct 2012-SP2 bugfix release
SQL	Fixed crash when performing an INSERT on a table with string-like column defaulting to NULL and omitting that column from VALUES, bug #3168
Java Module	Implemented type map support of Connection to allow custom mapping of UDTs to Java classes. By default the INET and URL UDTs are now mapped to nl.cwi.monetdb.jdbc.types.{INET,URL}. Most notably, ResultSet.getObject() and PreparedStatement.setObject() deal with the type map. Fixed a problem in PreparedStatement where the prepared statement's ResultSetMetaData (on its columns to be produced) incorrectly threw exceptions about non existing columns. Bug #3192
Bug Fixes	2579: SQL: incorrect cast from double to int 2659: tuples INTERSECT vs count (... INTERSECT ...) differs 2805: row_number doesn't work in complex query 2886: warning: Catalog parameter '' ignored 2977: timestamp minus date results in program error 2978: now() + 1 results in illegal argument 2999: Up-cast from SQL drops order information? 3048: algebra.join undefined when using multiple arithmetic operations in SQL where-clause 3124: insert into table with two foreign keys crashes mserver5 3136: Memleak (in querycache?) while querying 3143: Type resolution error in SQL procedural code 3163: mserver5 crashes with large number of concurrent clients 3168: crash when insert varchar/char/clob with default NULL 3169: php driver uses call-time pass-by-reference 3173: SQL queries with limit and offset clauses are unable to use bind parameters for either 3181: Join query returns zero results when there is an index on the tables 3185: inconsistency between documentation and implementation of command line options 3186: case evaluted to early 3188: Storage of client history 3189: Overflow in calculation 3190: When subtracting two columns of type "timestamp", no results are returned 3191: rel_optimizer.c:5596: _rel_optimizer: Assertion `0' failed. 3192: "No such column with index" error if CASE in SELECT clause 3199: Inconspicuous sequence of prepare, execute, commit and close produces error 3205: Inet datatype handling inconsistent between int, cast( as inet) and string

 

Oct 2012-SP3 bugfix release

Bug Fixes

2579: SQL: incorrect cast from double to int 2659: tuples INTERSECT vs count (... INTERSECT ...) differs 2805: row_number doesn't work in complex query 2886: warning: Catalog parameter '' ignored 2977: timestamp minus date results in program error 2978: now() + 1 results in illegal argument 2999: Up-cast from SQL drops order information? 3048: algebra.join undefined when using multiple arithmetic operations in SQL where-clause 3124: insert into table with two foreign keys crashes mserver5 3136: Memleak (in querycache?) while querying 3143: Type resolution error in SQL procedural code 3163: mserver5 crashes with large number of concurrent clients 3168: crash when insert varchar/char/clob with default NULL 3169: php driver uses call-time pass-by-reference 3173: SQL queries with limit and offset clauses are unable to use bind parameters for either 3176: Same query and different results 3181: Join query returns zero results when there is an index on the tables 3185: inconsistency between documentation and implementation of command line options 3186: case evaluted to early 3188: Storage of client history 3189: Overflow in calculation 3190: When subtracting two columns of type "timestamp", no results are returned 3191: rel_optimizer.c:5596: _rel_optimizer: Assertion `0' failed. 3192: "No such column with index" error if CASE in SELECT clause 3199: Inconspicuous sequence of prepare, execute, commit and close produces error 3205: Inet datatype handling inconsistent between int, cast( as inet) and string 3208: Unable to execute prepared statements with a large number of arguments 3209: exp_bin: Assertion `0' failed 3212: Query cache does not work properly

 

Jul 2012 feature release
Build Environment	Created seperate RPM and DEB packages for MonetDB/JAQL. Removed --enable-bits option from configure. Split the MonetDB-client-ruby RPM package into two and named them in accordance with the Fedora packaging guidelines as rubygem-<gem-name>.
SQL	COPY INTO now accepts optional parenthesis for file argument. Binary COPY INTO now requires 'COPY BINARY INTO'.
Client Package	ODBC: Fixed a bug where SQLNativeSql expected a statment handle instead of a connection handle.
MonetDB5 Server	Crackers code has been removed. Development continues in the holindex branch.
Merovingian	Fixed misc memory leaks, which caused monetdbd to grow in memory size over time. Removed erroneously (re-)added master and slave properties, this functionality is currently not working.
MonetDB Common	Fixed intermittent problem that joins and selects return incorrect results and possibly inconsistent databases. The problems only occurred after a series of queries and updates, therefore it was hard to reproduce. The sorted property, which was used to maintain whether a column in a BAT was sorted or reverse sorted, has been replaced by a pair of properties, sorted and revsorted. These new properties can be set independently (unlike the old sorted property), and so if both are set, the column must be constant. In addition, internal property checking has been overhauled. Now, when a property is set incorrectly, and when assertions are enabled, an assertion will go off. There is also a function which can derive properties. Implemented proper overflow checking on all arithmetic operations.
Bug Fixes	3016: Overflowing double multiplication leads to incorrect result 3091: Since Apr2012-SP1 !TypeException:user.s11_1[59]:'bat.reverse' undefined in: _89:any := bat.reverse(_86:wrd) 3099: ambiguous attributes in query with WITH clause and EXISTS clause 3100: charset problem with the odbc driver for windows 3103: display_size and internal_size of Cursor.description or incorrect 3105: Leaking using Python 3109: Wrong order in autoload mal script causes warning on startup 3111: memory leak, monetdb set/get command 3112: Median over temporal domains 3113: Cannot set value of indexed column when doing UPDATE 3114: Foreign Key prblems 3116: Segfault in SQL

 

Jul 2012-SP1 bugfix release
Java Module	Fixed adaptive cache size used when retrieving results, not to cause divide by zero errors when memory gets short, bug #3119.
Client Package	mclient no longer prints the SQLSTATE at the start of each error returned by the SQL-server.
MonetDB5 Server	The server now distinguishes between starting and started states, such that monetdbd can wait for it to finish starting.
Merovingian	Starting a server now waits for as long as the server needs to possibly recover, bug #3134. In case of a long wait, the monetdbd logfile gives extra information on what the server is doing to recover. Fixed a crash of monetdbd when local databases were unshared, bug #3135 Resolved a problem where automatic starting of a database initiated by multiple clients at the same time could cause failed starts. Bug #3107
Bug Fixes	3075: inconsistent declaration of algebra.markH with 3 input arguments 3090: crashed if using single identifier for where condition 3093: sql fail if use scalar subquery with alias 3107: concurrent connections to the same stopped database yield in multiple mserver5 starts by monetdbd 3119: MonetConnection$ResultSetResponse throws java.lang.ArithmeticException: divide by zero 3132: test/BugTracker-2011/func_iter_vs_bulk.Bug-2826 fails 3134: Database gets killed by timeout during startup 3135: monetdbd crash while creating & loading database

 

 

Jul 2012-SP2 bugfix release
Java Module	Fixed a bug where DatabaseMetaData.getURL() did return null:0 for hostname:port.
MonetDB5 Server	Changed the way nclients maximum was calculated to avoid 'out of client slots' errors way before the maximum was reached.
Merovingian	Resolved a problem where monetdb could fail to start a database with the message "database 'X' started up, but failed to open up a communication channel". Bug #3134, comment #7.
MonetDB Common	Fixed a bug in BATantijoin when either side is a singleton BAT. This fixes bug 3139.
Bug Fixes	3117: test sql/test/BugTracker-2011/Tests/history.Bug-2909.sql produces extra profiling output since changeset 817328b74e2f 3134: Database gets killed by timeout during startup 3139: Not equal evaluation does not work 3144: Memleak when connecting to a database (mclient and mapi) 3145: mclient error when enabling history 3150: Multiple string equality tests fail to match (Test_JdbcClient)

Apr 2012-SP2 bugfix release
Client Package	Resolved a cosmetical error where tab-characters would cause misalignments in tabular result views. For the time being, tabs are now represented as a single space in tabular view.
MonetDB Common	Limit number of detected CPU cores to 16 on 32-bits systems to avoid running quickly out of addressable resources followed by a kill from the OS.
Bug Fixes	2594: NULL cannot automatically be casted to every other type 3085: assertion fails upon execution of a valid SQL query 3086: Modulo function does not work in select from table 3087: query causes "Not yet implemented" message 3089: mserver5 crashed if join condition has two column and incorrect order 3090: crashed if using single identifier for where condition 3096: Type of median error

 

 

Apr 2012-SP1 bugfix release

Build Environment	Windows: we now use OpenSSL 1.0.1b.
SQL	Databases that were upgraded from the Aug2011 release have an error in the catalog for SQL procedures. This is now fixed.
Merovingian	Fixed a bug where connecting to a stopped multiplex-funnel would result in a 'there are no available connections' error.
MonetDB Common	Implemented MT_getrss for Mac OS X systems, this allows the server to know about how much memory is currently in use.
Bug Fixes	2454: SQL: add various statistical functions 2916: non admin user cannot create temporary tables 2949: SQL exception during execution of a correct user defined function that just returns a relation table constructed by a simple SQL select query 2958: Grant does't work with schema syntax 2960: mclient terminates connection during insertion of a large number of records from a file 2972: SQL URL functionality contains errors 2987: Broken type inference when using sql round 2992: SQL module truncates DECIMALS when passing integers to MAL module 3000: Inconsistency between read-only and read-write modes after dropping a schema 3028: Segmentation fault in seqscan_eq_bte_bte_tloc_void_void 3046: mserver5 terminates with segmentation fault 3063: datacell functions no longer work after restart of database 3072: Database upgrade from Dec2011 to Apr2012 broken for storage() 3073: VERY slow foreign key check (due to missing batcalc.isnil() ?) 3074: Name resolution in procedures 3077: No errors on UDF returning table with too many columns 3079: Converting an Oracle schema DDL for MonetDB fails to create a column named "no".

 

 

Apr 2012 feature release
Build Environment	Fixed some of the package names for Debian/Ubuntu. Packages for libraries should contain the major number of the library version number. This was not always the case. On Debian and Ubuntu, install Python modules in dist-packages instead of site-packages. This fixed bug 2997.
SQL	Fixed a crash that happened at the end of a database upgrade to the Dec2011 database scheme. The crash happened during cleanup after the database was upgraded, so it was merely inconvenient. Stripped off implementation-specific parts from error messages before they get presented to the user.
Java Module	Password reading by JdbcClient no longer results in strange artifacts JdbcClient now returns exit code 1 in case of failures Implemented missing Number types support in PreparedStatement.setObject() Fixed bug in DatabaseMetaData.getSchemas() method that caused an SQL error when called with catalog and schema argument. Resolved a bug where JDBC and Control connections could terminate abruptly with 'Connection closed' messages
Client Package	ODBC: Implemented the SQL_ATTR_CONNECTION_TIMEOUT attribute. mclient now has a -a (--autocommit) option to turn off autocommit mode.
MonetDB5 Server	The type "chr" has been removed. chr has long been superseded by bte for 1 byte arithmetic plus it is pretty useless to hold single characters since we use Unicode and thus only a tiny subset of the supported character set would fit. Fixed potential crash by dealing better with non-standard types. A couple of memory leaks have been plugged.
Merovingian	The logfile and pidfile monetdbd properties are now displayed with dbfarm path when relative Fixed a bug in the multiplex-funnel where certain clients would abort on responses for update queries. Fixed problem where version and mserver properties for monetdbd were not always successfully retrieved. Bug #2982. Fixed problem where shutdown of monetdbd would lead to shutting down database 'control' which does not exist. Bug #2983. Fixed issue causing (harmless) 'error reading from control channel' messages. Bug #2984. Resolved problem where remote start/stop/etc. commands with monetdb would report error 'OK'. Bug #2984.
MonetDB Common	The type "chr" has been removed. chr has long been superseded by bte for 1 byte arithmetic plus it is pretty useless to hold single characters since we use Unicode and thus only a tiny subset of the supported character set would fit.
Bug Fixes	2994: test monetdb5/modules/kernel/Tests/vacuum.mal fails since changeset ada288cef37c 3023: SQL tests fail as error messages now go to stdout iso. stderr 3030: optimizer tests fail with new optimizer framework 3032: numerous M5/MAL tests fail on SunOS/Solaris as TypeExecptions do no longer show arguments in command signatures

 

MonetDB Dec2011 Release

 

Dec 2011-SP2 bugfix release
Java Module	Implemented missing Number types support in PreparedStatement.setObject()
Bug Fixes	2915: Query (mclient or jdbc) with 2 outer joins appears to crash connectionor server 2947: Queries where a "is null" clause is combined with two equals clauses fail to produce the expected result. 2968: Memory Leaks 2974: thetajoin produces invalid/inconsistent bat 2988: Subquery in select part goes wrong 2998: Equivalent multi-way join queries give different results 3002: String plus interval result in program errors upon comparison 3011: count(*) in subquery results in 1 for query without results 3012: bitwise comparison fails after a cast 3029: test sql/test/Dependencies/Tests/Dependencies.sql fails after recent changes 3031: Virtually all SQL tests fail on Windows 3036: optimizers are not getting set correctly through set optimizer 3038: ~190 SQL tests fail on Darwin, FreeBSD & Solaris (only) since introduction of "(extended) relational algebra" 3043: WITH clause is not allowed in derived table 3045: algebra.project(0@0,bat) fails at propcheck 3052: Wrong Error Code on Rollback Exception

 

Dec 2011-SP1 bugfix release
Build Environment	Fixed some of the package names for Debian/Ubuntu. Packages for libraries should contain the major number of the library version number. This was not always the case. On Debian and Ubuntu, install Python modules in dist-packages instead of site-packages. This fixed bug 2997.
SQL	Fixed a crash that happened at the end of a database upgrade to the Dec2011 database scheme. The crash happened during cleanup after the database was upgraded, so it was merely inconvenient. Stripped off implementation-specific parts from error messages before they get presented to the user.
Java Module	Fixed bug in DatabaseMetaData.getSchemas() method that caused an SQL error when called with catalog and schema argument. Resolved a bug where JDBC and Control connections could terminate abruptly with 'Connection closed' messages
MonetDB5 Server	Fixed potential crash by dealing better with non-standard types.
Merovingian	Fixed a bug in the multiplex-funnel where certain clients would abort on responses for update queries. Fixed problem where version and mserver properties for monetdbd were not always successfully retrieved. Bug #2982. Fixed problem where shutdown of monetdbd would lead to shutting down database 'control' which does not exist. Bug #2983. Fixed issue causing (harmless) 'error reading from control channel' messages. Bug #2984. Resolved problem where remote start/stop/etc. commands with monetdb would report error 'OK'. Bug #2984.
Bug Fixes	2862: foreign key self references cause violation upon alter table add constraint... 2933: "TypeException: algebra.join undefined" when evaluating EXISTS (SELECT ...) predicate 2946: Invalid BAT if left join condition has only columns from right table 2964: LEFT OUTER JOIN with constants returns errors 2969: Result precision issue with Decimal data type 2973: Date coercion functionality tests 2979: misc merovingian control problems 2982: monetdbd status and mserver properties have random values 2983: monetdbd attempts to stop non-existing database 'control' 2984: monetdb create/release/start/stop returns error "OK" for remote connections 2995: Test geom/sql/Tests/example.sql crashes since changeset 6399077ff8a4 2997: monetdb-python packages installed in wrong dir on Ubuntu 10.04 LTS (lucid) 3004: Incorrect evaluation of conditions in WHERE part of SQL statement 3009: Segmentation fault on simple query 3013: calc.and is not yet defined for dbl 3014: dayofweek comment is wrong in mtime.mx:700 3017: rel_bin.c:611: exp_bin: Assertion `0' failed. with between on multicolumns 3020: Assertion error in debugging mode 3034: INNER JOIN with Subquery produces mal errors 3041: WHERE condition ignored after 2 LEFT JOINs with subqueries

 

Dec 2011 feature release
SQL	mclient: The csv output format can now also be of the form csv=c and csv+c where c is the column separator. The form with csv+ produces a single header line with column names. Changed a bug where the sign() function returned the same type as its argument instead of always an INTEGER. Added a fix for bug #2834, which caused weird (failing) behaviour with PreparedStatements. Fixed a bug when using default values for interval columns. Bug 2877. A bug was fixed where deleted rows weren't properly accounted for in all operations. This was bug 2882. A bug was fixed which caused an update to an internal table to happen too soon. The bug could be observed on a multicore system with a query INSERT INTO t (SELECT * FROM t) when the table t is "large enough". This was bug 2883.
Java Module	Implemented getMetaData() method of PreparedStatement. Fixed an AssertionError for special results from e.g. EXPLAIN queries. Fixed overflow error when batching large statements, bug #2952 Resolved a concurrency problem where ResultSet's date-related getters could cause odd stack traces when used by multiple threads at the same time, bug #2950. JDBC now implements JDBCv4.1 which makes it possible to be built with Java 7 and up. JDBCv4.1 is a maintenance release of JDBCv4, and hence can be compiled with Java 6, although the added methods obviously are not part of the java.sql interfaces. SQLExceptions thrown now carry a SQLSTATE. Until the server starts sending correct SQLSTATEs for all errors, server originated errors without SQLSTATE are considered generic data exceptions (22000). Fixed a bug where closing ResultSets and PreparedStatements could lead to errors on concurrent running queries using the same Connection due to a race condition. Changed version scheme of JDBC driver and MCL jar to be more standard, from monetdb-X.Y-<thing>.jar to monetdb-<thing>-X.Y.jar, bug #2943 The embedded monet instance capability of MonetConnection was removed. Bump JDBC version to 2.0 (Liberica). JDBC now implements JDBCv4 which makes it possible to be built with Java 6 and up. Java 5 and before are no longer supported, and can use the 1.x releases of the driver. INTERVAL columns are now treated as decimals, since they can have sub-second precision.
Client Package	Fix making connections with stethoscope to hosts without monetdbd. Bug #2944. Fixed some bugs in the PHP interface affecting the mapi_ping(), monetdb_insert_id() and auto_commit() functions. Bugs #2936, #2937, #2938. ODBC: Fixed SQLNumResultCols and SQLDescribeCol to return useful information after a call to SQLPrepare and before any SQLExecute. mclient: The exact interpretation of the -i (--interactive) option and the - filename argument have changed. The - filename argument means read from standard input and no longer implies that no \ interpretation is done on the input. Instead, \ interpretation is done if either standard input is a terminal, or if the -i option is given. The -i option no longer causes mclient to read from standard input. It only means to do \ interpretation when reading from standard input. Use the - filename argument to read from standard input. Note that if no -s option is specified and no filename arguments are present, mclient still reads from standard input. ODBC: Improved internal query for SQLSpecialColumns. Before, the query returned all columns taking part in a PRIMARY KEY *and* all columns taking part in a UNIQUE constraint. Now it returns only one or the other set. ODBC: The database name is now used as the catalog name throughout. Functions that return a catalog name return the database name, and functions that match on catalog name match it with the database name. ODBC: Implemented an easier way to create a log file of interactions with the ODBC driver. You can now add a connection attribute "LOGFILE=filename" to the connection string parameter of SQLBrowseConnect and SQLDriverConnect, and to the relevant part of the Windows registry or odbc.ini file. This value is only used if there is no environment variable ODBCDEBUG. Quoting of object names for mclient's \d command is now more flexible and consistent with standard SQL quoting rules, bug #2846. ODBC: Implemented SQLColumnPrivileges function. Install new program, mnc, which provides netcat functionality, but based on MonetDB's communication libraries Report full URI to database instead of just database when available in mclient. The time format of the timer output can now be controlled with an optional argument to the -i option. ms, s and m force the time to be formatted as milliseconds, seconds or minutes + seconds respectively. ODBC: Implemented a workaround in SQLTables for bug 2908. Small improvement to mclient's table rendering for tables without any rows. Previously, the column names in the header could be squeezed to very small widths, degrading readability. Python DB API connect() function now supports PEP 249-style arguments user and host, bug #2901 mclient now checks the result of encoding conversions using the iconv library. Fixed a source of crashes in mclient when a query on the command line using the -s option is combined with input on standard input (e.g. in the construct mclient -s 'COPY INTO t FROM STDIN ...' < file.csv). Fixed bug 2897 where slow (network) reads could cause blocks to not be fully read in one go, causing errors in the subsequent use of those blocks. With thanks to Rémy Chibois. Fixed a bug in mclient where processing queries from files could result in ghost empty results to be reported in the output Fixed Perl DBD rowcount for larger results, bug #2889 Perl: We now distinguish properly between TABLE and GLOBAL TEMPORARY (the latter are recognized by being in the "tmp" schema). Perl: fixed a bunch of syntax errors. This fixes bug 2884. With thanks to Rémy Chibois. Perl: Fixed DBD::monetdb table_info and tabletype_info. This fixes bug 2885. With thanks to Rémy Chibois. mclient: fix display of varchar columns with only NULL values. Fixed a bug in mclient/msqldump where an internal error occurred during dump when there are GLOBAL TEMPORARY tables.
MonetDB5 Server	A couple of memory leaks have been plugged. Introduced new variable max_clients that allows to define how many concurrent connections are allowed to be made against the database. mal_interpreter.mx Protect against concurrent exceptions If multiple parallel blocks access the exception variables and perform GDKfree on old messages, then we may enter a case that one thread attempts a second free. Simple lock-based protection is the first line of defense. bpm and partitions have been moved to the attic. It is replaced by the partition optimizer, still under construction. The MAL debugger next/step operations semantics have been fixed. The profiler also now clearly shows entrance/exit of a MAL function. The dataflow optimizer uses a less strict side-effect test for BAT new operations and better recognizes trivial plans. Fixed a problem where MAL variables weren't properly cleared before reuse of the data strucutre. This problem could cause the data flow scheduler to generate dependencies between instructions that didn't actually exist, which in turn could cause circular dependencies among instructions with deadlock as a result. Bugs 2865 and 2888.
Merovingian	Fixed crash in monetdb when an invalid property was retrieved using the get command, bug #2953. monetdb get output is now grouped by database instead of by property Unlike in previous releases, 'monetdbd get mserver' now returns the path to the mserver5 binary only for dbfarms that have a running monetdbd, instead of only for those that are not served by a monetdbd. Add nclients property that controls the maximum number of concurrent clients allowed to access the database Starting monetdbd without any arguments or without dbfarm is no longer supported. A dbfarm now must be provided for each command. The control passphrase has been turned into a hash of the password, for a more stronger authorisation model as used by mclients. On upgrade, any existing passphrase is converted to the hashed version of the passphrase. The monetdbd controlport option has been removed in favour of a boolean control option. On upgrade, when controlport was set to a non-zero value and a passphrase was set, control is set to true, or false otherwise. The monetdbd discoveryport option has been removed in favour of a boolean discovery option. On upgrade, when discoveryport was set to a non-zero value, discovery is set to true, or false otherwise. Fixed a bug where monetdbd's socket files from /tmp were removed when a second monetdbd was attempted to be started using the same port. Resolved problem where monetdbd would terminate abnormally when databases named 'control', 'discovery' or 'merovingian' were stopped. monetdbd get status now also reports the version of the running monetdbd Improved response time of 'monetdb start' when the database fails to start. Fixed a bug in monetdbd where starting a failing database could incorrectly be reported as a 'running but dead' database. To avoid confusion, all occurrences of merovingian were changed into monetdbd for error messages sent to a client.
MonetDB Common	Fixed memory detection on 32-bits Solaris systems with more memory than can be addressed in 32-bits mode, bug #2935
Bug Fixes	2740: Segmentation fault using a column inside of a functioncall 2802: update JDBC driver to JDBC4 2846: \d command fails if the schema is provided explicitly 2873: SQL: time with precision 2891: Make MapiPP.pm fetch results progressively 2910: SQLSetStmtOption(..., SQL_CONCURRENCY, SQL_CONCUR_READ_ONLY) should be allowed 2914: mserver5 crash for huge_nr_of_columns.Bug-2609 on ppc-macos 2934: Database upgrade with SQL catalog upgrade messes up privileges 2935: mserver5 doesn't detect memory on Solaris/Sparc 2936: bug into mapi_ping() 2937: minor bug into monetdb_insert_id() (php_monetdb.php) 2938: minor bug into auto_commit() (php_monetdb.php) 2939: "PHP Notice" in mapi_execute() 2940: PHP split() function deprecated 2944: stethoscope on can't connect to database without monetdbd (discovery enabled) 2950: MonetResultSet use of SimpleDateFormat not threadsafe 2952: utf-8 characters in long string 2953: segfault for monetdb get with invalid property 2959: Nested SELECT with EXISTS causes the server to crash 2963: invalid tuple received from server, got 17 columns, expected 24, ignoring 2965: joinPath runs indeterminately 2975: CSV file is kept write-locked after mclient failed to COPY 2981: Different answer from querys in the ODBC compared to mcliente or other DBMSs

The Aug2011 release is the first release where the default size of OIDs on 64-bit WIndows systems is 64 bits.  Before, the default on Windows was 32 bits.  On all other 64-bit platforms, the default size of OIDs was also 64 bits.  This means that, on 64-bit Windows systems, the database needs to be converted.  This is done automatically when the database is first opened, but as always with automatic conversions, it is a good idea to make a backup first.

The Aug2011 uses a slightly different catalog from earlier releases.  The database is automatically updated when first opened.

This release is the last release that comes with a JDBC driver that can be used with Java 1.4 and 1.5, also known as JDBCv3.

 

 

 

Aug 2011-SP3 bugfix release
SQL	Added a fix for bug #2834, which caused weird (failing) behaviour with PreparedStatements.
Merovingian	Fixed a bug where monetdbd's socket files from /tmp were removed when a second monetdbd was attempted to be started using the same port.
Bug Fixes	2875: incomplete database initialization 2912: Function body disappears after loading data - error message: SQLException:sql.resultSet:Cannot access descriptor 2919: Add column and select in same transaction 2920: Updated database from Aug2011 to Dec2011 make queries fail with "Subquery result missing". 2921: complex where clause broken 2923: Multiplication and cast from decimal to integer leads to 0 value 2924: use LIKE with ESCAPE ignores OR operator 2928: Cannot update a field with a count query based on an other table

 

 

Aug 2011-SP2 bugfix release
Client Package	ODBC: Implemented a workaround in SQLTables for bug 2908. Small improvement to mclient's table rendering for tables without any rows. Previously, the column names in the header could be squeezed to very small widths, degrading readability. Python DB API connect() function now supports PEP 249-style arguments user and host, bug #2901 mclient now checks the result of encoding conversions using the iconv library. Fixed a source of crashes in mclient when a query on the command line using the -s option is combined with input on standard input (e.g. in the construct mclient -s 'COPY INTO t FROM STDIN ...' < file.csv). Fixed bug 2897 where slow (network) reads could cause blocks to not be fully read in one go, causing errors in the subsequent use of those blocks. With thanks to Rémy Chibois. Fixed a bug in mclient where processing queries from files could result in ghost empty results to be reported in the output
Merovingian	Resolved problem where monetdbd would terminate abnormally when databases named 'control', 'discovery' or 'merovingian' were stopped. monetdbd get status now also reports the version of the running monetdbd Improved response time of 'monetdb start' when the database fails to start. Fixed a bug in monetdbd where starting a failing database could incorrectly be reported as a 'running but dead' database. To avoid confusion, all occurrences of merovingian were changed into monetdbd for error messages sent to a client.
Bug Fixes	2346: mkey.bulk_rotate_xor_hash MAL exception 2712: un-rolled-back savepoints cause crash upon scenario exit 2774: COPY INTO of double (~1.0e-310) cannot be imported. 2890: Slow execution of COUNT in combination with DISTINCT 2894: No consistent behavior of implicit cast of integer to double 2897: Mapi.pm: short reads not handled when reading blocks from server 2898: order by for multiple (two) columns fails for large tables when second ordering is 'desc': "mat.refine_reverse' undefined" 2900: rand(int) should not return NULL 2901: Make keyword arguments in Python DB API connect() function more standard 2904: TypeException after calling UDF in other than sys schema (when it is initialised) 2906: "ALTER TABLE mytable SET READ ONLY" results in server segfault 2908: SQL query produces error with MAL statements 2909: query result is not displayed when history = true

 

Aug 2011-SP1 bugfix release
SQL	Fixed a bug when using default values for interval columns. Bug 2877. A bug was fixed where deleted rows weren't properly accounted for in all operations. This was bug 2882. A bug was fixed which caused an update to an internal table to happen too soon. The bug could be observed on a multicore system with a query INSERT INTO t (SELECT * FROM t) when the table t is "large enough". This was bug 2883.
Client Package	Fixed Perl DBD rowcount for larger results, bug #2889 Perl: We now distinguish properly between TABLE and GLOBAL TEMPORARY (the latter are recognized by being in the "tmp" schema). Perl: fixed a bunch of syntax errors. This fixes bug 2884. With thanks to Rémy Chibois. Perl: Fixed DBD::monetdb table_info and tabletype_info. This fixes bug 2885. With thanks to Rémy Chibois. mclient: fix display of varchar columns with only NULL values. Fixed a bug in mclient/msqldump where an internal error occurred during dump when there are GLOBAL TEMPORARY tables.
MonetDB5 Server	Fixed a problem where MAL variables weren't properly cleared before reuse of the data strucutre. This problem could cause the data flow scheduler to generate dependencies between instructions that didn't actually exist, which in turn could cause circular dependencies among instructions with deadlock as a result. Bugs 2865 and 2888.
Bug Fixes	2865: mserver5 --set gdk_nr_threads=2 --forcemito: deadlock during first SQL client connect on virgin (empty) DB 2877: assertion failed when creating table with default interval 2879: A SELECT query fails to produce some results 2882: Delete of some records don't work on Monetdb 2883: insert into t (select * from t) bogus in multithreaded execution 2884: Mapi.pm and GetInfo.pm syntax fixes for array usage 2885: DBD::monetdb table_info error 2887: Division by zero error, when should not divide at 0 2888: test/null/null_in_case.sql hangs server 2889: Mapi::getBlock is erasing query count on second call

 

 

Aug 2011 feature release
Build Environment	Fixed Fedora 15 (and presumably later) configuration that uses a tmpfs file system for /var/run. This fixes bug 2850. The default OID size for 64-bit Windows is now 64 bits. Databases with 32 bit OIDs are converted automatically.
Java Module	Add so_timeout Driver property to specify a SO_TIMEOUT value for the socket in use to the database. Setting this property to a value in milliseconds defines the timeout for read calls, which may 'unlock' the driver if the server hangs, bug #2828 Added a naive implementation for PreparedStatement.setCharacterStream The obsolete Java-based implementation for PreparedStatements (formerly activated using the java_prepared_statements property) has been dropped
Client Package	msqldump now also accepts the database name as last argument on the command line (i.e. without -d). Made error messages from the server in mclient go to stderr, instead of stdout. Implemented SQL_ATTR_METADATA_ID attribute. The attribute is used in SQLColumns, SQLProcedures, and SQLTablePrivileges. Implemented SQLTablePrivileges in the ODBC driver. mclient now automatically sets the SQL `TIME ZONE' variable to its (the client's) time zone. Removed perl/Cimpl, MonetDB-CLI-MapiLib and MonetDB-CLI-MapiXS Switched implementation of MonetDB::CLI::MapiPP to Mapi.pm, and made it the default MonetDB::CLI provider. Made Mapi.pm usable with current versions of MonetDB again When the first non-option argument of mclient does not refer to an exising file, it now is taken as database name. This allows to simply do `mclient mydb`.
MonetDB5 Server	Make crackers optional and disable by default, since it wasn't used normally Authorisation no longer takes scenarios into account. Access for only sql or mal is no longer possible. Any credentials now mean access to all scenarios that the server has available.
Geom Module	Implemented NULL checks in the geom module. Now when given NULL as input, the module functions return NULL instead of an exception. This fixes bug 2814.
Merovingian	Fixed confusing 'Success' error message for monetdb commands where an invalid hostname was given The path to the mserver5 binary is no longer returned for the mserver property with monetdbd get for a dbfarm which is currently served by a monetdbd. Since the called monetdbd needs not to be the same as the running monetdbd, the reported mserver5 binary may be incorrect, and obviously lead to confusing situations. Refer to the running monetdbd's logfile to determine the mserver5 binary location instead. Added -n option to monetdbd start command, which prevents monetdbd from forking into the background. Fix incorrect (misleading) path for pidfile in pidfile error message, bug #2851 added status property to get command
MonetDB Common	Removed conversion code for databases that still used the (more than two year) old format of "string heaps". On Windows and Linux/Unix we can now read databases built on the other O/S, as long as the hardware-related architecture (bit size, floating point format, etc.) is identical. Implemented automatic conversion of a 64-bit database with 32-bit OIDs to one with 64-bit OIDs.
Bug Fixes	2336: ANSI Client clobbered data ::SQLTables 2579: SQL: incorrect cast from double to int 2701: Divide by zero in OPTmitosis with select name from tables 2814: Throws exception if a SQL/GIS takes argument null 2817: Failed to find a table producing function 2828: No Socket Timeout: MonetDB Java client hangs forever on reads and consume threads when the database is crashed 2836: compilation fails when --with-valgrind in effect 2838: Nested case results in message: "Predicate CASE 42: time to implement some more" 2839: test monetdb5/optimizer/Tests/inlineFunction2 fails 2840: Test monetdb5/optimizer/Tests/reorder01 fails 2841: Tests monetdb5/optimizer/Tests/inline0{2,3,6} fail 2842: Tests monetdb5/optimizer/Tests/dataflow{,3,4} lack stable output 2845: INSERT INTO fails (hangs) when values are derived from SELECT, in combination with a FOREIGN KEY constraint 2847: SELECT statement throws str_cast exception 2848: Problem when reusing column name as CASE... WHEN alias in subquery 2849: Multi-column constraints cause data corruption 2850: monetdbd cannot open pidfile on Fedora 15 2851: Incorrect error message when pidfile cannot be opened 2852: Query with CASE and nested functions results in very long execution 2855: test monetdb5/modules/mal/Tests/inspect00 fails 2856: test monetdb5/optimizer/Tests/dataflow fails 2857: obsolete conditional HAVE_LIBXML2 still used on Windows 2859: prepare statement tests fail with number of arguments mismatch 2860: /etc/init.d/monetdb5-sql uses incorrect PIDFILE to test if monetdb is running 2862: foreign key self references cause violation upon alter table add constraint... 2866: test monetdb5/optimizer/Tests/dataflow.mal fails on all Windows & Solaris (only) 2868: tests tst285 & pithole fail on some Windows platforms 2869: SQL tests with "LIKE" crash on Windows2008 & WindowsXP while working fine on Windows7

 

This is the first release of the MonetDB suite that does not contain MonetDB4 and MonetDB/XQuery. As already mentioned in the Mar2011 release notes, work on the Pathfinder compiler, which forms the basis of MonetDB/XQuery, continues at The University of Tübingen. An upgrade from the Mar2011 release should be smooth, even if MonetDB/XQuery is installed. The Windows installers for MonetDB/SQL and MonetDB/XQuery have always been independent, and the Fedora, Debian, and Ubuntu versions have been made independent in the Mar2011 release. The Apr2011 release uses a slightly changed SQL catalog than previous releases. The catalog will be updated automatically the first time an SQL database is opened. Note that because of the incompatibility, once updated, the database cannot be read by older versions. Also note that a database from the Feb2010 release or before cannot be read by this release. First update the database with the Mar2011 release. This is also the first release where we have created FreeBSD binaries. These binaries are highly experimental. The software should run as well as on any other system, but the packaging is obviously different from other systems. As before, we also provide binaries for Windows, Fedora (13 and 14), Debian (5.0 and 6.0), Ubuntu (9.10, 10.04, and 10.10), Mac OS X, and Solaris.     Apr 2011-SP2 bugfix release Build Environment We can now build RPMs on CentOS 6.0. Since there is no geos library on CentOS, we do not support the geom modules there. We can now also build on Fedora 15. This required some very minor changes. Changed configure check for OpenSSL so that we can also build on CentOS 5.6. We now no longer demand that OpenSSL is at least version 0.9.8f, but instead we require that the hash functions we need are supported. Client Package Fix stethoscope's mod.fcn filter when using multiple targets, bug #2827 The separate Python distribution now uses the same version number as the main package. MonetDB Common Fixed a problem where appending string BATs could cause enormous growth of the string heap. This fixes bug 2820. Fixes to memory detection on FreeBSD. Java Module Return false from Statement.getMoreResults() instead of a NullPointerException when no query has been performed on the Statement yet, bug #2833 Fixed read-only interpretation. Connection.isReadOnly now always returns false, setReadOnly now generates a warning when called with true. Partly from bug #2818 Allow readonly to be set when autocommit is disabled as well. Bug #2818 SQL Fixed regression where the superuser password could no longer be changed, bug #2844 Fixed incorrect insert counts. Fixed bug 2823: MAL exeption on SQL query with subquery in the where part. Redirect error from create scripts back to the first client. This fixes bug 2813. Added joinidx based semijoin; push join through union (using joinidx). Fixed pushing select down. Bug Fixes 2058: scoping issue of functions and with 2734: MALException:mkey.bulk_rotate_xor_hash on complex SQL query 2767: `make uninstall` fails in clients/rubyadapter 2797: mserver crashes at initialization when the disk is full 2811: mserver crashes 2812: Subqueries in FROM clause not accepted 2813: SQL: catalog is built even error has occurred 2818: JDBC: exception on calling Connection.setReadOnly 2819: make text readable 2820: SIGSEGV in BATins_kdiff triggered by a DELETE statement 2821: segmentation fault when a select query (with group by) is run on server with "--single-user" option 2822: A query with CASE and division results in ArithmeticException 2823: MAL exeption on SQL query with subquery in the where part 2825: "Insert into ... select distinct ..." fails with MALException:mkey.bulk_rotate_xor_hash:operation 2826: sql function iterates instead of bulk processing 2827: Incorrect filter setting when listening to several servers 2829: logging issue 2830: select 30.0 - 0.025; conversion to decimal type gives error 2831: Crash in query analysis 2832: Server terminates upon UPDATE 2833: NullPointerException when calling getMoreResults() on Statement without a query executed 2835: UNIQUE Constraint is not enforced 2837: Server terminates after CREATE FUNCTION in multi-user scenario 2844: ALTER USER SET PASSWORD doesn't recognise username   Apr 2011-SP1 bugfix release MonetDB Common Fixed a bug where large files (> 2GB) didn't always get deleted on Windows. Java Module Insertion via PreparedStatement and retrieval via ResultSet of timestamp and time fields with and without timezones was improved to better respect timezones, as partly indicated in bug #2781. MonetDB5 Server Fixed a bug in conversion from string to the URL type.  The bug was an incorrect call to free(). Bug Fixes 2784: configure does not (properly) check for yacc/bison 2798: Query did not produce a result set 2806: Wrong logic in predicates over views defined with an open-interval predicate 2807: wrong results for group by queries in the presence of primary keys 2808: wrong results for count()/count(distinct) on the same attribute 2810: Copy into should rollback correctly (i.e. restore the files) on a failure   Apr 2011 feature release
Client Package	Plugged a small memory leak occurring upon redirects by the server (e.g. via monetdbd) Remove XQuery related code from Ruby adapter, PHP driver and Perl Mapi library Make SQL the default language for mclient, e.g. to use when --language= or -l is omitted
MonetDB Common	Fixed bugs in antiselect which gave the incorrect result when upper and lower bound were equal. This bug could be triggered by the SQL query SELECT * FROM t WHERE x NOT BETWEEN y AND y
Java Module	Clarify exception messages for unsupported methods Removed XQuery related XRPC wrapper and XML:DB code, removed support for language=xquery and language=mil from JDBC.
MonetDB5 Server	Mserver5 no longer reads monetdb5.conf upon startup by default. Use --config=file to have mserver5 read a configuration on startup
SQL	Some names in the SQL catalog were changed. This means that the database in the Apr2011 release is not compatible with pre-Apr2011 databases. The database is converted automatically when opened the first time. This database can then no longer be read by an older release.
Merovingian	Fix monetdb return code upon failure to start/stop a database. The forward property for databases has been removed. Instead, only a global proxy or redirect mode can be set using monetdbd. Monetdbd can no longer log error and normal messages to separate logfiles, logging to stdout and stderr is no longer possible either. The .merovingian_pass file is no longer in use, and replaced by the .merovingian_properties file. Use monetdbd (get|set) passphrase to view/edit the control passphrase. Existing .merovingian_pass files will automatically be migrated upon startup of monetdbd. Monetdbd now understands commands that allow to create, start, stop, get and set properties on a given dbfarm. This behaviour is intended as primary way to start a MonetDB Database Server, on a given location of choice. monetdbd get and set are the replacement of editing the monetdb5.conf file (which is no longer in use as of the Apr2011 release). See monetdbd(1).
Bug Fixes	1956: Geom: relation and spatial tests fail 2547: create schema leads to sql_stmt symbol(x)->token = CREATE_SCHEMA 2588: test sql/src/test/BugTracker/Tests/authorization.SF-1430616.SQL.sh shows different error message 2590: test sql/src/test/BugTracker/Tests/mdb_starts_with_sql_debug_64.SF-1999354.py fails with "too many digits" 2639: Readline auto-complete reads from 'tables' instead of 'sys.tables' 2766: SQL: Compute modulo of rand() in an UPDATE stmt crashes mserver5 2779: Make building GDK optional 2793: EXTRACT(SECOND FROM timestamp) always returns 0 integer seconds part. 2795: SQL: ALTER TABLE with incorrect syntax crashes mserver5 2796: latest antiselect changes cause ~20 SQL tests to fail 2800: Test_PStypes fails due to ambiguous identifier 2801: Nested SELECT expression with JOIN causes outer WHERE to be disregarded sometimes

 

 

 

MonetDB Mar2011 release

This is the last planned release for MonetDB4 and MonetDB/XQuery. MonetDB4 has served us well, but we need to move on. MonetDB/XQuery, being implemented on top of MonetDB4, is a casualty of this decision. The work on the Pathfinder compiler, which forms the basis of MonetDB/XQuery, continues at The University of Tübingen.

As of this "Mar2011" release (version 11.1.1), all components of MonetDB have been gathered into one single package, mainly to simplify the compilation process. If you're using the Mercurial clone, use the Mar2011 branch and the Mar2011_1 tag.

Contents

• Mar 2011 feature release
  • Packaging
  • MonetDB Common
  • MonetDB Server 4
  • MonetDB Server 5
  • SQL
  • XQuery
  • Merovingian
  • Client Packages
  • Geom Module
  • Java Module
  • Bug fixes

Mar 2011 feature release

Packaging

The packaging of MonetDB has been completely overhauled. In the past, there were various subpackages that had to be configured and built individually, but that had to be done in the correct order. From this release onward, the process of building MonetDB has been greatly simplified. There is only a single configuration/build/install cycle for the whole suite. On Linux/Unix systems this means one call to configure (when building from the Mercurial sources this has to be preceded by one call to the bootstrap script), one call to make, and one call to make install. On Windows, the bootstrap and configure stages can be skipped (and you need to use nmake instead of make).

The directory structure of the source tree was also completely overhauled. The MonetDB4 and MonetDB5 directories were renamed monetdb4 and monetdb5 respectively, the MonetDB directory was renamed gdk, src subdirectories were removed, and several subsystems were moved to the new common directory.

The version numbers have changed since we now have a single package. The version number of this release is 11.1.1.

Linux

We now package for Fedora 13 and 14, for Debian 5.0 (Lenny) and 6.0 (Squeeze), and for Ubuntu 9.10 (Karmic Koala), 10.04 (Lucid Lynx), and 10.10 (Maverick Meerkat).

On Debian and Ubuntu, updating to this release is a bit more involved than simply using apt-get upgrade or the "Mark All Upgrades" button in the Synaptic Package Manager. When using these commands, they will "keep back" the monetdb packages, so in order to actually update them you will need to give the apt-get install command and specify the relevant monetdb packages, or in Synaptic first select the relevant monetdb packages and then select Package->Mark for Upgrade.

Windows

Binaries are available.

Mac OS X

Binaries are available.

Solaris

Binaries are available.

MonetDB Common

• Fixed memory detection on Darwin (Mac OS X) systems not to return bogus values
• MonetDB/src/gdk was moved to gdk

MonetDB Server 4

MonetDB Server 5

• sphinx module: update, adding limit/max_results support
• Improve the performance of remote.put for BAT arguments. The put speed is now roughly equal to the speed of get on a BAT.

SQL

• Fix bug #2648, do not allow restarting a sequence with NULL via the result of a sub-query.
• Allow clients to release prepared handles using Xrelease commands

XQuery

Client Packages

• Fixed bug 2677: SQL_DESC_OCTET_LENGTH should give the size in bytes required to copy the data.
• Disable timer functionality for non-XQuery languages since it is incorrect, bug #2705
• Added mapi_get_uri function to retrieve mapi URI for the connection
• Allow to dump table data using INSERT INTO statements, rather than COPY INTO + CSV data using the -N/--inserts flag of mclient and msqldump. Bug #2727
• Added support for \dn to list schemas or describe a specific one
• Added support for \df to list functions or describe a specific one
• Added support for \ds to list sequences or describe a specific one
• Added support for wildcards * and ? in object names given to \d commands, such that pattern matching is possible, e.g. \d my*
• Added support for \dS that lists also system tables
• object names given to \d are now lowercased, unless quoted by either single or double quotes
• Strip any trailing whitespace with the \d command

Geom Module

• Set endianness for wkb en/decoding.

Java Module

• PreparedStatements now free the server-side resources attached to them when closed. This implements bug #2720

Bug fixes

The following bug tracker items were fixed since the previous "Oct 2010" release:

• 104: SQL: extremely long and complex query causes SEGFAULT
• 2351: SQL: transaction aborts on prepared statements in autocommit
• 2358: Unrelevant error message when copying bad length field
• 2586: times.sql: procedure times() crashes the server
• 2593: test sql/src/test/Tests/load_dec_as_int.sql.src fails a file sql/src/test/load_dec_as_int.sql is missing
• 2609: insert into large number of columns
• 2634: !ERROR: CTrefine: both BATs must have the same cardinality ...
• 2640: VOC test fails multi-threaded
• 2647: M5 breaks with huge SQL query.
• 2648: ALTER SEQUENCE ... RESTART WITH (SELECT NULL) NO CYCLE is accepted
• 2673: select a string column from a view over a table with primary key crashes
• 2677: SQLColAttribute using SQL_DESC_OCTET_LENGTH doesn't return the maximum length
• 2679: timestamp selection based on now() takes too long
• 2680: make clean too eager
• 2698: test sql/src/test/bugs/Tests/except-union-intersect-bug-sf-1146079.SQL.sh produces wrong result with multi-threaded execution
• 2704: temporary tables on readonly mode
• 2705: Timer information problems
• 2708: function with different value for arguments crashes mserver
• 2710: implement \d{t,v,s,S,...} to describe other (or only) objects in the database
• 2713: wrong error message when dropping an un-existing view with the same as an existent table.
• 2715: Join with 2 AND-ed conditions where one is LIKE/ILIKE returns too many results
• 2717: function call in readonly mode.
• 2720: Allow a client to close a prepared handle
• 2725: mclient/mapilib aborts during connect
• 2726: join on two literal tables leads to
• 2727: allow dumping database with INSERT INTO statements
• 2729: mserver5: ../../../src/server/rel_bin.mx:537: exp_bin: Assertion `0' failed.
• 2730: select * from sequence crashes mserver5
• 2733: MAL parser accepts 1 + 1 + 1, MAL interpreter only computes 1 + 1
• 2735: complex SQL query leads to multiple TypeExceptions
• 2737: Sphinx module update, adding limit/max_results support
• 2738: MonetDB 'allows' binding and connecting to ports above 65535 (has patch)
• 2743: mapi_next_result skips over results (losing output)
• 2744: UNLOCKED causes mserver to work indefinitely
• 2745: foreign key type mismatch causes MAL stacktrace
• 2748: COPY INTO does not work correctly with backslashes in VARCHARs
• 2750: Function arguments of type string described not completely in SQL catalog
• 2752: CALL function without warning
• 2753: check existence of external names for SQL functions/procedures at creation time
• 2757: ORDER BY DESC + LIMIT incorrectly yields empty result
• 2768: "no such column 'u.name'" in users system view
• 2773: starting database 'test'... FAILED
• 2786: crash when first connection is with another user than monetdb
• 2788: stop -a complains about already stopped databases
• 2790: View returns erroneous result for squared average function as compared to query

The detailed list of changes can be found in the source code Mercurial logs.

MonetDB Jun2010 release

Release "Jun2010" consists of the following updated components. If you're using the Mercurial clone, use the Jun2010 branch and the Jun2010_1 tag.

 

Name	Version	Subdirectory	Note/Comment
MonetDB Common	1.38.1	MonetDB
MonetDB Clients	1.38.1	clients
MonetDB4 Server	4.38.1	MonetDB4
MonetDB4/XQuery	0.38.1	pathfinder
MonetDB5 Server	5.20.1	MonetDB5
MonetDB5/SQL	2.38.1	sql
MonetDB5/SQL/GIS	0.18.1	geom
MonetDB Java	1.38.1	java
MonetDB Testing	1.38.1	testing	(developers only)

NOTE:
Before upgrading to the latest "Jun2010" release from a release of MonetDB older than the Nov2008 release, please make a dump (SQL, XQuery) of your database, and rename/move/backup the database directories (dbfarm, sql_logs, xquery_logs). After the upgrade, restore (SQL, XQuery) your database, again. Once you have checked and confirmed the correctness of the restore, you can delete the above renamed/moved/backed-up old database directories.

When upgrading from a release of MonetDB older than the May2009 release, you need to move your write-ahead log following the "recommended" instructions for SQL and/or XQuery.

Contents

• Jun 2010 feature release
  • Packaging
  • MonetDB Common
  • MonetDB Server 4
  • MonetDB Server 5
  • SQL
  • XQuery
  • Client Packages
  • Geom Module

Jun 2010 feature release

Packaging

 

Linux

We now package for Fedora 11, 12, and 13, for Debian 5.0 (Lenny), and for Ubuntu 9.10 (Karmic Koala) and 10.04 (Lucid Lynx).

MonetDB Common

 

• Updated Vendor information.
• Made compilation of "testing" (and "java") independent of MonetDB. This is mainly for Windows, but also on other systems, "testing" can now be built independently of (and hence before) "MonetDB". Files that mimic configure functionality on Windows were moved from "MonetDB" to "buildtools"; hence, this affects all packages on Windows, requiring a complete rebuild from scratch on Windows. getopt() support in testing has changed; hence, (most probably) requiring a rebuild from scratch of testing on other systems.
• Implemented build directory support for Windows, i.e., like on Unix/Linux also on Windows we can now build in a separate build directory as alternative to ...<package>NT, and thus keep the latter clean from files generated during the build. On Windows, the build directory must be a sibling of ...<package>NT .

MonetDB Server 4

 

• Updated Vendor information.
• Made compilation of "testing" (and "java") independent of MonetDB. This is mainly for Windows, but also on other systems, "testing" can now be built independently of (and hence before) "MonetDB". Files that mimic configure functionality on Windows were moved from "MonetDB" to "buildtools"; hence, this affects all packages on Windows, requiring a complete rebuild from scratch on Windows. getopt() support in testing has changed; hence, (most probably) requiring a rebuild from scratch of testing on other systems.
• Implemented build directory support for Windows, i.e., like on Unix/Linux also on Windows we can now build in a separate build directory as alternative to ...<package>NT, and thus keep the latter clean from files generated during the build. On Windows, the build directory must be a sibling of ...<package>NT .

MonetDB Server 5

 

• When libxml2 is available, the XML module is automatically loaded.
• Updated Vendor information.
• Created a new RPM MonetDB5-server-rdf for the optional MonetDB/RDF module.
• Added the Z-order module to simplify future manipulation of arrays.
• fixed BUG #2994521 "mat.slice unable to cope with only empty BAT arguments" by making MATpackSliceInternal() handle empty input BATs correctly
• Select <col> from <t> limit <n> has been improved by introducing mat.slice().
• Made compilation of "testing" (and "java") independent of MonetDB. This is mainly for Windows, but also on other systems, "testing" can now be built independently of (and hence before) "MonetDB". Files that mimic configure functionality on Windows were moved from "MonetDB" to "buildtools"; hence, this affects all packages on Windows, requiring a complete rebuild from scratch on Windows. getopt() support in testing has changed; hence, (most probably) requiring a rebuild from scratch of testing on other systems.
• Implemented build directory support for Windows, i.e., like on Unix/Linux also on Windows we can now build in a separate build directory as alternative to ...<package>NT, and thus keep the latter clean from files generated during the build. On Windows, the build directory must be a sibling of ...<package>NT .
• The MAL debugger list command has been extended with an optional hash '#', which produces line numbers for each of reference and analysis of variable span.
• The dataflow scheduler has been revamped to allow for more parallelism to be exploited.
• The garbage collection administration has been changed. Every variable record now comes with an end-of-life field (eolife), which denotes the instruction after which the BAT variable reference counter can be decremented. The garbage collector is never called automagically on MAL functions, because there may be other pressing needs to retain them. This means that MAL functions defined and used in the context of SQL, and which are not inlined, may cause a leakage. The garbage collection has become part of each interpreter step. The new approach makes the SQL/MAL plans half the size as before.
• Removed stethoscope from MonetDB5 sources. New location is in the clients repository.
• Added the compression optimizer as an example of how to gain access to foreign file formats deep down in the kernel and transfer them just in time into a temporary BAT.
• Renamed configure argument --with-console to --enable-console. Default remains console being enabled.
• Fixed cleaning the user module context upon session end. Fixes bug #2956664
• The config variable mapi_usock can now be used to instruct the server to listen for connections on a local UNIX domain socket on UNIX-like systems.
• The MAL interpreter has been extended with an operation admission policy to control the memory claims of all concurrent running interpreters. Instructions are hold up unto there is sufficient resource or the query plan can not avoid its execution anymore.
• The join path optimizer has been extended with searching for join*, semijoin*, and leftjoin* paths. Furthermore, it avoids duplicate work by factoring out all common simple join paths.

SQL

 

• make it possible to use '?' in offset and limit
• Added include and .lib files to Windows installers that are needed to compile client programs.
• Updated Vendor information.
• Made compilation of "testing" (and "java") independent of MonetDB. This is mainly for Windows, but also on other systems, "testing" can now be built independently of (and hence before) "MonetDB". Files that mimic configure functionality on Windows were moved from "MonetDB" to "buildtools"; hence, this affects all packages on Windows, requiring a complete rebuild from scratch on Windows. getopt() support in testing has changed; hence, (most probably) requiring a rebuild from scratch of testing on other systems.
• Implemented build directory support for Windows, i.e., like on Unix/Linux also on Windows we can now build in a separate build directory as alternative to ...<package>NT, and thus keep the latter clean from files generated during the build. On Windows, the build directory must be a sibling of ...<package>NT .

XQuery

 

• Updated Vendor information.
• Made compilation of "testing" (and "java") independent of MonetDB. This is mainly for Windows, but also on other systems, "testing" can now be built independently of (and hence before) "MonetDB". Files that mimic configure functionality on Windows were moved from "MonetDB" to "buildtools"; hence, this affects all packages on Windows, requiring a complete rebuild from scratch on Windows. getopt() support in testing has changed; hence, (most probably) requiring a rebuild from scratch of testing on other systems.
• Implemented build directory support for Windows, i.e., like on Unix/Linux also on Windows we can now build in a separate build directory as alternative to ...<package>NT, and thus keep the latter clean from files generated during the build. On Windows, the build directory must be a sibling of ...<package>NT .

Client Packages

 

• Updated Vendor information.
• The MonetDB ODBC driver now no longer depends on its own copy of the standard ODBC include files but instead depends on system include files (unixODBC on Linux and Microsoft SDK on Windows).
• Made compilation of "testing" (and "java") independent of MonetDB. This is mainly for Windows, but also on other systems, "testing" can now be built independently of (and hence before) "MonetDB". Files that mimic configure functionality on Windows were moved from "MonetDB" to "buildtools"; hence, this affects all packages on Windows, requiring a complete rebuild from scratch on Windows. getopt() support in testing has changed; hence, (most probably) requiring a rebuild from scratch of testing on other systems.
• Implemented build directory support for Windows, i.e., like on Unix/Linux also on Windows we can now build in a separate build directory as alternative to ...<package>NT, and thus keep the latter clean from files generated during the build. On Windows, the build directory must be a sibling of ...<package>NT .
• Changed d output of mclient no longer to list (internal use only) system tables. Administrator users who like to inspect these tables can use SELECT * FROM tables; instead
• Implemented the changes required for properly running ODBC on a 64-bit platform.
• Added stethoscope, an utility to profile MonetDB5 instances, originally from the MonetDB5 repository.

Geom Module

 

• Updated Vendor information.
• Made compilation of "testing" (and "java") independent of MonetDB. This is mainly for Windows, but also on other systems, "testing" can now be built independently of (and hence before) "MonetDB". Files that mimic configure functionality on Windows were moved from "MonetDB" to "buildtools"; hence, this affects all packages on Windows, requiring a complete rebuild from scratch on Windows. getopt() support in testing has changed; hence, (most probably) requiring a rebuild from scratch of testing on other systems.
• Implemented build directory support for Windows, i.e., like on Unix/Linux also on Windows we can now build in a separate build directory as alternative to ...<package>NT, and thus keep the latter clean from files generated during the build. On Windows, the build directory must be a sibling of ...<package>NT .

Release "Feb2010" consists of the following updated components

Name	Version	CVS module	CVS branch	Note/Comment
MonetDB Common	1.36.1	MonetDB	Feb2010
MonetDB Clients	1.36.1	clients	Feb2010
MonetDB4 Server	4.36.1	MonetDB4	Feb2010
MonetDB4/XQuery	0.36.1	pathfinder	Feb2010
MonetDB5 Server	5.18.1	MonetDB5	Feb2010
MonetDB5/SQL	2.36.1	sql	Feb2010
MonetDB5/SQL/GIS	0.16.1	geom	Feb2010
MonetDB Java	1.36.1	java	Feb2010
MonetDB Testing	1.36.1	testing	Feb2010	(developers only)

NOTE:
Before upgrading to the latest "Feb2010" release from a release of MonetDB older than the Nov2008 release, please make a dump (SQL, XQuery) of your database, and rename/move/backup the database directories (dbfarm, sql_logs, xquery_logs). After the upgrade, restore (SQL, XQuery) your database, again. Once you have checked and confirmed the correctness of the restore, you can delete the above renamed/moved/backed-up old database directories.

When upgrading from a release of MonetDB older than the May2009 release, you need to move your write-ahead log following the "recommended" instructions for SQL and/or XQuery.

Contents

• Feb 2010 feature release
  • Packaging
  • MonetDB Common
  • MonetDB Server 5
  • SQL
  • XQuery
  • Client Packages
  • Bug fixes

Feb 2010 feature release

Packaging

We now package for Fedora 11 and 12.

MonetDB Common

• Various concurrency bugs were fixed.
• Various changes were made to run better on systems that don't have enough memory to keep everything in core that was touched during query processing. This is done by having the higher layers giving hints to the database kernel about future use, and the database kernel giving hings to the operating system kernel about how (virtual) memory is going to be used.
• Fixed bug in mergejoin implementation. This fixes bug #2952191.
• Added support for compiling on Windows using the Cygwin-provided version of flex.
• Fix compilation issue when configured with --with-curl. This fixes bug #2924999.
• Added implementation of MT_getrss() for Solaris. This yields in the kernel knowing about its (approximate) memory usage to try and help the operating system to free that memory that is best to free, instead of a random page, e.g. the work of the vmtrim thread.
• Implemented a "fast" string BAT append: Under certain conditions, instead of inserting values one-by-one, we now concatenate the string heap wholesale and just manipulate the offsets. This works both for BATins and BATappend.
• Changed the string heap implementation to also contain the hashes of strings.
• Changed the implementation of the string offset columns to be dynamically sized.

MonetDB Server 5

• The MonetDB 5 code base underwent a series of small/medium changes:
  • Solving name conflicts with external libraries in mal_box.
  • Dependency graph generation in MAL debugger fixed.
  • Code hardening against out of memory errors and based on Coverity checks.
  • Recycler improved to deal with SQL plans from different sessions.
  • Profiler extended to report the argument types, user, and thread id.
  • MAL interpreter reports an event before and after the instruction.
  • Dataflow also allowed for updates on temporary BATs.
  • Reorder optimizer better respects the dataflow.
  • All update instructions return their target to mark the dataflow.
  • Clean up of (bat)calc module.
  • Packing pieces together simplified and sped up.
• The MAL interpreter has been extended with an operator admission scheme, which is active during parallel execution. It blocks threads if the total amount of memory needed for the operator can not be claimed. Only if there is one operation left to execute, it won't block. The admission level is controlled by a threshold, which is set to 90% of the physical memory.
• The ilike[u]select operations has been included in the repertoire recognized by the mergetable optimizer for push through of selections.
• The joinpath optimizer has been extended with recognition of the leftjoin. This way, series of leftjoin operations can be optimized by looking at the smallest starting point.
• Introduced a dictionary encoding option to the optimizer pipeline. Encoding is initiated with a SQL call compress(tablename).
• Removed the ability to redirect to other running databases in the same dbfarm. This functionality has been taken over completely by merovingian, and only results in confusement these days, bug #2891191.
• Added --enable-console configure argument, defaulting to 'yes' for now. Disabling the server console increases security by avoiding local access exploits. This is not the default since our Testing setup cannot deal with a console-less server yet.
• Report detected amount of main memory and cpu cores in output of `mserver5 --version`.

SQL

Nothing to report.

XQuery

Nothing to report

Client Packages

• Fixed a bug in the Mapi libary when sending very large queries to the server.
• Implemented BLOB and CLOB support in ODBC driver.
• Fixed crash upon connect to a server that is under maintenance but has the requested language (scenario) not loaded.
• Fixed bug in the ODBC driver where a non-ASCII character at the end of a string caused an error when the string was converted to wide characters.
• mclient timer information (-t option) is now written to standard error instead of standard output. This makes getting timer information easier when there is a large amout of regular output (which can be redirected to a file or /dev/null).
• Implemented new function mapi_fetch_field_len() which returns the length (excluding trailing NULL byte) of the field returned by mapi_fetch_field().

Bug fixes

The following bug tracker items were fixed since the previous "Nov 2009" release: The detailed list of changes can be found in the source code CVS logs.

MonetDB Aug2009-SP2 release

Release "Aug2009-SP2" consists of the following updated components

Name	Version	CVS module	CVS branch	Note/Comment
MonetDB Common	1.32.4	MonetDB	Aug2009
MonetDB Clients	1.32.4	clients	Aug2009
MonetDB4 Server	4.32.4	MonetDB4	Aug2009
MonetDB4/XQuery	0.32.4	pathfinder	Aug2009
MonetDB5 Server	5.14.4	MonetDB5	Aug2009
MonetDB5/SQL	2.32.4	sql	Aug2009
MonetDB5/SQL/GIS	0.12.4	geom	Aug2009
MonetDB Java	1.32.4	java	Aug2009
MonetDB Testing	1.32.4	testing	Aug2009	(developers only)

NOTE:
Before upgrading to the latest "Aug2009-SP2" release from a release of MonetDB older than the Nov2008 release, please make a dump (SQL, XQuery) of your database, and rename/move/backup the database directories (dbfarm, sql_logs, xquery_logs). After the upgrade, restore (SQL, XQuery) your database, again. Once you have checked and confirmed the correctness of the restore, you can delete the above renamed/moved/backed-up old database directories.

When upgrading from a release of MonetDB older than the May2009 release, you need to move your write-ahead log following the "recommended" instructions for SQL and/or XQuery.

Contents

• Aug 2009 SP2 bug-fix release
• Aug 2009 SP1 bug-fix release
• Aug 2009 feature release
  • Packaging
  • MonetDB Common
  •  
  • MonetDB Server 5
  • SQL
  • XQuery
  • Client Packages

Aug 2009 SP2 bug-fix release

This is the second bug fix release of the Aug2009 version. This version can be installed "on top of" the Aug2009 release. The database format is fully compatible, so there is no need to do a dump and restore. (Although it is *always* a good idea to make backups.)

Some of the bugs that were fixed:

• Changes in the memory mapping code to less zealously allocate space.
• Changes to the use of the way the operating system is informed about how memory is used, resulting in a better fit between the predictions the operating system makes and the reality of what MonetDB actually does.
• Fix to exit code.
• Performance fix releated to anonymous virtual memory.
• Performance fix for when a single item is joined to a table. (MonetDB4/XQuery)
• When extracting fails, don't leave a partial output file (Mx, SF bug 2840225).
• Added support for GCC version 4.4.* by disabling the strict-aiasing optimization in the compiler.
• Various improvements to the Ruby interface.
• Fix warning messages to disappear when a connection to merovingian is made.
• Fix physical plan for merge-adjacent-text-nodes. (MonetDB/XQuery)
• For some plans with aggregates, lax checking resulted in incorrect rewrites. (MonetDB/XQuery)
• Fix bug in order lists merging. In case the replaced column is sorted in descending order we need to reverse the order of all replacement order criteria. (MonetDB/XQuery)
• Fixes so that larger plans can be handled. (MonetDB/XQuery, SF bug 2860037)
• Fixes so that less stack space is used during processing, and so that running out of stack space is recognized in more places. (MonetDB/SQL)
• Fixed bug in IN handling. (MonetDB/SQL, SF bug 2869818)
• Fixed crash when attempting to create a function in read-only mode by disallowing this. (MonetDB/SQL, SF bug 2872421)
• Added partial support for compiling with bison++. Bison++ is not supported and can't be used out of the box, but only minor tweaks are needed to use it. (SF bug 2872565)
• Fix warning messages to disappear when a connection to merovingian is made. (JDBC driver)

The following bug tracker items were fixed since the Aug2009-SP1 release:

• [ 2812963 ] JDBC connection hangs if result is too long
• [ 2839063 ] M5: property error in src/modules/mal/Tests/chopper03.mal
• [ 2840225 ] Mx creates output files even on errors
• [ 2847738 ] SuperBall build fails with PHP 5.3.0
• [ 2860037 ] PF: fatal error: aborted too deep recursion on linux
• [ 2860574 ] PF: fatal error: column not found / BATappend access denied
• [ 2865631 ] SQL: incref assertion
• [ 2869818 ] SQL: Program contains errors
• [ 2871426 ] SQL: mclient-lsql-D fails everywhere but on 64-bit Fedora
• [ 2872421 ] SQL: Crash when trying to create function to read only db
• [ 2872880 ] mclient shows prompt with "-s" option without "-i&qu ot; option
• [ 2872884 ] mclient -t -s... does not show "Timer" info
• [ 2876948 ] PF: error "We do not support the value NaN"
• [ 2879833 ] SQL: SIMILAR keyword does not exist

Aug 2009 SP1 bug-fix release

This is a bug fix release of the Aug2009 version. This version can be installed "on top of" the Aug2009 release. The database format is fully compatible, so there is no need to do a dump and restore. (Although it is *always* a good idea to make backups.)

Some of the bugs that were fixed:

• On Windows, we switched to newer versions of the iconv and geos libraries (iconv 1.11.1 and geos 3.1.1, from iconv 1.11 and geos 3.0).
• On Windows, MonetDB4/XQuery was compiled with a newer version of the Intel compiler (version 11.1.046 from version 10.1.022).
• Better error handling if BATs are growing too large. (SF bug #2850513)
• The MAL parser was made more robust against syntax errors.
• A recursion depth check was added to the MAL interpreter. (SF bug #2852707)
• The PHP client is not compatible with PHP 5.3.0 or later. We now check for that. (Note that in a future release, the PHP client code will be replaced by a pure PHP implementation where this restriction does not apply.) (SF bug #284773)
• The SQL dump code (msqldump and mclient --dump) now specify all delimiters and the quote character on the generated COPY INTO query. This is needed to be compatible with the next feature release.
• The SQL dump code (msqldump and mclient --dump) now also knows about CLOBs (Character Large Objects) and properly quotes the values.
• The problem that occurred under certain circumstances that the JDBC driver would throw an SQLException when large result sets were retrieved is fixed. (SF bug #2812963)
• Fixed a bug in MonetDB/XQuery where the comparison between values of types strings and untypedAtomic was not done correctly. (SF bug #2853441)
• Fixed a bug in MonetDB/XQuery where the dead code elimination was a bit too eager. (SF bug #2852928)
• Fixed a bug in MonetDB/XQuery where a warning was generated that a pin count was below zero. (SF bug #2863458)
• Fixed a bug in PF/Tijah where OR queries were treated as AND queries. (SF bug #2836870)
• Fixed a bug in PF/Tijah where the nexi scanner would split strings on '_'.
• Fixed overflow checks in MonetDB/SQL. (SF bug #2853458)
• Fixes to get the embedded SQL example working again.
• Fixed startup of MonetDB/SQL server so that a client cannot connect before the server is ready. (SF bug #2855021)
• Fixed a bug in MonetDB/SQL where old and dropped view definitions were sometimes incorrectly reused. (SF bug #2863804)
• Fixed a bug in merovingian where the mero_port in the config file was not properly recognized. (SF bug #2844202)
• Fixed a bug in MonetDB/SQL where it was not possible to add month intervals to a timestamp. (SF bug #2854949)
• Fixed various bugs in the SQL engine. (SF bugs #2847106, #2862146)
• Use bigint for RESTART value in MonetDB/SQL. (SF bug #2846238)
• Fixed a bug in MonetDB/SQL with parsing hex values.
• Fixed a bug in MonetDB/SQL when attempting to make changes to the schema when the database is in read-only mode. (SF bug #2830238)
• Fixed a bug in MonetDB/SQL where adding a primary key constraint to a column containing a NULL value was allowed. (SF bug #2075241)
• Fixed a bug in MonetDB/SQL where updating values in one column could affect the values in another column. (SF bug #2864313)

The following bug tracker items were fixed since the Aug2009 release:

• [ 2808453 ] SQL: Trying a copy into in read only database segfault
• [ 2830238 ] SQL: Crash when trying to write on read-only DB
• [ 2831472 ] tijah:queryall crashes Mserver
• [ 2836870 ] PFtijah: Use of 'or' in phrase search yelds incorrect result
• [ 2840310 ] SQL: "Cannot access descriptor" on big endian
• [ 2844202 ] Merovingian: mero_port configuration option ignored
• [ 2846238 ] alter sequence restart on large integer
• [ 2847106 ] SQL: crash upon join with advanced between
• [ 2848525 ] mac10.6 and monetdb installation
• [ 2850341 ] SQL: bit_and returns invalid values
• [ 2852928 ] PF: updates broken using algebra front-end?
• [ 2853441 ] PF: distinct values on two sequences
• [ 2853458 ] SQL: cast exceeding maximum value, returns null
• [ 2854949 ] SQL: + interval '1' year; Program contains errors
• [ 2855021 ] SQL: mserver accepts sql connections before scenario load
• [ 2855152 ] mac10.6 and monetdb installation
• [ 2862146 ] Two COUNT(*) with LIMIT doesn't execute
• [ 2863458 ] PF/TIJAH: tokenize in a for-loop
• [ 2863804 ] SQL: old views incorrectly reused
• [ 2864313 ] SQL: Update changes all columns with same type and default

Aug 2009 feature release

Packaging

Fedora

We now provide RPMs for Fedora 10 & 11, but no longer for Fedora 9. However, a first check indicates that the Fedora 10 RPMs also install and work fine on Fedora 9.

MonetDB Common

 

• A bug-fix in the May2009 release had unnoticedly introduced a potential performance degradation, caused by non-optimal use of memory-mapped files, with workload that exceed physical memory, in particular with bulk-loading ("COPY INTO") large tables in MonetDB/SQL. This performance problem has been fixed.
• Various problems related to sloppy type usage and type casting had surfaced with testing on big-endian hardware as well as on 64-bit systems when using 32-bit OIDs. These bugs have been fixed.
• The compilation process has been improved by splitting the "infamous" large (generated) "gdk_scanselect_defs.c" into multiple files.

MonetDB Server 5

 

• auto-sensing is no longer implicit based on the port number being 0, but can now be enabled and disabled via mapi_autosense in monetdb5.conf. When set to enabled, it will try to find a free port starting from the port given by mapi_port.

SQL

 

• fix crash with select * from 'whatever' = bla, select * where bla = "whatever", Bug #2825974
• Ignore escapes for non-escape sequences, instead of inserting a literal backslash. Bug #2830900
• Decimals now are properly detected in PreparedStatements, allowing to use them. Bug #2831994
• Fixed hang after invalid statement + COMMIT outside auto-commit mode. Bug #2814622
• Fixed NUMERIC type detection. Bug #2818176
• No longer allow SELECT in value lists, causing confusing error messages. Bug #2807609
• Allow for NULL assignment and NULL as argument of a function. Bug #2812729
• Added missing power (^) operator. Bug #2812721
• DISTINCT was not properly recognised, causing incorrect count results. Bug #2826015
• When encountering ambigious columns, fail instead of crashing. Bug #2807336
• GROUP BY on a view with constants no longer yields in a crash. Bug #2837561
• PLAN output is fixed to no longer violate the protocol and to be tuples instead of comments.
• Optimisating pipelines can now be set in monetdb5.conf and changed without compiling SQL.
• sql_logdir can no longer be set and defaults to a location close to the actual data. This approach was chosen since the logs are useless without the data.
• The ILIKE operator was added. This operator is a case-insensitive LIKE operator, as also found in PostgreSQL.

Merovingian

The following features were implemented in the merovingian and monetdb programs:

• merovingian: Implemented a configurable port for the neighbour discovery service. This also allows to entirely disable neighbour discovery by setting mero_discoveryport to 0. Feature request #2800964
• merovingian: Implemented full per-database property support. monetdb now can manage the following per-database properties: forward (whether to proxy or redirect), shared (if it will be available in the network and how) and nthreads (number of working threads the mserver process uses). Feature Request #1837591.
• merovingian: Increased the default mero_exittimeout from 7 to 60 seconds. This makes merovingian by default wait longer for mservers to exit before forcefully terminating them.
• merovingian: Implemented sharing of databases with tags. Using these tags, extra information is available on a remote merovingian to select a database during connect. For this, a client can now connect to a database using a glob expression, which is matched on the database plus tag. When multiple databases match such expression, a load-balancing technique is applied. See the merovingian man-page for details.
• monetdb: Implemented dynamic width of monetdb status output. If a database name is wider than 14 chars, the name column is made wider as long as the terminal window allows. If a column cannot be made any wider this way, it is abbreviated as before.
• monetdb: Upon monetdb destroy, avoid multiple questions by merging them into a single confirmation for the destroy command. This accidently also fixed a bug where a response such as 'yes' would abort the next database removal.
• monetdb: Databases created by monetdb now can only have names restricted to the pattern [A-Za-z0-9-_]+.
• monetdb: Allow setting the name of a database (a.k.a. renaming it).
• monetdb: Allow an expression to be used to match against the list of discovered or local databases as argument in the commands discover and status.
• monetdb: destroy now allows to remove a database under maintenance. This way an administrator can prevent users connecting to the database and hence starting it on demand, which in turn prevents removal of the database.

XQuery

 

• xquery_logdir can no longer be set and defaults to a location close to the actual data. This approach was chosen since the logs are useless without the data.

Client Packages

mclient

The rendering of the mclient utility for SQL has been improved to better utilise the available space on the screen. Main improvement is that columns are no longer wrapped if they contain only NULLs, even though enough space on the terminal is availble.

Python

 

• The python API now supports connections to Merovingian.
• Transaction support was added.
• Added compatibility with python3k.
• Full Unicode/UTF-8 support.
• autocommit, commit and revert functions were added.

Ruby

 

• Fixed bugs that prevented the correct working of activerecords' migration; The activerecord connector now supports insertion, update and alter table operations; Type casting is working in activerecord; Added a rubygem and rakefile for activerecord-monetdb-adapter; Added a new usage example for activerecord to the README file; Added an example directory to the cvs tree; The driver now correctly works with Merovingian.
• Support for ruby 1.9.
• Both the standalone driver and activerecord now support transactions, nested transactions are simulated via savepoints in activerecord.
• Type conversion in the standalone driver is now performed after data has been retrieved and can be executed on single fields.

JDBC

 

• blockmode_blocksize option, and the -Xblksize option from JdbcClient were removed, as they are meaningless and inimplemented in the current JDBC driver.
• Full redirection support was added to support Merovingian's tag-based connections for localisation and loadbalancing.

MonetDB May2009-SP2 release

Release "May2009-SP2" consists of the following updated components

Name	Version	CVS module	CVS branch	Note/Comment
MonetDB Common	1.30.4	MonetDB	May2009
MonetDB Clients	1.30.4	clients	May2009
MonetDB4 Server	4.30.4	MonetDB4	May2009
MonetDB4/XQuery	0.30.4	pathfinder	May2009
MonetDB5 Server	5.12.4	MonetDB5	May2009
MonetDB5/SQL	2.30.4	sql	May2009
MonetDB5/SQL/GIS	0.10.4	geom	May2009
MonetDB Java	1.30.4	java	May2009
MonetDB Testing	1.30.4	testing	May2009	(developers only)

IMPORTANT NOTE:
Before upgrading to the latest "May2009-SP2" release from a release of MonetDB older than the Nov2008 release, please make a dump (SQL, XQuery) of your database, and rename/move/backup the database directories (dbfarm, sql_logs, xquery_logs). After the upgrade, restore (SQL, XQuery) your database, again. Once you have checked and confirmed the correctness of the restore, you can delete the above renamed/moved/backed-up old database directories.

Contents

• May 2009 SP2 bug-fix release
• May 2009 SP1 bug-fix release
• May 2009 feature release
  • Packaging
  • MonetDB Common
  •  
  • MonetDB Server 5
  • SQL
  • XQuery
  • Client Packages

May 2009 SP2 bug-fix release

This is the second bug fix release of the May2009 version. This version can be installed "on top of" the May2009 release. The database format is fully compatible, so there is no need to do a dump and restore. (Although it is *always* a good idea to make backups.)

Some of the bugs that were fixed:

• Fixed a problem where in certain condition you could get an error on Windows saying "GDKremovedir: rmdir(bat\DELETE_ME) failed".
• Various improvements to the Python and Ruby clients.
• Fix mserver5 crash when started on demand by merovingian (SF bug #2805340).
• Check for write errors during dump and abort when they occur.
• Some innocuous looking XQuery queries could produce large incorrect results and take a relatively long time to do so. SF bug #2815820.
• Fixed some potential buffer overflow problems in the Mapi library.
• Improved help message from mclient (SF bug 2818536).
• Fixed certain SQL LIKE queries on some O/S.
• Fixed SQL dump in mclient and msqldump to dump values of some less used types correctly.
• The Debian/Ubuntu installation now also contains the MonetDB PHP client support files.
• Fixed several performance issues (among others: SF bug 2821251).
• The various client libraries now also recognize the wrd type which may be emitted by the SQL server.
• In SQL, the char and varchar types now use Unicode code points (characters to you and me) as the unit for counting instead of bytes in the UTF-8 representation (SF bug 2822855).
• Fixed some UI issues with the XQuery administrator console.
• Fixed a bug in SQL when handling IN and OR lists with equal values.
• Fixed a stack overflow condition in SQL with certain types of quite complex queries (SD bug 2827784).

The following bug tracker items were fixed since the May2009 SP1 release:

• [ 1840239 ] BAT capacity checking needed
• [ 2805340 ] Merovingian (?): mserver5 segfaults if started by a client
• [ 2815820 ] PF: Endless loop in innocent xquery
• [ 2818536 ] improve mclient -u / -p help
• [ 2818931 ] MAL: segv on mortal soul's syntax error
• [ 2819972 ] function problom ----return and elseif
• [ 2821251 ] PF: Complex optimization overly complex
• [ 2822855 ] utf8 real size upon insert

May 2009 SP1 bug-fix release

This is a bug fix release of the May2009 version. This version can be installed "on top of" the May2009 release. The database format is fully compatible, so there is no need to do a dump and restore. (Although it is *always* a good idea to make backups.)

Some of the bugs that were fixed:

• mclient and msqldump would cache all results. This is fixed.
• The Fedora RPM for the Ruby clients package is now included.
• Some files were missing in the Python client directory. They are now included in the client source tar ball.
• mclient now shows the results of all subqueries in a query batch that was passed on the command line using the -s option if there was an error in one of them.
• mclient \d and \D interactive commands now obey the \| command.
• Fixed naming of the document and collection when both the --input and --collection arguments are given to mclient (XQuery).
• Various fixes to the low level memory mapped file support.
• Fixes to the geom module.

The following bug tracker items were fixed since the May2009 release:

• [ 2603876 ] XQ: fn:put() tests segfault with ALG back-end
• [ 2687549 ] SQL: test type_dump_test.SF-989257.SQL.sh fails since Feb 28
• [ 2737889 ] rand() is not executed for every row when using alias
• [ 2782202 ] SQL: dump&restore fully usable only by admin
• [ 2791356 ] Floating point exception
• [ 2797767 ] mclient --dump caches dump
• [ 2803477 ] alter table drop contraint error
• [ 2804823 ] The Mapi Client Utility
• [ 2805513 ] fn:trace() produces output _and_ error
• [ 2807038 ] tab + ) + mal function = seg fault

May 2009 feature release

Packaging

Windows

The Windows binaries for SQL and ODBC were built using Microsoft Visual Studio 2008, unlike previous releases which were built using Microsoft Visual Studio 2005. The main implication of this change is that the ODBC driver should now also install on Windows Vista.

The Perl, PHP, and Python clients were removed from the Windows installers. The clients are still available in the source distribution (although the Python client has changed significantly).

Fedora

There is no longer an RPM for MonetDB-client-ruby. In order to update to this release, you will first have to remove MonetDB-client-ruby after which you can update to this version. (A new version of the RPM will probably appear in a future release.)

Ubuntu

The Ubuntu version was built for Ubuntu 9.04, the Jaunty Jackalope. The Feb2009 release was built for Ubuntu 8.10, the Intrepid Ibex.

MonetDB Common

The way string data is stored in the database has changed for systems with a 64-bit word size and 32-bit OIDs. When the server first opens a database that was created using the old format, the database is updated automatically. This change affect only the 64-bit Windows build. The other 64-bit builds that are provided by us use 64-bit OIDs and are unaffected by the change.

MonetDB Server 5

Parallel loading has been disabled since there was an inherent problem with the implementation. This also affects SQL.

SQL

Write-ahead log

The default location of the write-ahead log has changed. At startup, when the server sees there is an SQL catalog but no write-ahead log, the server will exit immediately with an error message. You can do one of two things (not counting deleting your whole database):

• move the log files to the new location (recommended): create the directory that is mentioned by the error message and move the log files (two files, log and log.n where n is some integer) there;
• start the server with the --set sql_logdir=... option (where you replace ... with the appropriate path) to point the server to the old location.

In either case you need to know the old location of the write-ahead logs. This old location was $prefix/var/MonetDB5/sql_logs/<dbname>/sql/ where $prefix is the prefix under which MonetDB was installed and <dbname> the database name. The error message mentioned above gives the expected location of the write-ahead logs.

Note that when you use the Windows installer and you start the server using the Start menu, the write-ahead log is moved to the new location automatically.

Authentication

The authentication mechanism has changed somewhat so that it is now possible to dump and restore password information. Note that the passwords themselves cannot be retrieved: the dump/restore mechanism uses a hash of the password.

Merovingian

The following features were implemented in the merovingian and monetdb programs:

• The logging of merovingian has changed. TME records are no longer issued, instead each line is prefixed with a timestamp, syslog style. Feature Request #2782776
• merovingian now reopens its logfiles when being sent the SIGHUP signal. This allows utilities like e.g. logrotate to rotate the merovingian.log file. Feature Request #2782776
• The merovingian default log file location has changed from /var/log/merovingian.log into /var/log/MonetDB/merovingian.log.

XQuery

Like the previous "Feb 2009 SP2" release, the default Algebra-based Pathfinder compiler ("ALG") still does not support a few XQuery features and extensions. A detailed list and temporary work-arounds by resorting to the deprecated "milprint_summer" ("MPS") version of the Pathfinder compiler can be found here.

Write-ahead log

The default location of the write-ahead log has changed. At startup, when the server sees there is an XQuery catalog but no write-ahead log, the server will exit immediately with an error message. You can do one of two things (not counting deleting your whole database):

• move the log files to the new location (recommended): create the directory that is mentioned by the error message and move the log files (two files, log and log.n where n is some integer) there;
• start the server with the --set xquery_logdir=... option (where you replace ... with the appropriate path) to point the server to the old location.

In either case you need to know the old location of the write-ahead logs. This old location was $prefix/var/MonetDB4/xquery_logs/<dbname>/xquery/ where $prefix is the prefix under which MonetDB was installed and <dbname> the database name. The error message mentioned above gives the expected location of the write-ahead logs.

Note that when you use the Windows installer and you start the server using the Start menu, the write-ahead log is moved to the new location automatically.

Client Packages

Mapi

Support for SSL-encrypted connections with the server has been discontinued.

The protocol between client and server has changed slightly. This means you cannot use an old client with a new server.

Python

The Python client modules were rewritten and are now in pure Python and so do not depend on any other part of MonetDB. The new module has support for Python 2.5, 2.6 and 3.0, and it is Python DBAPI 2.0 compatible.

Ruby

The Ruby client modules were rewritten and are now in pure Ruby and so do not depend on any other part of MonetDB.

MonetDB Nov2008-SP2 release

Release "Nov2008-SP2" consists of the following updated components

Name	Version	CVS module	CVS branch
MonetDB Common	1.26.2	MonetDB	Nov2008
MonetDB Clients	1.26.2	clients	Nov2008
MonetDB4 Server	4.26.2	MonetDB4	Nov2008
MonetDB4/XQuery	0.26.2	pathfinder	Nov2008
MonetDB5 Server	5.8.2	MonetDB5	Nov2008
MonetDB5/SQL	2.26.2	sql	Nov2008
MonetDB5/SQL/GIS	0.6.2	geom	Nov2008
MonetDB Java	20081118	java	(none; CVS trunk)

IMPORTANT NOTE:
Before upgrading to the latest "Nov2008-SP2" release from a release of MonetDB older than the Nov2008 release, please make a dump (SQL, XQuery) of your database, and rename/move/backup the database directories (dbfarm, sql_logs, xquery_logs). After the upgrade, restore (SQL, XQuery) your database, again. Once you have checked and confirmed the correctness of the restore, you can delete the above renamed/moved/backed-up old database directories.

Contents

• Nov 2008 SP2 bug-fix release
• Nov 2008 SP1 bug-fix release
• Nov 2008 feature release
  • Copyright & License
  • Windows Installers
  • MonetDB Common (GDK)
  • MonetDB Server 5
  • SQL
  • XQuery
  • Bug fixes
  • Known Issues in this Release

Nov 2008 SP2 bug-fix release

This is the second bug fix release of the Nov2008 version. This version can be installed "on top of" the Nov2008 and Nov2008-SP1 releases. The database format is fully compatible, so there is no need to do a dump and restore. (Although it is *always* a good idea to make backups.)

Some of the bugs that were fixed:

• MonetDB/XQuery: There was a bug when shredding documents into an existing updatable collection when the existing collection happened to end at a page boundary. (Bug [ 2141012 ] XQ: unclear error).
• General: Fix to configure to not complain when the raptor library isn't found, unless it was specifically asked for.
• MonetDB/SQL: significantly speeded up loading of data when lines are extremely long (e.g. when loading BLOBs).
• XRPC: Since the change of the XRPC protocol (by June 2008), the XRPC server also sends response messages (with empty nodes) for updating function calls. However, the XRPC serever failed to send response messages for calls to the Pathfinder document management functions, because they are handled differently than normal updating functions. This is now fixed.
• XRPC: The function XRPCHTTPConnection.sendReceive() did not check if HttpURLConnection.getResponseCode() returns -1 or not. If the return value is indeed -1, subsequent use of the stream returned by HttpURLConnection.getInputStream() or HttpURLConnection.getErrorStream() might cause a NullPointerException. This is now fixed. The return value and the stream returned by those two functions are now checked and a proper error message is given.

The following bug tracker items were fixed since the Nov2008-SP1 release:

• [ 1964658 ] Problem with names of reverse()'d BATs.
• [ 2141012 ] XQ: unclear error
• [ 2233581 ] MonetDB does not like '\0' in its input
• [ 2388542 ] SQL bad blob insert causes crash
• [ 2405952 ] Merovingian takes too much CPU resources
• [ 2430201 ] Is distinct broken?
• [ 2457195 ] Broken query results in OOM, working one has no output
• [ 2460935 ] Pathfinder fails to compile with icc
• [ 2460953 ] `make dist` (and hence `make rpm`) fails for pathfinder
• [ 2466725 ] SubSelect with Outer Reference, result is wrong
• [ 2468513 ] SQL: copy into does not detect syntax errors
• [ 2471371 ] SQL: varchar bounds should be obeyed

Nov 2008 SP1 bug-fix release

This is the first bug fix release of the Nov2008 version. This version can be installed "on top of" the Nov2008 release. The database format is fully compatible, so there is no need to do a dump and restore. (Although it is *always* a good idea to make backups.)

Some of the bugs that were fixed:

• Fixed formatted printing from MAL.
• mclient now exits with an error code if one of the queries in a non-interactive session failed. [2388047]
• Fixed a deadlock condition when multiple queries simultaneously update the same part of a document (XQuery).
• When writing to a file, mclient now flushes the output when the query is done instead of waiting until the file is closed. [2344251]
• Fixed a race condition in document management (XQuery). [2338542]

The following bug tracker items were fixed since the Nov2008 release:

• [ 1911209 ] XQ: MonetDB 'hangs' after requesting non-existing doc
• [ 1981762 ] PF/alg: "Mapping unique name to an original name failed."
• [ 2049320 ] pf:collection doc count wrong (again?)
• [ 2075180 ] the unique column can't insert more than 1 row with null
• [ 2075241 ] When the NULL is in a column,it shouldn't add primary key
• [ 2103348 ] unique problem with nulls
• [ 2142522 ] XQ: Duplicates when not using intermediate result
• [ 2144639 ] XQ: fn:collection() in algebra version
• [ 2209871 ] most GEOM test fail after error handling changes
• [ 2224233 ] M4 test ID.914935 fails after warning changes
• [ 2233677 ] SQL Command kills DB
• [ 2276427 ] CMDmulti_merged_union: operation failed
• [ 2277120 ] XQ: following(-sibling)/preceding cause Mserver to crash
• [ 2305746 ] merovingian crashes when starting a vaultkeyless database
• [ 2338524 ] Deadlock on barrier
• [ 2344251 ] Output isn't complete
• [ 2388047 ] Mclient exit code
• [ 2388424 ] ERROR: XML Generation: NULL BAT has a 24 head, but tail is N
• [ 2420768 ] col from another schema inside function body

Nov 2008 feature release

Copyright & License

The copyright for the MonetDB suite was transferred to a newly formed company MonetDB B.V. One of the most important conditions for this transfer was that the software remains open source, and so the license under which this software is being distributed has not changed (apart from the copyright notice).

Windows Installers

The default installation location has been changed from C:\Program Files\CWI to C:\Program Files\MonetDB.

MonetDB Common (GDK)

The kernel administration for BATs has been changed to prepare for future enhancements. All descriptors are now initialized and kept in memory, which reduces the number of files and related IO accessing the descriptors.

Memory management for large database has been improved, with the scavenging thread becoming more aggressive in moving dirty pages to disk when we reach 70% of the physical memory.

Code cleaning and simplification continues to provide ease of compilation on many platforms.

MonetDB Server 5

The MonetDB 5 kernel underwent a code clean up too, e.g. datatypes have been more strictly applied and all kernel errors are propagated to MAL exceptions.

The number of experimental modules have been developed. The 'recycler' provides support for automatic re-use of intermediate results. It improves query throughput for long sequences with partially overlapping queries.

The system supports intra-query parallelism using the multi-core systems capabilities using the 'dataflow' optimizer.

The 'mergetable' optimizer has been extended to perform symbolic reasoning over oid-reasoning. Together with the MAT and dataflow optimizers it provides a basis to improve multi-core performance.

Several bugs were fixed in Merovingian, the MonetDB Database Server daemon, including a file descriptor leak, which caused Merovingian to terminate after a certain amount of connections using the proxy method. Merovingian has been extended with neighbour discovery features, allowing to redirect to discovered neighbours. The monetdb tool has been improved to give better feedback on starting and stopping databases. Most notably it now waits for completion of the start operation, and gives error feedback when starting failed for some reason.

SQL

This release comes with a few more optimizers enable by default. The most important one is the inter-query parallelism, which leads to better use of a multi-core platforms. The effects can be seen in our TPCH experiments.

The dataloading code has been improved during our effort to load a 2.6TB astronomical database. Upon request we can provide a detailed report on this experiment.

A query log can be captured with the history.sql script. It provides a basis for post-session analysis of execution times.

Several bugs reported have been solved and functionality in many other areas has been improved.

XQuery

Like the previous "Jun 2008" release, the default Algebra-based Pathfinder compiler ("ALG") still does not support a few XQuery features and extensions. A detailed list and temporary work-arounds by resorting to the deprecated "milprint_summer" ("MPS") version of the Pathfinder compiler can be found here.

Bug fixes

A detailed list of the 116 bug reports that have been closed since the previous "Jun 2008" release can be found in the MonetDB Bug Tracker at SourceForge: page 1/3, page 2/3, page 3/3.
The detailed list of changes can be found in the source code CVS logs.

Known Issues in this Release

• Merovingian crashes when starting a vaultkeyless database. Actual workaround is not to try and start databases not created using monetdb create.
  Details: SF bug tracker #2305746
• Parallel loading is known to be buggy. Actual workaround is to start the server using gdk_nr_threads=1 set in your monetdb5.conf file (or given via --set gdk_nr_threads=1 on your mserver5's command line) during the bulkload using COPY INTO FROM FILE with mclient.
  Details: (no known bug tracker reference)

Release "Jun2008" consists of the following updated components

Name	Version	CVS module	CVS branch
MonetDB Common	1.24	MonetDB	MonetDB_1-24
MonetDB Clients	1.24	clients	Clients_1-24
MonetDB4 Server	4.24	MonetDB4	MonetDB_4-24
MonetDB4/XQuery	0.24	pathfinder	XQuery_0-24
MonetDB5 Server	5.6	MonetDB5	MonetDB_5-6
MonetDB5/SQL	2.24	sql	SQL_2-24
MonetDB5/SQL/GIS	0.4	geom	Geom_0-4

and the jars for JDBC and XRPC.

IMPORTANT NOTE:
Before upgrading to the latest "Jun 2008" release from any previous release of MonetDB, please make a dump (SQL, XQuery) of your database, and rename/move/backup the database directories (dbfarm, sql_logs, xquery_logs). After the upgrade, restore (SQL, XQuery) your database, again. Once you have checked and confirmed the correctness of the restore, you can delete the above renamed/moved/backed-up old database directories.

Contents

 

• Compilation from sources
• Windows Installers
• MonetDB Common (GDK)
• SQL
• Geometry (SQL/GIS)
• XQuery
• Bug fixes

Compilation from sources

For convenience of both developers and users as well as to comply even more with open source standards, we now set/use the following defaults for the configure options

	--enable-strict, --enable-assert, --enable-debug, --enable-optimize

When compiling from CVS sources (as mainly done by developers):

	strict=yes  assert=yes  debug=yes  optimize=no (*)

When compiling from packaged/distributed sources (i.e., tarballs) (as mainly done by users):

	strict=no   assert=no   debug=no   optimize=no (*)

For building binary distributions (RPMs, MSIs):

	strict=no   assert=no   debug=no   optimize=yes

With nightly testing for the "Stable" release branches:

	strict=yes  assert=no   debug=no   optimize=yes

With nightly testing for the "Current" development trunk:

	strict=yes  assert=yes  debug=no   optimize=yes

(*)

IMPORTANT NOTE:

 

Since

--enable-optimize=yes

is no longer the default for any case except binary packages, it is

strongly recommended

to (re)compile everything from scratch,

explicitly configured

with

	--enable-debug=no --enable-assert=no --enable-optimize=yes

in case you want/need to run any performance experiments with MonetDB!

Please note:
--enable-X=yes is equivalent to --enable-X, and
--enable-X=no is equivalent to --disable-X.

Windows Installers

There are some changes to the Windows installers since previous releases which are described in this section.

Backup and restore

It is recommended to dump the old database and remove it before installing the new version. After installation, the dump can be restored.

In order to dump the SQL database, start the MonetDB SQL Client program and type the commands

\>...\databasedump.sql
\D
\>

The path after

\>

should be an absolute path name (i.e. start with a drive letter) and be in a save location.

In order to dump the XQuery database, start the server and then connect to the administrative console http://127.0.0.1:50001/admin and use that to dump the database.

By default the database is located in %APPDATA%\MonetDB4 for the MonetDB4/XQuery server and %APPDATA%\MonetDB5 for the MonetDB5/SQL server. After having made a database dump, the latter folder can be removed, but the database dump for MonetDB4/XQuery is made inside the database folder, so remove anything except the folder backup and its contents. This folder is located inside the dbfarm\demo folder.

Restoring the SQL database can be done using the MonetDB SQL Client program with the following command

\<...\databasedump.sql

Restoring the XQuery database can be done from the administrative console.

Installation

The 32 bit Windows installers can be installed "over" the old installation. The installer will automatically remove the old version and install the new. The 64 bit Windows installer will not do this (this time), since the code identifying the package was changed. So on 64 bit Windows, first uninstall the previous version and then install the new. The reason for this change is so that the 32 bit version and 64 bit version can both be installed on a 64 bit platform.

The ODBC driver was removed from the MonetDB5-SQL installer. You now need to install the ODBC driver separately. On 64 bit Windows you should install both 32 and 64 bit versions of the ODBC driver.

Also, make sure that after you have uninstalled the old version or upgraded to the new, the files C:\WINDOWS\system32\lib*.dll are all gone. If not, remove them. The new version does not install any files in the system folder. (To be precise, the files are libstream.dll, libmutils.dll, libMonetODBC.dll, libMonetODBCs.dll, and libMapi.dll.)

The JDBC driver was removed from the Windows installers. The driver is now only available separately.

A new MonetDB5-Geom-Module installer is available which can get installed after the MonetDB5-SQL-Installer has been installed. This module provides GIS support to MonetDB/SQL.

MonetDB Common (GDK)

TODO: mmap, property checks, ...

SQL

TODO: ...

Geometry (SQL/GIS)

This release includes (partial) support for Simple Feature Specification of OpenGIS. Follow the GEOM link above for a small tutorial on it.

XQuery

This release marks the beginning of a new era. The default for MonetDB/XQuery has been changed from using the original Pathfinder compiler (nick-named "milprint_summer" or simply "MPS") to using the new improved and enhanced Algebra-based Pathfinder compiler ("ALG").

Compared to the "preview" in the previous "Feb 2008" release, the functionality of the Algebra-based Pathfinder compiler has been significantly extended to (almost) match that of the original MPS-based compiler. Only the following XQuery features and extensions are not yet supported with the new Algebra-based Pathfinder compiler:

• fn:put(), fn:collection()
  Please use the old MPS version as described below.
• recursive functions
  If the recursion depth is known at compile time, adding the option
  

  	declare option pf:recursion-depth "<number>";
  

  to the query text will ensure that the recursive function is unfolded <number> times. If the recursion is deeper than <number> invocations a runtime error is triggered.
  For arbitrary recursive functions, please use the old MPS version as described below.

• Arithmetic Functions
  Please use the old MPS version as described below.
• Prepared Queries
  When using prepared queries, MonetDB/XQuery will automatically switch back to the old MPS version.
• XRPC Extension
  In case MonetDB/XQuery's XRPC Server is contacted via XRPC (e.g., by the Administrative GUI), the MonetDB/XQuery server will automatically switch back to the old MPS version to handle the XRPC request.
  To use the execute at XRPC Syntax, please use the old MPS version as described below.
• StandOff Extension
  Pathfinder automatically falls back to the old MPS version, in case StandOff extensions are enabled at runtime via
  Mserver --set standoff=enabled and/or pf --enable-standoff

While support for the old MPS version will not be continued, it remains available for backward compatibility without recompilation, at least for the time being.
For the stand-alone XQuery compiler, the MonetDB/XQuery server, and/or the MonetDB/XQuery client, simply use the following command line options to revert the default to the old MPS version:

• pf -M
• Mserver --dbinit='module(pathfinder);' --set xquery_backend=milprint_summer
• mclient --language=xquery --no-algebra
• mclient -lxquery -g

In an interactive `

mclient -lxquery

` session, commands "

\g

" / "

\G

" enable / disable the old

MPS

-based compiler in the MonetDB/XQuery server.

Bug fixes

A detailed list of the 139 bug reports that have been closed since the previous "Feb2008" release can be found in the MonetDB Bug Tracker at SourceForge: page 1/3, page 2/3, page 3/3.
The detailed list of changes can be found in the source code CVS logs.