Release history
Release history mk Sun, 05/15/2011 - 11:55The trail of release notes as of the first time we made the code base available on SourceForge.
Our current release notes can be found in here.
June 2020
June 2020 sjoerd Mon, 10/19/2020 - 11:07Jun 2020-SP1 bugfix release (11.37.11)
MonetDB Common- Hash buckets come in variable widths. But if a BAT grows long enough so that the BAT indexes that are stored in the buckets don't fit anymore, the buckets need to be widened. This is now fixed.
- 6844: sys.getUser('https://me:pw@www.monetdb.org/Doc') does not return the user: me
- 6845: the url sys.get...(url) functions do not allow null as a parameter
- 6858: json.keyarray(json '{ "":0 }') fails with error: Could not allocate space
- 6859: only first character of the separator string in json.text(js json, sep string) is used
- 6873: sys.hot_snapshot() creates incomplete snapshots if the write-ahead log is very large
- 6876: tar files created by sys.hot_snapshot() produce warnings on some implementations of tar
- 6877: MonetDB produces malformed LZ4 files
- 6880: Left fuzzy queries are much slower than other fuzzy queries.
- 6882: cgroups limits no longer respected?
- 6883: SQLancer crash on delete query
- 6884: SQLancer generates query with unclear error message
- 6885: SQLancer causes assertion error on UTF8_strlen
- 6886: SQLancer alter table add unique gives strange error message
- 6887: SQLancer crash on complex query
- 6888: SQLancer crash on cross join on view
- 6889: SQLancer crash on long query
- 6892: SQLancer crash on query with HAVING
- 6893: SQLancer inner join reporting GDK error
- 6894: SQLancer crash on rtrim function
- 6895: SQLancer causing 'algebra.select' undefined error
- 6896: SQLancer algebra.select' undefined 2
- 6897: SQLancer distinct aggregate with error on group by constant
- 6898: SQLancer crash on join query
- 6899: SQLancer TLP query with wrong results
- 6900: SQLancer generated SIGFPE
- 6901: SQLancer TLP query with wrong results 2
- 6902: SQLancer query: batcalc.between undefined
- 6903: SQLancer calc.abs undefined
- 6904: SQLancer aggr.subavg undefined
- 6905: SQLancer TLP query with wrong results 3
- 6906: SQLancer crash on complex join
- 6907: SQLancer algebra.select undefined
- 6908: SQLancer inputs not the same size
- 6909: SQLancer query with wrong results
- 6911: SQLancer query: 'calc.bit' undefined
- 6917: Decimal parsing fails
- 6918: SQLancer query compilation error
- 6919: SQLancer insert function doesn't handle utf-8 strings
- 6920: SQLancer project_bte: does not match always
- 6922: Timestamp columns not migrated to new format
- 6923: Imprints data files for timestamp BAT not migrated to the new format
- 6924: SQLancer query copy on unique pair of columns fails and complex query with GDK error
- 6925: Count string rows in union of string tables leaks (RSS) memory
- 6926: SQLancer query with wrong results
- 6927: SQLancer inputs not the same size
- 6928: SQLancer crash on coalesce
- 6929: SQLancer calc.date undefined
- 6932: Syntax error while parsing JSON numbers with exponent
- 6934: sys.isauuid() returns wrong answer for some invalid uuid strings
Jun 2020 feature release (11.37.7)
MonetDB5 Server- The mserver5 option --verbose (-v) was removed. A similar effect can be had by issuing the query CALL logging.setcomplevel('SQL_TRANS', 'INFO'); as the monetdb user.
- The functions algebra.intersect, algebra.semijoin, and algebra.crossproduct have an extra argument, bool max_one, which indicates that there must be no more than one match in the join.
- The example module opt_sql_append is not installed in the binary packages anymore.
- Added mserver5 option (--dbtrace=<path>) in order to be able to specify the output file any produced traces.
- Removed function bat.setKey().
- algebra.rangejoin now has two extra arguments: anti:bit and symmetric:bit.
- Added session identifier, number of workers and memory claim to the sysmon queue.
- The worker (number of threads), memory (in MB) and optimizer pipeline limits can now be set per user session basis. The query and session timeouts are now set in seconds.
- With required privileges an user can set resource limits for a session.
- There are now versions of group.(sub)group(done) that produce a single output containing just the groups.
- algebra.join and algebra.leftjoin now have forms which return a single column. The column that is returned is the left column of the two column version.
- The MAL profiler now assigns the SQL TRACE output to the client record thereby avoiding the interaction with other queries, but loosing insight of competing queries. The stethoscope should be used for that.
- There was a problem with the MonetDB SELinux support on Fedora 32. That is fixed in this release. In order to do a proper upgrade of the package if you have already installed MonetDB-selinux on Fedora 32, you may need to uninstall (dnf remove) the old package and then install the new.
- A new system to deal with debug output has been implemented. There is now an option --dbtrace to mserver5 that takes a file argument to which debug output is written. The default value is the file mdbtrace.log inside the database directory. This option can also be set through the monetdb program.
- The home directory of the automatically created monetdb user was changed from /var/MonetDB to /var/lib/monetdb (RPM based systems only). This home directory is (currently) not used for anything, though.
- Python 2 support has been removed. There is now only support for using Python 3.
- Removed support for bam and sam files.
- Added mserver5 option (--set raw_strings=true|false) and monetdb database property (raw_strings=yes|no) to control interpretation of strings.
- On Fedora and RHEL systems (not RHEL 6), if monetdbd runs under systemd, when the package is updated, monetdbd (and hence any mserver5 process it runs) is restarted.
- Added dbtrace mserver5 option to the daemon in order to set mserver5's output directory for the produced traces.
- The monetdb-client-tools (Debian/Ubuntu) and MonetDB-client-tools (Fedora/RH) containing the stethoscope, tachograph, and tomograph has been removed. A completely new version of stethoscope will be released to replace the old version.
- Removed the possibility of using the MD5 checksum for authentication purposes. It was never actively used but was there as an option. Now the option has been removed.
- Add port and host as fields in the .monetdb file.
- Added 'sessionid' column to system function sys.queue(), so each query gets tagged with the current session identifier
- Allow monetdb user to control session and query time out and selectively stopping a client sessions with a soft termination request.
- The NUMERIC and FLOAT types are now handled fully. Before only DECIMAL, FLOAT, and DOUBLE were handled fully.
- Some bugs were fixed in the passing back and forth between application and server of values of type GUID (UUID).
- The functions BATintersect, BATsemijoin, and BATsubcross have an extra argument, bool max_one, which indicates that there must be no more than one match in the join.
- The "unique" property on BATs was removed. The property indicated that all values in a BAT *had* to be distinct, but this was not actually used.
- A new type of candidate list has been introduced. Candidate lists are used internally to specify which rows of a column participate in an operation. Before, candidate lists always contained a list of candidate row IDs. The new candidate list type specifies a list of row IDs that should NOT be considered (negative candidates).
- The maximum number of BATs in the system has been increased for 64 bit architectures.
- The hash tables used internally by the system now uses a technique based on Linear Hashing which allows them to grow gracefully. This means that hash tables aren't removed and recreated nearly as often anymore. This also meant that the hash table had to be split into two files, which means that after an upgrade the hash tables have to be recreated.
- Implemented a version of BATproject, called BATproject2, with two "right" arguments which conceptually follow each other.
- Removed MT_mmap and MT_munmap from the list of exported functions. Use GDKmmap and GDKmunmap with the same parameters instead.
- Changed the interface of the atom "fix" and "unfix" functions. They now return a value of type gdk_return to indicate success/failure.
- Removed the tunique property. The tunique property indicated that all values in the column had to be distinct. It was removed because it wasn't used.
- BATrangeselect now has two extra arguments: anti and symmetric (both bool).
- Updating the value of a sequence now requires privilege on its own schema.
- The sys.querylog_enable(threshold integer) now actually enables the querylog and uses a threshold in milliseconds.
- Removed UNION JOIN statements. They were dropped by the SQL:2003 standard, plus MonetDB implementation was not fully compliant.
- The OFFSET value in the COPY INTO query now counts uninterpreted newlines. Before it counted "unquoted record separators" which meant that if you had a single quote on a line that you want to skip, you could not use the feature.
- Added support for FROM RANGE MINVALUE TO RANGE MAXVALUE and FROM RANGE MINVALUE TO RANGE MAXVALUE WITH NULL VALUES cases in partitioned tables by range (before they weren't).
- Removed support for Python 2. Python 2 itself is no longer supported. Use Python 3 instead. Functions that were declared as LANGUAGE PYTHON2 or LANGUAGE PYTHON2_MAP are changed to LANGUAGE PYTHON and LANGUAGE PYTHON_MAP respectively (without changing the Python code).
- Implemented 'covar_pop' and 'covar_samp' aggregate functions, as well as their window function counterparts. Implemented 'stddev_samp', 'stddev_pop', 'var_samp', 'var_pop', 'corr' and 'group_concat' window function correspondents.
- Extended SQL catalog with CREATE WINDOW syntax for user-defined SQL:2003 window functions. At the moment, window functions must be defined on the backend engine, i.e. on this case MAL. In the current implementation, the backend code generation creates two additional columns of type lng with the start and end offsets for each row.
- Added prepared_statements_args view, which details the arguments for the prepared statements created in the current session.
- Added sys.prepared_statements view, which lists the available prepared statements in the current session.
- Added deallocate statements with the syntax 'DEALLOCATE [PREPARE] { number | ALL }', to close an existing prepared statement or all, through the SQL layer. Previously this feature was available via MAPI exclusively with the "release" command.
- Added support for raw strings using the syntax r'' or R''. This means that C-like escapes will remain uninterpreted within those strings. For instance SELECT r'\"' returns a string of length two. The user needs to escape single quotes by doubling them: SELECT r''''.
- Implemented ROLLUP, CUBE and GROUPING SETS from SQL:1999. They define grouping subsets used with the GROUP BY clause in order to compute partial groupings. Also, the GROUPING aggregate was added. This aggregate is a bitmask identifying the grouping columns not present in the generated grouping row when used with the operators described above.
- Updated user session procedures by adding the possibility to set properties based on a session identifier. Optimizer pipeline: sys.setoptimizer(int, string) Number of worker threads: sys.setworkerlimit(int, int) Memory limits (in MB): sys.setmemorylimit(int, int) Query timeout (in ms): sys.setquerytimeout(int, int) Session timeout (in ms): sys.setsessiontimeout(int, int) The first argument corresponds to the id of the session to modify, and these procedures are bound to the monetdb user exclusively. The versions of the mentioned procedures with just the second argument were added as well, where the changes are reflected in the current user session, and therefore every user can call them.
- The procedures sys.settimeout(bigint), sys.settimeout(bigint,bigint) and sys.session(bigint) are now deprecated. Instead use sys.setquerytimeout and sys.setsessiontimeout mentioned above.
- Added SQL procedure sys.hot_snapshot() which can be used to write a snapshot of the database to a tar file. For example, sys.hot_snapshot('/tmp/snapshot.tar'). If compression support is compiled in, snapshots can also be compressed ('/tmp/snapshot.tar.gz'). The tar file expands to a single directory with the same name as the database that was snapshotted. This directory can be passed directly as the --dbpath argument of mserver5 or it can be copied into an existing dbfarm and started from monetdbd.
- 6298: unexpectedly slow execution of SELECT length(fieldname) FROM tablename LIMIT 1 queries
- 6401: Suspected memory leak in mserver5 when creating/dropping tables
- 6687: Count distinct very slow and use too much the hard drive
- 6723: columns aliases duplicates should not be allowed. automatic aliasing required.
- 6731: Add system view to allow querying of available prepared statements and their parameters
- 6732: Add SQL command to close a specific prepared statement
- 6750: Executing a query on a non-existing column on a remote table crashes the remote server
- 6785: function sys.isaURL(url) should have been declared as sys.isaURL(string)
- 6808: reveal the alarm.sleep procedure in SQL
- 6813: function not_uniques(bigint) returns error when called
- 6818: usage of multiple column expressions in where-clause (f(a), f(b)) in (select a, b) causes assertion failure on mserver5
- 6821: Failed to start monetdb with embedded python
- 6828: Server crashes when executing a window query with ordering by EXTRACT date
- 6846: Global temporary table not accessible in other connections / sessions
- 6847: A simple way of speeding up impscheck for dense canditers
- 6850: Idle timestamp not set
- 6851: json parser doesn't parse integers correctly
- 6863: thash files not released upon drop table
- 6864: (I)LIKE with multiple % doen't find matches
November 2019
November 2019 sjoerd Wed, 06/03/2020 - 16:50Nov 2019-SP3 bugfix release (11.35.19)
The main fix in this release is the problem with running monetdbd under the control of systemd. There was also a related problem with the SELinux configuration that was fixed in this release.
Bug Fixes
- 6827: CUME_DIST window function returns incorrect results
- 6829: NTILE window function returns incorrect results
Nov 2019-SP2 bugfix release (11.35.15)
There is a known problem with running monetdbd under the control of systemd. The monetdbd service won't start. A new release to fix this problem is coming soon.
Bug Fixes
- 6805: Using the cascade operator in a drop table statement ends in an exit from the Monetdb shell.
- 6807: Median_avg and quantile_avg ignore NULL values
- 6815: query with ifthenelse() crashes mserver5
- 6816: Monetdb Crashes on INSERT statement after ALTER statement in another connection
- 6817: running analyze on a schema which contains a stream table stops with an error
- 6819: functions do not persist
Nov 2019-SP1 bugfix release (11.35.9)
Bug Fixes
- 6724: Prepare confuses types when more than one argument is used
- 6726: Python aggregation does not create aggr_group when aggregating over all rows
- 6765: GRANT SELECT privilege on a subset of table columns results in access denied error when selecting the same columns from the table
- 6790: Count distinct giving wrong results
- 6791: str_to_time('11:40', '%H:%M') creates wrong time value
- 6792: JSON path compiler accepts invalid input
- 6793: cast(interval second value to int or decimal) is wrong (by a factor of 1000), cast(interval month value to decimal or floating point) fails
- 6794: external name fits.listdir not bound (sys.listdir) Fatal error during initialization:
- 6796: Incorrect crash time reported by monetdb tool after crash of mserver5
- 6798: json.text off by one error
- 6804: DNS resolution of 0.0.0.0 fails on recent Ubuntus
Nov 2019 feature release (11.35.3)
MonetDB5 Server
- The server "console" has been removed, as has the --daemon option. The server now doesn't read from standard input anymore. The way to stop a server is by sending it a TERM signal (on Linux/Unix) or by sending it an interrupt signal (usually control-C -- on all systems).
- Implemented a function bat.diffcand to calculate difference of two candidate lists.
- The mtime module was completely rewritten, the atom types date, daytime, and timestamp were changed. Upgrade code for BATs containing these types has been implemented. The old daytime type used a 32 bit integer to record milliseconds since the start of the day. The new daytime type uses a 64 bit integer to record microseconds since the start of the day. The old date type recorded days since or before the year 1. The new daytime type records the day of the month and the number of months since the year -4712 separately in a single 32 bit integer of which only 26 bits are used. Dates now use the proleptic Gregorian calendar, meaning the normal Gregorian callendar has been extended backward, and before the year 1, we have the year 0. Both the old and new timestamp types are a combination of a daytime and a date value, but since those types have changed, the timestamp type has also changed. The new date type has a smaller range than the old. The new date range is from the year -4712 to the year 170049. During conversion of date and timestamp columns, the dates are clamped to this range.
- The tzone and rule atom types have been removed. They were not used by any code, and they were defined in a non-portable way.
- Added "mapi_ipv6" property to monet_options to force ipv6 address binding only. This property is inherited while forking from monetdbd if it is also set there.
- Removed (bat)calc.between_symmetric and changed (bat)calc.between by adding a number of extra arguments, all of type :bit: symmetric, low inclusive, high inclusive, nils false.
Merovingian
- Added "vmmaxsize" and "memmaxsize" mserver5 options to the daemon in order to set mserver5's maximum virtual and committed memory respectively.
- Added ipv6 property to monetdbd properties to force IPv6 addresses binding only. By default this property is false to allow IPv4 addresses as well.
MonetDB Common
- BATcalcbetween and all its variants now have an extra bool parameter "anti" to invert the test.
- All forms of BATcalcbetween and VARcalcbetween have extra bool arguments for low inclusive, high inclusive and nils false. The latter causes the result to be false instead of nil if the value being checked is nil.
SQL Frontend
- Removed functions json.text(string) returns string and json.text(int) returns string. Their MAL implementation didn't exist, so they were meaningless.
- There are new aggregate functions sys.median_avg and sys.quantile_avg that return the interpolated value if the median/quantile doesn't fall exactly on a particular row. These functions always return a value of type DOUBLE and only work for numeric types (various width integers and floating point).
- Added sys.deltas ("schema" string, "table" string, "column" string) returns table ("values" bigint) system function which returns a single column with 6 values: a flag indicating if the column's upper table is cleared or not, the count of the RDONLY, RD_INS and RD_UPD_ID deltas of the column itself, the number of deleted values of the column's table, as well as the level of the current transaction in the transaction level tree. It should be used for debugging purposes only.
- Added "VALUES row_list" statement as a top SQL projection statement.
- The implementation of in-expression now follows a join-based approach instead of using iterative union/selects. This greatly improves performance for large in-value-lists. Furthermore the old approach has large in-value-lists generate large MAL plans. This is now no longer the case.
- Strings are now limited to 1GB, double-quoted tokens are limited to 2kB. These sizes are bytes of (UTF-8 encoded) input data.
Bug Fixes
- 3533: SQL aggregate functions avg(), sum() and median() return an error when used on a column with datatype interval second
- 6134: Query produces error: HEAPalloc: Insufficient space for HEAP of 1168033427456 bytes.
- 6613: LATERAL crash /.../rel_bin.c:1473: rel2bin_table: Assertion `0' failed.
- 6683: Bug in subselect
- 6686: Bug in subselect (count function)
- 6688: Bug in subselect (or condition)
- 6689: Trying to improve the performance of SQL queries with a large list of members in IN clause.
- 6695: timestamp transformation
- 6700: Monetdb Bugs in Subselect statements:
- 6722: window functions issues
- 6740: while upgrading the database from verison (MonetDB-11.27.13) to (MonetDB-11.33.3) we are unable to bring up the database
- 6754: in mclient a strang msg is reported after issueing command: set schema sys;
- 6755: Assertion failure in rel_bin.c
- 6756: Error in optimizer garbageCollector on merge tables select
- 6757: Double free or corruption (out)
- 6758: SIGSEGV in __strcmp_sse2_unaligned()
- 6759: Python JSON loader creates invalid data type for strings
- 6761: Error: Program contains errors.:(NONE).multiplex
- 6762: mserver5 crashes on (re-)start
- 6764: mserver5 crashes with corruption, double free, invalid size or invalid pointer
- 6766: Missing bulk implementation for get_value and next_value calls
- 6769: ProfilerStart is not threadsafe
- 6771: R-devel
- 6772: TRACE includes information from all active sessions
- 6773: json.filter returns corrupted string when selecting JSON null value
- 6774: PROD aggregation gives wrong result
- 6775: NOT IN with an AND containing an OR gives wrong result
- 6776: Creating a table with a full outer join query gives type with wrong digits on the joined key.
- 6779: Using Windows Messages translation for errno error codes.
- 6780: Wrong value of the rank function
- 6781: Insert after index creation crash
- 6783: AVG changes scale of its results
- 6784: function sys.isauuid(string) should return false if string value cannot be converted to a UUID
April 2019
April 2019 sjoerd Thu, 11/28/2019 - 09:50Apr 2019-SP1 bugfix release (11.33.11)
Build Environment
- Removed restriction on using combinations of --enable-assert, --enable-debug, and --enable-optimize. --enable-debug adds a -g (or -even -g3) option, --enable-debug=gdb adds a -ggdb3 flag (for GCC); --enable-optimize adds a bunch of optimization flags; --disable-debug (or --enable-debug=no) removes any -g flags; --disable-optimize removes any -O flags; --enable-optimize=auto leaves the optimization flags untouched.
MonetDB Common
- We now look at the limits imposed by cgroups and the setrlimit system call to initialize some internal values related to how much (virtual) memory we think is available.
SQL Frontend
- Many fixes for using the server in a highly concurrent setting with many clients doing read-only queries mixed with clients doing updating queries.
- Changed the internal representation of RANGE MINVALUE and RANGE MAXVALUE in MERGE TABLE partitions. Before, the limits were represented by the smallest and largest value of the domain of the column, now these are represented by a NULL value. This has the added benefit that if you use TO RANGE MAXVALUE, the largest value of the domain is included in the partition (before it wasn't). Also, these new limits now work for any ordered type, so including VARCHAR.
Bug Fixes
- 6576: Sqlitelogictest aritmetic expressions with negative numbers handling
- 6697: Duplicate expressions not eliminated with long CASE statement
- 6701: When changing the schema name of a table, referencing rows from sys.columns, sys.keys and more tables are not removed
- 6703: SQL optimizer enters loop and goes into stack overflow
- 6706: prepare doesn't recognize merge statement
- 6712: Where clause with cast ignores sub-select
- 6713: COPY INTO FROM 'file.xz' does not work
- 6714: Assertion failure in rel_select.c for correlated subquery in aggregation query (with group by and having and order by and limit)
- 6715: Assertion failure in rel_bin.c for MERGE INTO command
- 6716: COPY INTO does not load UTF8 encoded text
- 6718: Can't copy table into another table with constraints (mkey.bulk_rotate_xor_hash)
- 6720: Compilation issues on gcc 9.1.1 [-Werror=stringop-truncation]
- 6721: Possibly incorrect call to pcre_exec
- 6725: Prepare statement on non-existing function crashes
- 6728: SELECT DISTINCT not removing duplicates
- 6729: Creating a table with duplicate column name should not be possible
- 6730: sys.stop or sys.pause a INSERT query triggers "BATproject: does not match always"
- 6736: Unexpected query result with merge tables and view after upgrade
- 6738: issue with window functions and hugeint type coercion
- 6741: report no such table when execute “alter table tb1 rename to tb2” according to using mapi
- 6744: CREATE SEQUENCE seq; fails
- 6748: MonetDB "forgets" columns after ALTER TABLE statements
- 6749: mserver5 restart aborts/segfaults after dropping column with constraint
- 6751: ODBC driver(11.33.3) Seg Faults when "fn ucase" is used in SQL
Apr 2019 feature release (11.33.3)
MonetDB5 Server
- Changed the way blobs are ordered. Before, shorter blobs came before longer, now the contents is compared first.
- Removed function blob.tostring() since it cannot guarantee that the resulting string is properly encoded in UTF-8.
- Removed the type "sqlblob", changed the external format of the type "blob" to be like what "sqlblob" used to be. In other words, the "blob" type is now SQL-compatible.
- The MAL functions algebra.sort and algebra.firstn now have a new, extra second-to-last argument, nilslast:bit, which indicates where NIL values are to be sorted: at the beginning, or at the end. The old behavior is when for algebra.sort, the value of nilslast is equal to the value or the reverse:bit parameter, for algebra.firstn when the value of nilslast is opposite to the value of the asc:bit parameter.
Build Environment
- We now use the C11 atomic operations if available, although libatomic_ops is still used if present and not disabled on the configure command line.
- We now also build the Python 3 integration packages for Fedora, Debian, Ubuntu, and Windows (not for Debian 8, Ubuntu 14.04, or EPEL 6 and 7 which are too old).
SELinux Support
- There is one tunable parameter, mserver5_can_read_home, which can be set using "setsebool -P mserver5_can_read_home=true" to allow an mserver5 process started by monetdbd under the control of systemd to read files in users' home directories.
Merovingian
- Added a new database property named `profilerbeatfreq`. Its value, <freq>, should be an integer and, if it is set the command `profilerstart` will call stethoscope with this value as the argument to -b. This will instruct stethoscope to collect heartbeat events from the MonetDB server every <freq> milliseconds. Please note that no type checking is done, and if <freq> is not an integer, stethoscope will silently ignore it.
Client Package
- The mclient program can now be quit by typing quit or exit on the command-line. This makes it easier for novice users to quit the program if they do not know that the standard command to quit is \q.
Mapi Library
- Removed function mapi_cache_shuffle.
- Removed function mapi_stream_query.
Stream Library
- The functions mnstr_write{Sht,Int,Lng,Hge} and their Array variants now swap bytes of the written values if the stream is set to swap bytes (i.e. big-endian on a little-endian machine and v.v.).
- Removed defines ST_ASCII, ST_BIN, ST_READ, ST_WRITE from stream.h.
- Changed function mnstr_set_byteorder(stream, char) to mnstr_set_bigendian(stream, bool) where the second argument should be `true' for specifying that the stream is bigendian. This sets an internal flag whether or not to swap bytes which can be retrieved with mnstr_get_swapbytes(stream).
- Changed function `int mnstr_byteorder(stream)' to `bool mnstr_get_swapbytes(stream)'; it now returns whether bytes are swapped.
- Changed function `int mnstr_type(stream)' to `bool mnstr_isbinary(stream)'.
MonetDB Common
- During processing, we now try to maintain the smallest and largest values that occur in a BAT. This information is not saved across server restarts.
- Added an extra argument of type `bool' to the atom to string and string to atom methods to indicate whether the string is for internal use (casting the value to or from a string) of external use (to be printed).
- Implemented a nilslast option for BATfirstn. If set, NILs come last in the ordering that BATfirstn simulates, so non-NIL values are preferentially returned. The old behavior can be obtained by setting nilslast to !asc(ending).
- Implemented a nilslast option for BATsort. This option should be equal to the reverse option for stable sort (it is not implemented for stable sort), but can be different from reverse for non-stable sort. The functions BATsort and GDKqsort have extra parameters, the function GDKqsort_rev has been removed (superseded by GDKqsort with the new `reverse' parameter).
- The BUNtail, BUNtvar, BUNtloc, and BUNtpos macros (and Tloc and Tpos) now return a `void *' instead of a `char *'.
- Function PROPdestroy now takes a BAT* parameter.
- A whole bunch of functions now take a bool argument instead of an int argument. Some functions now return a bool instead of an int. In all these cases, the int was used as a Boolean value.
SQL Frontend
- Implemented two-argument least/greatest functions. If one argument is NULL, the functions return the other value, otherwise they return the least/greatest of the two arguments.
- Implemented SQL2003 natural logarithm function: ln(num_expr).
- Function octet_length now also accepts a BLOB argument and returns the length of the BLOB (i.e. the same as length when called with a BLOB argument).
- Implemented length(blob) and comparison between blobs.
- Implemented a two argument function log which calculates the logarithm of the first argument using the second argument as base.
- Next to the functions log (natural logarithm) and log10 (base 10 logarithm), there is now also log2 (base 2 logarithm).
- Removed unused global SQL variable "history".
- Allow usage of custom expressions in GROUP BY and PARTITION BY clauses. The same expressions can be used in the projection phase as long as it is exactly the same used in the grouping clause (textual lookup is performed). As an example the query: SELECT col1*2 FROM t1 GROUP BY col1*2; is correct, while SELECT sin(col1+5) FROM t1 GROUP BY col1*2; is wrong.
- Allow usage of WITH clauses in insert, update, delete and merge statements.
- Implemented merge statements from SQL:2003 standard. Using a source relation R, a target table T is updating depending on the result of the merge. In case of a match, the table T's row is either updated or deleted with R's row. In a non-match case, R's row is inserted into T. The grammar is the follows: > MERGE INTO target [ [AS] ident ] USING source ON search_condition { WHEN MATCHED [ AND search_condition ] THEN { UPDATE SET assignment_list | DELETE } } | { WHEN NOT MATCHED [ AND search_condition ] THEN INSERT [ column_list ] [ { DEFAULT VALUES | VALUES row_values } ] }
- Added possibility to change the schema of a table with the syntax: > ALTER TABLE [ IF EXISTS ] qname SET SCHEMA ident
- Added optional alias option for target table in update and delete statements: (e.g. UPDATE target AS alias SET col1=1 WHERE alias.col2 < 5)
- Improved and extended storagemodel functionality. Changes include:
- Added views sys."tablestorage" and sys."schemastorage" for easy and quick assesment of aggregated storage usage per table or per schema.
- Excluded listing system tables in views sys."storage", sys."tablestorage" and sys."schemastorage". You can still use sys."storage"() to query storage of system tables and columns.
- In procedure sys.storagemodelinit() when populating table sys.storagemodelinput it now excludes system tables as system tables are not useful to be modeled for storagesize by application users. It now also computes and populates the atomwidth column more correctly for variable size data types (strings, blobs) when the table has representative data in those columns.
- System function sys.columnsize() has been corrected for types: tinyint, real, date, time, timetz, sec_interval, month_interval, decimal, uuid, mbr, char, clob, json, url, blob, geometry and geometrya. For variable size data types (varchar, char, clob, json, url, blob, geometry, geometrya) it now returns the columnsize excluding the variable heapsize. The heapsize is retrievable via sys.heapsize().
- System function sys.heapsize() has been corrected for all variable size data types: varchar, char, clob, json, url, blob, geometry and geometrya.
- System function sys.imprintsize() has been corrected. The imprintsize depends on the width (1 or 2 or 4 or 8 or 16 bytes) of the data type. Also instead of 12% it now uses 20%, which is a better estimate.
- System function sys.storagemodel() has been removed as it outputs the same data as view sys.storagemodel. Use view sys.storagemodel instead.
- Corrected views sys.storagemodel and sys.tablestoragemodel by returning a computed orderidxsize when the count has been changed in the sys.storagemodelinput table. Also the views now return data ordered by schema, table and column names.
- Extended view sys.tablestoragemodel with column: "storages". Besides columns also keys (primary, foreign and unique keys) and indexes (ordered, imprints) use storage, so the "storages" count can be higher than the number of columns per table.
- Corrected the data type of columns "schema", "table", "column", "type", "mode" and location in table sys.storagemodelinput and functions sys."storage"(), sys."storage"(sname), sys."storage"(sname, tname) and sys."storage"(sname, tname, cname) from string into varchar(1024). Consequently also the views based on the table or functions will be reporting varchar(1024) as column meta data instead of clob for those columns. This allows faster querying and reporting by generic SQL programs which treat clob querying different from varchar columns.
- Extended windowing functions catalog with SQL standard semantics. Standard aggregation functions can now be used in windowing functions: (avg,max,min,sum,prod,count). Other windowing specific functions were also implemented: (percent_rank,cume_dist,ntile, lag,lead,first_value,last_value,nth_value).
- The standard frame specification was implemented for aggregation functions as well as first_value, last_value and nth_value functions. The available frames are rows, range and groups. Bounds can be unbounded (partition limit), the current row, a fixed number of rows (constant), or variable (column as input). (e.g SELECT COUNT(col1) OVER (PARTITION BY col2 ORDER BY col3 RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) FROM t1;).
- Added WINDOW keyword which is optionally provided after the FROM clause, with window specifications used in the projection (e.g SELECT SUM(col1) OVER w1, LAST_VALUE(col2) OVER w2 FROM t1 WINDOW w1 AS (ROWS BETWEEN 5 PRECEDING AND 0 FOLLOWING), w2 AS (w1);).
- Our previous partitioning implementation didn’t impose order in the input. With this reexamination, partitioning now imposes ascending order by default, thus pairing with the industry standard implementation.
- Implemented X'...' style binary string literals.
- Implemented U&'...' Unicode character string literals and U&"..." Unicode delimited identifiers, including UESCAPE. For the string literals, you can have U&'...' '...' '...' UESCAPE '...' where the escape must be as single character and the other '...' strings are also Unicode character string literals. For now, these latter strings also undergo C-style backslash interpretation.
- Implemented PostgreSQL-like E'...' strings. The strings can contain C-style backslash escapes. The old format strings '...' currently still also accept C-style escapes, but that feature will be removed in a future release.
- Implemented the NULLS FIRST and NULLS LAST option to ORDER BY. The default is NULLS FIRST for ASC(ending) and NULLS LAST for DESC(ending).
- Added possibility to rename SQL schemas, tables and columns with ALTER statements: > ALTER SCHEMA [ IF EXISTS ] ident RENAME TO ident > ALTER TABLE [ IF EXISTS ] qname RENAME TO ident > ALTER TABLE [ IF EXISTS ] qname RENAME [ COLUMN ] ident TO ident Also added optional IF EXISTS clause to other existing ALTER TABLE statements.
- The sql SAMPLE syntax is extended to include an optional integral SEED parameter that sets the seed for the internal random number generator in the sample algorithm. When the seed is set, the user can obtain a reproducible sample set from a static dataset with the same sample query. The new syntax is SELECT ... FROM ... WHERE ... SAMPLE <expr> [SEED <integer>]
- Implemented a version of COPY INTO that reads/writes files from/to the client instead of doing it in the server. This has the advantage that COPY INTO is then no longer restricted to only the "super user" monetdb, nor only to absolute file names. The syntax to have the server communicate with the client for file content is COPY INTO table FROM file ON CLIENT ...; and COPY query INTO file ON CLIENT ...;. This also works for COPY BINARY INTO. There is also the possibility to specify that files are to be read/written by the server by using ON SERVER. This is also the default when ON CLIENT or ON SERVER is not specified.
- The sys.functions table now has an extra column called "system" of type BOOLEAN whose value is TRUE for system functions (i.e. functions that should not be dumped). The table sys.systemfunctions has been changed to a view and is now officially deprecated.
- Extended merge tables with partitioning using a predicate: > CREATE MERGE TABLE [ IF NOT EXISTS ] table_name (... columns ...) [ PARTITION BY { RANGE | VALUES } { ON '(' column_name ')' | USING '(' expression ')' } ] The partitioning can occur by range or list of values using one of the table's columns or an expression. The domain of the partitioning scheme will be covered by each child table added with alter statements depending if the table is partitioned by range or list of values: > ALTER TABLE merge_table_name { ADD | SET } TABLE child_table_name AS PARTITION IN '(' expression [ ',' ... ] ')' [ WITH NULL ] > ALTER TABLE merge_table_name { ADD | SET } TABLE child_table_name AS PARTITION BETWEEN { RANGE MINVALUE | expression } AND { RANGE MAXVALUE | expression } [ WITH NULL ] > ALTER TABLE merge_table_name { ADD | SET } TABLE child_table_name AS PARTITION WITH NULL When updating a single partition range/list of values, the SET keyword should be used instead of ADD. INSERT, UPDATE and DELETE statements are possible on partitioned tables with corresponding validation on each partition domain. The system tables sys.table_partitions, sys.range_partitions and sys.value_partitions were added to store meta-information about each existing partition.
Bug Fixes
- 2403: stream: http read support in streams
- 2416: GDK: file/dir creation mask
- 2478: SQL: rename columns
- 2496: SQL: implement greatest / least
- 3384: Auxiliary window functions
- 3416: log() SQL function only supports natural logarithm
- 3448: Make lbatomic_ops optional if installed
- 3520: Add support for scalar function LN(num)
- 3530: sigabort on window aggr function
- 3567: Add support for: ALTER TABLE [schema1.]oldtablename RENAME [TO] [schema2.]newtablename
- 3743: DELETE FROM merge table not supported.
- 3832: Cannot use expressions in GROUP BY clause
- 3946: Expose C log2 function to SQL
- 4056: mapi_timeout is not implemented
- 6181: comments cause error with return in function
- 6326: Eliminate unused UNION tables
- 6347: Mserver returns with "memory exhausted in" error if query size exceeds 2.5MB
- 6475: Remove unused SQL global variable "history"
- 6588: Table aliasing are not supported in SQL update queries
- 6591: Name mangling in the SQL catalog results in ambiguity
- 6608: Sqlitelogictest error message in group by with coalesce
- 6609: Rename Schema in MonetDB
- 6636: sys.queue() extension: allow DBA to see all running queries
- 6647: Add suport to Python 3 on Windows
- 6669: COPY [xxx RECORDS] INTO foo FROM STDIN ... doesn't work without specifying nr of to be copied records
- 6671: Error when running user function on merge table with remote part
- 6674: R UDF with Date type fails to convert RAPI
- 6676: Max data length of 2048 for column sys._tables.query is too small for the actual data (2811 for view sys.ids)
- 6678: Binding NULL parameter to parametrized query results in syntax error on execution
- 6680: Copy cannot open CSV file if systemctl enabled
- 6684: Inserting into a replica table crashes the server
- 6685: adding a view to a merge table gives unexpected behaviour
- 6690: Unable to fetch empty string with ODBC driver
- 6696: Re-use of MAL explain plans with complex type coercions
August 2018
August 2018 sjoerd Tue, 04/30/2019 - 12:27Aug 2018-SP2 bugfix release (11.31.13)
SQL Frontend- Disabled function sys.getcontent(url).
- 6643: schema name qualifier in create global temporary table json.table6643 ... is not honered
- 6645: optimizer treats all the functions with no or constant parameters as constant
- 6650: PREPARE SQL statement fails to compile user defined functions with parameter/s
- 6651: Multi-column IN clause for subquery produces wrong results
- 6653: CREATE TABLE accepts empty table/column name
- 6654: Incorrect handling of 'TRUE' in compound select
- 6657: Restart sequence with a non atomic sub-query cardinality gives wrong error message
- 6660: GRANTing a ROLE is not idempotent
- 6662: Concurrency Conflicts Exception string "!transaction is aborted because of concurrency conflicts, will ROLLBACK instead" not shown on prompt.
- 6664: mserver5 crash: infinite recursive happens at rel_bin.c:489
- 6665: Creation of serial types does not accept negative numbers
- 6666: COPY INTO from .. LOCKED doubles input data
- 6668: The SAMPLE key word doesn't work in a subquery.
- 6669: COPY [xxx RECORDS] INTO foo FROM STDIN ... doesn't work without specifying nr of to be copied records
- 6672: SQLGetData with SQL_C_WCHAR string truncation and invalid StrLen_or_Ind value
Aug 2018-SP1 bugfix release (11.31.11)
Build Environment- On Ubuntu 18.10 (Cosmic Cuttlefish), the libmonetdb5-server-bam package cannot be built because of an incompatibility in the libbam library (it cannot be used in a shared object).
- Some subtle dependencies between RPMs have been fixed.
- 6640: timestamp_to_str returning incorrectly adjusted results
- 6641: race condition in SQL UDF with update
- 6642: Hanging query
- 6648: key property potentially wrong after type conversion
- 6649: Projection inside within transaction gives wrong results
Aug 2018 feature release (11.31.7)
MonetDB5 Server- The lsst module was moved to a separate repository (https://dev.monetdb.org/hg/MonetDB-lsst/).
- Build the MonetDB-cfitsio RPM and libmonetdb5-server-cfitsio Debian/Ubuntu package.
- On Windows, the separate MonetDB5-Geom installer has been incorporated into the main MonetDB5-SQL installer and is therefore no longer available as a separate download.
- Added a "logrotate" configuration file. See /etc/logrotate.d/monetdbd.
- Changed the monetdb profilerstart command to be more robust. If the server or stethoscope crashed before, the pid file is still there, so the next time we try to start stethoscope, it will fail. Now the profilerstart command will check if a stethoscope process with the recorded pid is running. If not, we start stethoscope, assuming that something went wrong before.
- Changed the monetdb stop command to try to stop stethoscope before stoping the server. The error conditions that can arrise from attempting to stop stethoscope are:
- The database is not running.
- The profilerlogpath is not set.
- The profiler.pid file does not exist or is inaccessible.
- The contents of the profiler.pid are not valid.
- Shutdown of stethoscope did not succeed.
- Removing the profiler.pid file failed.
- ODBC: Implemented SQL_ATTR_QUERY_TIMEOUT parameter in SQLSetStmtAttr.
- ODBC SQLGetInfo now returns a positive numeric value for InfoTypes: SQL_MAX_COLUMN_NAME_LEN, SQL_MAX_DRIVER_CONNECTIONS, SQL_MAX_IDENTIFIER_LEN, SQL_MAX_PROCEDURE_NAME_LEN, SQL_MAX_SCHEMA_NAME_LEN, SQL_MAX_TABLE_NAME_LEN and SQL_MAX_USER_NAME_LEN.
- Added a '-f rowcount' option in mclient to repress printing the actual data of a resultset, but only print the number of returned tuples
- Added support for lz4 compressed files in the stream library
- Hash indexes are now persistent across server restarts.
- The macros bunfastapp and tfastins and variants no longer set the dirty flag of the heap they write to. This now needs to be done separately (and preferably outside of the inner loop).
- Removed batDirty flag from BAT record. Its function is completely superseded by batDirtydesc and the dirty flags on the various heaps.
- Removed "masksize" argument of function BAThash.
- A whole bunch of functions that took an int argument that was used as a Boolean (true/false) value now take a value of type bool. The functions BATkeyed, BATordered and BATordered_rev now return a bool instead of an int.
- Removed the tdense property: it's function is completely replaced by whether or not tseqbase is equal to oid_nil.
- Removed helper programs Mtimeout and MkillUsers: they have long been superseded by timeout handling by Mtest.py itself.
- Removed deprecated table producing system functions: sys.dependencies_columns_on_functions() sys.dependencies_columns_on_indexes() sys.dependencies_columns_on_keys() sys.dependencies_columns_on_triggers() sys.dependencies_columns_on_views() sys.dependencies_functions_on_functions() sys.dependencies_functions_on_triggers() sys.dependencies_keys_on_foreignkeys() sys.dependencies_owners_on_schemas() sys.dependencies_schemas_on_users() sys.dependencies_tables_on_foreignkeys() sys.dependencies_tables_on_functions() sys.dependencies_tables_on_indexes() sys.dependencies_tables_on_triggers() sys.dependencies_tables_on_views() sys.dependencies_views_on_functions() sys.dependencies_views_on_triggers() They are replaced by new system dependency_* views: sys.dependency_args_on_types sys.dependency_columns_on_functions sys.dependency_columns_on_indexes sys.dependency_columns_on_keys sys.dependency_columns_on_procedures sys.dependency_columns_on_triggers sys.dependency_columns_on_types sys.dependency_columns_on_views sys.dependency_functions_on_functions sys.dependency_functions_on_procedures sys.dependency_functions_on_triggers sys.dependency_functions_on_types sys.dependency_functions_on_views sys.dependency_keys_on_foreignkeys sys.dependency_owners_on_schemas sys.dependency_schemas_on_users sys.dependency_tables_on_foreignkeys sys.dependency_tables_on_functions sys.dependency_tables_on_indexes sys.dependency_tables_on_procedures sys.dependency_tables_on_triggers sys.dependency_tables_on_views sys.dependency_views_on_functions sys.dependency_views_on_procedures sys.dependency_views_on_views
- Implemented group_concat(X,Y) aggregate function which also concatenates a column of strings X, but using a supplied string Y as the separator. This function is also a SQL extension.
- Implemented group_concat(X) aggregate function which concatenates a column of strings using a comma as a separator. This function is not featured in the SQL standard.
- 4020: Importing timestamp with zone from copy into
- 6506: Improper performance counters
- 6556: Sqlitelogictest division by zero on COALESCE call
- 6564: Changes to the Remote Table definition
- 6575: Sqlitelogictest crash on groupby query with coalesce call
- 6579: Sqlitelogic test infinite loop while compiling SQL query
- 6586: Sqlitelogictest crash on complex aggregation query
- 6593: Poor performance with like operator and escape clause
- 6596: Multicolumn aggregation very slow after ANALYZE when persistent hashes are enabled
- 6605: Sqlitelogictest set queries with wrong results
- 6606: Misleading parameter name in generate_series function
- 6610: Sqlitelogictest algebra.rangejoin undefined
- 6611: Cannot compile with GCC 8.1 and --enable-debug=no
- 6612: Implement BLOB handling in python UDFs
- 6618: dependency column on sequence violated by DROP SEQUENCE
- 6621: SELECT FROM REMOTE TABLE WHERE <> returns wrong results
- 6624: "Cannot use non GROUP BY column in query results without an aggregate function" when using aggregate function in both HAVING and ORDER BY clauses.
- 6625: OR in subselect causes the server to crash with segmentation fault
- 6627: stddev_pop inconsistent behaviour
- 6628: User cannot insert into own local temporary table
- 6629: CREATE TABLE IF NOT EXISTS returns 42000!
- 6630: Sqlitelogictest cast NULL to integer failing
- 6632: Dataflow causes crash when THRnew fails
- 6633: ILIKE clauses don't work on certain characters
- 6635: monetdbd exits due to "Too many open files" error
- 6637: Within a transaction, \d after an error causes mclient to exit
- 6638: (sequences of) mkey.bulk_rotate_xor_hash() can generate NIL from non-NIL making multi-col joins return wrong results
- 6639: COMMENT ON TABLE abc IS NULL invalidly sets the remark column to null where remark column is defined as NOT NULLable
March 2018
March 2018 sjoerd Fri, 08/31/2018 - 13:14Mar 2018-SP1 bugfix release (11.29.7)
Build Environment
- On Linux and Unix, try not to link to libraries multiple times: that only causes the code (and worse, the variables in those libraries) to be included multiple times. On Windows, we do need to link to libraries multiple times (once for each DLL we create if the DLL references anything from the library) but that doesn't cause the linked library to be loaded multiple times.
SQL Frontend
- Corrected the definition of view: sys.ids.
Bug Fixes
- 6562: Sqlitelogictest crash on group by query with not in operator
- 6565: Sqlitelogictest crash on complex select query with coalesce call
- 6566: Sqlitelogictest unavailable calc.- MAL operations
- 6568: Sqlitelogictest crash on complex case query
- 6569: Sqlitelogictest select query with not between cause with wrong results
- 6570: Sqlitelogictest select coalesce undefined calc
- 6572: ordered index Error in optimizer garbageCollector
- 6573: Sqlitelogictest crash on complex select query with case statement
- 6574: server crashed could not find ordered index while creating a table
- 6576: Sqlitelogictest aritmetic expressions with negative numbers handling
- 6577: creating temp table kills performance of the original query
- 6578: One two-tuple insert gives different results than two single inserts
- 6581: Join condition errors.
- 6583: Fixed size string concatenation with integer results in fixed size string of size 0
- 6584: SELECT FROM REMOTE TABLE WHERE IS NOT NULL produces wrong result
- 6585: Nested Merge tables cause an infinite loop in rel_optimizer
- 6587: Sqlitelogictest crash on complex case statement
- 6589: Sqlitelogictest crash on complex on complex expression
- 6594: Sqlitelogictest crash on complex case statement
- 6595: Remote decimal division triggers assertion / returns wrong answer
- 6598: Python 3.4 not supported (due to usage of Py_DecodeLocale)
- 6600: Sqlitelogictest queries fail to execute
- 6601: "where is null" clause on remote table causes problem with next query
- 6602: Sqlitelogictest wrong results in IN query
- 6603: Sqlitelogictest: Aggregation query with distinct clause produces duplicated rows
Mar 2018 feature release (11.29.3)
MonetDB5 Server
- Implemented function pcre.replace_first which is like pcre.replace, except it only replaces the first match.
- The EXPLAIN command now shows all the MAL type resolutions, because in general users may not be aware of their signatures. It also simplifies programs to analyze such plans.
- Implemented versions of group.(sub)group(done) that don't return a histogram.
- Removed MAL functions streams.socketRead, streams.socketReadBytes, streams.socketWrite, and streams.socketWriteBytes.
- Removed MAL functions streams.openRead(s:streams):streams and streams.openWrite(s:streams):streams.
- Lots of changes to streamline the internal error handling. In principle, all errors should now include a SQLSTATE error code (see the SQL standard).
Build Environment
- Added the .pdb files needed for debug symbols to the Windows installer for MonetDB/SQL.
Whole System
- mclient's execution time profiling options and output format have been changed. Instead of implicitly via the "--interactive"/"-i" option (or when using an interactive mclient console), execution time profiling is now controlled via an explicit "--timer=timermode"/"-t timermode" command-line option, or a "\t timermode" command in the interactive mclient console. The default (also in the interactive mclient console) is now timermode "none", i.e., no timing information is given. Timermode "clock" activates client-side wall-clock timing ("clk") in "human-friendly" format much like the interactive mode did before. Timermode "performance" also provides detailed server-side timings: "sql" is the time to parse the SQL query, optimize the logical relational plan and create the initial physical (MAL) plan; "opt" is the time to optimize the physical (MAL) plan; "run" is the time to execute the physical (MAL) plan. With timermode "performance" all server-side timings and the client-side wall-clock time are given in milliseconds (ms). Note that the client-measured wall-clock time "clk" is reported per query only when options "--interactive" or "--echo" are used, because only then does mclient send individual lines (statements) of the SQL script to the server. Otherwise, mclient sends the SQL script in large(r) batch(es) to the server, and, thus, only the total wall-clock time per batch is measured and reported. The server-measured detailed performance timings "sql", "opt", "run" are always measured and reported per query. Also, all timing information is now given on a separate line and sent to stderr rather than stdout.
- Some types and constants were moved from configure (and hence monetdb_config.h) to gdk.h. In particular, the types "lng" and "ulng" have been moved and can therefore no longer be used by code that doesn't (ultimately) include gdk.h. Just use int64_t instead. A bunch of format defines have been removed: SZFMT, SSZFMT, PTRFMT, PDFMT. Just use the C standard codes for those (%zu, %zd, %p, %td). The define for printing a lng (LLFMT) was also moved. Use PRId64 for printing int64_t in code not using gdk.h. Removed all references to __int64 and long long (use int64_t instead).
Merovingian
- Add daemon commands for starting/stopping collection of profiler (stethoscope) logs.
Client Package
- ODBC: The driver function SQLProcedureColumns was implemented.
- ODBC: Changed table types as used by SQLTables from "LOCAL TEMPORARY" and "GLOBAL TEMPORARY" to "LOCAL TEMPORARY TABLE" and "GLOBAL TEMPORARY TABLE" respectively.
- ODBC: Initial support for the HUGEINT type in SQL was added. Any value with type HUGEINT can be retrieved in a C variable with type SQL_C_CHAR or SQL_C_WCHAR. A value of type HUGEINT can be retrieved in other C types as long as they fit, the largest C type supported being a 64 bit integer (equivalent to BIGINT).
- The functions in the mapi library that require 64 bit integers now use the standard type int64_t instead of the non-standard mapi_int64. This requires a compilation environment that has the stdint.h include file (standardized in C99). Compilation of the library also requires the inttypes.h include file (also standardized in C99).
- Add a new pretty printing option to stethoscope Running stethoscope with the flag -j will produce not pretty printed output (one json object per line). Running with the -y flag will produce pretty printed output. Running with neither will produce the legacy, line oriented format
mapilib
- The three mapi_explain* functions and mapi_trace don't return any useful information, so they now return void.
- The functions mapi_error_str and mapi_result_error now return const char * instead of plain char * to indicate that the returned data belongs to the library and should not be changed or freed by the application.
- New function const char *mapi_result_errorcode(MapiHdl) which returns the SQLSTATE code if available when an error has occurred.
stream
- The interface of mnstr_fgetpos and mnstr_fsetpos was changed to look more like the standard C functions fsetpos and fgetpos: they both have a second argument "pointer to fpos_t".
- Removed function wbstream.
- Removed functions udp_rastream and udp_wastream.
- Removed functions socket_rstream and socket_wstream.
- Removed functions append_wstream and append_wastream.
- Removed functions mnstr_rstream and mnstr_wstream.
utils
- The function mcrypt_getHashAlgorithms now returns a static, constant string, so the result should not be modified or freed.
MonetDB Common
- Changed return type of function void_replace_bat from BUN to gdk_return: it now only returns whether the operation succeeded or not.
- Changed the return type of BATroles from void to gdk_return: it can fail due to malloc failure.
- Removed functions ALIGNsetH, ALIGNsetT, and CREATEview_ (mind the underscore). The first can easily be replace by using BAThseqbase (that's all it did), the second was only used once, and the third can be replace by VIEWcreate.
- Removed unused functions BATmemsize and BATvmsize.
- Removed the tnodense property: it was maintained but never actually used, not even stored.
- The NIL representation of the internal flt and dbl types was changed from the smallest representable finite value to NaN (not-a-number).
- Changed the interface of ATOMformat and VALformat: they now return a pointer to the allocated string.
- The length "method" for atoms now returns a size_t, the "len" field of a ValRecord is now a size_t, the "size" field of the atomDesc structure is now unsigned short.
- Removed the "align" field from the ATOM descriptor (atomDesc) structure.
- The atomtostr and atomfromstr "methods" for atoms now return ssize_t and require a pointer to size_t for the size of the buffer.
- The atom tostr and fromstr "methods" now always return -1 on error. A return value greater than 0 is normal, a return value of 0 is not normal, but technically not an error.
Testing Environment
- Added a --data_path option to Mtest.py that defines an external data repository. See the commit message of c484932c7fd8 for more info.
SQL
- Extended support to use CREATE ORDERED INDEX on columns of type: char, varchar, clob, blob, url, json, inet and uuid.
- Added new system view: sys.ids which contains all database objects ids which can be used in sys.dependencies table.
- Added new system view: sys.dependencies_vw which shows all data of sys.dependencies including names on objects, object types and dependency types.
- Added 25 new system views for finding out dependencies between database objects. These new dependency views improve, extend and replace the 17 sys.dependencies_X_on_Y() functions as previously defined in 21_dependency_functions.sql. Those sys.dependencies_X_on_Y() functions are now marked as deprecated.
- Added new system view: sys.roles which contains all defined roles.
- Added new system view: sys.var_values which shows the values for system variables.
- Added support for COMMENT ON statements using SQL syntax: COMMENT ON { SCHEMA | TABLE | VIEW | COLUMN | INDEX | SEQUENCE | FUNCTION | PROCEDURE | AGGREGATE | FILTER FUNCTION | LOADER } qname IS { 'my description text' | NULL | '' } ; For COLUMN the qname can be "table_name"."column_name" or fully qualified as in: "schema_name"."table_name"."column_name". For FUNCTION, PROCEDURE, AGGREGATE, FILTER FUNCTION and LOADER the qname may need to include the signature (argument types) to be able to differentiate between multiple overloaded functions which have the same name and schema. By specifying IS NULL or IS '' you remove the comment for the database object. If a database object is dropped, the associated comment is also removed. Note: it is not allowed or possible to add comments for temporary tables or objects in schema "tmp". The sql catalog has been extended with system table: sys.comments. The keyword 'COMMENT' has now become a reserved keyword.
- Removed system function sys.environment(). It was a duplicate of system function sys.env().
- Implemented behavior for DROP SCHEMA my_schema RESTRICT command. Previously the RESTRICT keyword was accepted but not obeyed. It would always do a CASCADE operation. Also the default behavior of DROP SCHEMA my_schema command is now changed into RESTRICT behavior (was CASCADE).
- The internal NULL representation of the REAL (FLOAT) and DOUBLE types was changed from the smallest representable finite value to NaN (not-a-number).
- A column default value can be used in a UPDATE statement: UPDATE tname SET cname = DEFAULT, and INSERT statements: INSERT INTO tname VALUES (..., DEFAULT, ...)
- Added support for TRUNCATE statements conforming to the SQL:2008 standard: TRUNCATE [ TABLE ] qname [ CONTINUE IDENTITY | RESTART IDENTITY ] [ RESTRICT | CASCADE ] In a TRUNCATE statement a 'CONTINUE IDENTITY' or 'RESTART IDENTITY' clause can be passed to restart or not, being the former the default one. The 'CASCADE' option instructs to truncate referencing table(s) also if the referencing table(s) have foreign key references to this table. The default behavior is 'RESTRICT'. Note: it is possible to use TRUNCATE statements in a transaction and thus to rollback the effects of a truncate. The keywords 'TRUNCATE' and 'CONTINUE' have now become reserved keywords.
- Added possibility to GRANT or REVOKE a TRUNCATE privilege to a user or role.
- Added possibility to define a TRIGGER on a TRUNCATE event.
- Added possibility to specify 'OR REPLACE' in following CREATE commands: CREATE [ OR REPLACE ] VIEW qname ... CREATE [ OR REPLACE ] TRIGGER qname ...
- Added possibility to specify 'IF EXIST' in following DROP commands: DROP AGGREGATE [ IF EXISTS ] qname ... DROP FUNCTION [ IF EXISTS ] qname ... DROP FILTER FUNCTION [ IF EXISTS ] qname ... DROP LOADER [ IF EXISTS ] qname ... DROP PROCEDURE [ IF EXISTS ] qname ... DROP TRIGGER [ IF EXISTS ] qname ...
- Lots of changes to streamline the internal error handling. In principle, all errors should now include a SQLSTATE error code (see the SQL standard).
- Added support for extracting the quarter (number between 1 and 4) of a date or a timestamp or a timestamp with timezone in SQL: EXTRACT ( QUARTER FROM my_date_expr ). Added support for extracting the week (number between 1 and 53) of a date or a timestamp or a timestamp with timezone in SQL: EXTRACT ( WEEK FROM my_date_expr ). Added support for scalar functions: quarter(date_expr), quarter(timestamp_expr) and quarter(timestamptz_expr).
Bug Fixes
- 3574: Add support for: create OR REPLACE view ...
- 3824: Created table not visible from ODBC
- 3831: Extend date part extraction and date formating functions to support more formats like "quarter", "iso year/week"
- 6244: Add support for: TRUNCATE TABLE
- 6249: DEFAULT in row-values missing (sqlsmith)
- 6346: BATsort returns GDK_SUCCEED when **sorted bat is null
- 6438: Implement functionality to enforce the restrict option in: DROP SCHEMA xyz RESTRICT;
- 6507: Column Header coming with trailing spaces in compiled version of MonetDB 11.27.9/11
- 6513: Sqlitelogictest: Wrong MAL plan generation for column product
- 6526: Crash using aggregate function inside a case statement in having clause
- 6529: Sqlitelogictest crash in select query with IN operator and cast
- 6530: Sqlitelogictest: select query with NOT IN giving wrong results
- 6532: copy into ignore null as directive if first column doesn't come from file
- 6534: [Mar2018]: mclient -f tab / --format=tab complains about "unsupported formatter"
- 6535: [Mar2018]: mclient -t / --timer does not work as documented
- 6536: [Mar2018]: timing output of mclient -t / --timer= should go to stderr rather than stdout
- 6537: [Mar2018]: mclient's default timing mode should (again) be "none" rather than "clock"
- 6541: [Mar2018]: mclient reports incorrect wall-clock time
- 6542: assertion failure when querying: select count(*) from sys.commented_function_signatures;
- 6543: Mar2018: truncate on SQL system tables should NOT be allowed
- 6545: Sqlitelogictest crash in IN query
- 6546: Sqlitelogictest crash in IN query with division
- 6547: OS-dependent behaviour for ilike
- 6548: Select from remote table leaves session open
- 6549: Add log messages when listen fails
- 6550: Sqlitelogictest crash on complex CASE statement
- 6551: Sqlitelogictest wrong NULL value cast
- 6552: Sqlitelogictest crash on complex case statement
- 6553: Sqlitelogictest crash on aggregation with having statement
- 6554: Sqlitelogictest crash on complex case statement
- 6555: Sqlitelogictest wrong result set generated from complex case statement
- 6557: Sqlitelogictest crash on aggregation query with not in
- 6559: rows in sys.statistics are not removed when a temporary table is dropped.
- 6560: Sqlitelogictest crash on group by query with having in
- 6561: Sqlitelogictest crash on group by query with having not in clause
July 2017
July 2017 sjoerd Thu, 03/29/2018 - 15:07Jul 2017-SP4 bugfix release (11.27.13)
Build Environment
- Added the .lib and .h files needed for building extensions to the Windows installer.
Bug Fixes
- 3470: Support setClob without length restrictions
- 6468: JDBC 2.27 fails with year < 1000
- 6482: Query failures on order by on union
- 6483: Monetdb crashes, on query
- 6487: UNION of NULLs with several tables fails
- 6488: Semijoin returns duplicate values from a column with unique values
- 6489: Sqlitelogictest - Wrong result set of complex conditional query
- 6490: Sqlitelogictest - Select query with an IN clause parse error
- 6491: SELECT IN returns NULL instead of false when not found
- 6492: Persistent hashes stored and then ignored. Storage info not in sync with actual indices.
- 6493: Sqlitelogictest - Aggregation query on empty table with DISTINCT clause
- 6494: Sqlitelogictest - Algebra operators priority in select query
- 6495: Sqlitelogictest - Omitting AS in a result set column alias name
- 6496: Sqlitelogictest - Select interval comparisons between floating-points and NULL
- 6497: Sqlitelogictest - Select not between query producing wrong results
- 6498: large virtual memory spike on BLOB column COUNT
- 6499: Crash when trying to replace a function defined in sys from a different schema
- 6502: Query with multiple limit clauses does not return anything
- 6508: Segmentation fault in mserver5 on Python2 UDF with TIMESTAMP column input that has NULL values (conversion.c:438, PyNullMask_FromBAT)
- 6510: Sqlitelogictest: Wrong output in aggregation query
- 6512: Monetdb crashes on query with limit after sort with case
- 6514: Sqlitelogictest: Range query between NULL values not possible
- 6515: Insert null second interval value results in 0
- 6516: Sqlitelogictest unknown bat append operation
- 6517: Sqlitelogictest overflow in conversion during MAL plan execution
- 6518: Sqlitelogictest: count aggregation with not in operator
- 6519: Sqlitelogictest: algebra join between lng and int BATs undefined
- 6520: UPDATE with correlated subquery causes assertion (or segfault)
- 6522: Sqlitelogictest: IN operator return a single column
- 6523: Sqlitelogictest: Case statement subquery missing
- 6524: Sqlitelogictest: Crash in aggregation query with IN operator
- 6527: Crash using order by alias in subquery
Jul 2017-SP3 bugfix release (11.27.11)
MonetDB Common
- Reimplemented summing of a column of floating point (flt and dbl) values. The old code could give wildly inaccurate results when adding up lots and lots of values due to lack of precision. Try SELECT sum(c) FROM t; where t is 100,000,000 rows, c is of type REAL and all values are equal to 1.1. (The old code returned 33554432 instead of 1.1e8.)
Bug Fixes
- 3898: Deadlock on insertion
- 6429: ROUND produces wrong data type
- 6436: Query sequence with 2x ifthenelse() and next nullif() causes mserver5 Segmentation fault
- 6437: System schemas "profiler" and "json" shouldn't be allowed to be dropped.
- 6439: Invalid references to sys.columns.id from sys.statistics.column_id
- 6442: SEGFAULT with COPY INTO BEST EFFORT and skipping input columns
- 6443: complex(?) query forgets(?) column name
- 6444: Using 'with' keyword with table returning function crashes monetdb
- 6445: Sqlitelogictest crash in MySQL query
- 6446: sql_parser.y bug?
- 6448: 'insert into' with multiple rows containing subqueries crashes
- 6449: Assertion error in rel_dce_refs (sqlsmith)
- 6450: Assertion error in exp_bin (sqlsmith)
- 6451: Assertion error in sql_ref_dec (sqlsmith)
- 6453: Assertion error in rel_rename_exps (sqlsmith)
- 6454: SQL lexical error
- 6455: Assertion error in rel_apply_rewrite (sqlsmith)
- 6456: NULL becomes 0 in outer join
- 6459: Assertion error in exp_bin (sqlsmith)
- 6462: large virtual memory spike on BLOB column select
- 6465: appending to variables sized atom bats other than str bats with force flag may result in corrupted heap
- 6467: date_to_str formatter is wrong
- 6470: mitosis gets in the way of simple select
- 6471: calls to sys.generate_series should auto-convert arguments
- 6472: Assertion failure in rel_rename (Sqlsmith)
- 6477: assertion eror rel_push_project_up (sqlsmith)
- 6478: Crash with nested order by/ limit offset
- 6479: Mserver receives an assertion error on a procedure call
- 6480: Segfault in mvc_find_subexp (sqlsmith)
Jul 2017-SP2 bugfix release (11.27.9)
MonetDB Common
- A serious bug, possibly resulting in database corruption, having to do with appending data to a string BAT was fixed.
Bug Fixes
- 4017: server crashes when executing particular loopback query in embedded python
- 6207: identifier ambiguous when grouping and selecting the same column twice
- 6239: Incorrect profiling
- 6261: New handling of delta tables hurts badly reusage of bats
- 6287: should the CORR function return some numeric type that allows fractions?
- 6321: Two-column aggregation on join result extremely slow.
- 6335: Sqlitelogictest crash in complex SQL query
- 6343: MERGE TABLE issue: unable to find
- 6348: Interference of procedure/table name
- 6350: Carriage return and form feed in TEXT fields are returned as 'r' and 'f' in jdbcclient and ResultSets
- 6352: Scope resolution problem (sqlsmith)
- 6353: implicit NULL value not propagated in distributed/remote query
- 6374: Wrong answer from merge table after content changes
- 6379: Table UDF: SEGV raised when invoking a non existing function
- 6380: unable to create new databases from clean installation
- 6381: Parser misses error messages in conditional
- 6382: Can't set JSON fields via PreparedStatement
- 6384: crash when setting a wrong listenaddr
- 6385: AGGREGATE UDFs with more than 2 parameters incorrectly processed
- 6386: Unexpected error from server for query with long floats
- 6387: Performance degradation on multi column sort
- 6388: JDBC Connection via user voc produces errors when fetching certain meta data information
- 6392: SELECT EXISTS (empty table) returns 'true'
- 6395: BAT leak of scalar result sets
- 6397: Isolation of generating functions not correct
- 6398: Null Matches in outer join are not supported
- 6399: UDF crashes when subquery and scalar values are passed as pameters
- 6400: getCharacterStream() currently not supported
- 6404: COPY INTO crashes if table has primary key or foreign key constraint
- 6405: Compilation error if DISABLE_PARENT_HASH not defined
- 6409: sqllogictest crash on aggregation query with NOT IN clause in HAVING clause
- 6410: Sqlitelogictest crash on aggregation query with IN clause
- 6411: Sqlitelogictest crash in aggregation query
- 6412: Overflow in sys.epoch
- 6413: Wrong answer for TPC-H Q17
- 6414: Using RPAD returns: could not allocate space
- 6416: Sqlitelogictest crash in aggregation query with a NOT IN clause
- 6417: Segfault encountered (sqlsmith)
- 6418: Segfault in renaming (sqlsmith)
- 6419: segfault in rel_optimizer (sqlsmith)
- 6420: Assertion error in mergetable task (sqlsmith)
- 6422: Another assertion error in rel_or (sqlsmith)
- 6423: Dereference null pointer (sqlsmith)
- 6424: Assertion error in rel_rename_expr (sqlsmith)
- 6425: Assertion error in exp_bin (sqlsmith)
- 6426: Assertion error in rel_find_exp_ (sqlsmith)
- 6427: Assertion error in eq_typeswitchloop (sqlsmith)
- 6428: Sqlitelogictest crash in aggregation query
- 6430: Assertion raised in another eq_typeswitch error (sqlsmith)
- 6431: Sqlitelogictest crash in aggregation query with a long having clause
- 6432: Assertion error in exp_bin (sqlsmith)
- 6433: Sqlitelogictest crash in complex SELECT query with IN operator
- 6435: Sqlitelogictest crash in simple select query
- 6437: System schemas "profiler" and "json" shouldn't be allowed to be dropped.
- 6438: Implement functionality to enforce the restrict option in: DROP SCHEMA xyz RESTRICT;
- 6440: Faulty plan generated. Query returns more rows than expected or existing in the view sys.tables.
Important: we found a bug in the Jul2017 and Jul2017-SP1 releases (only) that can corrupt your database. We recommend to not upgrade to these two releases, but to skip directly to the Jul2017-SP2 release. See this email message.
Jul 2017-SP1 bugfix release (11.27.5)
Debian and Ubuntu uses may want to reimport the MonetDB GPG public key as per the instructions on the download page. See bug 6383 for more information.
Build Environment
- The Debian and Ubuntu installers have been fixed: there was a file missing in the Jul2017 release.
- Added a new RPM called MonetDB-selinux which provides the SELinux policy required to run MonetDB under systemd, especially on Fedora 26.
- The Windows installers (*.msi files) are now created using the WiX Toolset.
- The Windows binaries are now built using Visual Studio 2015. Because of this, you may need to install the Visual C++ Redistributable for Visual Studio 2015 before being able to run MonetDB.
Merovingian
- monetdbd was leaking open file descriptors to the mserver5 process it started. This has been fixed.
MonetDB Common
- Many functions in GDK are now annotated with the GCC attribute __warn_unused_result__ meaning that the compiler will issue a warning if the result of the function (usually an indication of an error) is not used.
Bug Fixes
- 6325: Merge table unusable in other connections
- 6328: Transactional/multi-connection issues with merge tables
- 6336: VALUES multiple inserts error
- 6339: Mserver5 crashes on nested SELECT
- 6340: sample operator takes effect after the execution of the query, expected before
- 6341: MERGE TABLE issue: Cannot register
- 6342: MERGE TABLE issue: hang
- 6344: Spurious errors and assertions (SQLsmith)
- 6375: MAL profiler truncates JSON objects larger than 8192 characters
Jul 2017 feature release (11.27.1)
On Fedora 26 there is a known issue with this release. When you use MonetDB under systemd (if you have enabled the system service with systemctl enable monetdbd.service), and you have SELinux (Security-Enhanced Linux) enabled, the database server processes will not start when you try to connect.
MonetDB5 Server
- The "sub" prefix of many functions, both at the MAL and the C level, has been removed.
- Changed the interfaces of the AUTH* functions: pass values, not pointers to values.
- Removed calc.setoid().
- group.subgroup is now called group.group if it is not refining a group. Both group.group and group.subgroup now also have variants with a candidate list.
- The allocation schemes for MAL blocks and Variables has been turned into block-based. This reduces the number of malloc()/free() calls.
.
- Added a new server-side protocol implementation. The new protocol is backwards compatible with the old protocol. Clients can choose whether they want to use the old or the new protocol during the initial handshake with the server. The new protocol is a binary column-based protocol that is significantly faster than the old protocol when transferring large result sets. In addition, the new protocol supports compression using Snappy or LZ4.
- Moved the sphinx extension module to its own repository. See https://dev.monetdb.org/hg/MonetDB-sphinx/.
- Removed GSL module: it's now a separate (extension) package. See https://dev.monetdb.org/hg/MonetDB-gsl/.
- The PCRE library is now optional for systems that support POSIX regular expressions.
- Added 5 new sys schema tables: function_languages, function_types, key_types, index_types and privilege_codes. They are pre-loaded with static content and contain descriptive names for the various integer type and code values. See also sql/scripts/51_sys_schema_extension.sql
Merovingian
- Added handling of a dbextra property per database at the daemon level. The user can set the dbextra property for a database using the command: $ monetdb set dbextra=<path> <database> and the daemon will make sure to start the new server using the correct --dbextra parameter.
Client Package
- The mclient and msqldump programs lost compatibility with old mserver5 versions (pre 2014) which didn't have a "system" column in the sys.schemas table.
- The mclient and msqldump programs lost compatibility with ancient mserver5 versions (pre 2011) which didn't have the sys.systemfunctions table.
- Removed the "array" and "quick" functions from the mapi library. To be precise, the removed functions are: mapi_execute_array, mapi_fetch_field_array, mapi_prepare_array, mapi_query_array, mapi_quick_query, mapi_quick_query_array, and mapi_quick_response.
- Added a more elaborate \help command for SQL expressions.
MonetDB Common
- Improved error checking in the logger code (dealing with the write-ahead log); changed return types a several functions from int to gdk_return (i.e., they now return GDK_SUCCEED or GDK_FAIL). The logger no longer calls GDKfatal on error. Instead the caller is responsible for dealing with errors.
- BATsort may now create an order index as a by product.
- Quantile calculations now use the order index if available (and use BATsort otherwise, producing an order index).
- Quantiles calculate a position in the sorted column. If this position is not an integer, we now choose the nearest position, favoring the lower if the distance to the two adjacent positions is equal (round down to nearest integer).
- Removed function BATprintf. Use BATprint or BATprintcolumns instead.
- Removed BATsave from the list of exported functions.
- Replaced BBPincref/BBPdecref with BBPfix/BBPunfix for physical reference count and BBPretain/BBPrelease for logical reference count maintenance.
- Removed automatic conversion of 32-bit OIDs to 64 bits on 64-bit architectures.
- Removed functions OIDbase() and OIDnew().
- Removed talign field from BAT descriptor.
- BATappend now takes an optional (NULL if not used) candidate list for the to-be-appended BAT.
- New function BATkeyed(BAT *b) that determines (possibly using a hash table) whether all values in b are distinct.
SQL
- Made the operator precedence of % equal to those of * and /. All three are evaluated from left to right.
- Removed table sys.connections. It was a remnant of an experimental change that had already been removed in 2012.
- Protect against runaway profiler events If you hit a barrier block during profiling, the JSON event log may quickly become unwieldy. Event production is protected using a high water mark, which ensures that never within the single execution of MAL block the instruction causes excessive event records.
Bug Fixes
- 3465: Request: add support for CREATE VIEW with ORDER BY clause
- 3545: monetdb commands don't work with -h -P -p options (locally and remotely)
- 3996: select * from sys.connections always returns 0 rows. Expected to see at least one row for the active connection.
- 6187: Nested WITH queries not supported
- 6225: Order of evaluation of the modulo operator
- 6289: Crashes and hangs with remote tables
- 6292: Runaway SQL optimizer in too many nested operators
- 6310: Name resolution error (sqlsmith)
- 6312: Object not found in LIMIT clause (sqlsmith)
- 6313: Null type resolution in disjunction fails (sqlsmith)
- 6319: Server crash on LATERAL (sqlsmith)
- 6322: Crash on disjunction with LIMIT (sqlsmith)
- 6323: Deadlock calling sys.bbp()
- 6324: Sqlitelogictest crash in a IN query (8th)
- 6327: The daemon does not respect the actual name of the mserver5 executable
- 6330: Sqlitelogictest crash on a complex SELECT query
- 6331: sys.statistics column "nils" always contains 0. Expected a positive value for columns that have one or more nils/NULLs
- 6332: Sqlitelogictest crash related to an undefined MAL function
December 2016
December 2016 sjoerd Mon, 07/17/2017 - 15:49Dec 2016-SP5 bugfix release
Bug Fixes |
|
Dec 2016-SP4 bugfix release
MonetDB5 Server |
|
MonetDB Common |
|
SQL |
|
Bug Fixes |
|
Dec 2016-SP3 bugfix release
MonetDB Common |
|
Bug Fixes |
|
Dec 2016-SP2 bugfix release
Bug Fixes |
|
Dec 2016-SP1 bugfix release
Bug Fixes |
|
Dec 2016 feature release
There are lots of changes in this (Dec2016) release and the previous (Jun2016). Here is an outline.
- The client interfaces that are not written in C, that is to say, the Java (JDBC driver), Perl, PHP, Python, and Ruby interfaces all now have their own repository and are no longer part of the main MonetDB repository. (Actually, the Ruby interface was already separate in the Jun2016 release.) These client interfaces are still supported but now have an independent release schedule. Their first releases are synchronized with this release. Debian and Ubuntu users, please note that the package names for the client interfaces have changed. If you have the old versions installed, they will not get updated automatically. Use: sudo apt-get install libdbd-monetdb-perl php-monetdb python-pymonetdb python3-pymonetdb removing the ones from the list above you don't want.
- For years we have been working towards a version that we've internally called "headless". This is now complete. This has had consequences for quite a few internal interfaces, also ones that could potentially be used by extensions written by third parties. The trickiest one is probable BATkey() which has the same calling interface but does something different. The compiler should complain about other functions since their arguments changed or the functions were completely removed.
- In the Jun2016 release it was possible to have user-defined functions (UDF) callable from SQL that were written in the R language. In this release, the same is now possible for UDFs written in Python. Also see this blog.
- The possibility of having 32 bit OID values in a system that was compiled for a 64 bit architecture has been removed. The size of the OID type is now always the same as the "natural" size of the architecture. This was a compile-time feature and we haven't made any binary releases in a long time in which this feature was used. The automatic conversion from a database in which the size of the OID type was 32 to one in which the size is 64 is still in place. This will be removed in a future version.
- The internal type "wrd" has been removed. The "wrd" type was an architecture-dependent integer type, 32 bits wide on a 32-bit system and 64 bits wide on a 64-bit system. Databases where this type is used will automatically have the type converted to either "int" or "lng", the internal types for 32-bit and 64-bit integers.
- Some old and unused code and interfaces have been removed.
- By making some adjustments to internal parameters, we have been able to get a significant improvements on query speed for many types of queries (in informal tests we have seen a 20% improvement compared to the previous release when running the queries from the TPC-H benchmark).
- A new type of experimental column index has been introduced: the ordered index. Since creating this index is expensive, it is only used when explicitly created. To create an index in SQL use CALL createorderindex(schema,table,column) where the procedure arguments are all strings.
- We have fixed quite a few memory leaks and other bugs.
MonetDB5 Server |
|
Build Environment |
|
. |
|
MonetDB Common |
|
SQL |
|
Bug Fixes |
|
June 2016
June 2016 panos Wed, 12/21/2016 - 12:37Jun 2016-SP2 bugfix release
Build Environment |
|
Bug Fixes |
|
Jun 2016-SP1 bugfix release
. |
|
Java Module |
|
Bug Fixes |
|
Jun 2016 feature release
MonetDB5 Server |
|
Build Environment |
|
Client Package |
|
Geom Module |
|
MonetDB Common |
|
Java Module |
|
SQL |
|
Bug Fixes |
|
July 2015
July 2015 sjoerd Thu, 06/23/2016 - 09:49Jul 2015-SP4 bugfix release
MonetDB Common |
|
Bug Fixes |
Jul 2015-SP3 bugfix release
MonetDB5 Server |
|
Bug Fixes |
|
Jul 2015-SP2 bugfix release
Bug Fixes |
|
Jul 2015-SP1 bugfix release
Client Package |
|
Bug Fixes |
|
Jul 2015 feature release
Do not skip upgrades! If you are running a release older than the Oct2014 series (11.19.X), do not skip upgrading to the Oct2014 release before upgrading to the Jul2015 release. In fact, you should first upgrade to the Oct2014-SP2 (11.19.9). Don't forget to start the server and to connect a client to the server for each of your databases.
One of the new features in this release is the possibility to use "best effort" when importing CSV files into the server. However, we have found a bug that causes this feature to misbehave when the CSV file cannot be read correctly (exactly when one might want to use best effort). Hence we recommend to not use the feature. See bug 3797.
MonetDB5 Server |
|
Build Environment |
|
Client Package |
|
MonetDB Common |
|
Java Module |
|
SQL |
|
Bug Fixes |
|
October 2014
October 2014 sjoerd Fri, 08/28/2015 - 13:11Important consideration when upgrading to this release. In this release we have fixed bug 3474. This bug made it possible that there were duplicates despite there being a unique or primary key constraint present. This could only happen for multi-column constraints. MonetDB maintains an internal column containing a hash for multi-column constraints, and there was a bug in the maintenance code of this column. During an upgrade, this column needs to be recreated. This is done by dropping and adding back the constraints. If there are no duplicates, adding back the constrains should work without problem, but if there are duplicates, adding the constraints will fail. There is unfortunately no easy way to tell the user, so after having made the first connection to the server (when the upgrade happens), check the merovingian log (if you use merovingian -- also known as monetdbd) or the server output window (if you don't use merovingian).
Jaql support is removed. The experimental code for Jaql support has been removed. If you have RPM or DEB packages for Jaql installed from an earlier version (MonetDB-jaql and monetdb5_jaql respectively), you will need to uninstall them before the upgrade.
Upgrade code fixed. The code to upgrade from previous releases still had some bugs. These have now been fixed. Upgrades directly from the Feb2013 release should work again. Also, upgrading to this version should fix some problems that were caused by the upgrade to the Oct2014 release.
Oct 2014-SP4 bugfix release
Due to a code conflict it is not possible to upgrade to this bug fix release from a release older than the Oct2014 release. If you are running an older release, first upgrade to the Oct2014 release, and then to Oct2014-SP4.
Bug Fixes |
|
Oct 2014-SP3 bugfix release
Build Environment |
|
MonetDB Common |
|
Bug Fixes |
|
Oct 2014-SP2 bugfix release
Client Package |
|
SQL |
|
Bug Fixes |
|
Oct 2014 SP1 bugfix release
MonetDB5 Server |
|
Build Environment |
|
MonetDB Common |
|
SQL |
|
Bug Fixes |
|
Known issue when upgrading old versions. We have noticed that upgrading directly from a version older than the Jan2014 release gave an error. Upgrading to Jan2014 first and then on to Oct2014 should work.
Backup. As always, make a backup of your databases before upgrading.
Oct 2014 feature release
MonetDB5 Server |
|
Build Environment |
|
Merovingian |
|
mapilib |
|
stream |
|
MonetDB Common |
|
SQL |
|
Bug Fixes |
|
January 2014
January 2014 sjoerd Fri, 10/31/2014 - 13:58
Jan 2014-SP3 bugfix release |
|
Build Environment |
|
Bug Fixes |
|
Jan 2014-SP2 bugfix release |
|
Build Environment |
|
Client Package |
|
Bug Fixes |
|
Jan 2014-SP1 bugfix release |
|
Build Environment |
|
Merovingian |
|
Bug Fixes |
|
Jan 2014 feature release |
|
Build Environment |
|
Client Package |
|
python2 |
|
python3 |
|
R |
|
MonetDB Common |
|
SQL |
|
Bug Fixes |
|
Februari 2013
Februari 2013 sjoerd Thu, 02/20/2014 - 14:40
Feb 2013 feature release |
|
Testing Environment |
|
Java Module |
|
Client Package |
|
MonetDB5 Server |
|
Merovingian |
|
Bug Fixes |
|
Feb 2013-SP1 bugfix release |
|
Bug Fixes |
|
In addition to these changes, we also fixed the database upgrade. Due to internal changes in the Feb2013 release, some changes had to be made to the SQL catalog. These changes were done incompletely in the Feb2013 release. This has now been fixed in the Feb2013-SP1 release. Both a direct upgrade from Oct2012 and from Feb2013 should work.
Feb 2013-SP2 bugfix release |
|
MonetDB5 Server |
|
MonetDB Common |
|
Java Module |
|
SQL |
|
Bug Fixes |
|
Feb 2013-SP3 bugfix release |
|
MonetDB Common |
|
Java Module | |
Bug Fixes |
|
Feb 2013-SP4 bugfix release |
|
Merovingian |
|
Bug Fixes |
|
Feb 2013-SP5 bugfix release |
|
Client Package |
|
MonetDB Common | |
Java Module |
|
Bug Fixes |
|
Feb 2013-SP6 bugfix release |
|
MonetDB5 Server |
|
Merovingian |
|
Client Package |
|
MonetDB Common |
|
SQL |
|
Bug Fixes |
|
October 2012
October 2012 sjoerd Tue, 02/19/2013 - 13:12
Oct 2012 feature release |
|
Build Environment |
|
Java Module |
|
Client Package |
|
MonetDB5 Server |
|
Merovingian |
|
MonetDB Common |
|
Bug Fixes |
No Debian and Ubuntu packages were built for this release (i.e. Oct2012) due to a packaging failure. We expect to have resolved that problem by Oct2012-SP1. We apologise for the inconvenience.
Oct 2012-SP1 bugfix release |
|
Bug Fixes |
For this release (i.e. Oct2012-SP1) there are once again Debian and Ubuntu packages. From this release onward, Ubuntu Lucid Lynx (10.04) is no longer supported.
Oct 2012-SP2 bugfix release |
|
SQL |
|
Java Module |
|
Bug Fixes |
|
Oct 2012-SP3 bugfix release |
|
Bug Fixes |
|
July 2012
July 2012 fabian Tue, 10/23/2012 - 11:34
Jul 2012 feature release |
|
Build Environment |
|
SQL |
|
Client Package |
|
MonetDB5 Server |
|
Merovingian |
|
MonetDB Common |
|
Bug Fixes |
|
Jul 2012-SP1 bugfix release |
|
Java Module |
|
Client Package |
|
MonetDB5 Server |
|
Merovingian |
|
Bug Fixes |
|
Jul 2012-SP2 bugfix release |
|
Java Module |
|
MonetDB5 Server |
|
Merovingian |
|
MonetDB Common |
|
Bug Fixes |
|
April 2012
April 2012 mk Thu, 04/28/2011 - 17:47
Apr 2012-SP2 bugfix release |
|
Client Package |
|
MonetDB Common |
|
Bug Fixes |
|
Apr 2012-SP1 bugfix release
Build Environment |
|
SQL |
|
Merovingian |
|
MonetDB Common |
|
Bug Fixes |
|
Apr 2012 feature release |
|
Build Environment |
|
SQL |
|
Java Module |
|
Client Package |
|
MonetDB5 Server |
|
Merovingian |
|
MonetDB Common |
|
Bug Fixes |
|
December 2011
December 2011 sjoerd Fri, 04/27/2012 - 14:22MonetDB Dec2011 Release
Dec 2011-SP2 bugfix release |
|
Java Module |
|
Bug Fixes |
|
Dec 2011-SP1 bugfix release |
|
Build Environment |
|
SQL |
|
Java Module |
|
MonetDB5 Server |
|
Merovingian |
|
Bug Fixes |
|
Dec 2011 feature release |
|
SQL |
|
Java Module |
|
Client Package |
|
MonetDB5 Server |
|
Merovingian |
|
MonetDB Common |
|
Bug Fixes |
|
August 2011
August 2011 sjoerd Thu, 01/26/2012 - 15:43The Aug2011 release is the first release where the default size of OIDs on 64-bit WIndows systems is 64 bits. Before, the default on Windows was 32 bits. On all other 64-bit platforms, the default size of OIDs was also 64 bits. This means that, on 64-bit Windows systems, the database needs to be converted. This is done automatically when the database is first opened, but as always with automatic conversions, it is a good idea to make a backup first.
The Aug2011 uses a slightly different catalog from earlier releases. The database is automatically updated when first opened.
This release is the last release that comes with a JDBC driver that can be used with Java 1.4 and 1.5, also known as JDBCv3.
Aug 2011-SP3 bugfix release |
|
SQL |
|
Merovingian |
|
Bug Fixes |
|
Aug 2011-SP2 bugfix release |
|
Client Package |
|
Merovingian |
|
Bug Fixes |
|
Aug 2011-SP1 bugfix release |
|
SQL |
|
Client Package |
|
MonetDB5 Server |
|
Bug Fixes |
|
Aug 2011 feature release |
|
Build Environment |
|
Java Module |
|
Client Package |
|
MonetDB5 Server |
|
Geom Module |
|
Merovingian |
|
MonetDB Common |
|
Bug Fixes |
|
April 2011
April 2011 mk Sun, 06/26/2011 - 11:12This is the first release of the MonetDB suite that does not contain MonetDB4 and MonetDB/XQuery. As already mentioned in the Mar2011 release notes, work on the Pathfinder compiler, which forms the basis of MonetDB/XQuery, continues at The University of Tübingen.
An upgrade from the Mar2011 release should be smooth, even if MonetDB/XQuery is installed. The Windows installers for MonetDB/SQL and MonetDB/XQuery have always been independent, and the Fedora, Debian, and Ubuntu versions have been made independent in the Mar2011 release. The Apr2011 release uses a slightly changed SQL catalog than previous releases. The catalog will be updated automatically the first time an SQL database is opened. Note that because of the incompatibility, once updated, the database cannot be read by older versions. Also note that a database from the Feb2010 release or before cannot be read by this release. First update the database with the Mar2011 release. This is also the first release where we have created FreeBSD binaries. These binaries are highly experimental. The software should run as well as on any other system, but the packaging is obviously different from other systems. As before, we also provide binaries for Windows, Fedora (13 and 14), Debian (5.0 and 6.0), Ubuntu (9.10, 10.04, and 10.10), Mac OS X, and Solaris.
Apr 2011 feature release |
|||||||||||||||||||||||||
Client Package |
Plugged a small memory leak occurring upon redirects by the server (e.g. via monetdbd) |
||||||||||||||||||||||||
MonetDB Common | Fixed bugs in antiselect which gave the incorrect result when upper and lower bound were equal. This bug could be triggered by the SQL query SELECT * FROM t WHERE x NOT BETWEEN y AND y | ||||||||||||||||||||||||
Java Module | Clarify exception messages for unsupported methods Removed XQuery related XRPC wrapper and XML:DB code, removed support for language=xquery and language=mil from JDBC. |
||||||||||||||||||||||||
MonetDB5 Server |
Mserver5 no longer reads monetdb5.conf upon startup by default. Use --config=file to have mserver5 read a configuration on startup | ||||||||||||||||||||||||
SQL | Some names in the SQL catalog were changed. This means that the database in the Apr2011 release is not compatible with pre-Apr2011 databases. The database is converted automatically when opened the first time. This database can then no longer be read by an older release. | ||||||||||||||||||||||||
Merovingian | Fix monetdb return code upon failure to start/stop a database. The forward property for databases has been removed. Instead, only a global proxy or redirect mode can be set using monetdbd. Monetdbd can no longer log error and normal messages to separate logfiles, logging to stdout and stderr is no longer possible either. The .merovingian_pass file is no longer in use, and replaced by the .merovingian_properties file. Use monetdbd (get|set) passphrase to view/edit the control passphrase. Existing .merovingian_pass files will automatically be migrated upon startup of monetdbd. Monetdbd now understands commands that allow to create, start, stop, get and set properties on a given dbfarm. This behaviour is intended as primary way to start a MonetDB Database Server, on a given location of choice. monetdbd get and set are the replacement of editing the monetdb5.conf file (which is no longer in use as of the Apr2011 release). See monetdbd(1). |
||||||||||||||||||||||||
Bug Fixes |
|
March 2011
March 2011 mk Sun, 05/15/2011 - 12:00MonetDB Mar2011 release
This is the last planned release for MonetDB4 and MonetDB/XQuery. MonetDB4 has served us well, but we need to move on. MonetDB/XQuery, being implemented on top of MonetDB4, is a casualty of this decision. The work on the Pathfinder compiler, which forms the basis of MonetDB/XQuery, continues at The University of Tübingen.
As of this "Mar2011" release (version 11.1.1), all components of MonetDB have been gathered into one single package, mainly to simplify the compilation process. If you're using the Mercurial clone, use the Mar2011 branch and the Mar2011_1 tag.
Contents
Mar 2011 feature release
The packaging of MonetDB has been completely overhauled. In the past, there were various subpackages that had to be configured and built individually, but that had to be done in the correct order. From this release onward, the process of building MonetDB has been greatly simplified. There is only a single configuration/build/install cycle for the whole suite. On Linux/Unix systems this means one call to configure (when building from the Mercurial sources this has to be preceded by one call to the bootstrap script), one call to make, and one call to make install. On Windows, the bootstrap and configure stages can be skipped (and you need to use nmake instead of make).
The directory structure of the source tree was also completely overhauled. The MonetDB4 and MonetDB5 directories were renamed monetdb4 and monetdb5 respectively, the MonetDB directory was renamed gdk, src subdirectories were removed, and several subsystems were moved to the new common directory.
The version numbers have changed since we now have a single package. The version number of this release is 11.1.1.
Linux
We now package for Fedora 13 and 14, for Debian 5.0 (Lenny) and 6.0 (Squeeze), and for Ubuntu 9.10 (Karmic Koala), 10.04 (Lucid Lynx), and 10.10 (Maverick Meerkat).
On Debian and Ubuntu, updating to this release is a bit more involved than simply using apt-get upgrade or the "Mark All Upgrades" button in the Synaptic Package Manager. When using these commands, they will "keep back" the monetdb packages, so in order to actually update them you will need to give the apt-get install command and specify the relevant monetdb packages, or in Synaptic first select the relevant monetdb packages and then select Package->Mark for Upgrade.
Windows
Binaries are available.
Mac OS X
Binaries are available.
Solaris
Binaries are available.
- Fixed memory detection on Darwin (Mac OS X) systems not to return bogus values
- MonetDB/src/gdk was moved to gdk
MonetDB Server 4
MonetDB Server 5
- sphinx module: update, adding limit/max_results support
- Improve the performance of remote.put for BAT arguments. The put speed is now roughly equal to the speed of get on a BAT.
- Fix bug #2648, do not allow restarting a sequence with NULL via the result of a sub-query.
- Allow clients to release prepared handles using Xrelease commands
- Fixed bug 2677: SQL_DESC_OCTET_LENGTH should give the size in bytes required to copy the data.
- Disable timer functionality for non-XQuery languages since it is incorrect, bug #2705
- Added mapi_get_uri function to retrieve mapi URI for the connection
- Allow to dump table data using INSERT INTO statements, rather than COPY INTO + CSV data using the -N/--inserts flag of mclient and msqldump. Bug #2727
- Added support for \dn to list schemas or describe a specific one
- Added support for \df to list functions or describe a specific one
- Added support for \ds to list sequences or describe a specific one
- Added support for wildcards * and ? in object names given to \d commands, such that pattern matching is possible, e.g. \d my*
- Added support for \dS that lists also system tables
- object names given to \d are now lowercased, unless quoted by either single or double quotes
- Strip any trailing whitespace with the \d command
- Set endianness for wkb en/decoding.
- PreparedStatements now free the server-side resources attached to them when closed. This implements bug #2720
The following bug tracker items were fixed since the previous "Oct 2010" release:
- 104: SQL: extremely long and complex query causes SEGFAULT
- 2351: SQL: transaction aborts on prepared statements in autocommit
- 2358: Unrelevant error message when copying bad length field
- 2586: times.sql: procedure times() crashes the server
- 2593: test sql/src/test/Tests/load_dec_as_int.sql.src fails a file sql/src/test/load_dec_as_int.sql is missing
- 2609: insert into large number of columns
- 2634: !ERROR: CTrefine: both BATs must have the same cardinality ...
- 2640: VOC test fails multi-threaded
- 2647: M5 breaks with huge SQL query.
- 2648: ALTER SEQUENCE ... RESTART WITH (SELECT NULL) NO CYCLE is accepted
- 2673: select a string column from a view over a table with primary key crashes
- 2677: SQLColAttribute using SQL_DESC_OCTET_LENGTH doesn't return the maximum length
- 2679: timestamp selection based on now() takes too long
- 2680: make clean too eager
- 2698: test sql/src/test/bugs/Tests/except-union-intersect-bug-sf-1146079.SQL.sh produces wrong result with multi-threaded execution
- 2704: temporary tables on readonly mode
- 2705: Timer information problems
- 2708: function with different value for arguments crashes mserver
- 2710: implement \d{t,v,s,S,...} to describe other (or only) objects in the database
- 2713: wrong error message when dropping an un-existing view with the same as an existent table.
- 2715: Join with 2 AND-ed conditions where one is LIKE/ILIKE returns too many results
- 2717: function call in readonly mode.
- 2720: Allow a client to close a prepared handle
- 2725: mclient/mapilib aborts during connect
- 2726: join on two literal tables leads to
- 2727: allow dumping database with INSERT INTO statements
- 2729: mserver5: ../../../src/server/rel_bin.mx:537: exp_bin: Assertion `0' failed.
- 2730: select * from sequence crashes mserver5
- 2733: MAL parser accepts 1 + 1 + 1, MAL interpreter only computes 1 + 1
- 2735: complex SQL query leads to multiple TypeExceptions
- 2737: Sphinx module update, adding limit/max_results support
- 2738: MonetDB 'allows' binding and connecting to ports above 65535 (has patch)
- 2743: mapi_next_result skips over results (losing output)
- 2744: UNLOCKED causes mserver to work indefinitely
- 2745: foreign key type mismatch causes MAL stacktrace
- 2748: COPY INTO does not work correctly with backslashes in VARCHARs
- 2750: Function arguments of type string described not completely in SQL catalog
- 2752: CALL function without warning
- 2753: check existence of external names for SQL functions/procedures at creation time
- 2757: ORDER BY DESC + LIMIT incorrectly yields empty result
- 2768: "no such column 'u.name'" in users system view
- 2773: starting database 'test'... FAILED
- 2786: crash when first connection is with another user than monetdb
- 2788: stop -a complains about already stopped databases
- 2790: View returns erroneous result for squared average function as compared to query
The detailed list of changes can be found in the source code Mercurial logs.
June 2010
June 2010 sjoerd Thu, 07/01/2010 - 16:54MonetDB Jun2010 release
Release "Jun2010" consists of the following updated components. If you're using the Mercurial clone, use the Jun2010 branch and the Jun2010_1 tag.
Name | Version | Subdirectory | Note/Comment |
MonetDB Common | 1.38.1 | MonetDB | |
MonetDB Clients | 1.38.1 | clients | |
MonetDB4 Server | 4.38.1 | MonetDB4 | |
MonetDB4/XQuery | 0.38.1 | pathfinder | |
MonetDB5 Server | 5.20.1 | MonetDB5 | |
MonetDB5/SQL | 2.38.1 | sql | |
MonetDB5/SQL/GIS | 0.18.1 | geom | |
MonetDB Java | 1.38.1 | java | |
MonetDB Testing | 1.38.1 | testing | (developers only) |
NOTE:
Before upgrading to the latest "Jun2010" release from a release of MonetDB older than the Nov2008 release, please make a dump (SQL, XQuery) of your database, and rename/move/backup the database directories (dbfarm, sql_logs, xquery_logs). After the upgrade, restore (SQL, XQuery) your database, again. Once you have checked and confirmed the correctness of the restore, you can delete the above renamed/moved/backed-up old database directories.
When upgrading from a release of MonetDB older than the May2009 release, you need to move your write-ahead log following the "recommended" instructions for SQL and/or XQuery.
Contents
Jun 2010 feature release
Packaging
Linux
We now package for Fedora 11, 12, and 13, for Debian 5.0 (Lenny), and for Ubuntu 9.10 (Karmic Koala) and 10.04 (Lucid Lynx).
MonetDB Common
- Updated Vendor information.
- Made compilation of "testing" (and "java") independent of MonetDB. This is mainly for Windows, but also on other systems, "testing" can now be built independently of (and hence before) "MonetDB". Files that mimic configure functionality on Windows were moved from "MonetDB" to "buildtools"; hence, this affects all packages on Windows, requiring a complete rebuild from scratch on Windows. getopt() support in testing has changed; hence, (most probably) requiring a rebuild from scratch of testing on other systems.
- Implemented build directory support for Windows, i.e., like on Unix/Linux also on Windows we can now build in a separate build directory as alternative to ...<package>NT, and thus keep the latter clean from files generated during the build. On Windows, the build directory must be a sibling of ...<package>NT .
MonetDB Server 4
- Updated Vendor information.
- Made compilation of "testing" (and "java") independent of MonetDB. This is mainly for Windows, but also on other systems, "testing" can now be built independently of (and hence before) "MonetDB". Files that mimic configure functionality on Windows were moved from "MonetDB" to "buildtools"; hence, this affects all packages on Windows, requiring a complete rebuild from scratch on Windows. getopt() support in testing has changed; hence, (most probably) requiring a rebuild from scratch of testing on other systems.
- Implemented build directory support for Windows, i.e., like on Unix/Linux also on Windows we can now build in a separate build directory as alternative to ...<package>NT, and thus keep the latter clean from files generated during the build. On Windows, the build directory must be a sibling of ...<package>NT .
MonetDB Server 5
- When libxml2 is available, the XML module is automatically loaded.
- Updated Vendor information.
- Created a new RPM MonetDB5-server-rdf for the optional MonetDB/RDF module.
- Added the Z-order module to simplify future manipulation of arrays.
- fixed BUG #2994521 "mat.slice unable to cope with only empty BAT arguments" by making MATpackSliceInternal() handle empty input BATs correctly
- Select <col> from <t> limit <n> has been improved by introducing mat.slice().
- Made compilation of "testing" (and "java") independent of MonetDB. This is mainly for Windows, but also on other systems, "testing" can now be built independently of (and hence before) "MonetDB". Files that mimic configure functionality on Windows were moved from "MonetDB" to "buildtools"; hence, this affects all packages on Windows, requiring a complete rebuild from scratch on Windows. getopt() support in testing has changed; hence, (most probably) requiring a rebuild from scratch of testing on other systems.
- Implemented build directory support for Windows, i.e., like on Unix/Linux also on Windows we can now build in a separate build directory as alternative to ...<package>NT, and thus keep the latter clean from files generated during the build. On Windows, the build directory must be a sibling of ...<package>NT .
- The MAL debugger list command has been extended with an optional hash '#', which produces line numbers for each of reference and analysis of variable span.
- The dataflow scheduler has been revamped to allow for more parallelism to be exploited.
- The garbage collection administration has been changed. Every variable record now comes with an end-of-life field (eolife), which denotes the instruction after which the BAT variable reference counter can be decremented. The garbage collector is never called automagically on MAL functions, because there may be other pressing needs to retain them. This means that MAL functions defined and used in the context of SQL, and which are not inlined, may cause a leakage. The garbage collection has become part of each interpreter step. The new approach makes the SQL/MAL plans half the size as before.
- Removed stethoscope from MonetDB5 sources. New location is in the clients repository.
- Added the compression optimizer as an example of how to gain access to foreign file formats deep down in the kernel and transfer them just in time into a temporary BAT.
- Renamed configure argument --with-console to --enable-console. Default remains console being enabled.
- Fixed cleaning the user module context upon session end. Fixes bug #2956664
- The config variable mapi_usock can now be used to instruct the server to listen for connections on a local UNIX domain socket on UNIX-like systems.
- The MAL interpreter has been extended with an operation admission policy to control the memory claims of all concurrent running interpreters. Instructions are hold up unto there is sufficient resource or the query plan can not avoid its execution anymore.
- The join path optimizer has been extended with searching for join*, semijoin*, and leftjoin* paths. Furthermore, it avoids duplicate work by factoring out all common simple join paths.
SQL
- make it possible to use '?' in offset and limit
- Added include and .lib files to Windows installers that are needed to compile client programs.
- Updated Vendor information.
- Made compilation of "testing" (and "java") independent of MonetDB. This is mainly for Windows, but also on other systems, "testing" can now be built independently of (and hence before) "MonetDB". Files that mimic configure functionality on Windows were moved from "MonetDB" to "buildtools"; hence, this affects all packages on Windows, requiring a complete rebuild from scratch on Windows. getopt() support in testing has changed; hence, (most probably) requiring a rebuild from scratch of testing on other systems.
- Implemented build directory support for Windows, i.e., like on Unix/Linux also on Windows we can now build in a separate build directory as alternative to ...<package>NT, and thus keep the latter clean from files generated during the build. On Windows, the build directory must be a sibling of ...<package>NT .
XQuery
- Updated Vendor information.
- Made compilation of "testing" (and "java") independent of MonetDB. This is mainly for Windows, but also on other systems, "testing" can now be built independently of (and hence before) "MonetDB". Files that mimic configure functionality on Windows were moved from "MonetDB" to "buildtools"; hence, this affects all packages on Windows, requiring a complete rebuild from scratch on Windows. getopt() support in testing has changed; hence, (most probably) requiring a rebuild from scratch of testing on other systems.
- Implemented build directory support for Windows, i.e., like on Unix/Linux also on Windows we can now build in a separate build directory as alternative to ...<package>NT, and thus keep the latter clean from files generated during the build. On Windows, the build directory must be a sibling of ...<package>NT .
Client Packages
- Updated Vendor information.
- The MonetDB ODBC driver now no longer depends on its own copy of the standard ODBC include files but instead depends on system include files (unixODBC on Linux and Microsoft SDK on Windows).
- Made compilation of "testing" (and "java") independent of MonetDB. This is mainly for Windows, but also on other systems, "testing" can now be built independently of (and hence before) "MonetDB". Files that mimic configure functionality on Windows were moved from "MonetDB" to "buildtools"; hence, this affects all packages on Windows, requiring a complete rebuild from scratch on Windows. getopt() support in testing has changed; hence, (most probably) requiring a rebuild from scratch of testing on other systems.
- Implemented build directory support for Windows, i.e., like on Unix/Linux also on Windows we can now build in a separate build directory as alternative to ...<package>NT, and thus keep the latter clean from files generated during the build. On Windows, the build directory must be a sibling of ...<package>NT .
- Changed d output of mclient no longer to list (internal use only) system tables. Administrator users who like to inspect these tables can use SELECT * FROM tables; instead
- Implemented the changes required for properly running ODBC on a 64-bit platform.
- Added stethoscope, an utility to profile MonetDB5 instances, originally from the MonetDB5 repository.
Geom Module
- Updated Vendor information.
- Made compilation of "testing" (and "java") independent of MonetDB. This is mainly for Windows, but also on other systems, "testing" can now be built independently of (and hence before) "MonetDB". Files that mimic configure functionality on Windows were moved from "MonetDB" to "buildtools"; hence, this affects all packages on Windows, requiring a complete rebuild from scratch on Windows. getopt() support in testing has changed; hence, (most probably) requiring a rebuild from scratch of testing on other systems.
- Implemented build directory support for Windows, i.e., like on Unix/Linux also on Windows we can now build in a separate build directory as alternative to ...<package>NT, and thus keep the latter clean from files generated during the build. On Windows, the build directory must be a sibling of ...<package>NT .
February 2010
February 2010 mk Wed, 03/24/2010 - 11:27Release "Feb2010" consists of the following updated components
Name | Version | CVS module | CVS branch | Note/Comment |
MonetDB Common | 1.36.1 | MonetDB | Feb2010 | |
MonetDB Clients | 1.36.1 | clients | Feb2010 | |
MonetDB4 Server | 4.36.1 | MonetDB4 | Feb2010 | |
MonetDB4/XQuery | 0.36.1 | pathfinder | Feb2010 | |
MonetDB5 Server | 5.18.1 | MonetDB5 | Feb2010 | |
MonetDB5/SQL | 2.36.1 | sql | Feb2010 | |
MonetDB5/SQL/GIS | 0.16.1 | geom | Feb2010 | |
MonetDB Java | 1.36.1 | java | Feb2010 | |
MonetDB Testing | 1.36.1 | testing | Feb2010 | (developers only) |
NOTE:
Before upgrading to the latest "Feb2010" release from a release of MonetDB older than the Nov2008 release, please make a dump (SQL, XQuery) of your database, and rename/move/backup the database directories (dbfarm, sql_logs, xquery_logs). After the upgrade, restore (SQL, XQuery) your database, again. Once you have checked and confirmed the correctness of the restore, you can delete the above renamed/moved/backed-up old database directories.
When upgrading from a release of MonetDB older than the May2009 release, you need to move your write-ahead log following the "recommended" instructions for SQL and/or XQuery.
Contents
Feb 2010 feature release
Packaging
- Various concurrency bugs were fixed.
- Various changes were made to run better on systems that don't have enough memory to keep everything in core that was touched during query processing. This is done by having the higher layers giving hints to the database kernel about future use, and the database kernel giving hings to the operating system kernel about how (virtual) memory is going to be used.
- Fixed bug in mergejoin implementation. This fixes bug #2952191.
- Added support for compiling on Windows using the Cygwin-provided version of flex.
- Fix compilation issue when configured with --with-curl. This fixes bug #2924999.
- Added implementation of MT_getrss() for Solaris. This yields in the kernel knowing about its (approximate) memory usage to try and help the operating system to free that memory that is best to free, instead of a random page, e.g. the work of the vmtrim thread.
- Implemented a "fast" string BAT append: Under certain conditions, instead of inserting values one-by-one, we now concatenate the string heap wholesale and just manipulate the offsets. This works both for BATins and BATappend.
- Changed the string heap implementation to also contain the hashes of strings.
- Changed the implementation of the string offset columns to be dynamically sized.
- The MonetDB 5 code base underwent a series of small/medium changes:
- Solving name conflicts with external libraries in mal_box.
- Dependency graph generation in MAL debugger fixed.
- Code hardening against out of memory errors and based on Coverity checks.
- Recycler improved to deal with SQL plans from different sessions.
- Profiler extended to report the argument types, user, and thread id.
- MAL interpreter reports an event before and after the instruction.
- Dataflow also allowed for updates on temporary BATs.
- Reorder optimizer better respects the dataflow.
- All update instructions return their target to mark the dataflow.
- Clean up of (bat)calc module.
- Packing pieces together simplified and sped up.
- The MAL interpreter has been extended with an operator admission scheme, which is active during parallel execution. It blocks threads if the total amount of memory needed for the operator can not be claimed. Only if there is one operation left to execute, it won't block. The admission level is controlled by a threshold, which is set to 90% of the physical memory.
- The ilike[u]select operations has been included in the repertoire recognized by the mergetable optimizer for push through of selections.
- The joinpath optimizer has been extended with recognition of the leftjoin. This way, series of leftjoin operations can be optimized by looking at the smallest starting point.
- Introduced a dictionary encoding option to the optimizer pipeline. Encoding is initiated with a SQL call compress(tablename).
- Removed the ability to redirect to other running databases in the same dbfarm. This functionality has been taken over completely by merovingian, and only results in confusement these days, bug #2891191.
- Added --enable-console configure argument, defaulting to 'yes' for now. Disabling the server console increases security by avoiding local access exploits. This is not the default since our Testing setup cannot deal with a console-less server yet.
- Report detected amount of main memory and cpu cores in output of `mserver5 --version`.
Nothing to report. Nothing to report SQL
XQuery
Client Packages
- Fixed a bug in the Mapi libary when sending very large queries to the server.
- Implemented BLOB and CLOB support in ODBC driver.
- Fixed crash upon connect to a server that is under maintenance but has the requested language (scenario) not loaded.
- Fixed bug in the ODBC driver where a non-ASCII character at the end of a string caused an error when the string was converted to wide characters.
- mclient timer information (-t option) is now written to standard error instead of standard output. This makes getting timer information easier when there is a large amout of regular output (which can be redirected to a file or /dev/null).
- Implemented new function mapi_fetch_field_len() which returns the length (excluding trailing NULL byte) of the field returned by mapi_fetch_field().
The following bug tracker items were fixed since the previous "Nov 2009" release: The detailed list of changes can be found in the source code CVS logs.
November 2009
November 2009 mk Tue, 03/30/2010 - 20:27
MonetDB Nov2009-SP2 releaseRelease "Nov2009-SP2" consists of the following updated components
NOTE: ContentsNov 2009 SP2 bug-fix releaseThis is the second bug fix release of the Nov2009 version. This version can be installed "on top of" the Nov2009 release. The database format is fully compatible, so there is no need to do a dump and restore. (Although it is *always* a good idea to make backups.) Some of the bugs that were fixed:
The following bug tracker items were fixed since the Nov2009-SP1 release:
Nov 2009 SP1 bug-fix releaseThis is a bug fix release of the Nov2009 version. This version can be installed "on top of" the Nov2009 release. The database format is fully compatible, so there is no need to do a dump and restore. (Although it is *always* a good idea to make backups.) Some of the bugs that were fixed:
The following bug tracker items were fixed since the Nov2009 release:
Nov 2009 feature releaseUbuntuWhile the Ubuntu version of the Aug2009 release was built for Ubuntu 9.04, the Jaunty Jackalope, the Ubuntu version of the Nov2009 release has now been built for Ubuntu 9.10, the Karmic Koala. FedoraIn addition to Fedora 10 & 11, we now also provide RPMs for Fedora 12. PythonPackaging of some Python support files which are only used for testing was changed. The MonetDB-python RPM has been removed, the files are now packaged in the MonetDB-testing RPM. Equally, the python-monetdb Debian/Ubuntu package has been removed. WindowsOn Windows, the bzip2 library is now also used. The server (e.g., using SQL's COPY INTO) can read files that are compressed using gzip and bzip2. The Coverity service has been used to uncover potential errors, such as dereference through NULL pointers and leakage. The warnings produced in the first run have been dealt with for >90%.
MerovingianThe following features were implemented in the merovingian and monetdb programs:
mclient
Python & Ruby
PHP
JDBC
The following bug tracker items were fixed since the previous "Aug 2009 SP2" release:
The detailed list of changes can be found in the source code CVS logs. |
|||||||||||||||||||||||||||||||||||||||||||
© 1994-2009 CWI | Contact us Legal |
Augustus 2009
Augustus 2009 mk Tue, 03/30/2010 - 20:30MonetDB Aug2009-SP2 release
Release "Aug2009-SP2" consists of the following updated components
Name | Version | CVS module | CVS branch | Note/Comment |
MonetDB Common | 1.32.4 | MonetDB | Aug2009 | |
MonetDB Clients | 1.32.4 | clients | Aug2009 | |
MonetDB4 Server | 4.32.4 | MonetDB4 | Aug2009 | |
MonetDB4/XQuery | 0.32.4 | pathfinder | Aug2009 | |
MonetDB5 Server | 5.14.4 | MonetDB5 | Aug2009 | |
MonetDB5/SQL | 2.32.4 | sql | Aug2009 | |
MonetDB5/SQL/GIS | 0.12.4 | geom | Aug2009 | |
MonetDB Java | 1.32.4 | java | Aug2009 | |
MonetDB Testing | 1.32.4 | testing | Aug2009 | (developers only) |
NOTE:
Before upgrading to the latest "Aug2009-SP2" release from a release of MonetDB older than the Nov2008 release, please make a dump (SQL, XQuery) of your database, and rename/move/backup the database directories (dbfarm, sql_logs, xquery_logs). After the upgrade, restore (SQL, XQuery) your database, again. Once you have checked and confirmed the correctness of the restore, you can delete the above renamed/moved/backed-up old database directories.
When upgrading from a release of MonetDB older than the May2009 release, you need to move your write-ahead log following the "recommended" instructions for SQL and/or XQuery.
Contents
Aug 2009 SP2 bug-fix release
This is the second bug fix release of the Aug2009 version. This version can be installed "on top of" the Aug2009 release. The database format is fully compatible, so there is no need to do a dump and restore. (Although it is *always* a good idea to make backups.)
Some of the bugs that were fixed:
- Changes in the memory mapping code to less zealously allocate space.
- Changes to the use of the way the operating system is informed about how memory is used, resulting in a better fit between the predictions the operating system makes and the reality of what MonetDB actually does.
- Fix to exit code.
- Performance fix releated to anonymous virtual memory.
- Performance fix for when a single item is joined to a table. (MonetDB4/XQuery)
- When extracting fails, don't leave a partial output file (Mx, SF bug 2840225).
- Added support for GCC version 4.4.* by disabling the strict-aiasing optimization in the compiler.
- Various improvements to the Ruby interface.
- Fix warning messages to disappear when a connection to merovingian is made.
- Fix physical plan for merge-adjacent-text-nodes. (MonetDB/XQuery)
- For some plans with aggregates, lax checking resulted in incorrect rewrites. (MonetDB/XQuery)
- Fix bug in order lists merging. In case the replaced column is sorted in descending order we need to reverse the order of all replacement order criteria. (MonetDB/XQuery)
- Fixes so that larger plans can be handled. (MonetDB/XQuery, SF bug 2860037)
- Fixes so that less stack space is used during processing, and so that running out of stack space is recognized in more places. (MonetDB/SQL)
- Fixed bug in IN handling. (MonetDB/SQL, SF bug 2869818)
- Fixed crash when attempting to create a function in read-only mode by disallowing this. (MonetDB/SQL, SF bug 2872421)
- Added partial support for compiling with bison++. Bison++ is not supported and can't be used out of the box, but only minor tweaks are needed to use it. (SF bug 2872565)
- Fix warning messages to disappear when a connection to merovingian is made. (JDBC driver)
The following bug tracker items were fixed since the Aug2009-SP1 release:
- [ 2812963 ] JDBC connection hangs if result is too long
- [ 2839063 ] M5: property error in src/modules/mal/Tests/chopper03.mal
- [ 2840225 ] Mx creates output files even on errors
- [ 2847738 ] SuperBall build fails with PHP 5.3.0
- [ 2860037 ] PF: fatal error: aborted too deep recursion on linux
- [ 2860574 ] PF: fatal error: column not found / BATappend access denied
- [ 2865631 ] SQL: incref assertion
- [ 2869818 ] SQL: Program contains errors
- [ 2871426 ] SQL: mclient-lsql-D fails everywhere but on 64-bit Fedora
- [ 2872421 ] SQL: Crash when trying to create function to read only db
- [ 2872880 ] mclient shows prompt with "-s" option without "-i&qu ot; option
- [ 2872884 ] mclient -t -s... does not show "Timer" info
- [ 2876948 ] PF: error "We do not support the value NaN"
- [ 2879833 ] SQL: SIMILAR keyword does not exist
Aug 2009 SP1 bug-fix release
This is a bug fix release of the Aug2009 version. This version can be installed "on top of" the Aug2009 release. The database format is fully compatible, so there is no need to do a dump and restore. (Although it is *always* a good idea to make backups.)
Some of the bugs that were fixed:
- On Windows, we switched to newer versions of the iconv and geos libraries (iconv 1.11.1 and geos 3.1.1, from iconv 1.11 and geos 3.0).
- On Windows, MonetDB4/XQuery was compiled with a newer version of the Intel compiler (version 11.1.046 from version 10.1.022).
- Better error handling if BATs are growing too large. (SF bug #2850513)
- The MAL parser was made more robust against syntax errors.
- A recursion depth check was added to the MAL interpreter. (SF bug #2852707)
- The PHP client is not compatible with PHP 5.3.0 or later. We now check for that. (Note that in a future release, the PHP client code will be replaced by a pure PHP implementation where this restriction does not apply.) (SF bug #284773)
- The SQL dump code (msqldump and mclient --dump) now specify all delimiters and the quote character on the generated COPY INTO query. This is needed to be compatible with the next feature release.
- The SQL dump code (msqldump and mclient --dump) now also knows about CLOBs (Character Large Objects) and properly quotes the values.
- The problem that occurred under certain circumstances that the JDBC driver would throw an SQLException when large result sets were retrieved is fixed. (SF bug #2812963)
- Fixed a bug in MonetDB/XQuery where the comparison between values of types strings and untypedAtomic was not done correctly. (SF bug #2853441)
- Fixed a bug in MonetDB/XQuery where the dead code elimination was a bit too eager. (SF bug #2852928)
- Fixed a bug in MonetDB/XQuery where a warning was generated that a pin count was below zero. (SF bug #2863458)
- Fixed a bug in PF/Tijah where OR queries were treated as AND queries. (SF bug #2836870)
- Fixed a bug in PF/Tijah where the nexi scanner would split strings on '_'.
- Fixed overflow checks in MonetDB/SQL. (SF bug #2853458)
- Fixes to get the embedded SQL example working again.
- Fixed startup of MonetDB/SQL server so that a client cannot connect before the server is ready. (SF bug #2855021)
- Fixed a bug in MonetDB/SQL where old and dropped view definitions were sometimes incorrectly reused. (SF bug #2863804)
- Fixed a bug in merovingian where the mero_port in the config file was not properly recognized. (SF bug #2844202)
- Fixed a bug in MonetDB/SQL where it was not possible to add month intervals to a timestamp. (SF bug #2854949)
- Fixed various bugs in the SQL engine. (SF bugs #2847106, #2862146)
- Use bigint for RESTART value in MonetDB/SQL. (SF bug #2846238)
- Fixed a bug in MonetDB/SQL with parsing hex values.
- Fixed a bug in MonetDB/SQL when attempting to make changes to the schema when the database is in read-only mode. (SF bug #2830238)
- Fixed a bug in MonetDB/SQL where adding a primary key constraint to a column containing a NULL value was allowed. (SF bug #2075241)
- Fixed a bug in MonetDB/SQL where updating values in one column could affect the values in another column. (SF bug #2864313)
The following bug tracker items were fixed since the Aug2009 release:
- [ 2808453 ] SQL: Trying a copy into in read only database segfault
- [ 2830238 ] SQL: Crash when trying to write on read-only DB
- [ 2831472 ] tijah:queryall crashes Mserver
- [ 2836870 ] PFtijah: Use of 'or' in phrase search yelds incorrect result
- [ 2840310 ] SQL: "Cannot access descriptor" on big endian
- [ 2844202 ] Merovingian: mero_port configuration option ignored
- [ 2846238 ] alter sequence restart on large integer
- [ 2847106 ] SQL: crash upon join with advanced between
- [ 2848525 ] mac10.6 and monetdb installation
- [ 2850341 ] SQL: bit_and returns invalid values
- [ 2852928 ] PF: updates broken using algebra front-end?
- [ 2853441 ] PF: distinct values on two sequences
- [ 2853458 ] SQL: cast exceeding maximum value, returns null
- [ 2854949 ] SQL: + interval '1' year; Program contains errors
- [ 2855021 ] SQL: mserver accepts sql connections before scenario load
- [ 2855152 ] mac10.6 and monetdb installation
- [ 2862146 ] Two COUNT(*) with LIMIT doesn't execute
- [ 2863458 ] PF/TIJAH: tokenize in a for-loop
- [ 2863804 ] SQL: old views incorrectly reused
- [ 2864313 ] SQL: Update changes all columns with same type and default
Aug 2009 feature release
Fedora
We now provide RPMs for Fedora 10 & 11, but no longer for Fedora 9. However, a first check indicates that the Fedora 10 RPMs also install and work fine on Fedora 9.
- A bug-fix in the May2009 release had unnoticedly introduced a potential performance degradation, caused by non-optimal use of memory-mapped files, with workload that exceed physical memory, in particular with bulk-loading ("COPY INTO") large tables in MonetDB/SQL. This performance problem has been fixed.
- Various problems related to sloppy type usage and type casting had surfaced with testing on big-endian hardware as well as on 64-bit systems when using 32-bit OIDs. These bugs have been fixed.
- The compilation process has been improved by splitting the "infamous" large (generated) "gdk_scanselect_defs.c" into multiple files.
- auto-sensing is no longer implicit based on the port number being 0, but can now be enabled and disabled via mapi_autosense in monetdb5.conf. When set to enabled, it will try to find a free port starting from the port given by mapi_port.
- fix crash with select * from 'whatever' = bla, select * where bla = "whatever", Bug #2825974
- Ignore escapes for non-escape sequences, instead of inserting a literal backslash. Bug #2830900
- Decimals now are properly detected in PreparedStatements, allowing to use them. Bug #2831994
- Fixed hang after invalid statement + COMMIT outside auto-commit mode. Bug #2814622
- Fixed NUMERIC type detection. Bug #2818176
- No longer allow SELECT in value lists, causing confusing error messages. Bug #2807609
- Allow for NULL assignment and NULL as argument of a function. Bug #2812729
- Added missing power (^) operator. Bug #2812721
- DISTINCT was not properly recognised, causing incorrect count results. Bug #2826015
- When encountering ambigious columns, fail instead of crashing. Bug #2807336
- GROUP BY on a view with constants no longer yields in a crash. Bug #2837561
- PLAN output is fixed to no longer violate the protocol and to be tuples instead of comments.
- Optimisating pipelines can now be set in monetdb5.conf and changed without compiling SQL.
- sql_logdir can no longer be set and defaults to a location close to the actual data. This approach was chosen since the logs are useless without the data.
- The ILIKE operator was added. This operator is a case-insensitive LIKE operator, as also found in PostgreSQL.
Merovingian
The following features were implemented in the merovingian and monetdb programs:
- merovingian: Implemented a configurable port for the neighbour discovery service. This also allows to entirely disable neighbour discovery by setting mero_discoveryport to 0. Feature request #2800964
- merovingian: Implemented full per-database property support. monetdb now can manage the following per-database properties: forward (whether to proxy or redirect), shared (if it will be available in the network and how) and nthreads (number of working threads the mserver process uses). Feature Request #1837591.
- merovingian: Increased the default mero_exittimeout from 7 to 60 seconds. This makes merovingian by default wait longer for mservers to exit before forcefully terminating them.
- merovingian: Implemented sharing of databases with tags. Using these tags, extra information is available on a remote merovingian to select a database during connect. For this, a client can now connect to a database using a glob expression, which is matched on the database plus tag. When multiple databases match such expression, a load-balancing technique is applied. See the merovingian man-page for details.
- monetdb: Implemented dynamic width of monetdb status output. If a database name is wider than 14 chars, the name column is made wider as long as the terminal window allows. If a column cannot be made any wider this way, it is abbreviated as before.
- monetdb: Upon monetdb destroy, avoid multiple questions by merging them into a single confirmation for the destroy command. This accidently also fixed a bug where a response such as 'yes' would abort the next database removal.
- monetdb: Databases created by monetdb now can only have names restricted to the pattern [A-Za-z0-9-_]+.
- monetdb: Allow setting the name of a database (a.k.a. renaming it).
- monetdb: Allow an expression to be used to match against the list of discovered or local databases as argument in the commands discover and status.
- monetdb: destroy now allows to remove a database under maintenance. This way an administrator can prevent users connecting to the database and hence starting it on demand, which in turn prevents removal of the database.
- xquery_logdir can no longer be set and defaults to a location close to the actual data. This approach was chosen since the logs are useless without the data.
mclient
The rendering of the mclient utility for SQL has been improved to better utilise the available space on the screen. Main improvement is that columns are no longer wrapped if they contain only NULLs, even though enough space on the terminal is availble.
Python
- The python API now supports connections to Merovingian.
- Transaction support was added.
- Added compatibility with python3k.
- Full Unicode/UTF-8 support.
- autocommit, commit and revert functions were added.
Ruby
- Fixed bugs that prevented the correct working of activerecords' migration; The activerecord connector now supports insertion, update and alter table operations; Type casting is working in activerecord; Added a rubygem and rakefile for activerecord-monetdb-adapter; Added a new usage example for activerecord to the README file; Added an example directory to the cvs tree; The driver now correctly works with Merovingian.
- Support for ruby 1.9.
- Both the standalone driver and activerecord now support transactions, nested transactions are simulated via savepoints in activerecord.
- Type conversion in the standalone driver is now performed after data has been retrieved and can be executed on single fields.
JDBC
- blockmode_blocksize option, and the -Xblksize option from JdbcClient were removed, as they are meaningless and inimplemented in the current JDBC driver.
- Full redirection support was added to support Merovingian's tag-based connections for localisation and loadbalancing.
May 2009
May 2009 mk Tue, 03/30/2010 - 20:31MonetDB May2009-SP2 release
Release "May2009-SP2" consists of the following updated components
Name | Version | CVS module | CVS branch | Note/Comment |
MonetDB Common | 1.30.4 | MonetDB | May2009 | |
MonetDB Clients | 1.30.4 | clients | May2009 | |
MonetDB4 Server | 4.30.4 | MonetDB4 | May2009 | |
MonetDB4/XQuery | 0.30.4 | pathfinder | May2009 | |
MonetDB5 Server | 5.12.4 | MonetDB5 | May2009 | |
MonetDB5/SQL | 2.30.4 | sql | May2009 | |
MonetDB5/SQL/GIS | 0.10.4 | geom | May2009 | |
MonetDB Java | 1.30.4 | java | May2009 | |
MonetDB Testing | 1.30.4 | testing | May2009 | (developers only) |
IMPORTANT NOTE:
Before upgrading to the latest "May2009-SP2" release from a release of MonetDB older than the Nov2008 release, please make a dump (SQL, XQuery) of your database, and rename/move/backup the database directories (dbfarm, sql_logs, xquery_logs). After the upgrade, restore (SQL, XQuery) your database, again. Once you have checked and confirmed the correctness of the restore, you can delete the above renamed/moved/backed-up old database directories.
Contents
May 2009 SP2 bug-fix release
This is the second bug fix release of the May2009 version. This version can be installed "on top of" the May2009 release. The database format is fully compatible, so there is no need to do a dump and restore. (Although it is *always* a good idea to make backups.)
Some of the bugs that were fixed:
- Fixed a problem where in certain condition you could get an error on Windows saying "GDKremovedir: rmdir(bat\DELETE_ME) failed".
- Various improvements to the Python and Ruby clients.
- Fix mserver5 crash when started on demand by merovingian (SF bug #2805340).
- Check for write errors during dump and abort when they occur.
- Some innocuous looking XQuery queries could produce large incorrect results and take a relatively long time to do so. SF bug #2815820.
- Fixed some potential buffer overflow problems in the Mapi library.
- Improved help message from mclient (SF bug 2818536).
- Fixed certain SQL LIKE queries on some O/S.
- Fixed SQL dump in mclient and msqldump to dump values of some less used types correctly.
- The Debian/Ubuntu installation now also contains the MonetDB PHP client support files.
- Fixed several performance issues (among others: SF bug 2821251).
- The various client libraries now also recognize the wrd type which may be emitted by the SQL server.
- In SQL, the char and varchar types now use Unicode code points (characters to you and me) as the unit for counting instead of bytes in the UTF-8 representation (SF bug 2822855).
- Fixed some UI issues with the XQuery administrator console.
- Fixed a bug in SQL when handling IN and OR lists with equal values.
- Fixed a stack overflow condition in SQL with certain types of quite complex queries (SD bug 2827784).
The following bug tracker items were fixed since the May2009 SP1 release:
- [ 1840239 ] BAT capacity checking needed
- [ 2805340 ] Merovingian (?): mserver5 segfaults if started by a client
- [ 2815820 ] PF: Endless loop in innocent xquery
- [ 2818536 ] improve mclient -u / -p help
- [ 2818931 ] MAL: segv on mortal soul's syntax error
- [ 2819972 ] function problom ----return and elseif
- [ 2821251 ] PF: Complex optimization overly complex
- [ 2822855 ] utf8 real size upon insert
May 2009 SP1 bug-fix release
This is a bug fix release of the May2009 version. This version can be installed "on top of" the May2009 release. The database format is fully compatible, so there is no need to do a dump and restore. (Although it is *always* a good idea to make backups.)
Some of the bugs that were fixed:
- mclient and msqldump would cache all results. This is fixed.
- The Fedora RPM for the Ruby clients package is now included.
- Some files were missing in the Python client directory. They are now included in the client source tar ball.
- mclient now shows the results of all subqueries in a query batch that was passed on the command line using the -s option if there was an error in one of them.
- mclient \d and \D interactive commands now obey the \| command.
- Fixed naming of the document and collection when both the --input and --collection arguments are given to mclient (XQuery).
- Various fixes to the low level memory mapped file support.
- Fixes to the geom module.
The following bug tracker items were fixed since the May2009 release:
- [ 2603876 ] XQ: fn:put() tests segfault with ALG back-end
- [ 2687549 ] SQL: test type_dump_test.SF-989257.SQL.sh fails since Feb 28
- [ 2737889 ] rand() is not executed for every row when using alias
- [ 2782202 ] SQL: dump&restore fully usable only by admin
- [ 2791356 ] Floating point exception
- [ 2797767 ] mclient --dump caches dump
- [ 2803477 ] alter table drop contraint error
- [ 2804823 ] The Mapi Client Utility
- [ 2805513 ] fn:trace() produces output _and_ error
- [ 2807038 ] tab + ) + mal function = seg fault
May 2009 feature release
Windows
The Windows binaries for SQL and ODBC were built using Microsoft Visual Studio 2008, unlike previous releases which were built using Microsoft Visual Studio 2005. The main implication of this change is that the ODBC driver should now also install on Windows Vista.
The Perl, PHP, and Python clients were removed from the Windows installers. The clients are still available in the source distribution (although the Python client has changed significantly).
Fedora
There is no longer an RPM for MonetDB-client-ruby. In order to update to this release, you will first have to remove MonetDB-client-ruby after which you can update to this version. (A new version of the RPM will probably appear in a future release.)
Ubuntu
The Ubuntu version was built for Ubuntu 9.04, the Jaunty Jackalope. The Feb2009 release was built for Ubuntu 8.10, the Intrepid Ibex.
The way string data is stored in the database has changed for systems with a 64-bit word size and 32-bit OIDs. When the server first opens a database that was created using the old format, the database is updated automatically. This change affect only the 64-bit Windows build. The other 64-bit builds that are provided by us use 64-bit OIDs and are unaffected by the change.
Parallel loading has been disabled since there was an inherent problem with the implementation. This also affects SQL.
Write-ahead log
The default location of the write-ahead log has changed. At startup, when the server sees there is an SQL catalog but no write-ahead log, the server will exit immediately with an error message. You can do one of two things (not counting deleting your whole database):
- move the log files to the new location (recommended): create the directory that is mentioned by the error message and move the log files (two files,
log
andlog.n
wheren
is some integer) there; - start the server with the
--set sql_logdir=...
option (where you replace...
with the appropriate path) to point the server to the old location.
In either case you need to know the old location of the write-ahead logs. This old location was $prefix/var/MonetDB5/sql_logs/<dbname>/sql/
where $prefix
is the prefix under which MonetDB was installed and <dbname>
the database name. The error message mentioned above gives the expected location of the write-ahead logs.
Note that when you use the Windows installer and you start the server using the Start menu, the write-ahead log is moved to the new location automatically.
Authentication
The authentication mechanism has changed somewhat so that it is now possible to dump and restore password information. Note that the passwords themselves cannot be retrieved: the dump/restore mechanism uses a hash of the password.
Merovingian
The following features were implemented in the merovingian and monetdb programs:
- The logging of merovingian has changed. TME records are no longer issued, instead each line is prefixed with a timestamp, syslog style. Feature Request #2782776
- merovingian now reopens its logfiles when being sent the SIGHUP signal. This allows utilities like e.g. logrotate to rotate the merovingian.log file. Feature Request #2782776
- The merovingian default log file location has changed from /var/log/merovingian.log into /var/log/MonetDB/merovingian.log.
Like the previous "Feb 2009 SP2" release, the default Algebra-based Pathfinder compiler ("ALG") still does not support a few XQuery features and extensions. A detailed list and temporary work-arounds by resorting to the deprecated "milprint_summer" ("MPS") version of the Pathfinder compiler can be found here.
Write-ahead log
The default location of the write-ahead log has changed. At startup, when the server sees there is an XQuery catalog but no write-ahead log, the server will exit immediately with an error message. You can do one of two things (not counting deleting your whole database):
- move the log files to the new location (recommended): create the directory that is mentioned by the error message and move the log files (two files,
log
andlog.n
wheren
is some integer) there; - start the server with the
--set xquery_logdir=...
option (where you replace...
with the appropriate path) to point the server to the old location.
In either case you need to know the old location of the write-ahead logs. This old location was $prefix/var/MonetDB4/xquery_logs/<dbname>/xquery/
where $prefix
is the prefix under which MonetDB was installed and <dbname>
the database name. The error message mentioned above gives the expected location of the write-ahead logs.
Note that when you use the Windows installer and you start the server using the Start menu, the write-ahead log is moved to the new location automatically.
Mapi
Support for SSL-encrypted connections with the server has been discontinued.
The protocol between client and server has changed slightly. This means you cannot use an old client with a new server.
Python
The Python client modules were rewritten and are now in pure Python and so do not depend on any other part of MonetDB. The new module has support for Python 2.5, 2.6 and 3.0, and it is Python DBAPI 2.0 compatible.
Ruby
The Ruby client modules were rewritten and are now in pure Ruby and so do not depend on any other part of MonetDB.
February 2009
February 2009 mk Tue, 03/30/2010 - 20:33
MonetDB Feb2009-SP2 releaseRelease "Feb2009-SP2" consists of the following updated components
IMPORTANT NOTE: ContentsFeb 2009 SP2 bug-fix releaseOne set of fixes that affects all components is that stack overflow checking has been improved. It is now much less likely that the server crashes when queries need too much stack space. Instead, the server will return an error. MonetDB/XQuery (tijah): fixes to UTF-8 handling; fixes towards being able to index more than 2**31 words on a 64 bit architecture. MonetDB5: fixed a crash on exit of the server (this crash was not harmful, just annoying). MonetDB: avoid integer overflow on very large tables. mclient/msqldump: fixes to the order in which tables, views, and functions are dumped; also dump triggers; fixed the Ubuntu and Debian installers to include mclient and msqldump. MonetDB/XQuery: fixed a bug where documents were not available when another document in the collection was loaded. merovingian/monetdb: fixes to the way the server is stopped. MonetDB/SQL: fixed a bad concurrency bug where updates were lost or data was incorrectly inserted. The following bug tracker items were fixed since the Feb2009 SP1 release:
Feb 2009 SP1 bug-fix releaseThe main fix that was done in this release that affects all parts of the code, but most especially MonetDB/SQL is that we have plugged a large number of memory leaks. MonetDB/SQL can now handle larger databases and more demanding queries. This is especially noticeable on 32 bit Windows systems where the memory management system (the system malloc library) is not as sophisticated as the one in Linux. mclient/msqldump: Fix to the order in which tables and views are dumped: tables must be dumped before views, since views may reference tables and the view creation will only work if the referenced tables already exist. mclient: Fixed potential deadlock situation when there is a large input file and several queries early on produce large results. MonetDB/XQuery: Fixed a bug where a replace operation could corrupt the document (bug #2642003). The following bug tracker items were fixed since the Feb2009 release:
The following things were fixed or changed in merovingian and monetdb:
The following was changed in the JDBC interface:
Feb 2009 feature releasePackagingWe now provide Debian (Lenny, i.e. 5.0) and Ubuntu (Intrepid, i.e. 8.10) packages. These packages have only been very lightly tested (the packaged software is tested extensively, though). For now we provide 32 and 64 bit versions for Debian and only a 64 bit version for Ubuntu. This of course in addition to the Fedora packages we already provided. MonetDB Server 5The code base has been tested for memory leaks reducing the change of running out of memory too quickly. The Merovingian has undergone more thorough stress testing making it much more stable. SQLThe optimizer of SQL has been overhauled. Rather than working on the binary plans, it performs optimization on the relational algebra tree before it produces the binary plans. XQueryLike the previous "Nov 2008 SP2" release, the default Algebra-based Pathfinder compiler ("ALG") still does not support a few XQuery features and extensions. A detailed list and temporary work-arounds by resorting to the deprecated "milprint_summer" ("MPS") version of the Pathfinder compiler can be found here. Time and date related types and the comparison and extraction functions on top of them have been implemented in MonetDB/XQuery. MtestMainly for MonetDB developers: The MonetDB testing tools (Mtest.py & friends) from MonetDB/src/testing/ have been moved into their own "testing" CVS module / MonetDB component. IMPORTANT NOTE:In case you plan to upgrade your existing CVS-based "MonetDB" installation, please run `make uninstall` in your "MonetDB" build directory, first, i.e., before running `cvs update` in your "MonetDB" source directory. Once you update your "MonetDB" to the latest CVS development trunk version, and plan to continue (or start) using Mtest.py, you also need to check out the new "testing" CVS module (from SF) and compile and install it any time after you compiled and installed "MonetDB", and before you plan to use Mtest.py. The new "testing" CVS module depends only on "MonetDB" (and hence requires that to be installed), but on no other MonetDB component (except from the "buildtools", in case your are compiling from CVS sources). No other MonetDB component depends on the new "testing" CVS module other than that you obviously need to have "testing" installed once you want/need to run tests using Mtest.py. Details:The MonetDB testing tools (Mtest.py & friends) are not part of the actual MonetDB functionality, but mainly for MonetDB development purposes (and hence only distributed in the source tarballs, but not in any (official) binary packages). To make them less dependent on release cycles (and vice versa), as well as to prepare making them usable without MonetDB --- for now, they actually still depend on MonetDB; see below) --- we decided to separate the MonetDB testing tools from the "MonetDB" CVS module into their own "testing" CVS module. For the time being (i.e., until further notice), "testing" depends on "MonetDB" --- more precisely only on some utils from MonetDB/src/common/, but not on the GDK kernel from MonetDB/src/gdk/, as
Python modules "subprocess26.py" & "trace.py" have been moved from <PYTHON_LIBDIR>/MonetDB/ to <PYTHON_LIBDIR>/MonetDBtesting/ . Bug fixesA detailed list of the 65 bug reports that have been closed since the previous "Nov 2008 SP2" release can be found in the MonetDB Bug Tracker at SourceForge: page 1/2, page 2/2. Known Issues in this Release
|
|||||||||||||||||||||||||||||||||||||||||||
© 1994-2009 CWI | Contact us Legal |
November 2008
November 2008 mk Tue, 03/30/2010 - 20:35MonetDB Nov2008-SP2 release
Release "Nov2008-SP2" consists of the following updated components
Name | Version | CVS module | CVS branch |
MonetDB Common | 1.26.2 | MonetDB | Nov2008 |
MonetDB Clients | 1.26.2 | clients | Nov2008 |
MonetDB4 Server | 4.26.2 | MonetDB4 | Nov2008 |
MonetDB4/XQuery | 0.26.2 | pathfinder | Nov2008 |
MonetDB5 Server | 5.8.2 | MonetDB5 | Nov2008 |
MonetDB5/SQL | 2.26.2 | sql | Nov2008 |
MonetDB5/SQL/GIS | 0.6.2 | geom | Nov2008 |
MonetDB Java | 20081118 | java | (none; CVS trunk) |
IMPORTANT NOTE:
Before upgrading to the latest "Nov2008-SP2" release from a release of MonetDB older than the Nov2008 release, please make a dump (SQL, XQuery) of your database, and rename/move/backup the database directories (dbfarm, sql_logs, xquery_logs). After the upgrade, restore (SQL, XQuery) your database, again. Once you have checked and confirmed the correctness of the restore, you can delete the above renamed/moved/backed-up old database directories.
Contents
Nov 2008 SP2 bug-fix release
This is the second bug fix release of the Nov2008 version. This version can be installed "on top of" the Nov2008 and Nov2008-SP1 releases. The database format is fully compatible, so there is no need to do a dump and restore. (Although it is *always* a good idea to make backups.)
Some of the bugs that were fixed:
- MonetDB/XQuery: There was a bug when shredding documents into an existing updatable collection when the existing collection happened to end at a page boundary. (Bug [ 2141012 ] XQ: unclear error).
- General: Fix to configure to not complain when the raptor library isn't found, unless it was specifically asked for.
- MonetDB/SQL: significantly speeded up loading of data when lines are extremely long (e.g. when loading BLOBs).
- XRPC: Since the change of the XRPC protocol (by June 2008), the XRPC server also sends response messages (with empty
nodes) for updating function calls. However, the XRPC serever failed to send response messages for calls to the Pathfinder document management functions, because they are handled differently than normal updating functions. This is now fixed. - XRPC: The function XRPCHTTPConnection.sendReceive() did not check if HttpURLConnection.getResponseCode() returns -1 or not. If the return value is indeed -1, subsequent use of the stream returned by HttpURLConnection.getInputStream() or HttpURLConnection.getErrorStream() might cause a NullPointerException. This is now fixed. The return value and the stream returned by those two functions are now checked and a proper error message is given.
The following bug tracker items were fixed since the Nov2008-SP1 release:
- [ 1964658 ] Problem with names of reverse()'d BATs.
- [ 2141012 ] XQ: unclear error
- [ 2233581 ] MonetDB does not like '\0' in its input
- [ 2388542 ] SQL bad blob insert causes crash
- [ 2405952 ] Merovingian takes too much CPU resources
- [ 2430201 ] Is distinct broken?
- [ 2457195 ] Broken query results in OOM, working one has no output
- [ 2460935 ] Pathfinder fails to compile with icc
- [ 2460953 ] `make dist` (and hence `make rpm`) fails for pathfinder
- [ 2466725 ] SubSelect with Outer Reference, result is wrong
- [ 2468513 ] SQL: copy into does not detect syntax errors
- [ 2471371 ] SQL: varchar bounds should be obeyed
Nov 2008 SP1 bug-fix release
This is the first bug fix release of the Nov2008 version. This version can be installed "on top of" the Nov2008 release. The database format is fully compatible, so there is no need to do a dump and restore. (Although it is *always* a good idea to make backups.)
Some of the bugs that were fixed:
- Fixed formatted printing from MAL.
- mclient now exits with an error code if one of the queries in a non-interactive session failed. [2388047]
- Fixed a deadlock condition when multiple queries simultaneously update the same part of a document (XQuery).
- When writing to a file, mclient now flushes the output when the query is done instead of waiting until the file is closed. [2344251]
- Fixed a race condition in document management (XQuery). [2338542]
The following bug tracker items were fixed since the Nov2008 release:
- [ 1911209 ] XQ: MonetDB 'hangs' after requesting non-existing doc
- [ 1981762 ] PF/alg: "Mapping unique name to an original name failed."
- [ 2049320 ] pf:collection doc count wrong (again?)
- [ 2075180 ] the unique column can't insert more than 1 row with null
- [ 2075241 ] When the NULL is in a column,it shouldn't add primary key
- [ 2103348 ] unique problem with nulls
- [ 2142522 ] XQ: Duplicates when not using intermediate result
- [ 2144639 ] XQ: fn:collection() in algebra version
- [ 2209871 ] most GEOM test fail after error handling changes
- [ 2224233 ] M4 test ID.914935 fails after warning changes
- [ 2233677 ] SQL Command kills DB
- [ 2276427 ] CMDmulti_merged_union: operation failed
- [ 2277120 ] XQ: following(-sibling)/preceding cause Mserver to crash
- [ 2305746 ] merovingian crashes when starting a vaultkeyless database
- [ 2338524 ] Deadlock on barrier
- [ 2344251 ] Output isn't complete
- [ 2388047 ] Mclient exit code
- [ 2388424 ] ERROR: XML Generation: NULL BAT has a 24 head, but tail is N
- [ 2420768 ] col from another schema inside function body
Nov 2008 feature release
Copyright & License
The copyright for the MonetDB suite was transferred to a newly formed company MonetDB B.V. One of the most important conditions for this transfer was that the software remains open source, and so the license under which this software is being distributed has not changed (apart from the copyright notice).
Windows Installers
The default installation location has been changed from C:\Program Files\CWI to C:\Program Files\MonetDB.
MonetDB Common (GDK)
The kernel administration for BATs has been changed to prepare for future enhancements. All descriptors are now initialized and kept in memory, which reduces the number of files and related IO accessing the descriptors.
Memory management for large database has been improved, with the scavenging thread becoming more aggressive in moving dirty pages to disk when we reach 70% of the physical memory.
Code cleaning and simplification continues to provide ease of compilation on many platforms.
MonetDB Server 5
The MonetDB 5 kernel underwent a code clean up too, e.g. datatypes have been more strictly applied and all kernel errors are propagated to MAL exceptions.
The number of experimental modules have been developed. The 'recycler' provides support for automatic re-use of intermediate results. It improves query throughput for long sequences with partially overlapping queries.
The system supports intra-query parallelism using the multi-core systems capabilities using the 'dataflow' optimizer.
The 'mergetable' optimizer has been extended to perform symbolic reasoning over oid-reasoning. Together with the MAT and dataflow optimizers it provides a basis to improve multi-core performance.
Several bugs were fixed in Merovingian, the MonetDB Database Server daemon, including a file descriptor leak, which caused Merovingian to terminate after a certain amount of connections using the proxy method. Merovingian has been extended with neighbour discovery features, allowing to redirect to discovered neighbours. The monetdb tool has been improved to give better feedback on starting and stopping databases. Most notably it now waits for completion of the start operation, and gives error feedback when starting failed for some reason.
SQL
This release comes with a few more optimizers enable by default. The most important one is the inter-query parallelism, which leads to better use of a multi-core platforms. The effects can be seen in our TPCH experiments.
The dataloading code has been improved during our effort to load a 2.6TB astronomical database. Upon request we can provide a detailed report on this experiment.
A query log can be captured with the history.sql script. It provides a basis for post-session analysis of execution times.
Several bugs reported have been solved and functionality in many other areas has been improved.
XQuery
Like the previous "Jun 2008" release, the default Algebra-based Pathfinder compiler ("ALG") still does not support a few XQuery features and extensions. A detailed list and temporary work-arounds by resorting to the deprecated "milprint_summer" ("MPS") version of the Pathfinder compiler can be found here.
Bug fixes
A detailed list of the 116 bug reports that have been closed since the previous "Jun 2008" release can be found in the MonetDB Bug Tracker at SourceForge: page 1/3, page 2/3, page 3/3.
The detailed list of changes can be found in the source code CVS logs.
Known Issues in this Release
- Merovingian crashes when starting a vaultkeyless database. Actual workaround is not to try and start databases not created using monetdb create.
Details: SF bug tracker #2305746 - Parallel loading is known to be buggy. Actual workaround is to start the server using gdk_nr_threads=1 set in your monetdb5.conf file (or given via --set gdk_nr_threads=1 on your mserver5's command line) during the bulkload using COPY INTO FROM FILE with mclient.
Details: (no known bug tracker reference)
June 2008
June 2008 mk Tue, 03/30/2010 - 20:35Release "Jun2008" consists of the following updated components
Name | Version | CVS module | CVS branch |
---|---|---|---|
MonetDB Common | 1.24 | MonetDB | MonetDB_1-24 |
MonetDB Clients | 1.24 | clients | Clients_1-24 |
MonetDB4 Server | 4.24 | MonetDB4 | MonetDB_4-24 |
MonetDB4/XQuery | 0.24 | pathfinder | XQuery_0-24 |
MonetDB5 Server | 5.6 | MonetDB5 | MonetDB_5-6 |
MonetDB5/SQL | 2.24 | sql | SQL_2-24 |
MonetDB5/SQL/GIS | 0.4 | geom | Geom_0-4 |
and the jars for JDBC and XRPC.
IMPORTANT NOTE:
Before upgrading to the latest "Jun 2008" release from any previous release of MonetDB, please make a dump (SQL, XQuery) of your database, and rename/move/backup the database directories (dbfarm, sql_logs, xquery_logs). After the upgrade, restore (SQL, XQuery) your database, again. Once you have checked and confirmed the correctness of the restore, you can delete the above renamed/moved/backed-up old database directories.
Contents
- Compilation from sources
- Windows Installers
- MonetDB Common (GDK)
- SQL
- Geometry (SQL/GIS)
- XQuery
- Bug fixes
Compilation from sources
For convenience of both developers and users as well as to comply even more with open source standards, we now set/use the following defaults for the configure options
--enable-strict, --enable-assert, --enable-debug, --enable-optimize
When compiling from CVS sources (as mainly done by developers):
strict=yes assert=yes debug=yes optimize=no (*)
When compiling from packaged/distributed sources (i.e., tarballs) (as mainly done by users):
strict=no assert=no debug=no optimize=no (*)
For building binary distributions (RPMs, MSIs):
strict=no assert=no debug=no optimize=yes
With nightly testing for the "Stable" release branches:
strict=yes assert=no debug=no optimize=yes
With nightly testing for the "Current" development trunk:
strict=yes assert=yes debug=no optimize=yes
(*)
IMPORTANT NOTE:
Since
--enable-optimize=yes
is no longer the default for any case except binary packages, it is
strongly recommended
to (re)compile everything from scratch,
explicitly configured
with
--enable-debug=no --enable-assert=no --enable-optimize=yes
in case you want/need to run any performance experiments with MonetDB!
Please note:
--enable-X=yes is equivalent to --enable-X, and
--enable-X=no is equivalent to --disable-X.
Windows Installers
There are some changes to the Windows installers since previous releases which are described in this section.
Backup and restore
It is recommended to dump the old database and remove it before installing the new version. After installation, the dump can be restored.
In order to dump the SQL database, start the MonetDB SQL Client program and type the commands
\>...\databasedump.sql \D \>
The path after
\>
should be an absolute path name (i.e. start with a drive letter) and be in a save location.
In order to dump the XQuery database, start the server and then connect to the administrative console http://127.0.0.1:50001/admin and use that to dump the database.
By default the database is located in %APPDATA%\MonetDB4 for the MonetDB4/XQuery server and %APPDATA%\MonetDB5 for the MonetDB5/SQL server. After having made a database dump, the latter folder can be removed, but the database dump for MonetDB4/XQuery is made inside the database folder, so remove anything except the folder backup and its contents. This folder is located inside the dbfarm\demo folder.
Restoring the SQL database can be done using the MonetDB SQL Client program with the following command
\<...\databasedump.sql
Restoring the XQuery database can be done from the administrative console.
Installation
The 32 bit Windows installers can be installed "over" the old installation. The installer will automatically remove the old version and install the new. The 64 bit Windows installer will not do this (this time), since the code identifying the package was changed. So on 64 bit Windows, first uninstall the previous version and then install the new. The reason for this change is so that the 32 bit version and 64 bit version can both be installed on a 64 bit platform.
The ODBC driver was removed from the MonetDB5-SQL installer. You now need to install the ODBC driver separately. On 64 bit Windows you should install both 32 and 64 bit versions of the ODBC driver.
Also, make sure that after you have uninstalled the old version or upgraded to the new, the files C:\WINDOWS\system32\lib*.dll are all gone. If not, remove them. The new version does not install any files in the system folder. (To be precise, the files are libstream.dll, libmutils.dll, libMonetODBC.dll, libMonetODBCs.dll, and libMapi.dll.)
The JDBC driver was removed from the Windows installers. The driver is now only available separately.
A new MonetDB5-Geom-Module installer is available which can get installed after the MonetDB5-SQL-Installer has been installed. This module provides GIS support to MonetDB/SQL.
MonetDB Common (GDK)
TODO: mmap, property checks, ...
SQL
TODO: ...
Geometry (SQL/GIS)
This release includes (partial) support for Simple Feature Specification of OpenGIS. Follow the GEOM link above for a small tutorial on it.
XQuery
This release marks the beginning of a new era. The default for MonetDB/XQuery has been changed from using the original Pathfinder compiler (nick-named "milprint_summer" or simply "MPS") to using the new improved and enhanced Algebra-based Pathfinder compiler ("ALG").
Compared to the "preview" in the previous "Feb 2008" release, the functionality of the Algebra-based Pathfinder compiler has been significantly extended to (almost) match that of the original MPS-based compiler. Only the following XQuery features and extensions are not yet supported with the new Algebra-based Pathfinder compiler:
- fn:put(), fn:collection()
Please use the old MPS version as described below. - recursive functions
If the recursion depth is known at compile time, adding the optiondeclare option pf:recursion-depth "<number>";
to the query text will ensure that the recursive function is unfolded <number> times. If the recursion is deeper than <number> invocations a runtime error is triggered.
For arbitrary recursive functions, please use the old MPS version as described below. - Arithmetic Functions
Please use the old MPS version as described below. - Prepared Queries
When using prepared queries, MonetDB/XQuery will automatically switch back to the old MPS version. - XRPC Extension
In case MonetDB/XQuery's XRPC Server is contacted via XRPC (e.g., by the Administrative GUI), the MonetDB/XQuery server will automatically switch back to the old MPS version to handle the XRPC request.
To use the execute at XRPC Syntax, please use the old MPS version as described below. - StandOff Extension
Pathfinder automatically falls back to the old MPS version, in case StandOff extensions are enabled at runtime via
Mserver --set standoff=enabled and/or pf --enable-standoff
While support for the old MPS version will not be continued, it remains available for backward compatibility without recompilation, at least for the time being.
For the stand-alone XQuery compiler, the MonetDB/XQuery server, and/or the MonetDB/XQuery client, simply use the following command line options to revert the default to the old MPS version:
- pf -M
- Mserver --dbinit='module(pathfinder);' --set xquery_backend=milprint_summer
- mclient --language=xquery --no-algebra
- mclient -lxquery -g
In an interactive `
mclient -lxquery
` session, commands "
\g
" / "
\G
" enable / disable the old
MPS
-based compiler in the MonetDB/XQuery server.
Bug fixes
A detailed list of the 139 bug reports that have been closed since the previous "Feb2008" release can be found in the MonetDB Bug Tracker at SourceForge: page 1/3, page 2/3, page 3/3.
The detailed list of changes can be found in the source code CVS logs.
February 2008
February 2008 mk Tue, 03/30/2010 - 20:36Release "Feb2008" consists of the following updated components MonetDB Common 1.22, MonetDB Clients 1.22, MonetDB4 Server 4.22, MonetDB4/XQuery 0.22, MonetDB5 Server 5.4, MonetDB5/SQL 2.22, and the jars for JDBC and XRPC.
IMPORTANT NOTE:
The database format of this release is not compatible with that of any previous release. Before installing this release, make a dump of your database, and do a restore afterwards.
MonetDB Server This release marks the roll forward of the binary storage layout to a single column storage layout. This involved a major overhaul of the MonetDB Server code. Memory leaks have been addressed.
JDBC & XRPCwrapper The Java code (JDBC & XRPCwrapper) has been moved from the clients and pathfinder source trees to a dedicated java source tree in CVS, which can simply be compiled using "ant". Pre-compiled ".jar" are available for Download.
SQL The functional extensions to SQL are limited. The mostly concern minor bugs and some of the optimizers have been improved to better exploit re-occurring joins. The core cracker code has been cleaned up for better inspection and sharing in the community.
XQuery Next to the original Pathfinder compiler (nick-named "milprint_summer" or simply "MPS"), the development of the new improved and enhanced Algebra-based Pathfinder compiler ("ALG") is progressing quickly. While the original MPS version remains the default in this release of MonetDB/XQuery, a preview of the new ALG version is now available without recompilation. For the stand-alone XQuery compiler, the MonetDB/XQuery server, and/or the MonetDB/XQuery client, simply use the following command line options to select the Algebra version as default:
- pf -A
- Mserver --dbinit='module(pathfinder);' --set xquery_backend=algebra
- mclient --language=xquery --algebra
- mclient -lxquery -G
In an interactive `
mclient -lxquery
` session, commands "
\G
" / "
\g
" enable / disable the
Algebra
-based compiler in the MonetDB/XQuery server.
The
Algebra
-based version of Pathfinder will become the default in the next release of MonetDB/XQuery.
Additionally, the following features have been added:
- Algebra & MPS:
-
- New Feature: (Restricted) optimistic type checking for functions.
For every function argument whose type is not promotable to the type in the function definition we check whether the type would be promotable if we would apply fn:zero-or-one to the function argument. If that makes the types compatible we introduce an fn:zero-or-one function call for the argument.
The effect is that a query writer now can quite often ignore adding fn:zero-or-one function calls (just to make the compiler happy :)).
- New Feature: (Restricted) optimistic type checking for functions.
- Algebra:
-
- XML import/export for logical algebra plans (pf -I / pf -X). There is however no official description of the used dialect.
- New Feature: (Restricted) support for recursive functions in the algebra.
If at query compile time the recursion depth is known the query can be extended with the following option:declare option pf:recursion-depth "<number>";
which triggers the compiler to unfold recursive functions <number> times -- thus supporting recursive functions until depth <number>. If the recursion is deeper than <number> invocations a runtime error is triggered.
- Implementations of the following functions have been added:
op:to (e.g., "1 to 10"), fn:concat, fn:string-join, fn:substring, fn:string-length, fn:normalize-space, fn:upper-case, fn:lower-case, fn:translate, fn:contains, fn:starts-with, fn:ends-with, fn:substring-before, fn:substring-after, fn:matches, fn:replace, fn:name, fn:local-name, fn:namespace-uri, fn:error, pf:add-doc, pf:del-doc.
Bug fixes
A summary with the highlights is given below. A detailed list of
can be found in the MonetDB Bug Tracker at SourceForge. The detailed list of changes can be found in the source code CVS change logs.
SQL
- 1884382 table dependency inside union...
- 1865953 local temp table
- 1855673 bug in xml attribute parsing
- 1850809 create the same temp table after its drop...
- 1850789 add and then drop the same constraint...
- 1850781 alter column to set the same constraint
- 1850187 length function is not working properly
- 1848541 select sign...
- 1848511 select insert....
- 1848500 octet_length...
- 1848499 difference function crashes
- 1842550 Better error reports on bulk load (include line num)...
- 1841759 insert text with repeat...
- 1841754 select with full join...
- 1841749 creation of exception for select crash...
- 1841734 select with coalesce
- 1841727 pkey in temp table
- 1838816 SQL: can't use variable name staring with _digit in SQL func
- 1838813 SQL: select 5 % 3 crashes
- 1838785 column with date type
- 1765919 API:Incomplete block read from stream (Mserver still alive?)
XQuery
- 1883569 XQ: Algebra
- 1881018 PF: value clash in compiler/include/algebra.h
- 1877300 XQ: reduce: Assertion `rule' failed
- 1872806 PF: weird results of fn:id()
- 1871790 XQ: uncompilable comparison
- 1871789 XQ: substitution fails
- 1866944 Failed to compile current pathfinder
MonetDB Server
- 1856038 Memory leaks when querying
- 1856034 Memory leaks when bulk loading
- 1852518 GDK/M4/PF: GDK assertion in PF/XQ tests since
- 1842019 BULK load failing for NUMERIC columns
Miscellaneous
- 1845266 Python converters don't quote Unicode strings...
November 2007
November 2007 mk Tue, 03/30/2010 - 20:37Release "2007-11-22" (released on Nov 22 2007) consists of the following updatedparts MonetDB Clients 1.20.2, MonetDB5 Server 5.2.2, MonetDB5/SQL 2.20.2.
This release is a bug fix release, covering the following items:
- Clients: The APIs for python, perl and php have been synchronized by the latest state of the Mapi protocol and implementation. JDBC contains a non-blocking IO fix for slow network connections.
- A performance fix of the group module. Improved grouping functionality significantly through a better estimate of the intermediate sizes.
- The MonetDB/SQL embedded version has been made to work again.
- Memory leakage in both the Server and SQL module have been addressed.
Bug fixes
List of fixed bugs
- 1834941 cursor.executemany (python) fails
- 1834442 Compiling clients (Stable) failed on GNU.32.32.d-Gentoo1.12.
- 1833800 Python errors not getting passed through MAPI
- 1831630 Clients: Debian gcj/ant doesn't work.
- 1828314 make failed on MonetDB5
- 1827627 --enable-bits doesn't work with icc
- 1826708 Merovingian: cannot start out of the box, as dbfarm doesn't
- 1823387 Cascade operations in different sessions...
- 1823378 Incorrect return type of "with - seeded by - recurse"
- 1819168 sql/src/test/mapi/python_dbapi test fails
- 1814768 Python Client problems
Release "2007-10-17" (released on Oct 22 2007) consists of MonetDB Common 1.20, MonetDB Clients 1.20, MonetDB4 Server 4.20, MonetDB4/XQuery 0.20, MonetDB5 Server 5.2, MonetDB5/SQL 2.20.
This release marks a long summer of mostly core software development and maintenance. The restructuring of the source tree was partly successful. Semantic dependencies amongst packages, however, forced us to release more components together than originally planned. Close interaction with some large users provided valuable input for bug fixes and performance improvements.
SQL
We added a way to bulk request sequence numbers. When loading (copying) data into tables which have an auto_increment (or alike) column lots of time was spend in the many calls to get all the sequence numbers (observed a performance in crease of about 10x when loading pieces of 100K rows) TINYINT was added to the standard repertoire. We changed how selections over multiple columns of a single table are handled. Furthermore, several bugs stemming from large-scale deployment have been resolved.
XQuery In the XQuery area many issues have been dealt related to the transaction safe update functionality. Furthermore, a value based indexing scheme has been added to speedup the search.
Clients
The mclient utility has undergone a major overhaul. It moved from a Spartan rendering of SQL results towards a more user friendly tabular rendering on ascii terminals. In passing, the option structure and command options has been stratified and extended with a few use full functions, e.g. piping the tabular result as a CSV to an external program. Use the --help for an overview.
MonetDB 5
Merovingian has been added as the gatekeeper of all SQL database services in a particular farm. It redirect clients to the proper connection. The accompanying Monetdb toolkit provides a textual frontend for assessing the status of a collection of MonetDB servers.
Bug fixes
A summary with the highlights is given below. The detailed list of changes can be found in the source code CVS change logs.
SQL
- 1806869 SQL: prepare output has unqualified table_name header
- 1790101 SQL: parser warnings
- 1786908 SQL+PF: loading both "sql_server" and "pathfinder" fails
- 1775550 SQL: implied conversion from numeric to varchar disconnect
- 1774312 SQL: conversion from decimal to varchar error
- 1774217 SQL: order by timestamps column with limit crash
- 1757923 SQL: Prepared statement type mismatch
- 1737777 select id not in another select...
- 1737668 Alter table to add Fkey....
- 1734143 Update fkey with a values not in the pkey...
- 1733712 Update pKey with the same value for different rows...
XQuery
- 1786397 XQ: SAX serialization not working properly
- 1771532 MIL-optimizer fails sometimes for big queries
- 1766414 XQ: third pf-add into collection fails
- 1763575 XQ: insert attribute gives ERROR in merged_union
- 1763495 XQ: illegal offsets in swizzle (2)
- 1760811 XQ: illegal offsets in swizzle
- 1758902 XQUF replace function seems to corrupt the database
- 1757327 XQ: attributes in 2nd doc don't show
- 1755466 XQ: printing special characters in comment nodes
- 1751684 XQ: merged_union error with fn:sum()
- 1747017 XQ: admin console images not installed
- 1743442 XQ: illegal XML syntax in serialized attributes
- 1743433 XQ: problem with ampersand in attribute value
- 1766173 Double for / position / sub-sequence bug
- 1760445 PF: test attribute_access_mposjoin_error.SF-1726928 fails
- 1760168 PF: wrong use of leftfetchjoin causes segfault in bat_cast
- 1741459 PF: ws_create inconsistently adds a collection node
- 1740502 PF: live-variable-limit-handling in mil_opt.c buggy
- 1737735 128MB RAM needed to compile '1'?
Miscellaneous
- 1774575 Single page manual damaged
- 1754452 NullPointerException in MonetConnection
- 1739356 TRACE,PROFILE,DEBUG should be impossible for JDBC and ODBC
- 1739353 EXPLAIN support from JDBC
- 1733744 Cannot connect via ODBC (from Powerbuilder 10)
June 2007
June 2007 mk Tue, 03/30/2010 - 20:38MonetDB 4.18, MonetDB/XQuery 0.18, MonetDB 5.0, MonetDB 5.0/SQL 2.18. The Mars release marks the official release of the MonetDB server 5 line as the basis for future investments in application development. This affects first and foremost the maintenance of SQL code base, where the MIL version is not being maintained actively anymore by the MonetDB team. A synopsis of the major improvements
- implementation of delete/update with cascading
- PSM procedure call implementation
- support for polymorphic functions
- autoloading of sql scripts upon server restart
WarningThe SQL catalog structure has been changed, which requires a database dump before you install the new version.WarningThe old Window installers do not always remove MonetDB DLLs. You may have to manually remove them from system32 before (re-)installing the new version. It concerns:
- libstream.dll
- libmutils.dll
- libMonetODBCs.dll
- libMonetODBC.dll
- libMapi.dll
MonetDB XQuery XQuery Update Facility has undergone a large series of stress tests to improve correctness, performance and robustness under concurrent updates. In many areas the functionality has been improved:
- a completely revised and enhanced type checking code
- the administrative console for XQuery
- automatic use of value indexing
- improved and extended XRpc functionality (javascript library)
- all XPath axes looplifted (siblings will come soon)
- Intel-based native Windows build (i.e., 64-bit)
This release containsPF/Tijah,, a text search extension for XQuery. It provides a flexible environment for setting up search systems by supporting out-of-the-box solutions for common tasks, like index creation and result ranking of XML elements. Work continuous on the algebra version compiler and its SQL backend code generator.Release model The months passed since the bug-fix release of February 2007 showed that the MonetDB product family has grown to a size that more frequent releases are only possible if we continue our decomposition process. The SQL and XQuery release dependencies have now been cut. Source distribution The Super Source Tarball has become a stable means to access and install the packages on Linux platforms. Bug fixes A detailed list of changes can be found in the source code CVS change logs. A summary with the highlights is given below.MonetDB/SQLBug fixes SQL/Core:
- 1734143 Update fkey with a values not in the pkey...
- 1733712 Update pKey with the same value for different rows...
- 1730987 cast of expression hangs
- 1725349 large COPY INTO with primary key
- 1724151 SQL: mixing auto-commit with explicit transactions
- 1723863 SQL Query Crashes Server - CRITICAL
- 1723791 rand() is not executed for every row
- 1720934 Bind Variable with CAST function Crashes Server
- 1720293 WITH queries works only once!
- 1718253 SQL: decimal handling errors
- 1715679 CREATE TABLE WITH DATA...
- 1715672 LOCAL AND GLOBAL temporary table
- 1715669 Temporary tables....
- 1715659 LIKE for time values...
- 1715553 Set default value.....
- 1714829 SQL: CHECK CONSTRAINT...
- 1714814 SQL: View creation using a function....
- 1714809 Infinite loop....
- 1711947 src/test/Update_Delete_action: missing stable output files
- 1711251 M5: mserver5 segfaults after the second query
- 1711242 M5/SQL: wrong result for range predicate
- 1710678 CREATE PROCEDURE...
- 1710677 DROP PROCEDURE.....
- 1709182 SQL: M4 and M5 crash on Solaris/x86 when SQL module is loaded
- 1703497 The mserve crash due the null control...
- 1701823 copy into from 'filename' not closed
- 1695290 SQL changing column default gives error
- 1691646 REVOKE authorization crash
- 1691344 SQL UNION with ORDER BY crash
- 1691098 Mserver/SQL crashes on empty window_specification
- 1688915 SQL: crash on nested query
- 1684798 LIKE clause
- 1670739 SQL: Corrupt database after restart
- 1659889 SQL: sequence not restored properly
Bug fixes XQuery:
- 1731850 Error in preceding axis
- 1730556 XQ/JDBC: 250 results
- 1730547 XQ: Mserver crashes on concatenated query
- 1726954 XQuery: inserting (large) doc into other doc
- 1726928 XQuery: Attribute access causes mposjoin error
- 1726599 XQuery: collection management broken
- 1726548 XQuery: Administrative Console
- 1725352 PFtijah: failing alloc() causes assertion failure (segfault)
- 1724118 PF: coreopt optimization eats all available memory and swap
- 1723931 XQuery compiler warning treated as error
- 1723217 PF: compiler/sql/lalg2sql.brg fails to compile w/o assertion
- 1722078 PF: WebSite tests books[123] fail since 2007.05.03
- 1721701 PF: "!ERROR: Division by zero" after latest checkins
- 1719164 PF: new loop-lifted following step still buggy?
- 1718709 PF: select deleted node results in error
- 1718622 PF: strange behaviour of "rename"
- 1718066 PF: put() to /tmp crashes Mserver
- 1716158 PF: compiler/algebra/opt/opt_thetajoin.c icc-compil. fails
- 1712578 Mix-up in nested element construction
- 1706520 XQuery: boolean every
- 1705898 "forgot to cope with type 'double' in cast to integer"
- 1682059 XQ: internal integer representation
- 1698498 id-function fails to locate some kinds of nodes
- 1684416 Pathfinder does not compile due to undefined LLFMT
- 1718709 PF: select deleted node results in error
- 1718635 PF: node's sibling not accessable after deletion of the node
- 1718622 PF: strange behaviour of "rename"
- 1718066 PF: put() to /tmp crashes Mserver
- 1714603 PF: XRpc tests fail on MacOS (Darwin) since
- 1714399 XQuery/XRPC: refresh admin-page causes segmentation fault
- 1714360 PF/runtime: find_qn_bulk() adds new qnames twice
- 1712595 Error mposjoin: missing matches
- 1711754 shredder msync() bug
- 1706438 PF: property error when adding a doc. to an existing coll.
- 1683730 XQuery: mserver (0.16.2) sometimes crashes on complex query
- 1681921 XQ: Character escaping is bogus
- 1678593 XQ: document added/deleted with error occured
- 1676443 XQ: Interleave of put and doc via mod. func. crashes
- 1719164 PF: new loop-lifted following step still buggy?
- 1712463 PF: various tests fail since 2007.05.04
- 1712461 PF: several tests fail since 2007.05.03
- 1712459 PF: several tests fail since 2007.04.28
- 1712458 PF: prop. err. in update tests since 2007.04.26 (64-bit OIDs
- 1712456 PF: various tests fail since 2007.04.04
- 1712455 PF: test fails since 2007.04.03
- 1712454 PF: tests/W3C_use_cases/XQ/XMP/Q04alg fails since 2007.04.03
- 1712449 PF: "bogus_text_step" test fails since 2007.03.08
- 1714133 PFtijah: "/tmp/TIJAH_LOGGER" never cleaned => fills /tmp/ !
- 1714129 PFtijah: hard-wired "/tmp/TIJAH_LOGGER" not portable
Bug fixes APIs:
- 1733744 Cannot connect via ODBC (from Powerbuilder 10)
- 1675795 Float to decimal conversion still broken in v5.0.0_beta1_2
- 1671737 JDBC: exception in getChallengeResponse
- 1670732 XQ/JDBC: escaping backslash
- 1693694 ODBC driver not honoring host, username, or password
- 1473288 SQLTables fails if previous statement failed
- 1702698 Mapi < problem
Bug fixes Kernel:
- 1713584 delete on committed dense persistent BAT causes prop. error
- 1709519 M5: Linux process killed
- 1706713 Database corruption after out-of-memory condition
- 1705726 crash in leftfetchjoin due to lack of range checking
- 1704972 GDK change causes FATAL BATpropcheck with XQuery
- 1686756 Implicit declaration of function .fdatasync.
- 1679800 Mserver segfaults during recursive module loading
- 1673410 STREAM_DEBUG causes compilation error in stream.mx
January 2007
January 2007 mk Tue, 03/30/2010 - 20:39MonetDB 4.16, MonetDB/SQL 2.16, MonetDB/XQuery 0.16, MonetDB 5.0.0_beta1, MonetDB 5.0.0_beta1/SQL 2.16.
A bug fix release was made available on Feb 26, 2007. See the notes at the end of this paper.
The Venus release represents a major shift in functionality. First, we released the long awaited for XQuery Update Facility. As far as we know, it is the first W3C compliant XUF implementation available and provides full transaction control.
The next step was the
big bang
change in the packaging of the server layer. The successor, MonetDB Version 5 is moved to SourceForge. The MonetDB/SQL software line is available with both back-ends, but for Version 4 it is the final release. Finally, redesign over the website has lead to a better navigational structure and much more information on the product lines. A detailed list of changes can be found in the source code CVS change logs. A summary with the highlights is given below.
Source distribution MonetDB Version 5.0.0_beta1 was prepared and shipped to SourceForge.
SQL front-end The SQL functionality has been extended to support SQL triggers and fully functional Persistent Stored Modules. This extension was inspired by our ongoing port of the Skyserver application. Dependencies between SQL objects are stored now and dropping objectsrespect the 'strict' object dependency rule. Loading large tables have been significantly improved and the heuristic query optimizer covers a much larger space.
XQuery This release contains an implementation of XUF (XQuery Update Facility) based on the W3C working draft. In addition, the release adds a number of significant other features to XQuery:
- "multi-document collections" allow much more efficient management of XML databases that consist of *many* (small) documents, a weakpoint of the previous releases.
- XRPC: distributed query processing between multiple MonetDB servers; also useful as SOAP API.
- document management interface: the XML document database can now be controlled fully from XQuery (no MIL needed anymore).
- improvements in query performance.
Kernel improvements The memory mapped file functionality has been signficantly changed to accomodate efficient processing of XQuery/XUF.
Documentation The revamped website became alive.
Bugfix release Feb 26,2007
New versions of all components of the MonetDB suite are released.
- MonetDB/XQuery
The handling of updates was incomplete. Several memory mapping and concurrency problems have been fixed. Moreover, empty pages that resulted from deletion of nodes are now properly handed back to the system for reuse. - MonetDB/XRPC
Security and concurrency handling has been improved. - MonetDB/SQL
Foreign key constraint checking fixed; View in view dependencies bug fixed; Resolved crashes caused by large views; Prepared statements with more than 30 arguments now work. - MonetDB/API
Haskell and TCL libraries were broken and have been removed; Perl and Python libraries were broken and have been fixed. The current version of the protocol is not compatible with the previous version. - Bugs fixed:
[ 1579510 ] XQ: update error for "(1,2)/."
[ 1590523 ] XQ: substring-after
[ 1636828 ] fork() behaves strangely
[ 1637867 ] XQ: seqcast int? to numeric unsupported
[ 1639108 ] mserver crashes on delete query
[ 1640248 ] PF: ~500 tests randomly fail on Solaris x86
[ 1644245 ] M5 & SQL5: various tests hang/timeout on Darwin
[ 1648303 ] database dump does not save/restore auto-increment sequences
[ 1650469 ] SQL: Armada sinks MonetDB ships
[ 1651567 ] yet another delete bug, foreign key related
[ 1651599 ] caching of inserts causes extreme memory consumption
[ 1655818 ] SQL:Error inserting float in numeric column using JDBC
[ 1655857 ] XQ: call to builtin function crashes XRPC client
[ 1659889 ] SQL: sequence not restored properly
[ 1644610 ] Conflict between normal query and XRPC query
(and many more that hadn't been reported)
November 2006
November 2006 mk Tue, 03/30/2010 - 20:39MonetDB 4.14, MonetDB/SQL 2.14, MonetDB/XQuery 0.14, MonetDB 5.0_beta1
This release was created in November 2006, but abandoned a few weeks later as the code base was too unstable to warrant a public release.
June 2006
June 2006 mk Tue, 03/30/2010 - 20:40MonetDB 4.12, MonetDB/SQL 2.12, MonetDB/XQuery 0.12, and MonetDB/Five Alpha Release 1 were released on June 23, 2006.
This MonetDB release marks the end of a period where much attention was given to extend the XQuery/Xupdate functionality and efficiency. Although not finished, the many code improvements are valuable to the extended client base. Furthermore, this release marks a feature freeze for XUpdate functionality, claiming enough time to asses its stability and performance before an official release is announced. You may encounter some loose ends and rough edges. A detailed list of changes can be found in the change log of the CVS repository. A summary with the highlights is given below.
Source distribution The MonetDB Version 5 Alpha Release was prepared. It shares the SQL front-end with version 4.
SQL front-end Preliminary support for triggers was added. Now local and global temporary tables are supported. They caused a large rewrite of the session/transaction management. Next to these improvements, many small and large bugs have been fixed. Most notably are fixes to key constraint handling and concurrent transactions.
XQuery All open bugs of XQuery have been addressed, significantly improving its functionality. Serialization, a critical component in any XQuery implementation, has been revamped to obtain a 5-10x speedup in this area. Special attention has been given to parent/ancestor steps. Now they work many times faster inside for-loops (have been loop-lifted) Furthermore, we added XPath extensions for XML standoff annotations (--set standoff=enabled as runtime option to Mserver)
The development of XUpdate was set back by the necessary changes to the underlying MonetDB engine to support BATs with holes in the void sequences. More testing is needed to remove the rough edges.
APIs
The PHP library on Linux has been revamped. It is now based on the PostgreSQL PHP library code.
Kernel improvements The BAT recycle cache has been completed. This greatly improve the performance for applications that create/destroy an abundance of small BATs.
The transaction semantics of the kernel has been extended to support a sub-commit of BATs. This prepares the road to combine SQL and XQuery in one Mserver in the future.
Windows port Cross- and native- compilation for 64-bit Windows platforms versions has somewhat be improved, but is not complete yet. The distribution section will shortly contain a 64-bit version for download.
Documentation improvements Some minor changes, but still an area with a potential great return-on-investment.
January 2006
January 2006 mk Tue, 03/30/2010 - 20:40MonetDB 4.10, MonetDB/SQL 2.10 and MonetDB/XQuery 0.10 were released on Jan 24, 2006. This MonetDB release marks the end of a period where much attention was given to support simple SQL and XQuery applications efficiently. These improvements came from careful analysis of student programs and participating in the c't Magazine DVDstore benchmark.
A detailed list of changes can be found in the change log of the CVS repository. A summary with the highlights is given below.
Source distribution The MonetDB software base has been split to ease source distribution. The tools to prepare the source compilation (Mx), source code generators (MEL, Burg) and makefile generators (autogen) are assembled in the buildtools package. They are not needed anymore to recompile the source code distributed. It involved major changes to the configuarion scripts. A complete clean rebuild is advised.
SQL front end The SQL catalog management has been completely overhauled. It provides much better performance for lightweight queries and provides a cleaner interface for transaction management. All changes to a base table are assembled in insertion/deletion BATs, which can be merged at transaction commit with the base tables.
Sequences support has been added along the specification of SQL 2003. This allows for identity columns, serials (PostgreSQL) or auto_increments (MySQL) in MonetDB. Several minor changes to the build-in operators and syntax recognition, e.g. uchr has been dropped and temporal support has been improved.
SQL server's default transaction mode is now AUTO COMMIT. Support for SET AUTO_COMMIT=true/false has been removed and has been replaced with the lower level Xauto_commit statement issued from the communication libraries. AUTO COMMIT can be switched on and off using the SQL statements START TRANSACTION and COMMIT or ROLLBACK.
There were many small changes to the SQL back-end, such as private name space for cached queries and a large cleanup of the SQL result set code. The protocol for sending results set has been optimized to reduce the communication overhead.
Most BAT related code has been moved to their own directory to prepare for future different storage back-ends. This directory also contains the transaction and logger code. The transaction management has been inproved to allow for reusing the structures which store the transactions catalog information.
XQuery MonetDB/XQuery performance has been boosted using an index node and 'prepared' query/result caching. It now also supports XQuery module definitions. The performance of small XQueries has noticeably improved.
APIs
The experiments with the c't Magazine benchmark led to a re-design of parts of the MAPI protocol, aiming for less overhead in header analysis and falling back to a single exchange mode (blocked) for JDBC. It proofed to increase performance with at least a factor two.
JDBC has been largely revamped for additional speed improvements. Support for retrieval of more metadata has been added as well. The stage has been prepared to also support server-side redirections for future use.
The Perl DBI library has greatly be improved. It uses the MAPI functionality directly now. The module has been registered at the official Perl archive (CPAN)
ODBC has been upgraded to align with the protocol changes and also supports native (server side) prepared statements.
PHP, Python: No changes.
MIL The MIL internal representation (YYTREE) has been overhauled to improve execution of small XQuery operations. Minor language changes, e.g. the module statement is limited to a single argument.
Kernel improvements The BAT buffer pool re-uses the file names of BAT locations and the default temporary names. This improves handling of small BAT creations. The code has been prepared for re-use-garbage in the near future.
A constant module has been added. It supports BAT operations with a column designated by a scalar value.
Windows port A new windows installer preparation scheme under Linux has been deployed.
Documentation improvements Some minor changes, but still an area with a potential great return-on-investment.
May 2005
May 2005 mk Tue, 03/30/2010 - 20:41MonetDB 4.8 & MonetDB/SQL 2.8 were released on May 29, 2005. They were accompanied by the first MonetDB/XQuery 0.8 release (the Mercury release). A detailed list of changes can be found in the change log of the CVS repository. A summary with the highlights is given below.
SQL front end Added SQL statement modifiers EXPLAIN, PROFILE and PREPARE/EXEC. Explain returns the query execution plan as a MIL program and the query is not executed. PROFILE executes the query and stores profiling information in the 'history' table. The prepare and execute work together and can be used by the client programs to implement PREPARE and EXECUTE functionality. The SQL module functionality has been included . They can be used to introduce a module written in C or a plain SQL module. Modules can add new types, functions and tables to the system. [TODO] See for example the 'url' example module.
JDBC driver
- JdbcClient
- support for reading (gzipped) files from http web locations
- better dumping support for time zones on temporal types
- improved warning feedback system
- Connection
- client time zone support for support of INSERT INTO queries which deal with time zones
- update to intermediate Mapi authorisation scheme with support for multiple catalogs if the server supports it
- sliding window fetch size which improves speed on large result sets
- various small changes to improve overall processing speed (not necessarily simplifying the code)
- DatabaseMetaData
- updated actual limits of the database measured in maximum column lengths
- Driver
- added support for multiple catalogs
- PreparedStatement
- fixed bug with Float processing (returned null)
- ResultSet
- various small fixes to improve performance
- preliminary support for retrieving Clobs
- fixed bug with Float processing
- fixed time zone behaviour (also made it more efficient)
- MonetSocketBlockMode
- made small optimalisation in internal buffer management
- Statement
- if fetchSize is not set, a sliding window fetch size is now being used
- Clob
- preliminary implementation added
Application interfaces The PHP interface has been finalized. The Perl DBI library interface has reached a mature level.
Kernel improvements The kernel underwent some renovation to improve its usefulness in future releases. The BAT storage structures have been trimmed to consume just 1/3th of the previous space, which can be used to reduce the overhead of short queries by caching all descriptors in main memory. The BAT structure also underwent a revision to accomodate a more flexible property maintenance list. The changes have be shown to improve processing of 'compiled' small MIL queries to run up to 30\% faster.
Windows port The windows port can now be build both using Microsofts Compiler suites and using MinGW.
Documentation improvements The documentation for PHP5 installation and interaction has been added.
February 2005
February 2005 mk Tue, 03/30/2010 - 20:42MonetDB 4.6.0 & MonetDB/SQL 2.6.0 was released on February 14th, 2005, (the Valentijn release). Since the previous release, many improvements have been made. A detailed list of changes can be found in the change log of the CVS repository. A summary with the highlights is given below.
SQL front end MonetDB/SQL now survives the famous 'crashme' program used by MySQL to analyze the features of a DBMS implementation.
Several improvements with regard to the SQL integrity and compliance with the standard were made, as well as new supporting features:
- The LIKE operator has been overhauled for better support for search expressions
- Closer conformance to the SQL standard for types
- INT(x) is no longer allowed/supported
- SMALLINT, INT and BIGINT are now supported 'native' types
- convenience aliases for TINYINT (mapped on SMALLINT) and MEDIUMINT & INTEGER (mapped on INT)
- CHARACTER VARYING and VARCHAR now need a mandatory length specification, ie. VARCHAR(x)
- added support for CHARACTER LARGE OBJECT next to CLOB
- Constraints that are not given a name explicitly now get an explanatory name, ie. mytable_mycolumn_pkey instead of c102773
- Improved dumping support for MapiClient: now constraints and indexes are also dumped
- Various fixes and optimizations in JDBC and JdbcClient
- resolved a bug where the JDBC driver would hang if the server would stop sending results prematurely
- improved support for types, ie. better and more consistent mapping from server to JDBC types
- Newly implemented java.sql.ResultSet methods: getCatalogName(), isCaseSensitive(), isSearchable(), isCurrency(), isSigned() and getColumnClassName()
- JdbcClient now correctly escapes identifier names in database dumps and describe actions
- added -e of --echo flag to JdbcClient to echo input while reading from a file: results in a clear dialog, instead of answers only
- small optimizations in the socket code, amongst others: avoidance of unnecessary memory copy, skipping of empty blocks.
- SQLWarnings are now added to the warning lists of the Connection, Statement, and ResultSet objects for various problems or inconsistencies encountered by the driver, that are not fatal.
- INTERVAL works with TIME and TIMESTAMP
- Improvement of SQL parser errors: can lead to exact specification of what is expected in certain cases
Mbedded The code base has been reduced and the configuration parameterized to allow MonetDB to be linked with your application. The smallest system we have been able to run MonetDB was a single-board Linux computer, called the Gumstixs.
Application interfaces The MapiClient command line tool has been extended to enable automatic timing. Reading scripts as part of the interaction is now supported.
Steffen Goeldner joined our team and completely revamped the Perl DBI library interface.
JDBC: see above (section "SQL front end").
MIL cleanup The MIL syntax cleanup, started in version 4.4, was finished. The parse now denies all "legacy" (i.e., pre 4.4.0_rc0) syntax alternatives.
Kernel improvements MonetDB was known to crash on for example large cross-product calculations. This has been prevented by better boundary checking code at several places.
Documentation improvements The MonetDB source code has been largely moved to texi as a preparation step to improve documentation generation.
Code base improvements
Source code indentation has been normalized throughout the code base.
September 2004
September 2004 mk Tue, 03/30/2010 - 20:42Version 4.4.2 was released on October 26, 2004. It is primarily a bug-fix release.
SQL front end Several bugs affecting the SQL integrity were fixed:
- Unique constraints were not checked in auto-commit mode
- The LIKE operator returned wrong matches on patterns starting with %
- Deleted rows reappeared on conditional selections
- Removal of legacy variables in system tables
- ALTER TABLE in auto-commit mode generated error "Error in result"
- Leakage of BATs caused crashes after heavy load with updates
ODBC and JDBC JDBC suffered from a bug when non-ASCII characters were used in queries. Due to an incorrect charset handling the driver locked up. Another bug returned the incorrect username when being asked for the current logged in user using JDBC's DatabaseMetaData.getUser().
Synopsis of features available in Version 4.4.0
Version 4.4 was released on September 30, 2004 and the first official release. It was preceded by version 3.14, 3.16 and 3.18 delivered in 2004, which were all focused on reaching the point where researchers and casual users would benefit from the MonetDB solution. The feature list below summarizes the efforts spent.
SQL front end A fairly complete SQL-99 front end has been constructed. Amongst others, it provides schema management, user authorization, view definitions, referential constraint enforcement, time zone support, sub-queries, and query caching to speedup processing.
ODBC and JDBC The predominant application programming languages are supported using the MonetDB Application Programming Interface (MAPI) library. Amongst others, it is used to construct a functional rich ODBC implementation, which allows you to retrieve data from a MonetDB server from inside an Excel spreadsheet. The JDBC code is a class 4 implementation. It has been used to link MonetDB with other database application development environments.
MIL cleanup The MonetDB Interface Language (MIL) was originally designed as an intermediate and debugging language. Its definition has been brought more in line of a real programming language, without loosing its flexibility to write concise scripts.
Kernel improvements Heavy internal use revealed bugs and performance issues. In combination with the lengthly bug list compiled by our commercial partner over the years, we have reached the point that serious crashes have become a rarity.
Documentation improvements Documentation in a research setting often comes last, but we are confident that the information provided is sufficient for both a casual end-user, building an application like the VOC, and the demanding researcher, interested in our research on cache-conscious code.
Testing MonetDB is tested daily on 17 different platforms, configured with widely different hardware profiles, but also a variety of compilers. The MonetDB code base was cleaned up to the point that all compiler warnings were dealt with. Currently, it compiles with the strongest compilation options available. Testing has reached the point that a day with 24 hours is not enough to run and analyze the test suite.
Packaging Distribution of a software package as rich and complex as MonetDB calls for major (human) investments in software engineering tasks. The software heavily relies on open-source packages such as automake, auto-config. For the Windows platform the necessary files are included as well.