- developers-list - monetdb.org

Announcement: New Oct2020-SP5 Bugfix release of MonetDB suite
by Sjoerd Mullender 10 May '21

10 May '21

The MonetDB team at MonetDB BV is pleased to announce the Oct2020-SP5 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://www.monetdb.org/downloads/>. Oct2020-SP5 Bugfix Release (11.39.17) Windows Packaging * Added the monetdbe library to the Windows installer. MonetDB Common * A bug that would very occasionally produce an error "strPut: incorrectly encoded UTF-8", even when no incorrectly coded characters are used at all, has been fixed. It was the result of a rare combination of strings having been added to the column that caused essentially an off-by-one type of error to occur. SQL Frontend * A bug where a sequence of TRUNCATE TABLE and COPY INTO the just truncated table would result in success being reported to both queries, but the table still being empty afterwards, has been fixed. Merovingian * When stopping monetdbd using the `monetdbd stop' command, this command now waits for 5 seconds longer than the exittimeout value before it kills the monetdbd daemon instead of only 30 seconds total (or until that daemon stops earlier). This gives the daemon enough time to terminate the mserver5 processes that it is managing. If exittimeout is negative, the daemon and the monetdbd process initiating the stop wait indefinitely until the mserver5 processes have stopped. Bug Fixes * 3336: DB files not removed if all rows are deleted, even after restart * 7104: Monetdbe NTILE function does not produce correct ordering * 7108: Monetdb crashes on query execution * 7109: MERGE Statement incorrectly reports that input relation matches multiple rows * 7110: Monetdb Query parsing consistency issues in the latest release (Remote Table)

1 0

Announcement: New Oct2020-SP4 Bugfix release of MonetDB suite
by Sjoerd Mullender 12 Apr '21

12 Apr '21

The MonetDB team at MonetDB BV is pleased to announce the Oct2020-SP4 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://www.monetdb.org/downloads/>. Oct2020-SP4 Bugfix Release (11.39.15) ODBC Driver * When connecting using a DSN (Data Source Name), information about the data source is retrieved from the ODBC.INI file. Now we also get the location of the LOGFILE from this file. The logfile can be used to log all calls to the MonetDB ODBC driver to a file which can be used for debugging. * The ODBC driver now only passes on information about HUGEINT columns as HUGEINT when the application has indicated interest by querying about the SQL_HUGEINT extension type using the SQLGetTypeInfo function or by specifying the type in a call to SQLSetDescField. Otherwise the driver silently translates the HUGEINT type to BIGINT. This means that most application will see BIGINT columns when the server produced a HUGEINT column and only give an error if the value in the HUGEINT column didn't fit into a BIGINT. Bug Fixes * 6786: function json.isvalid(js json) is not useful, could be removed * 7016: Database crashes when use similarity function on a table with more than 200k records * 7037: Clearer err msg for ALTER USER with insufficient privileges * 7042: AddressSanitizer:DEADLYSIGNAL in Oct2020/gdk/gdk_tracer.c:494 * 7050: file descriptor leak when forward=redirect * 7057: ODBC driver installer on Windows is missing some DLLs * 7058: MonetDBe: COPY INTO csv file does not produce any output * 7059: MonetDBe: 'reverse' C UDF crashes * 7061: Have bulk load support combined gzip files * 7064: Temporary hashes created in hash and unique logic should try to use transient data farm first * 7066: percent_rank function with wrong results * 7070: double free error when running MonetDBe Example * 7076: mserver5 ignores memory.low from cgroups v2 * 7077: Oct2020: new default privileges not effectively communicated * 7083: MonetDBe C++ Compiling Error * 7085: Mitosis and filter functions * 7087: SIGSEGV caused by error in subquery's function being ignored by top-level query * 7089: Data consistency problem of query results in the latest release of Monetdb (Remote Table)

1 0

monetdbe, fetching date, daytime, timestamp in their MonetDB native memory storage format
by Daniel Zvinca 02 Mar '21

02 Mar '21

Hello, Since the first version of monetdblite (monetdbe now), passing/fetching the date and time columns (in monetdbe or capi code for instance) is done using intermediary structures, instead of native types (int, __int64). typedef struct { unsigned char day; unsigned char month; short year; } monetdbe_data_date; typedef struct { unsigned int ms; unsigned char seconds; unsigned char minutes; unsigned char hours; } monetdbe_data_time; typedef struct { monetdbe_data_date date; monetdbe_data_time time; } monetdbe_data_timestamp; typedef struct { unsigned char day; unsigned char month; int year; } cudf_data_date; typedef struct { unsigned int ms; unsigned char seconds; unsigned char minutes; unsigned char hours; } cudf_data_time; typedef struct { cudf_data_date date; cudf_data_time time; } cudf_data_timestamp; I am wondering if this conversion is truly necessary, considering that this effort is usually doubled by either converting back to native format (for append operations, for instance) or extra conversion in other flavours of date/time structures some other C code might need. The very few macros required for time information extraction from the native types (is_nil, extract) can be done on the client side when needed. Related to this, a low level function "monetdbe_result_scan" alternative to "monetdbe_result_fetch" would avoid extra memory allocation in case no storage is required or if a different storage for the result (user list, compressed, straight on disk, ...) is considered by a developer. For integers, doubles, float, date, daytime, timestamp types passing the data address Tloc(b, 0) in the same way GENERATE_BAT_INPUT is doing it, looks good enough, assuming the developer knows to use that memory address for reading purposes. For the rest of the types (str, blob, others), invoking a pointer to a provided user function (void *data, size_t data_size, size_t j) inside the BATloop would suffice. This "monetdbe_result_scan" part can be adjusted by a custom development, sure, but passing date-time types columns in native format (which can come from a remote connection as well) should be uniform. Having a version ID for the monetdbe_result structure would also help in case a remote connection can be instructed to provide results in different ways. Dan

2 1

Announcement: New Oct2020-SP3 Bugfix release of MonetDB suite
by Sjoerd Mullender 19 Feb '21

19 Feb '21

The MonetDB team at MonetDB BV is pleased to announce the Oct2020-SP3 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://www.monetdb.org/downloads/>. Oct2020-SP3 Bugfix Release (11.39.13) Bug Fixes * 7049: Implement DISTINCT for GROUP_CONCAT

1 0

Announcement: New Oct2020-SP2 Bugfix release of MonetDB suite
by Sjoerd Mullender 25 Jan '21

25 Jan '21

The MonetDB team at MonetDB BV is pleased to announce the Oct2020-SP2 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://www.monetdb.org/downloads/>. Oct2020-SP2 Bugfix Release (11.39.11) SQL Frontend * CREATE [OR REPLACE] TRIGGER schema_name.trigger_name is now disallowed, because the trigger will be stored on the same schema as the table it refers to. Use a schema-qualified on the table reference (ie ON clause) when necessary. Bug Fixes * 6862: mserver5: crashes under update_table() when calling lib_sql.so ( max_clients = 2048) * 7002: monetdb stop fails * 7012: mclient enters an infinite loop when a file on the command line does not exist * 7013: Select * on grouped view: wrong error "cannot use non GROUP BY column 'a1' in query results without an aggregate function" * 7017: mal seems to leak in functions * 7020: release an older savepoint causes "BATproject2: does not match always" * 7021: savepoints crash mserver5 * 7022: transaction with an unreleased savepoint not properly persisted * 7023: CREATE VIEW: SELECT: cannot use non GROUP BY column '%1' in query results without an aggregate function * 7024: DELETE FROM or TRUNCATE on freshly created table leads to loosing all further inserts in same transaction * 7030: DROP TABLE with AUTO_INCREMENT doesn't drop sequence causing left-over dependency * 7034: User with sysadmin role cannot create another user * 7035: UPDATE and SELECT column privileges * 3772: Any user can grant a role.

1 0

MonetDB bug tracker has moved to github
by Sjoerd Mullender 01 Dec '20

01 Dec '20

The MonetDB bug tracker has been moved from our self-hosted bugzilla instance to github issues. All bug IDs have remained the same in the transition, making referencing bugs easier. The new bug tracker can be found at https://github.com/MonetDB/MonetDB/issues Don't forget to follow this tracker (if you're interested in doing that) or follow individual bugs. Due to the way we had to do the move, the old stars that you may have given to the MonetDB github repo have been lost, so please, star it again (or for the first time) if you feel so inclined. The old bugzilla tracker will remain on-line for the time being but entry of new bugs has been turned off. Shortcuts to bugs are forwarded to github. (Shortcuts are URLs in the form of https://www.monetdb.org/bugzilla/<bugid> and https://bugs.monetdb.org/<bugid>.) -- Sjoerd Mullender

1 0

MonetDB bug tracker is going to move
by Sjoerd Mullender 29 Nov '20

29 Nov '20

LS, The MonetDB team wishes to announce the end of our reliance on Bugzilla for bug tracking and enhancement requests. Instead, we switch over to the MonetDB Github issues repo (https://github.com/monetdb/monetdb), which is an hourly based mirror of the Mercurial repository managed at the development group. The move will take place on Friday, November 27, 2020. The actual transfer of the Bugzilla archive into the Github Issues may take a number of hours. A redirect will be in place to send people from Bugzilla towards the MonetDB GitHub repository instead. -- Sjoerd Mullender

1 1

Announcement: New Oct2020-SP1 Bugfix release of MonetDB suite
by Sjoerd Mullender 24 Nov '20

24 Nov '20

The MonetDB team at MonetDB BV is pleased to announce the Oct2020-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://www.monetdb.org/downloads/>. Oct 2020-SP1 bugfix release (11.39.7) Bug Fixes * 6890: Add support of xz/lzma (de)compression on MS Windows * 6891: Add support of lz4 (de)compression on MS Windows * 6971: Parsing table returning function on remote server fails * 6981: Oct2020: PREPARE DDL statement silently fails * 6983: monetdb allows to use non-existing optimizer pipe * 6998: MAL profiler buffer limitations * 7001: crossproduct generated for a simple (semi-)join * 7003: Segfault on large chain of constant decimal multiplication * 7005: Dropping a STREAM TABLE does not remove the associated column info from sys._columns * 7010: deallocate <id> results in all prepared statements being deallocated (not error-related) * 7011: uuid() called only once when used in projection list

1 0

Announcement: New Oct2020 Feature release of MonetDB suite
by Sjoerd Mullender 19 Oct '20

19 Oct '20

The MonetDB team at MonetDB BV is pleased to announce the Oct2020 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://www.monetdb.org/downloads/>. Oct 2020 feature release (11.39.5) MonetDB5 Server * The settings for specifying how mserver5 should listen to "The Internet" have been overhauled. See the manual for details. In brief, mapi_autosense, mapi_ipv6 and mapi_open are gone. If mapi_listenaddr equals "localhost" or "all", we listen to both IPv4 and IPv6 (if available), if "127.0.0.1" or "0.0.0.0", we listen to IPv4 only, if "::1" or "::" we listen to IPv6 only. The first of each pair is loopback interface only, the second is all interfaces. If mapi_listenaddr is "none", then no IP port is opened, you need to use a UNIX domain socket. If mapi_port is 0, we let the operating system choose a free port (like mapi_autosense). Default behavior has not changed. Whole System * Finished a first version of the new monitoring function user_statistics(), which is only intended for the DBAs. For each database user who has logged in during the current mserver5 session, it returns "username": login name of the database user, "querycount": the number of queries this user has executed since his/her first login, "totalticks": the total execution time (in microsecond) of the queries ran by this user, "maxquery": the query with the longest execution time (if two queries have the same execution time, the newer overwrites the older), "maxticks": the execution time of the 'maxquery' (in microsecond), "started": the start timestamp of the 'maxquery', "finished": the finish timestamp of the 'maxquery'. * Removed support for LiDAR data, that is the SQL procedures sys.lidarattach, sys.lidarload, and sys.lidarexport. * The sys.queue() has been turned into a circular buffer to allow for inspection of both active, paused and recently executed queries. Client Package * mclient and msqldump now also look in $XDG_CONFIG_HOME for the monetdb configuration file. SQL Frontend * Extended the system monitor with a table-returning function user_statistics() which keeps some statistics for each SQL user, e.g. the user's query count, total time spent, and maximal query seen. * Removed '@' syntax used to refer into a variable in a query. It was a non-standard method, which was replaced by a schema addition to variables. Variables in the global scope now have schema. All default global variables are set under schema "sys". However variables inside PSM don't have a schema, because there are no transaction semantics inside PSM at the moment. * Removed declared variables and tables from the global scope. They were transaction agnostic and incompatible with the SQL standard, i.e. they are valid exclusively under PSM (e.g. functions, procedures and triggers). * Scoping semantics were added for both variables and tables. Variables with the same name at a query are now resolved under the following precedence rules: 1. Tables, Views and CTEs at the FROM clause. 2. Variable declared in the body of function/procedure, i.e. local variable. 3. Function/procedure parameter. 4. Variable from the global scope. Tables with the same name now have the following precedence rules at a SQL query: 1. Table declared in the body of function/procedure, ie local table. 2. Temporary table. 3. Table from the current session schema. This means the query: SELECT * FROM "keys"; will list keys from temporary tables instead of persisted ones, because "keys" table is available for both "sys" and "tmp" schemas. * The table returning function "var" was extended with more details about globally declared variables, namely their schema, type and current value. * Removed obsolete sys.getContent(url) function. * Removed obsolete json.output(json) function. * Removed compatibility between interval types and other numeric types in favor for a more strict SQL standard compliance. This means operations between temporal types and other numeric types such as INT and DECIMAL are no longer possible, instead use interval types. e.g. SELECT date '2020-01-01' + 1; now gives the error. Instead do: SELECT date '2020-01-01' + interval '1' day; if 1 was meant to be a day interval. Setting an interval variable such as the session's current timezone with a number e.g. SET current_timezone = 1; is no longer possible. Instead do SET current_timezone = interval '1' hour; Casting between interval and other numeric types is no longer possible as well, because they are not compatible. * Because of incompatibilities this change may create, if a user intends to convert a numeric value to an interval, the multiplication function can be used in the form: <numeric value> * interval '1' <interval length> e.g. 10 * interval '1' second = interval '10' second. As for the other way around, the 'EPOCH' option was added to the extract syntax. This option returns the number of milliseconds since the UNIX epoch 1970-01-01 00:00:00 UTC for date, timestamp and time values (it can be negative). Meanwhile, for day and second intervals, it returns the total number of milliseconds in the interval. As a side note, the 'EPOCH' option is not available for month intervals, because this conversion is not transparent for this type. * Made general logarithm function log(x,base) compliant with the SQL standard, by swapping the input parameters. Instead of log(x,base), now is log(base,x). Bug Fixes * 3553: All schema access to ubiquitous functions * 3815: Incorrect results when expression contains implicit float/integer conversions * 6415: Date arithmetic types are inconsistent * 6814: provide native implementations for scalar functions sys.degrees(rad) and sys.radians(deg) * 6843: function sys.getcontent(url) always returns "Feature not supported" * 6857: remove not implemented aggregate function json.output(js json) * 6870: Missing bulk operators * 6878: SQL Connection Error when running SELECT queries containing AND command * 6910: SQLancer query: 'bat.append' undefined * 6930: SQLancer crash on join with coalesce * 6931: Allow EDITOR to be used for the current command in mclient * 6935: Wrong result when dividing interval by literal float * 6937: Lost the microsecond precisions * 6938: Segmentation fault in MalOptimizer * 6939: Error in optimizer multiplex when selecting profiler.getlimit() or wlc.clock() or wlc.tick() or wlr.clock() or wlr.tick() * 6941: SELECT queries on remote table fail when using LIKE in WHERE conditions * 6943: JSON parser is too permissive * 6948: msqldump with Empty BLOBs cannot be imported * 6949: Loosing timing precision * 6950: redundant/replicated code line in gdk/gdk_hash.c * 6951: Use a different naming scheme for MAL blocks * 6954: FILTER functions no longer find their implementation * 6955: ROUND(DECIMAL, PRECISION) gives incorrect result with non-scalar precision parameter * 6960: implementation of log(arg1,arg2) function is not compliant with the SQL standard, arguments are switched * 6962: "SELECT * FROM ids LIMIT 1" produces: exp_bin: !ERROR: Could not find %173.id * 6964: Table returning function: Cannot access column descriptor * 6965: Crash when using distinct on the result of a table returning function * 6974: Oct2020-branch cannot attach and load FITS files * 6976: Oct2020: default dbfarm cannot be started * 6978: Oct2020: \d shows empty result in schema created by include sql script * 6979: timestamp add integer * 6980: Oct2020: wrong mel definition for str.epilogue

1 0

New Oct2020 stable branch created
by Sjoerd Mullender 21 Aug '20

21 Aug '20

I have created a new stable branch for the next feature release. This branch was created in order to stabilize the code before the feature release. >From now on *only* bug fixes may go to this new stable branch. I reserve the right to undo any changes that do not comply with this rule. I also reserve the right to admit changes that are not strictly bug fixes. The name for the branch is Oct2020. So in each of your stable checkout directories you can do hg pull hg update -rOct2020 to update to this new branch. -- Sjoerd Mullender

1 0