Published on MonetDB (http://www.monetdb.org)

Home > User guide

User guide

A MonetDB database instance manages a single database on a single system. It is perfectly possible to have multiple database instances run concurrently on a single system, but its resource claim may negatively affect the performance perceived.

All instances on a system are governed by a MonetDB deamon process. It provides a single point of access to any of the database instances locally, but also to those running in a local network.  The deamon process ensures automatic (re) start of database instances when users request access, it can act as a proxy for clients, it can transparently handle fail-over  and it can exploit data-parallel processing.

Getting started

 VOCThe MonetDB/SQL tutorial is based on the material published in the book J.R. Bruijn, F.S. Gaastra and I. Schaar, Dutch-Asiatic Shipping in the 17th and 18th Centuries, which gives an account of the trips made to the East and ships returned safely (or wrecked on the way). A total of 8000 records are provided. They include information about ship name and type, captain, the arrival/departure of harbors along the route, personnel accounts, and anecdotal information.

Get going quickly on Linux involves starting the MonetDB daemon monetdbd, code name meriovingian. It runs in the background and facilitates, amongst others, management of local/remote servers.  The daemon is controlled by the application  monetdb. See its documentation for all details. Here we demonstrate a simple session involving creation of a database, loading data, and querying (Windows is described at the end).

shell> monetdbd create /path/to/mydbfarm
shell> monetdbd start /path/to/mydbfarm
shell> monetdb create voc
shell> monetdb release voc
shell> mclient -u monetdb -d voc
password:<monetdb>
Welcome to mclient, the MonetDB/SQL interactive terminal (Aug2011-SP2)
Database: MonetDB v11.5.4, 'voc'
Type \q to quit, \? for a list of available commands
auto commit mode: on
sql>select 'hello world';
+---------------+
| single_value  |
+===============+
| hello world   |
+---------------+
1 tuple (0.530ms)
sql>

The command \q or  end-of-file <Ctrl-d> signal terminates the connection with the server.

Exploring the wealth of functionality offered by MonetDB/SQL is best started using a toy database. For this we use the VOC database which provides a peephole view into the administrative system of an early multi-national company, the Vereenigde geoctrooieerde Oostindische Compagnie (VOC for short - The (Dutch) East Indian Company) established on March 20, 1602.  Download the VOC data set voc.sql.gz (519K) [1] bz2 (371K) [2] which is a compressed file with SQL statements. After the file has been extracted, load its contents into MonetDB using mclient.

Before you load the VOC data set, it is advised to introduce a user different from the omnipresent default monetdb. The new user is given his own schema for the MonetDB database. Assuming you have started MonetDB with SQL module, proceed as follows:

shell> mclient -u monetdb -d voc
password:<monetdb>
sql>CREATE USER "voc" WITH PASSWORD 'voc' NAME 'VOC Explorer' SCHEMA "sys";
sql>CREATE SCHEMA "voc" AUTHORIZATION "voc";
sql>ALTER USER "voc" SET SCHEMA "voc";
sql>\q

To illustrate the use of the newly created account and schema, the following example connects and creates a table, which is dropped afterwards by aborting the transaction.

shell> mclient -u voc -d voc
password:<voc>
sql>START TRANSACTION;
sql>CREATE TABLE test (
more>     id int,
more>     data varchar(30)
more> );
sql>\d
+------+
| name |
+======+
| test |
+------+
sql>\d test
CREATE TABLE "voc"."test" (
        "id" int,
        "data" varchar(30)
);
sql>ROLLBACK;

Importing the voc_dump.sql file into the database can be done using the textual client interface. Some alternative ways are as follows:

1:
shell> mclient -u voc -d voc voc_dump.sql
password:<voc>

2:
shell> mclient -u voc -d voc < voc_dump.sql
password:<voc>

3:
shell> mclient -u voc -d voc
password:<voc>
sql> \< voc_dump.sql

The VOC data set contains data for around 8000 voyages.

sql>\d
+------------+
| name       |
+============+
| craftsmen  |
| impotenten |
| invoices   |
| passengers |
| seafarers  |
| soldiers   |
| total      |
| voyages    |
+------------+
sql>select count(*) from voyages;
+--------+
| count_ |
+========+
|   8115 |
+--------+
The set consists of 8 tables, which are all bound to each other using FOREIGN KEY relationships. The voyages table is the main table, which all others refer to. Every table, except invoices has a PRIMARY KEY defined over the columns number and number_sup. Since the invoices table holds zero or more invoices per voyage (identified by number, number_sub) a PRIMARY KEY constraint is not possible. The tables craftsmen, impotenten, passengers, seafarers, and soldiers all share the same columns. We can define a VIEW that combines them all into one big table, to make them easier to access.

sql>CREATE VIEW onboard_people AS
more>SELECT * FROM (
more>SELECT 'craftsmen' AS type, craftsmen.* FROM craftsmen
more>UNION ALL
more>SELECT 'impotenten' AS type, impotenten.* FROM impotenten
more>UNION ALL
more>SELECT 'passengers' AS type, passengers.* FROM passengers
more>UNION ALL
more>SELECT 'seafarers' AS type, seafarers.* FROM seafarers
more>UNION ALL
more>SELECT 'soldiers' AS type, soldiers.* FROM soldiers
more>UNION ALL
more> SELECT 'total' AS type, total.* FROM total
more> ) AS onboard_people_table;
sql>

The new view will show up and we can just use it as a normal table, to for instance calculate the number of records for each group of people:
sql>\d
+----------------+
| name           |
+================+
| craftsmen      |
| impotenten     |
| invoices       |
| onboard_people |
| passengers     |
| seafarers      |
| soldiers       |
| total          |
| voyages        |
+----------------+
sql> SELECT type, COUNT(*) AS total
more> FROM onboard_people GROUP BY type ORDER BY type;
+------------+-------+
| type       | total |
+============+=======+
| craftsmen  |  2349 |
| impotenten |   938 |
| passengers |  2813 |
| seafarers  |  4468 |
| soldiers   |  4177 |
| total      |  2454 |
+------------+-------+
sql> select count(*) from impotenten;
+--------+
| count_ |
+========+
|    938 |
+--------+
It is possible to play with the set in many ways, to find out several things that took place during the voyages of the ships, or the money that was earned. A few examples are shown below:

sql>SELECT COUNT(*) FROM voyages
more>WHERE particulars LIKE '%_recked%';
+--------+
| count_ |
+========+
|    358 |
+--------+
sql> SELECT chamber, CAST(AVG(invoice) AS integer) AS average
more> FROM invoices
more> WHERE invoice IS NOT NULL
more> GROUP BY chamber
more> ORDER BY average DESC;
+---------+---------+
| chamber | average |
+=========+=========+
| A       |  282996 |
| Z       |  259300 |
| H       |  150182 |
| R       |  149628 |
| D       |  149522 |
| E       |  149518 |
| null    |   83309 |
+---------+---------+
sql>CREATE VIEW extended_onboard AS
more>SELECT number, number_sup, trip, trip_sup,
more>    onboard_at_departure, death_at_cape,
more>    left_at_cape, onboard_at_cape,
more>    death_during_voyage, onboard_at_arrival,
more>    death_during_voyage - left_at_cape AS death_at_arrival
more>FROM onboard_people;

WINDOWS

On Windows the first step is to initialize a MonetDB server by simply clicking: 'Start' -> 'Programs' -> 'MonetDB ' -> 'Start server'.  Subsequently you can start the textual interface by clicking 'Start'->'Programs'->MonetDB'->'SQL client' .  The commands entered there are identical to those found on other platforms. To stop the server, you can simply close the MonetDB SQL Server window.  Note the server by default only accepts connections originating from the local host. If you need other machines to access your database, change the configuration file by setting mapi_open=yes.

The graphical user interfaces based on JDBC connectivity provide an alternative client interface. See Squirrel and dbvisualiser installation pages for details.

Exploring the wealth of functionality offered by MonetDB/SQL is best started using a toy database. For this we use the VOC database which provides a peephole view into the administrative system of an early multi-national company, the Vereenigde geoctrooieerde Oostindische Compagnie (VOC for short - The (Dutch) East Indian Company) established on March 20, 1602.
Download the VOC data set voc.sql  gz (519K) [1] bz2 (371K) [2] which is a compressed file with SQL statements. After the file has been extracted, load its contents into MonetDB using the mclient.exe file, located in the C:\Program Files\CWI\MonetDB5\bin.

MonetDB daemon

Starting a MonetDB database is as simple as starting the deamon program monetdbd, also known by its previous code name merovingian.  At the time of this writing, monetdbdis only available on UNIX-like platforms. The MonetDB daemon program encapsulates an arbitrary number of database servers, and takes care of their configuration.  The general concept of the MonetDB daemon program is that it builds upon a given directory, often referred to as the dbfarm.  There can be at most one instance of monetdbd monitoring such dbfarm, but multiple monetdbd processes can be running on the same system.

In the simple case, monetdbd is started by the system through some init script, and databases are created by the local DBA, and assigned to users.  A more flexible approach for users, however, is the case where a user runs monetdbd itself.  Because a dbfarm can be anywhere in the filesystem, and needs nothing more than just being an ordinary directory, a user can use monetdbd to initialise his/her own dbfarm in an accessible place.  For example:

% monetdbd create ~/my-dbfarm

% monetdbd get all ~/my-dbfarm
property         value
hostname         vomac.ins.cwi.nl
dbfarm           /Users/fabian/my-dbfarm
status           no monetdbd is serving this dbfarm
mserver          unknown (monetdbd not running)
logfile          merovingian.log
pidfile          merovingian.pid
sockdir          /tmp
port             50000
exittimeout      60
forward          proxy
discovery        true
discoveryttl     600
control          no
passphrase       <unknown>
mapisock         /tmp/.s.monetdb.50000
controlsock      /tmp/.s.merovingian.50000

After creation of the personal dbfarm directory, the get command can be used to inspect the dbfarm-wide settings of the newly created environment.  Note that the above output differs per system, per user and per dbfarm.  Important setting is the port to use to open up services for clients.  In particular for a user installation, these often may conflict with another running monetdbd, hence it is wise to change these to a private port in situations where such conflicts may arise:

% monetdbd set port=54321 ~/my-dbfarm

% monetdbd get all ~/my-dbfarm
property         value
hostname         vomac.ins.cwi.nl
dbfarm           /Users/fabian/my-dbfarm
status           no monetdbd is serving this dbfarm
mserver          unknown (monetdbd not running)
logfile          merovingian.log
pidfile          merovingian.pid
sockdir          /tmp
port             54321
exittimeout      60
forward          proxy
discovery        true
discoveryttl     600
control          no
passphrase       <unknown>
mapisock         /tmp/.s.monetdb.54321
controlsock      /tmp/.s.merovingian.54321

Note that control defaults to no.  For remote control connections, control has to be set to yes, and a passphrase has to be set as well.  When set, it will show as a hash in get's output.

When the settings look ok, the next step is to simply start monetdbd:

% monetdbd start ~/my-dbfarm

If the above command returned without any message, the monetdbd process was successfully started.  The logfile (default merovingian.log) should show that the daemon has started, bound some ports, and discovered itself.

From this part on, monetdbd can be given commands through the monetdbcommand (without the trailing 'd').  The monetdb client utility knows commands much like monetdbd, but it is much more focussed towards giving information about what is going on inside monetdbd.  The monetdb client utility is the main tool to use to manage the dbfarm.

The dbfarm we created is empty, as such there are no databases inside it.  With monetdb we can create a new database:

% monetdb create my-first-db
created database in maintenance mode: my-first-db

This simple command created the necessary bits inside the dbfarm for a database called my-first-db.  It is created in maintenance mode, since otherwise anyone could access it before it is properly setup.  If the command gives an access-denied error, most probably another monetdbd is running on the system.  In that case try adding -p54321(the port) as arguments to monetdb in order to tell it which monetdbd to contact as in the status example.  In the status view of monetdb the newly created database will show up as a locked database:

% monetdb -p54321 status
     name        state     uptime       health
my-first-db     locked             

Albeit the database is locked, it can be started, such that the monetdb superuser can access the database:

% monetdb start my-first-db
starting database 'my-first-db'... done

% monetdb status
     name        state     uptime       health
my-first-db     locked           18s  100%,  0s

The database remains locked, but the uptime field indicates the database is actuallty running.  If setting up the database is ready, the database can be made available for other users (if any, see VOC demo [3]) to connect to it using the release command:

% monetdb release my-first-db
taken database out of maintenance mode: my-first-db

% monetdb status
     name        state     uptime       health
my-first-db     running       2m 48s  100%,  0s

Next thing, making a connection to the database using mclientrequires the -dargument to specify which database to connect to.  It will ask for username and password when none given or provided via a .monetdb file.  The default username/password is monetdb/monetdb:

% mclient -dmy-first-db
user(fabian):monetdb
password:<monetdb>
Welcome to mclient, the MonetDB/SQL interactive terminal (Dec2011-SP1)
Database: MonetDB v11.7.7 (Dec2011-SP1), 'mapi:monetdb://vomac.ins.cwi.nl:54321/my-first-db'
Type \q to quit, \? for a list of available commands
auto commit mode: on
sql>

Like for the monetdb command, if this fails with some database does not exist error, try giving the port (not the controlport!) to mclient such that it finds the correct monetdbd, e.g.:

% mclient -p54321 -dmy-first-db

It may be necessary to stop individual databases, or even make them unavailable.  The monetdb commands stopand lockcan perform these tasks.  To completely stop the monetdbd daemon process for a dbfarm, stop it using the stop command of monetdbd:

% monetdbd stop ~/my-dbfarm

This should stop the monetdbd and all the databases that are started inside it.  To verify that monetdbd has indeed stopped, review the end of the merovingian.log file.  It should report all databases being shut down, and the deamon shutting down.

mclient man-page

NAME

mclient − the MonetDB command-line tool

SYNOPSIS

mclient [ options ] [ file or database [ file ... ] ]
mclient −−help

DESCRIPTION

MonetDB is a database management system that is developed from a main-memory perspective with use of a fully decomposed storage model, automatic index management, extensibility of data types and search accelerators, SQL- and JAQL- frontends.

mclient is the command-line interface to the MonetDB server.

If the −−statement=query (−s query) option is given, the query is executed. If any files are listed after the options, queries are read from the files and executed. The special filename − refers to standard input. Note that if there is both a −−statement option and filename arguments, the query given with −−statement is executed first. If no −−statement option is given and no files are specified on the command line, mclient reads queries from standard input.

When reading from standard input, if standard input is a terminal or if the −−interactive (−i) option is given, mclient interprets lines starting with \ (backslash) specially. See the section BACKSLASH COMMANDS below.

Before mclient starts parsing command line options, it reads a .monetdb file. If the environment variable DOTMONETDBFILE is set, it reads the file pointed to by that variable instead. When unset, mclient searches for a .monetdb file in the current working directory, and if that doesn’t exist, in the current user’s home directory. This file can contain defaults for the flags user, password, language, save_history, format and width. For example, an entry in a .monetdb file that sets the default language for mclient to mal looks like this: language=mal. To disable reading the .monetdb file, set the variable DOTMONETDBFILE to the empty string in the environment.

OPTIONS

General Options
−−help (−?)

Print usage information and exit.

−−version (−v)

Print version information and exit.

−−encoding=encoding (−E encoding)

Specify the character encoding of the input. The option applies to both the standard input of mclient and to the argument of the −−statement (−s) option but not to the contents of files specified on the command line (except for − which refers to standard input) or files specified using the \< command (those must be encoded using UTF-8). The default encoding is taken from the locale.

−−language=language (−l language)

Specify the query language. The following languages are recognized: mal and sql. A unique prefix suffices. When the −−language option is omitted, the default of sql is assumed.

−−database=database (−d database)

Specify the name or URI of the database to connect to. The -d can be omitted if an equally named file does not exist in the current directory. As such, the first non-option argument will be interpreted as database to connect to if the argument does not exist as file. Valid URIs are as returned by ‘monetdb discover‘, see monetdb(1), and look like mapi:monetdb://hostname:port/database.

−−host=hostname (−h hostname)

Specify the name of the host on which the server runs (default: localhost). When the argument starts with a forward slash (/), host is assumed to be the directory where the UNIX sockets are stored for platforms where these are supported.

−−port=portnr (−p portnr)

Specify the portnumber of the server (default: 50000).

−−interactive[=timermode] (−i[timermode])

When reading from standard input, interpret lines starting with \ (backslash) specially. See the section BACKSLASH COMMANDS below. This is the default if standard input is a terminal. The optional timermode argument controls the format of the time reported for queries. Note that no space is allowed between −i and timermode. The default mode is human which adjusts the time precision to the measured value. The modes ms, s and m force millisecond, second and minute + second precision respectively.

−−user=user (−u user)

Specify the user to connect as. If this flag is absent, the client will ask for a user name, unless a default was found in .monetdb file. Note that user must follow immediately after the option.

−−format=format (−f format)

Specify the output format. The possible values are sql, csv, tab, raw, and xml. csv is comma-separated values, tab is tab-separated values, raw is no special formatting (data is dumped the way the server sends it to the client), sql is a pretty format which is meant for human consumption, and xml is a valid (in the XML sense) document. In addition to plain csv, two other forms are possible. csv=c uses c as column separator; csv+c uses c as column separator and produces a single header line in addition to the data.

−−echo (−e)

Echo the query. Note that using this option slows down processing.

−−history (−H)

Load and save the command line history (default off).

−−log=logfile (−L logfile)

Save client/server interaction in the specified file.

−−statement=stmt (−s stmt)

Execute the specified query. The query is run before any queries from files specified on the command line are run.

−−timezone (−z)

Do not tell the client’s timezone to the server.

−−Xdebug (−X)

Trace network interaction between mclient and the server.

−−pager=cmd (−| cmd)

Send query output through the specified cmd. One cmd is started for each query. Note that the | will have to be quoted or else the shell will interpret it.

SQL Options
−−null=nullstr (−n nullstr)

Set the string to be used as NULL representation when using the sql, csv, or tab output formats. If not used, NULL values are represented by the string "null" in the sql output format, and as the empty string in the csv and tab output formats. Note that an argument is required, so in order to use the empty string, use −n "" (with the space) or −−null=.

−−autocommit (−a)

Switch autocommit mode off. By default, autocommit mode is on.

−−rows=nr (−r nr)

If specified, query results will be paged by an internal pager at the specified number of lines.

−−width=nr (−w nr)

Specify the width of the screen. The default is the (initial) width of the terminal.

−−dump (−D)

Create an SQL dump.

−−inserts (−N)

Use INSERT INTO statements instead of COPY INTO + CSV values when dumping the data of a table. This option can be used when trying to load data from MonetDB into another database, or when e.g. JDBC applications are used to reload the dump.

BACKSLASH COMMANDS

General Commands

 

\?

  

Show a help message explaining the backslash commands.
 

\q

  

Exit mclient.

\< file

Read input from the named file.

\> file

Write output to the named file. If no file is specified, write to standard output.

\| command

Pipe output to the given command. Each query is piped to a new invocation of the command. If no command is given, revert to writing output to standard output.

 

\h

  

Show the readline(3) history.

  

\L file

Log client/server interaction in the given file. If no file is specified, stop logging information.

 

\X

  

Trace what mclient is doing. This is mostly for debugging purposes.
 

\e

  

Echo the query in SQL formatting mode.

\f format

Use the specified format mode to format the output. Possible modes the same as for the −−format (−f) option.

\w width

Set the maximum page width for rendering in the sql formatting mode. If width is −1, the page width is unlimited, when width is 0, use the terminal width. If width is greater than 0, use the given width.

\r rows

Use an internal pager using rows per page. If rows is −1, stop using the internal pager.

SQL Commands

 

\D

  

Dump the complete database. This is equivalent to using the program msqldump(1).

\D table

Dump the given table.

 

\d

  

Alias for \dvt.

  

\d[Stvsfn]+

List database objects of the given type. Multiple type specifiers can be used at the same time. The specifiers S, t, v, s, f and n stand for System, table, view, sequence, function and schema respectively. Note that S simply switches on viewing system catalog objects, which is orthogonal on the other specifiers.

\d[Stvsfn]+ object

Describe the given object in the database using SQL statements that reconstruct the object. The same specifiers as above can be used, following the same rules. When no specifiers are given, vt is assumed. The object can be given with or without a schema, separated by a dot. The object name can contain the wildcard characters * and _ that represent zero or more, and exactly one character respectively. An object name is converted to lowercase, unless the object name is quoted by double quotes ("). Examples of this, are e.g. *.mytable, tabletype* or "myschema.FOO". Note that wildcard characters do not work in quoted objects. Quoting follows SQL quoting rules. Arbitrary parts can be quoted, and two quotes following each other in a quoted string represent the quote itself.

 

\A

  

Enable auto commit mode.

  
 

\a

  

Disable auto commit mode.

  

EXAMPLES

Efficiently import data from a CSV (comma-separated values) file into a table. The file must be readable by the server. $file is the absolute path name of the file, $table is the name of the table, $db is the name of the database.

mclient -d $db -s "COPY INTO $table FROM ’$file’ USING DELIMITERS ’,’,’\\n’,’\"’"

Efficiently import data from a CSV file into a table when the file is to be read by mclient (e.g. the server has no access to the file). $file is the (absolute or relative) path name of the file, $table is the name of the table, $db is the name of the database.

mclient -d $db -s "COPY INTO $table FROM STDIN USING DELIMITERS ’,’,’\\n’,’\"’" - < $file

Note that in this latter case, if a count of records is supplied, it should be at least as large as the number of records actually present in the CSV file. This, because otherwise the remainder of the file will be interpreted as SQL queries.

See http://www.monetdb.org/Documentation/Manuals/SQLreference/CopyInto [4] for more information about the COPY INTO query.

SEE ALSO

msqldump(1) [5]

monetdb man-page

NAME

monetdb − control a MonetDB Database Server instance

SYNOPSIS

monetdb [monetdb_options] command [command_options] [command_args]

DESCRIPTION

monetdb allows an administrator of the MonetDB Database Server to perform various operations on the databases in the server. It relies on monetdbd(1) running in the background for all operations.

OPTIONS

monetdb_options affect all commands and control the general behaviour of monetdb.

 

−q

  

Supresses all standard progress messages, only writing output to stderr if an error occurred.

−h hostname

Connect to hostname instead of attempting a connection over the local UNIX socket. This allows monetdb to connect to a remote monetdbd(1). The use of this option requires −P (see below).

−p port

Connects to the given portnumber instead of the default (50000). Requires −h to be given as option too.

−P passphrase

Specifies the passphrase necessary to login to a remote monetdbd(1). This option requires −h to be given as well. A bad passphrase causes monetdb to fail to login, and hence fail to perform any remote action.

 

−v

  

Show version, equal to monetdb version.

  

COMMANDS

The commands for the monetdb utility are create, destroy, lock, release, status, start, stop, kill, set, get, inherit, discover, help and version. The commands facilitate adding, removing, maintaining, starting and stopping a database inside the MonetDB Database Server.

For all commands, database arguments can be glob-like expressions. This allows to do wildcard matches. For details on the syntax, see EXPRESSIONS.
create [−m pattern] database [database ...]

Initialises a new database in the MonetDB Database Server. A database created with this command makes it available under its database name, but not yet for use by clients, as the database is put into maintenance mode. This allows the database administrator to perform initialisation steps before releasing it to users. See also monetdb lock. The name of the database must match the expression [A−Za−z0−9−_]+.

−m pattern

With the −m flag, instead of creating a database, a multiplex-funnel is created. See section MULTIPLEX-FUNNEL in monetdbd(1). The pattern argument is not fully the same as a pattern for connecting or discovery. Each parallel target for the multiplex-funnel is given as username+password@pattern sequence, separated by commas. Here the pattern is an ordinary pattern as would be used for connecting to a database, and can hence also be just the name of a database.

destroy [−f] database [database ...]

Removes the given database, including all its data and logfiles. Once destroy has completed, all data is lost. Be careful when using this command.

 

−f

  

By default, a confirmation question is asked, however the −f option, when provided, suppresses this question and removal is executed right away. Note that you cannot destroy a running database, bring it down first using the stop command.

lock database [database ...]

Puts the given database in maintenance mode. A database under maintenance can only be connected to by an administrator account (by default the monetdb account). A database which is under maintenance is not started automatically by monetdbd(1), the MonetDB Database Server, when clients request for it. Use the release command to bring the database back for normal usage. To start a database which is under maintenance for administrator access, the start command can be used.

release database [database ...]

Brings back a database from maintenance mode. A released database is available again for normal use by any client, and is started on demand. Use the lock command to take a database under maintenance.

status [−lc] [−s states] [database ...]

Shows the state of the given database, or, when none given, all known databases. Three modes control the level of detail in the displayed output. By default a condensed one-line output per database format is used. This output resembles pretty much the output of various xxxstat programs, and is ideal for quickly gaining an overview of the system state. The output is divided into four columns, name, state, health and remarks. The state column contains two characters that identify the state of the database, based on Booting (starting up), Running, Stopped, Crashed and Locked (under maintenance). This is followed by the uptime when running. The health column contains the percentage of successful starts and stops, followed by the average uptime. The remarks column can contain arbitrary information about the database state, but usually contains the URI the database can be connected to.

 

−c

  

The −c flag shows the most used properties of a database. This includes the state of the database (running, crashed, stopped), whether it is under maintenance or not, the crash averages and uptime statistics. The crash average is the number of times the database has crashed over the last 1, 15 or 30 starts. The lower the average, the healthier the database is.
 

−l

  

Triggered by the −l flag, a long listing is used. This listing spans many rows with on each row one property and its value separated by a colon (‘:’). The long listing includes all information that is available.
 

−s

  

The −s flag controls which databases are being shown, matching their state. The required argument to this flag can be a combination of any of the following characters. Note that the order in which they are put also controls the order in which the databases are printed. b, r, s, c and l are used to print a starting up (booting), started (running), stopped, crashed and locked database respectively. The default order which is used when the −s flag is absent, is rbscl.

start [−a] database [database ...]
stop [−a] database [database ...]
kill [−a] database [database ...]

Starts, stops or kills the given database, or, when −a is supplied, all known databases. The kill command immediately sends a SIGKILL and should only be used as last resort for a database that doesn’t respond any more. Killing a database may result in (partial) data loss. It is more common to use the stop command to stop a database. It will first attempt to stop the database, waiting for mero_exittimeout seconds and if that fails, kill the database. When using the start command, monetdb(1) will output diagnostic messages if the requested action failed. When encountering an error, one should always consult the logfile of monetdbd(1) for more details. For the kill command a diagnostic message indicating the database has crashed is always emitted, due to the nature of that command. Note that in combination with −a the return code of monetdb(1) indicates failure if one of the databases had a failure, even though the operation on other databases was successful.

get <all | property[,property[,..]]> [database ...]

Prints the requested properties, or all known properties, for the given database. For each property its source and value are printed. Source indicates where the current value comes from, e.g. the configuration file, or a local override.

set property=value database [database ...]

Sets property to value for the given database. For a list of properties, run monetdb get all. Most properties require the database to be stopped when set.

shared=<yes|no|tag>

Defines if and how the database is being announced to other monetdbds or not. If not set to yes or no the database is simply announced or not. Using a string, called tag the database is shared using that tag, allowing for more sophisticated usage. For information about the tag format and use, see section REMOTE DATABASES in the monetdbd(1) manpage. Note that this property can be set for a running database, and that a change takes immediate effect in the network.

nthreads=<number>

Defines how many worker threads the server should use to perform main processing. Normally, this number equals the number of available CPU cores in the system. Reducing this number forces the server to use less parallelism when executing queries, or none at all if set to 1.

optpipe=<string>

Each server operates with a given optimiser pipeline. While the default usually is the best setting, for some experimental uses the pipeline can be changed. See the mserver5(1) manpage for available pipelines. Changing this setting is discouraged at all times.

readonly=<yes|no>

Defines if the database has to be started in readonly mode. Updates are rejected in this mode, and the server employs some read-only optimisations that can lead to improved performance.

nclients=<number>

Sets the maximum amount of clients that can connect to this database at the same time. Setting this to a high value is discouraged. A multiplex-funnel may be more performant, see MULTIPLEX-FUNNEL below.

inherit property database [database ...]

Like set, but unsets the database-local value, and reverts to inherit from the default again.

discover [expression]

Returns a list of remote monetdbds and database URIs that were discovered by monetdbd(1). All databases listed can be connected to via the local MonetDB Database Server as if it were local databases using their database name. The connection is redirected or proxied based on configuration settings. If expression is given, only those discovered databases are returned for which their URI matches the expression. The expression syntax is described in the section EXPRESSIONS. Next to database URIs the hostnames and ports for monetdbds that allow to be controlled remotely can be found in the discover list masked with an asterisk. These entries can easily be filtered out using an expression (e.g. "mapi:monetdb:*") if desired. The control entries come in handy when one wants to get an overview of available monetdbds in e.g. a local cluster. Note that for monetdbd to announce its control port, the mero_controlport setting for that monetdbd must be enabled in the configuration file.

 

−h

  

help [command]

Shows general help, or short help for a given command.

 

−v

  

version

Shows the version of the monetdb utility.

EXPRESSIONS

For various options, typically database names, expressions can be used. These expressions are limited shell-globbing like, where the * in any position is expanded to an arbitrary string. The * can occur multiple times in the expression, allowing for more advanced matches. Note that the empty string also matches the *, hence "de*mo" can return "demo" as match. To match the literal ’*’ character, one has to escape it using a backslash, e.g. "\*".

RETURN VALUE

The monetdb utility returns exit code 0 if it successfully performed the requested command. An error caused by user input or database state is indicated by exit code 1. If an internal error in the utility occurs, exit code 2 is returned.

SEE ALSO

monetdbd(1) [6] mserver5(1) [7]

monetdbd man-page

NAME

monetdbd − the MonetDB Database Server daemon

SYNOPSIS

monetdbd command [command_args] dbfarm

DESCRIPTION

monetdbd is the MonetDB Database Server daemon. The program is mainly meant to be used as daemon, but it also allows to setup and change the configuration of a dbfarm. The use of monetdbd is either as user-oriented way to configure, start and stop a database farm, or to be started from a startup script, such as from /etc/init.d/ on Linux systems or smf(5) on Solaris systems, as part of a system startup.

monetdbd is the system formerly known as merovingian. It was renamed to monetdbd since the name merovingian proved to be confusing to most regular end-users. Internally, monetdbd uses the name merovingian at many places for historical reasons.

A monetdbd instance manages one local cluster based, which is a directory in the system, referred to as the dbfarm. Nowadays, the dbfarm location always has to be given as argument to monetdbd.

Within its local cluster monetdbd takes care of starting up databases when necessary, and stopping them either upon request via monetdb(1) [8] or when being shut down. Client database connections are made against monetdbd initially which redirects or proxies the client to the appropriate database process, started on the fly when necessary.

When started, monetdbd runs by default in the background, sending log messages to merovingian.log, until being sent a stop, terminate or interrupt signal, possibly using the stop command of monetdbd.

monetdbd uses a neighbour discovery scheme to detect other monetdbd processes running in the local network. Databases from those remote instances are made available to a locally connecting client. Remote databases never override local databases, and their availability is controlled by the remote monetdbd process. See also the sharing capabilities of monetdb(1) [8] and the REMOTE DATABASES section below.

COMMANDS

The commands for monetdbd are create, start, stop, get, set, version, and help. The commands facilitate initialising a dbfarm, starting and stopping the MonetDB Database Server, and retrieving or setting options.
create dbfarm

Initialises a new database farm, such that a MonetDB Database Server can be started on that location. All necessary directories are attempted to be created, and an initial properties file is created in the directory itself. dbfarm must be a location addressable in the local filesystem hierarchy.

start [-n] <dbfarm>

Starts monetdbd, the MonetDB Database Server, on the given dbfarm. When the -n flag is given, monetdbd will not fork into the background, but instead remain attached to the calling environment, until given a stop signal.

stop <dbfarm>

Sends a stop signal to the monetdbd process responsible for the given dbfarm.

get <all | property[,property[,..]]> <dbfarm>

Prints the requested properties, or all known properties, for the given dbfarm. For each property, its value is printed. Some properties are virtual, and given for information purposes only, they cannot be modified using the set command.

set property=value <dbfarm>

Sets property to value for the given database. For a list of properties, run monetdbd get all. Some properties require a restart of the MonetDB Database Server in order to take effect. The set command, will however always write the property, and tell the running monetdbd to reload the properties file (if running). For an explanation of the properties, see the CONFIGURATION section below.

CONFIGURATION

monetdbd reads its properties from the .merovingian_properties file inside the dbfarm. This file is created by the create command. This file is not meant to be editted manually, instead it should be updated using the set command. The following properties can be set:
logfile

This property points to the file where all log messages are written to. It is relative to the dbfarm directory, but can be absolute to point to e.g. another medium. Changing this property takes effect immediately at runtime.

pidfile

monetdbd stores the process ID of the background server in the file pointed to by this property. The same rules apply as for the logfile property.

sockdir

For faster access, monetdbd uses UNIX domain sockets for its control mechanism and regular database connections. The sockets are placed as files in the filesystem hierarchy. The sockdir property controls in which directory they are placed. In general this setting should not be changed.

 

port

  

This property specifies which TCP port monetdbd should listen to for connection requests. Defaults to 50000.

control

For remote management of monetdbd, the control property specifies whether or not to enable remote management. Note that for remote management, a passphrase is required, see below. It defaults to false for security reasons. Changing this property takes effect immediately at runtime.

passphrase

To control monetdbd from a remote machine, a passphrase is necessary, to be given to monetdb(1) [8]. The passphrase can be either given as hashed value prefixed by the hash type in curly braces (e.g. {SHA512}xxx...) or as plain text value which will be hashed automatically. Note that the only hash accepted is the one specified at configure time, which is SHA512. Changing this property takes effect immediately at runtime.

discovery

Specifies whether neighbour discovery is to be enabled using UDP broadcasts or not. The broadcasts are done on the same portnumber as the port setting.

discoveryttl

monetdbd publishes locally available databases to others periodically. The interval used here, defined in seconds, depends on the time-to-live of the databases before they need to get refreshed. The default is 600 seconds (10 minutes), which should keep traffic in your network fairly low. Additions and removals are processed immediately regardless of this timeout. If you are in a network environment where physical network links disappear often, you may want to decrease this value to more quickly remove no longer reachable databases.

exittimeout

mservers that were started by the MonetDB Database Server are shut down when monetdbd is shut down. Setting the exittimeout property to a positive non-zero value will shut down each running mserver with the given time-out in seconds. If the time-out expires, the mserver process is killed using the SIGKILL signal. A time-out value of 0 means no mservers will be shut down, and hence they will continue to run after monetdbd has shut down. Note that this particular configuration is extremely inconvenient. The default time-out is 60 seconds. If your databases are rather large and find your databases consistently being killed by monetdbd upon shutdown, you may want to increase this time-out. Changing this property takes effect immediately at runtime.

forward

monetdbd has two ways in which it can "attach" a connecting client to the target database. The first method, redirect, uses a redirect sent to the client with the responsible mserver process. The second method, proxy, proxies the client to the mserver over monetdbd. While redirect is more efficient, it requires the connecting client to be able to connect to the mserver. In many settings this may be undesirable or even impossible, since a wide range of open ports and routing are necessary for this. In such case the proxy technique of monetdbd is a good solution, which also allows a monetdbd instance on the border of a network to serve requests to nodes in the local (unreachable) network. Note that for local databases, the proxy method uses a UNIX domain socket feature to pass file-descriptors to the local mserver. This effectively is as efficient as the redirect approach, but still hides away the mservers properly behind monetdbd. Hence, in practice it is only relevant for connections to remote databases to use redirects instead of proxies. Changing this property takes effect immediately at runtime.

REMOTE DATABASES

The neighbour discovery capabilities of monetdbd allow a user to contact a remote database transparently, as if it were a local database. By default, all local databases are announced in the network, such that neighbours can pick them up to make them available for their local users. This feature can be disabled globally, or on database level. For the latter, the monetdb(1) [8] utility can be used to change the share property of a database.

While neighbour discovery in itself is sufficient to locate a database in a cluster, it is limited in expressiveness. For instance, database names are assumed to be unique throughout the entire system. This means local databases overshadow remote ones, and duplicate remote entries cannot be distinguished. To compensate for this, monetdbd allows to adds a tag to each database that is being shared. This tag is sent in addition to the database name, and only understood by other monetdbds.

Tags are arbitrary ASCII-strings matching the pattern [A−Za−z0−9./]+. There are no assumed semantics in the tag, which allows for multiple approaches when using the tag. The tag is always used in combination with the database name. For this, the ‘/’ character is used as separator, which hence suggests the user to use that character as separator for multilevel tags. monetdbd allows common path globbing using ‘*’ on tags, which allows for many use-cases. Consider for instance the following three databases with their tag:

dbX/master/tableQ
dbY/slave/tableQ
dbZ/slave/tableQ

A default match has implicit ‘/*’ added to the search, making more generic search strings match more specific ones. Hence, a connect with database dbX is the same as dbX/* and hence matches dbX/master/tableQ. Similar, a database connect for */master matches the same database as before. Note that the implicit ‘/*’ is not added if that would cause no matches, such as for */master/tableQ which would return all masters for tableQ, which in our hypothetical example is only dbX. In contrast, a database connect for */slave/tableQ matches with either dbY or dbZ. monetdbd returns the two options to the client in a round-robin fashion, such that subsequent connects for the same pattern result in a load-balanced connect to either of both databases.

With tags in use, one can possibly make distinction between databases, if setup like that. The previous example could hence also be setup like this:

tableQ/master
tableQ/slave
tableQ/slave

Connecting to tableQ/slave would now return either of both databases even though they are not unique (apart from the host they are located on, which is not shown in the example). While being confusing for humans, for monetdbd it is the same situation as in the previous example. However, because globbing allows to make things easier to understand, tags for both slaves could be changed to slaveX or slave/X and use the necessary pattern to match them. It is up to the user to decide how to use the tags.

MULTIPLEX-FUNNELS

monetdbd implements multiplex-funnel capabilities. As the name suggests two techniques are combined, the multiplexer and the funnel.

The funnel capability limits the access to the database to one client at a time. That is, if multiple clients connect to the funnel, their queries will be serialised such that they are executed one after the other. An effect of this approach is that clients no longer have an exclusive channel to the database, which means that individual queries from one client may have been interleaved with queries from others. This most notably makes SQL transaction blocks unreliable with a funnel. The funnel, hence, is meant to scale down a large amount of clients that perform short-running (read-only) queries, as typically seen in web-based query loads.

When a funnel is defined to use multiple databases, the funnel adds a multiplexer to its query channel. A multiplex-funnel sends each query to all of the defined databases. This behaviour can be quite confusing at first, but proves to be useful in typical sharding configurations, where in particular simple selection queries have to be performed on each of the shards. The multiplexer combines the answers from all defined databases in one single answer that it sends back to the client. However, this combining is without any smart logic, that is, the multiplexer does not evaluate the query it is running, but just combines all answers it receives from the databases. This results in e.g. as many return tuples for a SELECT COUNT(*) query, as there are databases defined.

Due to the two above mentioned characteristics, a multiplex-funnel has some limitations. As mentioned before, transactions over multiple queries are likely not to result in the desired behaviour. This is due to each query to the funnel is required to be self-contained. Further, since for each query, the results from multiple servers have to be combined into one, that query must only return a single response, i.e. multi-statement queries are most likely causing the funnel to respond with an error, or return garbled results. Last, the size of each query is limited to currently about 80K. While this size should be sufficient for most queries, it is likely not enough for e.g. COPY INTO statements. Apart from the data transfer implications, such statements should not be used with the funnel, as the results will be undefined due to the limited query buffer. Applications using the funnel should aim for short and single-statement queries that require no transactions.

See the create command in the monetdb(1) [8] man-page for details on how to setup a multiplex-funnel.

SIGNALS

monetdbd acts upon a number of signals as is common for a daemon.
SIGINT, SIGTERM, SIGQUIT

Any of these signals make monetdbd enter the shutdown sequence. This sequence involves cleanly shutting down listener sockets, shutting down all started databases and finally terminating itself.

 

SIGHUP

  

When this signal is received by monetdbd it will reopen the logfile as pointed to by the logfile setting. Before it reopens the logfile, it will re-read the properties file from the dbfarm, which might result in opening a different file to continue logging.

RETURN VALUE

monetdbd returns exit code 0 if it was able to successfully perform the requested action, e.g. start, stop, etc. When an error occurs during the action, that prevents monetdbd from successfully performing the action, the exit code 1 is returned.

SEE ALSO

monetdb(1) [8] mserver5(1) [7]

mserver5 man-page

NAME

mserver5 − the MonetDB server version 5

DESCRIPTION

mserver5 is the current MonetDB server that performs all processing on request of clients for a certain database.

Note that while mserver5 is the process that does the actual work, it is usually more common to start, monitor and connect to the mserver5 process through monetdbd(1) [6].

This man-page describes the options that mserver5 understands. It is intended for people who really need to work with mserver5 itself. In regular cases, the programs monetdbd(1) [6] and monetdb(1) [8] control the many options, and allow to adjust them to appropriate values where sensible. For normal usage, it is preferred to apply any configuration through these programs.

OPERATION

When the build-time configuration did not disable this, a mserver5 process presents the user with a console prompt. On this prompt, MAL commands can be executed. The architecture is setup to handle multiple streams of requests. The first thread started represents the server, which is the console prompt, reading from standard input and writing to standard output.

The server thread started remains in existence until all other threads die. The server is stopped by Ctrl-D on its console, typing quit() or by sending it a termination signal (SIGINT, SIGTERM, SIGQUIT).

MSERVER5 OPTIONS

mserver5 can be started with options and scripts as arguments. The MAL scripts will be executed directly after startup on the console, which eases e.g. testing of MAL scripts directly, without starting a client.
--dbpath=<path>

Path where mserver5 should find a database. Shorthand for option gdk_dbpath. Default value: @localstatedir@/monetdb5/dbfarm/demo.

--dbinit=<stmt>

MAL statement to execute as part of the startup of the server.

--config=<file>

Config file to read options from. This file can contain all options as can be set with the --set flag. See CONFIG FILE FORMAT.

--daemon=<yes|no>

Disable the console prompt, do not read commands from standard input. Default: no

--set <option>=<value>

Set individual configuration option. For possible options, see PARAMETERSsections.

 

--help

  

Print list of options.

  

--version

Print version and compile configuration.

GDK PARAMETERS

GDK (Goblin Database Kernel) is the current columnar storage kernel engine of the MonetDB 5 database. It is the component that manages and performs operations on BATs (Binary Association Tables), single columns. The parameters here affect the behaviour of GDK which may nagatively impact performance if set wrongly. The kernel tries to choose the values for optimal performance. Changing these parameters is discouraged.
gdk_mem_bigsize

Memory chunks of size >= gdk_mem_bigsize (in bytes) will be mmaped anonymously. Default: 1<<20 == 1024576 == 1 MiB

gdk_vmtrim

Enable or disable the vmtrim thread which tries to unload memory that is not in use. Default: yes

gdk_debug

You can enable debug output for specific kernel operations. By default debug is switched off for obvious reasons. The value of gdk_debug is an integer, which value can be (a combination of):

          1 = THRDMASK     = thread-specific debug output
          2 = CHECKMASK    = property enforcing on new BATs
          4 = MEMMASK      = memory allocation
          8 = PROPMASK     = property checking on all values:
                             tells about wrongly set properties
         16 = IOMASK       = major IO activity
         32 = BATMASK      = BAT handling
        128 = PARMASK      = Thread management
        256 = HEADLESSMASK = Warn about BAT heads that are not "headless-ready"
        512 = TMMASK       = Transaction management
       1024 = TEMMASK      = Locks and Triggers
       4096 = PERFMASK     = BBP Performance (?)
       8192 = DELTAMASK    = Delta debugging (?)
      16384 = LOADMASK     = Module loading
    2097152 = ALGOMASK     = show join/select algorithm chosen
    4194304 = ESTIMASK     = show result size estimations
                             (for join, select)
   16777216 = JOINPROPMASK = disable property checking with
                             join & outerjoin (e.g., for
                             performance measurements)
   33554432 = DEADBEEFMASK = disable "cleaning" of freed memory
                             in GDKfree() (e.g., for performance
                             measurements)
   67108864 = ALLOCMASK    = exhaustive GDK malloc & free tracing
                             for debugging (GDK developers, only)
  134217728 = OPTMASK      = trace the actions, decisions and
                             effects of MAL optimizers
  268435456 = HEAPMASK     = trace/debug HEAPextend;
                             used only for development & debugging
  536870912 = FORCEMITOMASK = forcefully activate mitosis even on
                              small tables, i.e., split small tables
                              in as many (tiny) pieces as there are
                              cores (threads) available;
                              this allows us to test mitosis
                              functionality without requiring large
                              data sets (--- at the expense of a
                              potentially significant interpretation
                              overhead for unnecessary large plans);
                              used only for development & testing;
                              set automatically by Mtest.py

Note that mserver5 recognizes a series of command line options that sets one or more of these debug flags as well:

  --threads       (THRDMASK | PARMASK)
  --memory        (MEMMASK | ALLOCMASK)
  --properties    (CHECKMASK | PROPMASK | BATMASK)
  --io            (IOMASK | PERFMASK)
  --heaps         (HEAPMASK)
  --transactions  (TMMASK | DELTAMASK | TEMMASK)
  --modules       (LOADMASK)
  --algorithms    (ALGOMASK | ESTIMASK)
  --performance   (JOINPROPMASK | DEADBEEFMASK)
  --optimizers    (OPTMASK)
  --forcemito     (FORCEMITOMASK)

Default: 0

MSERVER5 PARAMETERS

mserver5 instructs the GDK kernel through the MAL (MonetDB Assembler Language) language. MonetDB 5 contains an extensive optimiser framework to transform MAL plans into more optimal or functional (e.g. distributed) plans. These parameters control behaviour on the MAL level.
mal_listing

You can enable the server listing the parsed MAL program for any script parsed on the command line. The value of mal_listing is an integer that have the following possible values:

   0 = Disable
   1 = List the original input
   2 = List the MAL instruction
   4 = List the MAL type information
   8 = List the MAL UDF type
  16 = List the MAL properties
  32 = List the hidden details
  64 = List the bat tuple count

Default: 0

monet_vault_key

The authorisation tables inside mserver5 can be encrypted with a key, such that reading the BATs does not directly disclose any credentials. The monet_vault_key setting points to a file that stores a secret key to unlock the password vault. It can contain anything. The file is read up to the first null-byte (’ ’), hence it can be padded to any length with trailing null-bytes to obfuscate the key length. Generating a key can be done for example by using a tool such as pwgen and adding a few of the passwords generated. Make sure not to chose a too small key. Note that on absence of a vault key file, some default key is used to encrypt the authorisation tables. Changing this setting (effectively changing the key) for an existing database makes that database unusable as noone is any longer able to login. If you use monetdbd(1) [6], a per-database vault key is set.

max_clients

Controls how many client slots are allocated for clients to connect. This settings limits the maximum number of connected clients at the same time. Note that MonetDB is not designed to handle massive amounts of connected clients. The funnel capability from monetdbd(1) [6] might be a more suitable solution for such workloads.

Default 64.

SQL PARAMETERS

The SQL component of MonetDB 5 runs on top of the MAL environment. It has its own SQL-level specific settings.
sql_debug

Enable debugging using a mask. This option should normally be disabled (0). Default: 0

sql_optimizer

The default SQL optimizer pipeline can be set per server. See the optpipe setting in monetdb(1) [8] when using monetdbd. During SQL initialization, the optimizer pipeline is checked against the dependency information maintained in the optimizer library to ensure there are no conflicts and at least the pre-requisite optimizers are used. The setting of sql_optimizer can be either the list of optimizers to run, or one or more variables containing the optimizer pipeline to run. The latter is provided for readability purposes only. Default: default_pipe

The following are possible pipes to use:
minimal_pipe

The minimal pipeline necessary by the server to operate correctly. minimal_pipe=inline,remap,deadcode,multiplex,garbageCollector

default_pipe

The default pipe line contains as of Feb2010 mitosis-mergetable-reorder, aimed at large tables and improved access locality. default_pipe=inline,remap,costModel,coercions,evaluate,emptySet,aliases,pushselect,mitosis,mergetable,deadcode,commonTerms,joinPath,reorder,deadcode,reduce,matpack,dataflow,history,multiplex,garbageCollector

no_mitosis_pipe

The no_mitosis pipe line is identical to the default pipeline, except that optimizer mitosis is omitted. It is used mainly to make some tests work deterministically, and to check/debug whether "unexpected" problems are related to mitosis (and/or mergetable). no_mitosis_pipe=inline,remap,costModel,coercions,evaluate,emptySet,aliases,pushselect,mergetable,deadcode,commonTerms,joinPath,reorder,deadcode,reduce,matpack,dataflow,history,multiplex,garbageCollector

sequential_pipe

The sequential pipe line is identical to the default pipeline, except that optimizers mitosis & dataflow are omitted. It is use mainly to make some tests work deterministically, i.e., avoid ambigious output, by avoiding parallelism. sequential_pipe=inline,remap,costModel,coercions,evaluate,emptySet,aliases,pushselect,mergetable,deadcode,commonTerms,joinPath,reorder,deadcode,reduce,matpack,history,multiplex,garbageCollector

nov2009_pipe

The default pipeline used in the November 2009 release. nov2009_pipe=inline,remap,evaluate,costModel,coercions,emptySet,aliases,mergetable,deadcode,constants,commonTerms,joinPath,deadcode,reduce,dataflow,history,multiplex,garbageCollector

Debugging the optimizer pipeline The best way is to use mdb and inspect the information gathered during the optimization phase. Several optimizers produce more intermediate information, which may shed light on the details. The opt_debug bitvector controls their output. It can be set to a pipeline or a comma separated list of optimizers you would like to trace. It is a server wide property and can not be set dynamically, as it is intended for internal use.

CONFIG FILE FORMAT

The conf-file readable by mserver5 consists of parameters of the form "name=value".

The file is line-based, each newline-terminated line represents either a comment or a parameter.

Only the first equals sign in a parameter is significant. Whitespace before or after the first equals sign is not stripped. Trailing whitespace in a parameter value is retained verbatim.

Any line beginning with a hash (#) is ignored, as are lines containing only whitespace.

The values following the equals sign in parameters are all a string where quotes are not needed, and if written be part of the string.

SEE ALSO

monetdbd(1) [6], monetdb(1) [8], mclient(1) [9]

msqldump man-page

NAME

msqldump − dump a MonetDB/SQL database

SYNOPSIS

msqldump [ options ] [ dbname ]

DESCRIPTION

MonetDB is a database management system that is developed from a main-memory perspective with use of a fully decomposed storage model, automatic index management, extensibility of data types and search accelerators, SQL- and XML- frontends.

Msqldump is the program to dump an MonetDB/SQL database. The dump can be used to populate a new MonetDB/SQL database.

OPTIONS

−−help (−?)

Print usage information and exit.

−−database=database (−d database)

Specify the name of the database to connect to. The −d can be omitted if it is the last option.

−−host=hostname (−h hostname)

Specify the name of the host on which the server runs (default: localhost).

−−port=portnr (−p portnr)

Specify the portnumber of the server (default: 50000).

−−user=user (−u user)

Specify the user to connect as. If this flag is absent, the client will ask for a user name. Note that user must follow immediately after the option.

−−describe (−D)

Only dump the database schema.

−−inserts (−N)

When dumping the table data, use INSERT INTO statements, rather than COPY INTO + CSV values. INSERT INTO statements are better portable, and necessary when the load of the dump is processed by e.g. a JDBC application.

−−functions (−f)

Only dump functions definitions.

−−table=table (−t table)

Only dump the specified table.

−−quiet (−q)

Don’t print the welcome message.

−−Xdebug (−X)

Trace network interaction between mclient and the server.

SEE ALSO

mclient(1) [9]

System resources

Memory consumption.

MonetDB requires all data that needs to be active at any given point in time to fit into the address space --- and of course to fit on the storage device, (i.e., your disk system). On 32-bit systems, the address space is at most 32-bit ( 4 GB); in practice, it is actually limited to 3 GB or even 2 GB on most systems. On 64-bit systems, the address space can theoritcally be 64-bit, but in pactice is often "limited" to 48-bit or so --- not that that makes any difference ...

MonetDB excessively uses main memory for processing, but does not require that all data fit in the available physical memory. To handle a dataset that exceed the available physical memory, MonetDB does not (only) rely on the available swap space, but (also) uses memory-mapped files to exploit disk storage beyond the swap space as virtual memory.

For example, while bulk-loading data (preferably via a COPY INTO statements from a (possibly compressed) CSV file), MonetDB need to have all columns of the table that is currently being loaded "active", i.e., accessable in the address space. However, during loading, parts of the data are continuously written to the persistent files on disk, i.e., the whole table does not have to fit into main memory. E.g., loading a 100 GB table works fine on a system with 8 GB RAM and 16 GB swap -- provided there is sufficient free disk space.

During query processing, MonetDB requires for each single MAL operation during the query execution that all its inputs, its outputs, and possible temporary data structure fit in the address space. MonetDB automatically resorts to virtual memory and memory mapped files for large intermedate results. Also (large) persistent tables are accessed using memory mapping.

While running, you might see your mserver5 process' virtual size grow well beyond the available physical memory and possibly also well beyond your swap space.  In principle, this is not a problem at all.  Most of this virtual size is due to the fact that large base tables (or intermediate results) that recide as files on disk are memory-mapped into the address space. Those parts of the data that are currently not accessed do not consume any physical memory (except possible for caching purposes).

However, if individual columns of your table(s) and/or individual columns of intermediate results exceed the size of the available physical memory, the performance of MonetDB might (will) decrease due to increased I/O requirements.

Disk space

The disk space footprint is determined by the way columns are being stored. MonetDB uses dictionary encoding for string columns, but aside from this there is no default compression applied to reduce the disk footprint. The prime reason being the two-headed sword of compression. It safes (cheap) disk space and IO bandwidth at the cost of expensive CPU (de)compression overhead (See compression [10]). Since all columns are memory mapped upon access, i.e. they need not be decompressed.  If disk space comes at a premium and memory residency can be guaranteed for a long time, then one of the compression optimizers may become helpful.

The disk footprint can be assessed using the (Linux) command 'du' on the dbfarm directory or to run the query 'select * from storage();', provided the sql extensions are pre-loaded into your database. (See storage model [11])

Linux OOM Killer

The MonetDB server may become a victim of Linux kernel functionality  called Out Of Memory Killer (or OOM Killer) responsible for dealing with excessive memory requirements.
If the system reaches a point where it may  run out of memory, OOM Killer looks for victim process and ends its life the hard way.

In most cases, a simple restart of the server will suffice, but when the killer enters during a complex update transaction then the database may end-up in an inconsistent state.

Compression

A distinctive feature of column stores is to apply aggressive data compression. However, compression is often a two-edged sword, where movement of large data files over relative slow networks, disk access or memory interconnects is compensated for by applying CPU cycles. The effectiveness of which strongly depends on the mapping from database schema into data structures,  their high maintenance cost, the relational algorithms,  the system architecture, and the data distribution. The compression ratios cited  depends on the input size, which is commonly assumed to be in CSV format, the data distribution, and the database storage footprint, with or without auxiliary data structures like indices.

MonetDB applies different compression techniques automatically at many levels.

The column store representation is highly optimized, where the basic storage structure is a dense array, i.e. without holes to accommodate future insertions or overhead caused by the data structure itself (e.g. B-trees). This dense representation allows for direct mapping of the database files into memory. The storage width ranges from 1 (byte) to 8 bytes (doubles).  NULL values are part of the domain space, which avoids auxiliary bit masks at the expensive of 'loosing' a single value from the domain.

All strings are stored using dictionary encoding. This significantly reduces their storage space, but with larger dictionaries the maintenance cost may become expensive. Therefore for really large dictionary tables, MonetDB resort to non-compressed string representation. The references into the dictionary table occupy anywhere from 1 to 8 bytes, depending on the number of elements.

During query evaluation, a dense range of results is represented by a column view. This is a small footprint representation of the result set. It avoids both copying the result and storing it in its private column structure.

Readonly databases

When working in a regular mode, the query processing spans the basic column storage and the data changed by transactions (inserts, updates, and deletes). For instance, the inserted data, kept in a separate delta structure, are added to the column before other operations in the plan. In a 'read-only mode' modifying transactions are not allowed and the query processing spans only over the basic column storage. One of the consequences is that query execution may become faster, but temporary tables can not be created..

The readonly mode is administered with a boolean variable, which  can be set by the system administrator using the monetdb tool:

shell> monetdb set readonly=yes <mydatabasename>

It ensures that all tables are accessed in read only mode.  The user can not even create a temporary table for keeping an intermediate result around.

The alternative route is to deploy SQL schemas in combination with user authentication and access control grants to selectively allow users access to the database. See the corresponding description in the SQL manual [12].

Dump and restore

An ASCII-based database dump is a safe scheme to transport a database to another platform or to migrate to an (incompatible) new version of MonetDB. This feature is standard available in mclient.

Linux platforms

Consider you have already installed the SQL tutorial database voc and wish to transport it to another machine. Then the following client command line option generates the dump file.

     shell> mclient -lsql --database=voc --dump >/tmp/voc.sql

You can inspect the file /tmp/voc.sql to confirm that indeed a readable database dump is available. Move this file over to the new machine. The  monetdb [8] tool can be used to create the database. Once done, it suffices to feed the dump file to mclient to populate the database.

     shell> mclient -lsql --database=voc /tmp/voc.sql

Migration of a database from other system follows the same route, but be aware that SQL dialects often differ. A manual or scripted patch of a foreign SQL dump is often needed.

Fast backups on Linux

Aside from the ASCII dumps, one can also take the database server out of production using commands

shell> monetdb stop demo
shell> monetdb lock demo

After the database has been securely stopped, we can create a copy of the database directory in the dbfarm and put it aside in a safe place. Alternatively, incremental file-system dumps can be used to reduce the time and storage space for a recovery point. Finally, the database is released for production again

shell> monetdb release demo

Windows platforms

It is recommended to always dump the old database into ASCII before installing a new MonetDB release. Subsequently remove the dbfarm. After installation, the dump can be restored.

To dump the SQL database, start the MonetDB SQL Client program and type the command

\>...\databasedump.sql
\D
\>

The path after \>should be an absolute path name (i.e. start with a drive letter) and be in a save location. By default the database is located in  %APPDATA%\MonetDB5. After having made a database dump it can be removed. This folder is located inside the dbfarm\demofolder.

Restoring the SQL database can be done using the MonetDB SQL Client program with the following command

     \<...\databasedump.sql

 

Server debugging

In most cases, the system produces  informative error messages. However, there are situations where MonetDB enters an area not covered by the test suite or previous use and a segmentation fault occurs. These cases are hard to analyze outside the development lab. To isolate and resolve the issue we need at least the following information.

Sent us the result of the command mserver5 --version --dbname=<databasename> or the equivalent using monetdb --version <databasename>

Is the error reproducible with a small (5-10 line) script/query/program? Trim your experiment to the minimal size that demonstrates the erroneous behavior. Such a script is the best we can hope for, because it will end up in the nightly testing.

In addition, follow the steps, assuming that you are logged onto the same Linux (!) machine as where the server will run:

  1. Sent the tail of the merovingian log (100 lines) located in the dbfarm directory.
  2. Start the database server either with monetdb  or as a full command line mserver --dbinit="..." .
  3. Lookup the process id for this server using the command  ps -al |grep mserver5
  4. Attach the debugger with gdb mserver5 <processid>
  5. Run your experiment again until it hits the segmentation fault, which you see emerging in window.
  6. Then obtain a stack trace with the command where. It shows of the context of the crash using the list command and the variables used in the context of the crash.
  7. Collect the status of all threads using the command thr app all bt.

 

Release notes

The trail of release notes as of the first time we made the code base available on SourceForge.

Our current release notes can be found in here [13].

October 2012

 

Oct 2012 feature release
Build Environment
  • Removed --enable-noexpand configure option.
Java Module
  • Fixed problem with PreparedStatements and setXXX() methods using column numbers instead of names, bug #3158 [14]
Client Package
  • mclient now accepts URIs as database to connect to.
  • all strings returned by python2 are unicode, removed use_unicode option
  • python2 and 3 type convertion speed improvements
  • python2 uses new style objects now (bug #3104 [15])
  • split python2 and python3
MonetDB5 Server
  • Removed module attach since it wasn't used or even tested.
  • The MAL-to-C Compiler (mcc) was removed. The code wasn't tested and most likely non-functional.
Merovingian
  • Fixed problem where monetdbd would refuse to startup when discovery was set to false, bug #3155 [16]
MonetDB Common
  • Removed the gdk_embedded (and embedded) option. The code wasn't tested and most likely non-functional.
  • BAT-of-BATs is no longer allowed. It was already not allowed to make these types of BATs persistent, but now they can't be created at all anymore.
Bug Fixes
  • 3084 [17]: Timestamp arithmetic very slow (especially on Windows)
  • 3125 [18]: Python tests fail after recent Python API changes
  • 3155 [16]: Provide an option for setting discovery to false

 

No Debian and Ubuntu packages were built for this release (i.e. Oct2012) due to a packaging failure.  We expect to have resolved that problem by Oct2012-SP1.  We apologise for the inconvenience.

 

Oct 2012-SP1 bugfix release
Bug Fixes
  • 3155 [16]: Provide an option for setting discovery to false
  • 3161 [19]: invalid tuple received from server, got 138 columns, expected 476, ignoring
  • 3182 [20]: When using aggregate SQL functions, precision & decimal metadata for numeric/decimal type returned is 0,0

For this release (i.e. Oct2012-SP1) there are once again Debian and Ubuntu packages.  From this release onward, Ubuntu Lucid Lynx (10.04) is no longer supported.

 

Oct 2012-SP2 bugfix release
SQL
  • Fixed crash when performing an INSERT on a table with string-like column defaulting to NULL and omitting that column from VALUES, bug #3168 [21]
Java Module
  • Implemented type map support of Connection to allow custom mapping of UDTs to Java classes. By default the INET and URL UDTs are now mapped to nl.cwi.monetdb.jdbc.types.{INET,URL}. Most notably, ResultSet.getObject() and PreparedStatement.setObject() deal with the type map.
  • Fixed a problem in PreparedStatement where the prepared statement's ResultSetMetaData (on its columns to be produced) incorrectly threw exceptions about non existing columns. Bug #3192 [22]
Bug Fixes
  • 2579 [23]: SQL: incorrect cast from double to int
  • 2659 [24]: tuples INTERSECT vs count (... INTERSECT ...) differs
  • 2805 [25]: row_number doesn't work in complex query
  • 2886 [26]: warning: Catalog parameter '' ignored
  • 2977 [27]: timestamp minus date results in program error
  • 2978 [28]: now() + 1 results in illegal argument
  • 2999 [29]: Up-cast from SQL drops order information?
  • 3048 [30]: algebra.join undefined when using multiple arithmetic operations in SQL where-clause
  • 3124 [31]: insert into table with two foreign keys crashes mserver5
  • 3136 [32]: Memleak (in querycache?) while querying
  • 3143 [33]: Type resolution error in SQL procedural code
  • 3163 [34]: mserver5 crashes with large number of concurrent clients
  • 3168 [21]: crash when insert varchar/char/clob with default NULL
  • 3169 [35]: php driver uses call-time pass-by-reference
  • 3173 [36]: SQL queries with limit and offset clauses are unable to use bind parameters for either
  • 3181 [37]: Join query returns zero results when there is an index on the tables
  • 3185 [38]: inconsistency between documentation and implementation of command line options
  • 3186 [39]: case evaluted to early
  • 3188 [40]: Storage of client history
  • 3189 [41]: Overflow in calculation
  • 3190 [42]: When subtracting two columns of type "timestamp", no results are returned
  • 3191 [43]: rel_optimizer.c:5596: _rel_optimizer: Assertion `0' failed.
  • 3192 [22]: "No such column with index" error if CASE in SELECT clause
  • 3199 [44]: Inconspicuous sequence of prepare, execute, commit and close produces error
  • 3205 [45]: Inet datatype handling inconsistent between int, cast( as inet) and string

 

Oct 2012-SP3 bugfix release
Bug Fixes
  • 2579 [23]: SQL: incorrect cast from double to int
  • 2659 [24]: tuples INTERSECT vs count (... INTERSECT ...) differs
  • 2805 [25]: row_number doesn't work in complex query
  • 2886 [26]: warning: Catalog parameter '' ignored
  • 2977 [27]: timestamp minus date results in program error
  • 2978 [28]: now() + 1 results in illegal argument
  • 2999 [29]: Up-cast from SQL drops order information?
  • 3048 [30]: algebra.join undefined when using multiple arithmetic operations in SQL where-clause
  • 3124 [31]: insert into table with two foreign keys crashes mserver5
  • 3136 [32]: Memleak (in querycache?) while querying
  • 3143 [33]: Type resolution error in SQL procedural code
  • 3163 [34]: mserver5 crashes with large number of concurrent clients
  • 3168 [21]: crash when insert varchar/char/clob with default NULL
  • 3169 [35]: php driver uses call-time pass-by-reference
  • 3173 [36]: SQL queries with limit and offset clauses are unable to use bind parameters for either
  • 3176 [46]: Same query and different results
  • 3181 [37]: Join query returns zero results when there is an index on the tables
  • 3185 [38]: inconsistency between documentation and implementation of command line options
  • 3186 [39]: case evaluted to early
  • 3188 [40]: Storage of client history
  • 3189 [41]: Overflow in calculation
  • 3190 [42]: When subtracting two columns of type "timestamp", no results are returned
  • 3191 [43]: rel_optimizer.c:5596: _rel_optimizer: Assertion `0' failed.
  • 3192 [22]: "No such column with index" error if CASE in SELECT clause
  • 3199 [44]: Inconspicuous sequence of prepare, execute, commit and close produces error
  • 3205 [45]: Inet datatype handling inconsistent between int, cast( as inet) and string
  • 3208 [47]: Unable to execute prepared statements with a large number of arguments
  • 3209 [48]: exp_bin: Assertion `0' failed
  • 3212 [49]: Query cache does not work properly

 

July 2012

Jul 2012 feature release
Build Environment
  • Created seperate RPM and DEB packages for MonetDB/JAQL.
  • Removed --enable-bits option from configure.
  • Split the MonetDB-client-ruby RPM package into two and named them in accordance with the Fedora packaging guidelines as rubygem-<gem-name>.
SQL
  • COPY INTO now accepts optional parenthesis for file argument. Binary COPY INTO now requires 'COPY BINARY INTO'.
Client Package
  • ODBC: Fixed a bug where SQLNativeSql expected a statment handle instead of a connection handle.
MonetDB5 Server
  • Crackers code has been removed. Development continues in the holindex branch.
Merovingian
  • Fixed misc memory leaks, which caused monetdbd to grow in memory size over time.
  • Removed erroneously (re-)added master and slave properties, this functionality is currently not working.
MonetDB Common
  • Fixed intermittent problem that joins and selects return incorrect results and possibly inconsistent databases. The problems only occurred after a series of queries and updates, therefore it was hard to reproduce.
  • The sorted property, which was used to maintain whether a column in a BAT was sorted or reverse sorted, has been replaced by a pair of properties, sorted and revsorted. These new properties can be set independently (unlike the old sorted property), and so if both are set, the column must be constant. In addition, internal property checking has been overhauled. Now, when a property is set incorrectly, and when assertions are enabled, an assertion will go off. There is also a function which can derive properties.
  • Implemented proper overflow checking on all arithmetic operations.
Bug Fixes
  • 3016 [50]: Overflowing double multiplication leads to incorrect result
  • 3091 [51]: Since Apr2012-SP1 !TypeException:user.s11_1[59]:'bat.reverse' undefined in: _89:any := bat.reverse(_86:wrd)
  • 3099 [52]: ambiguous attributes in query with WITH clause and EXISTS clause
  • 3100 [53]: charset problem with the odbc driver for windows
  • 3103 [54]: display_size and internal_size of Cursor.description or incorrect
  • 3105 [55]: Leaking using Python
  • 3109 [56]: Wrong order in autoload mal script causes warning on startup
  • 3111 [57]: memory leak, monetdb set/get command
  • 3112 [58]: Median over temporal domains
  • 3113 [59]: Cannot set value of indexed column when doing UPDATE
  • 3114 [60]: Foreign Key prblems
  • 3116 [61]: Segfault in SQL

 

Jul 2012-SP1 bugfix release
Java Module
  • Fixed adaptive cache size used when retrieving results, not to cause divide by zero errors when memory gets short, bug #3119 [62].
Client Package
  • mclient no longer prints the SQLSTATE at the start of each error returned by the SQL-server.
MonetDB5 Server
  • The server now distinguishes between starting and started states, such that monetdbd can wait for it to finish starting.
Merovingian
  • Starting a server now waits for as long as the server needs to possibly recover, bug #3134 [63]. In case of a long wait, the monetdbd logfile gives extra information on what the server is doing to recover.
  • Fixed a crash of monetdbd when local databases were unshared, bug #3135 [64]
  • Resolved a problem where automatic starting of a database initiated by multiple clients at the same time could cause failed starts. Bug #3107 [65]
Bug Fixes
  • 3075 [66]: inconsistent declaration of algebra.markH with 3 input arguments
  • 3090 [67]: crashed if using single identifier for where condition
  • 3093 [68]: sql fail if use scalar subquery with alias
  • 3107 [65]: concurrent connections to the same stopped database yield in multiple mserver5 starts by monetdbd
  • 3119 [62]: MonetConnection$ResultSetResponse throws java.lang.ArithmeticException: divide by zero
  • 3132 [69]: test/BugTracker-2011/func_iter_vs_bulk.Bug-2826 fails
  • 3134 [63]: Database gets killed by timeout during startup
  • 3135 [64]: monetdbd crash while creating & loading database

 

 

Jul 2012-SP2 bugfix release
Java Module
  • Fixed a bug where DatabaseMetaData.getURL() did return null:0 for hostname:port.
MonetDB5 Server
  • Changed the way nclients maximum was calculated to avoid 'out of client slots' errors way before the maximum was reached.
Merovingian
  • Resolved a problem where monetdb could fail to start a database with the message "database 'X' started up, but failed to open up a communication channel". Bug #3134 [63], comment #7.
MonetDB Common
  • Fixed a bug in BATantijoin when either side is a singleton BAT. This fixes bug 3139 [70].
Bug Fixes
  • 3117 [71]: test sql/test/BugTracker-2011/Tests/history.Bug-2909.sql produces extra profiling output since changeset 817328b74e2f
  • 3134 [63]: Database gets killed by timeout during startup
  • 3139 [70]: Not equal evaluation does not work
  • 3144 [72]: Memleak when connecting to a database (mclient and mapi)
  • 3145 [73]: mclient error when enabling history
  • 3150 [74]: Multiple string equality tests fail to match (Test_JdbcClient)

April 2012

Apr 2012-SP2 bugfix release
Client Package
  • Resolved a cosmetical error where tab-characters would cause misalignments in tabular result views. For the time being, tabs are now represented as a single space in tabular view.
MonetDB Common
  • Limit number of detected CPU cores to 16 on 32-bits systems to avoid running quickly out of addressable resources followed by a kill from the OS.
Bug Fixes
  • 2594 [75]: NULL cannot automatically be casted to every other type
  • 3085 [76]: assertion fails upon execution of a valid SQL query
  • 3086 [77]: Modulo function does not work in select from table
  • 3087 [78]: query causes "Not yet implemented" message
  • 3089 [79]: mserver5 crashed if join condition has two column and incorrect order
  • 3090 [67]: crashed if using single identifier for where condition
  • 3096 [80]: Type of median error

 

 

Apr 2012-SP1 bugfix release

Build Environment
  • Windows: we now use OpenSSL 1.0.1b.
SQL
  • Databases that were upgraded from the Aug2011 release have an error in the catalog for SQL procedures. This is now fixed.
Merovingian
  • Fixed a bug where connecting to a stopped multiplex-funnel would result in a 'there are no available connections' error.
MonetDB Common
  • Implemented MT_getrss for Mac OS X systems, this allows the server to know about how much memory is currently in use.
Bug Fixes
  • 2454 [81]: SQL: add various statistical functions
  • 2916 [82]: non admin user cannot create temporary tables
  • 2949 [83]: SQL exception during execution of a correct user defined function that just returns a relation table constructed by a simple SQL select query
  • 2958 [84]: Grant does't work with schema syntax
  • 2960 [85]: mclient terminates connection during insertion of a large number of records from a file
  • 2972 [86]: SQL URL functionality contains errors
  • 2987 [87]: Broken type inference when using sql round
  • 2992 [88]: SQL module truncates DECIMALS when passing integers to MAL module
  • 3000 [89]: Inconsistency between read-only and read-write modes after dropping a schema
  • 3028 [90]: Segmentation fault in seqscan_eq_bte_bte_tloc_void_void
  • 3046 [91]: mserver5 terminates with segmentation fault
  • 3063 [92]: datacell functions no longer work after restart of database
  • 3072 [93]: Database upgrade from Dec2011 to Apr2012 broken for storage()
  • 3073 [94]: VERY slow foreign key check (due to missing batcalc.isnil() ?)
  • 3074 [95]: Name resolution in procedures
  • 3077 [96]: No errors on UDF returning table with too many columns
  • 3079 [97]: Converting an Oracle schema DDL for MonetDB fails to create a column named "no".

 

 

Apr 2012 feature release
Build Environment

  • Fixed some of the package names for Debian/Ubuntu. Packages for libraries should contain the major number of the library version number. This was not always the case.

  • On Debian and Ubuntu, install Python modules in dist-packages instead of site-packages. This fixed bug 2997 [98].
SQL

  • Fixed a crash that happened at the end of a database upgrade to the Dec2011 database scheme. The crash happened during cleanup after the database was upgraded, so it was merely inconvenient.

  • Stripped off implementation-specific parts from error messages before they get presented to the user.
Java Module

  • Password reading by JdbcClient no longer results in strange artifacts

  • JdbcClient now returns exit code 1 in case of failures

  • Implemented missing Number types support in PreparedStatement.setObject()

  • Fixed bug in DatabaseMetaData.getSchemas() method that caused an SQL error when called with catalog and schema argument.

  • Resolved a bug where JDBC and Control connections could terminate abruptly with 'Connection closed' messages
Client Package

  • ODBC: Implemented the SQL_ATTR_CONNECTION_TIMEOUT attribute.

  • mclient now has a -a (--autocommit) option to turn off autocommit mode.
MonetDB5 Server

  • The type "chr" has been removed. chr has long been superseded by bte for 1 byte arithmetic plus it is pretty useless to hold single characters since we use Unicode and thus only a tiny subset of the supported character set would fit.

  • Fixed potential crash by dealing better with non-standard types.

  • A couple of memory leaks have been plugged.
Merovingian

  • The logfile and pidfile monetdbd properties are now displayed with dbfarm path when relative

  • Fixed a bug in the multiplex-funnel where certain clients would abort on responses for update queries.

  • Fixed problem where version and mserver properties for monetdbd were not always successfully retrieved. Bug #2982 [99].

  • Fixed problem where shutdown of monetdbd would lead to shutting down database 'control' which does not exist. Bug #2983 [100].

  • Fixed issue causing (harmless) 'error reading from control channel' messages. Bug #2984 [101].

  • Resolved problem where remote start/stop/etc. commands with monetdb would report error 'OK'. Bug #2984 [101].
MonetDB Common

  • The type "chr" has been removed. chr has long been superseded by bte for 1 byte arithmetic plus it is pretty useless to hold single characters since we use Unicode and thus only a tiny subset of the supported character set would fit.
Bug Fixes
  • 2994 [102]: test monetdb5/modules/kernel/Tests/vacuum.mal fails since changeset ada288cef37c
  • 3023 [103]: SQL tests fail as error messages now go to stdout iso. stderr
  • 3030 [104]: optimizer tests fail with new optimizer framework
  • 3032 [105]: numerous M5/MAL tests fail on SunOS/Solaris as TypeExecptions do no longer show arguments in command signatures

 

December 2011

MonetDB Dec2011 Release

 

 

Dec 2011-SP2 bugfix release
Java Module
  • Implemented missing Number types support in PreparedStatement.setObject()
Bug Fixes
  • 2915 [106]: Query (mclient or jdbc) with 2 outer joins appears to crash connectionor server
  • 2947 [107]: Queries where a "is null" clause is combined with two equals clauses fail to produce the expected result.
  • 2968 [108]: Memory Leaks
  • 2974 [109]: thetajoin produces invalid/inconsistent bat
  • 2988 [110]: Subquery in select part goes wrong
  • 2998 [111]: Equivalent multi-way join queries give different results
  • 3002 [112]: String plus interval result in program errors upon comparison
  • 3011 [113]: count(*) in subquery results in 1 for query without results
  • 3012 [114]: bitwise comparison fails after a cast
  • 3029 [115]: test sql/test/Dependencies/Tests/Dependencies.sql fails after recent changes
  • 3031 [116]: Virtually all SQL tests fail on Windows
  • 3036 [117]: optimizers are not getting set correctly through set optimizer
  • 3038 [118]: ~190 SQL tests fail on Darwin, FreeBSD & Solaris (only) since introduction of "(extended) relational algebra"
  • 3043 [119]: WITH clause is not allowed in derived table
  • 3045 [120]: algebra.project(0@0,bat) fails at propcheck
  • 3052 [121]: Wrong Error Code on Rollback Exception

 

Dec 2011-SP1 bugfix release
Build Environment
  • Fixed some of the package names for Debian/Ubuntu. Packages for libraries should contain the major number of the library version number. This was not always the case.
  • On Debian and Ubuntu, install Python modules in dist-packages instead of site-packages. This fixed bug 2997 [98].
SQL
  • Fixed a crash that happened at the end of a database upgrade to the Dec2011 database scheme. The crash happened during cleanup after the database was upgraded, so it was merely inconvenient.
  • Stripped off implementation-specific parts from error messages before they get presented to the user.
Java Module
  • Fixed bug in DatabaseMetaData.getSchemas() method that caused an SQL error when called with catalog and schema argument.
  • Resolved a bug where JDBC and Control connections could terminate abruptly with 'Connection closed' messages
MonetDB5 Server
  • Fixed potential crash by dealing better with non-standard types.
Merovingian
  • Fixed a bug in the multiplex-funnel where certain clients would abort on responses for update queries.
  • Fixed problem where version and mserver properties for monetdbd were not always successfully retrieved. Bug #2982 [99].
  • Fixed problem where shutdown of monetdbd would lead to shutting down database 'control' which does not exist. Bug #2983 [100].
  • Fixed issue causing (harmless) 'error reading from control channel' messages. Bug #2984 [101].
  • Resolved problem where remote start/stop/etc. commands with monetdb would report error 'OK'. Bug #2984 [101].
Bug Fixes
  • 2862 [122]: foreign key self references cause violation upon alter table add constraint...
  • 2933 [123]: "TypeException: algebra.join undefined" when evaluating EXISTS (SELECT ...) predicate
  • 2946 [124]: Invalid BAT if left join condition has only columns from right table
  • 2964 [125]: LEFT OUTER JOIN with constants returns errors
  • 2969 [126]: Result precision issue with Decimal data type
  • 2973 [127]: Date coercion functionality tests
  • 2979 [128]: misc merovingian control problems
  • 2982 [99]: monetdbd status and mserver properties have random values
  • 2983 [100]: monetdbd attempts to stop non-existing database 'control'
  • 2984 [101]: monetdb create/release/start/stop returns error "OK" for remote connections
  • 2995 [129]: Test geom/sql/Tests/example.sql crashes since changeset 6399077ff8a4
  • 2997 [98]: monetdb-python packages installed in wrong dir on Ubuntu 10.04 LTS (lucid)
  • 3004 [130]: Incorrect evaluation of conditions in WHERE part of SQL statement
  • 3009 [131]: Segmentation fault on simple query
  • 3013 [132]: calc.and is not yet defined for dbl
  • 3014 [133]: dayofweek comment is wrong in mtime.mx:700
  • 3017 [134]: rel_bin.c:611: exp_bin: Assertion `0' failed. with between on multicolumns
  • 3020 [135]: Assertion error in debugging mode
  • 3034 [136]: INNER JOIN with Subquery produces mal errors
  • 3041 [137]: WHERE condition ignored after 2 LEFT JOINs with subqueries

 

Dec 2011 feature release
SQL
  • mclient: The csv output format can now also be of the form csv=c and csv+c where c is the column separator. The form with csv+ produces a single header line with column names.
  • Changed a bug where the sign() function returned the same type as its argument instead of always an INTEGER.
  • Added a fix for bug #2834 [138], which caused weird (failing) behaviour with PreparedStatements.
  • Fixed a bug when using default values for interval columns. Bug 2877 [139].
  • A bug was fixed where deleted rows weren't properly accounted for in all operations. This was bug 2882 [140].
  • A bug was fixed which caused an update to an internal table to happen too soon. The bug could be observed on a multicore system with a query INSERT INTO t (SELECT * FROM t) when the table t is "large enough". This was bug 2883 [141].
Java Module
  • Implemented getMetaData() method of PreparedStatement.
  • Fixed an AssertionError for special results from e.g. EXPLAIN queries.
  • Fixed overflow error when batching large statements, bug #2952 [142]
  • Resolved a concurrency problem where ResultSet's date-related getters could cause odd stack traces when used by multiple threads at the same time, bug #2950 [143].
  • JDBC now implements JDBCv4.1 which makes it possible to be built with Java 7 and up. JDBCv4.1 is a maintenance release of JDBCv4, and hence can be compiled with Java 6, although the added methods obviously are not part of the java.sql interfaces.
  • SQLExceptions thrown now carry a SQLSTATE. Until the server starts sending correct SQLSTATEs for all errors, server originated errors without SQLSTATE are considered generic data exceptions (22000).
  • Fixed a bug where closing ResultSets and PreparedStatements could lead to errors on concurrent running queries using the same Connection due to a race condition.
  • Changed version scheme of JDBC driver and MCL jar to be more standard, from monetdb-X.Y-<thing>.jar to monetdb-<thing>-X.Y.jar, bug #2943 [144]
  • The embedded monet instance capability of MonetConnection was removed.
  • Bump JDBC version to 2.0 (Liberica). JDBC now implements JDBCv4 which makes it possible to be built with Java 6 and up. Java 5 and before are no longer supported, and can use the 1.x releases of the driver.
  • INTERVAL columns are now treated as decimals, since they can have sub-second precision.
Client Package
  • Fix making connections with stethoscope to hosts without monetdbd. Bug #2944 [145].
  • Fixed some bugs in the PHP interface affecting the mapi_ping(), monetdb_insert_id() and auto_commit() functions. Bugs #2936 [146], #2937 [147], #2938 [148].
  • ODBC: Fixed SQLNumResultCols and SQLDescribeCol to return useful information after a call to SQLPrepare and before any SQLExecute.
  • mclient: The exact interpretation of the -i (--interactive) option and the - filename argument have changed. The - filename argument means read from standard input and no longer implies that no \ interpretation is done on the input. Instead, \ interpretation is done if either standard input is a terminal, or if the -i option is given. The -i option no longer causes mclient to read from standard input. It only means to do \ interpretation when reading from standard input. Use the - filename argument to read from standard input. Note that if no -s option is specified and no filename arguments are present, mclient still reads from standard input.
  • ODBC: Improved internal query for SQLSpecialColumns. Before, the query returned all columns taking part in a PRIMARY KEY *and* all columns taking part in a UNIQUE constraint. Now it returns only one or the other set.
  • ODBC: The database name is now used as the catalog name throughout. Functions that return a catalog name return the database name, and functions that match on catalog name match it with the database name.
  • ODBC: Implemented an easier way to create a log file of interactions with the ODBC driver. You can now add a connection attribute "LOGFILE=filename" to the connection string parameter of SQLBrowseConnect and SQLDriverConnect, and to the relevant part of the Windows registry or odbc.ini file. This value is only used if there is no environment variable ODBCDEBUG.
  • Quoting of object names for mclient's \d command is now more flexible and consistent with standard SQL quoting rules, bug #2846 [149].
  • ODBC: Implemented SQLColumnPrivileges function.
  • Install new program, mnc, which provides netcat functionality, but based on MonetDB's communication libraries
  • Report full URI to database instead of just database when available in mclient.
  • The time format of the timer output can now be controlled with an optional argument to the -i option. ms, s and m force the time to be formatted as milliseconds, seconds or minutes + seconds respectively.
  • ODBC: Implemented a workaround in SQLTables for bug 2908 [150].
  • Small improvement to mclient's table rendering for tables without any rows. Previously, the column names in the header could be squeezed to very small widths, degrading readability.
  • Python DB API connect() function now supports PEP 249-style arguments user and host, bug #2901 [151]
  • mclient now checks the result of encoding conversions using the iconv library.
  • Fixed a source of crashes in mclient when a query on the command line using the -s option is combined with input on standard input (e.g. in the construct mclient -s 'COPY INTO t FROM STDIN ...' < file.csv).
  • Fixed bug 2897 [152] where slow (network) reads could cause blocks to not be fully read in one go, causing errors in the subsequent use of those blocks. With thanks to Rémy Chibois.
  • Fixed a bug in mclient where processing queries from files could result in ghost empty results to be reported in the output
  • Fixed Perl DBD rowcount for larger results, bug #2889 [153]
  • Perl: We now distinguish properly between TABLE and GLOBAL TEMPORARY (the latter are recognized by being in the "tmp" schema).
  • Perl: fixed a bunch of syntax errors. This fixes bug 2884 [154]. With thanks to Rémy Chibois.
  • Perl: Fixed DBD::monetdb table_info and tabletype_info. This fixes bug 2885 [155]. With thanks to Rémy Chibois.
  • mclient: fix display of varchar columns with only NULL values.
  • Fixed a bug in mclient/msqldump where an internal error occurred during dump when there are GLOBAL TEMPORARY tables.
MonetDB5 Server
  • A couple of memory leaks have been plugged.
  • Introduced new variable max_clients that allows to define how many concurrent connections are allowed to be made against the database.
  • mal_interpreter.mx Protect against concurrent exceptions If multiple parallel blocks access the exception variables and perform GDKfree on old messages, then we may enter a case that one thread attempts a second free. Simple lock-based protection is the first line of defense.
  • bpm and partitions have been moved to the attic. It is replaced by the partition optimizer, still under construction.
  • The MAL debugger next/step operations semantics have been fixed. The profiler also now clearly shows entrance/exit of a MAL function.
  • The dataflow optimizer uses a less strict side-effect test for BAT new operations and better recognizes trivial plans.
  • Fixed a problem where MAL variables weren't properly cleared before reuse of the data strucutre. This problem could cause the data flow scheduler to generate dependencies between instructions that didn't actually exist, which in turn could cause circular dependencies among instructions with deadlock as a result. Bugs 2865 [156] and 2888 [157].
Merovingian
  • Fixed crash in monetdb when an invalid property was retrieved using the get command, bug #2953 [158].
  • monetdb get output is now grouped by database instead of by property
  • Unlike in previous releases, 'monetdbd get mserver' now returns the path to the mserver5 binary only for dbfarms that have a running monetdbd, instead of only for those that are not served by a monetdbd.
  • Add nclients property that controls the maximum number of concurrent clients allowed to access the database
  • Starting monetdbd without any arguments or without dbfarm is no longer supported. A dbfarm now must be provided for each command.
  • The control passphrase has been turned into a hash of the password, for a more stronger authorisation model as used by mclients. On upgrade, any existing passphrase is converted to the hashed version of the passphrase.
  • The monetdbd controlport option has been removed in favour of a boolean control option. On upgrade, when controlport was set to a non-zero value and a passphrase was set, control is set to true, or false otherwise.
  • The monetdbd discoveryport option has been removed in favour of a boolean discovery option. On upgrade, when discoveryport was set to a non-zero value, discovery is set to true, or false otherwise.
  • Fixed a bug where monetdbd's socket files from /tmp were removed when a second monetdbd was attempted to be started using the same port.
  • Resolved problem where monetdbd would terminate abnormally when databases named 'control', 'discovery' or 'merovingian' were stopped.
  • monetdbd get status now also reports the version of the running monetdbd
  • Improved response time of 'monetdb start' when the database fails to start.
  • Fixed a bug in monetdbd where starting a failing database could incorrectly be reported as a 'running but dead' database.
  • To avoid confusion, all occurrences of merovingian were changed into monetdbd for error messages sent to a client.
MonetDB Common
  • Fixed memory detection on 32-bits Solaris systems with more memory than can be addressed in 32-bits mode, bug #2935 [159]
Bug Fixes
  • 2740 [160]: Segmentation fault using a column inside of a functioncall
  • 2802 [161]: update JDBC driver to JDBC4
  • 2846 [149]: \d command fails if the schema is provided explicitly
  • 2873 [162]: SQL: time with precision
  • 2891 [163]: Make MapiPP.pm fetch results progressively
  • 2910 [164]: SQLSetStmtOption(..., SQL_CONCURRENCY, SQL_CONCUR_READ_ONLY) should be allowed
  • 2914 [165]: mserver5 crash for huge_nr_of_columns.Bug-2609 on ppc-macos
  • 2934 [166]: Database upgrade with SQL catalog upgrade messes up privileges
  • 2935 [159]: mserver5 doesn't detect memory on Solaris/Sparc
  • 2936 [146]: bug into mapi_ping()
  • 2937 [147]: minor bug into monetdb_insert_id() (php_monetdb.php)
  • 2938 [148]: minor bug into auto_commit() (php_monetdb.php)
  • 2939 [167]: "PHP Notice" in mapi_execute()
  • 2940 [168]: PHP split() function deprecated
  • 2944 [145]: stethoscope on can't connect to database without monetdbd (discovery enabled)
  • 2950 [143]: MonetResultSet use of SimpleDateFormat not threadsafe
  • 2952 [142]: utf-8 characters in long string
  • 2953 [158]: segfault for monetdb get with invalid property
  • 2959 [169]: Nested SELECT with EXISTS causes the server to crash
  • 2963 [170]: invalid tuple received from server, got 17 columns, expected 24, ignoring
  • 2965 [171]: joinPath runs indeterminately
  • 2975 [172]: CSV file is kept write-locked after mclient failed to COPY
  • 2981 [173]: Different answer from querys in the ODBC compared to mcliente or other DBMSs

August 2011

The Aug2011 release is the first release where the default size of OIDs on 64-bit WIndows systems is 64 bits.  Before, the default on Windows was 32 bits.  On all other 64-bit platforms, the default size of OIDs was also 64 bits.  This means that, on 64-bit Windows systems, the database needs to be converted.  This is done automatically when the database is first opened, but as always with automatic conversions, it is a good idea to make a backup first.

The Aug2011 uses a slightly different catalog from earlier releases.  The database is automatically updated when first opened.

This release is the last release that comes with a JDBC driver that can be used with Java 1.4 and 1.5, also known as JDBCv3.

 

 

 

Aug 2011-SP3 bugfix release
SQL
  • Added a fix for bug #2834 [138], which caused weird (failing) behaviour with PreparedStatements.
Merovingian
  • Fixed a bug where monetdbd's socket files from /tmp were removed when a second monetdbd was attempted to be started using the same port.
Bug Fixes
  • 2875 [174]: incomplete database initialization
  • 2912 [175]: Function body disappears after loading data - error message: SQLException:sql.resultSet:Cannot access descriptor
  • 2919 [176]: Add column and select in same transaction
  • 2920 [177]: Updated database from Aug2011 to Dec2011 make queries fail with "Subquery result missing".
  • 2921 [178]: complex where clause broken
  • 2923 [179]: Multiplication and cast from decimal to integer leads to 0 value
  • 2924 [180]: use LIKE with ESCAPE ignores OR operator
  • 2928 [181]: Cannot update a field with a count query based on an other table

 

 

 

Aug 2011-SP2 bugfix release
Client Package
  • ODBC: Implemented a workaround in SQLTables for bug 2908 [150].
  • Small improvement to mclient's table rendering for tables without any rows. Previously, the column names in the header could be squeezed to very small widths, degrading readability.
  • Python DB API connect() function now supports PEP 249-style arguments user and host, bug #2901 [151]
  • mclient now checks the result of encoding conversions using the iconv library.
  • Fixed a source of crashes in mclient when a query on the command line using the -s option is combined with input on standard input (e.g. in the construct mclient -s 'COPY INTO t FROM STDIN ...' < file.csv).
  • Fixed bug 2897 [152] where slow (network) reads could cause blocks to not be fully read in one go, causing errors in the subsequent use of those blocks. With thanks to Rémy Chibois.
  • Fixed a bug in mclient where processing queries from files could result in ghost empty results to be reported in the output
Merovingian
  • Resolved problem where monetdbd would terminate abnormally when databases named 'control', 'discovery' or 'merovingian' were stopped.
  • monetdbd get status now also reports the version of the running monetdbd
  • Improved response time of 'monetdb start' when the database fails to start.
  • Fixed a bug in monetdbd where starting a failing database could incorrectly be reported as a 'running but dead' database.
  • To avoid confusion, all occurrences of merovingian were changed into monetdbd for error messages sent to a client.
Bug Fixes
  • 2346 [182]: mkey.bulk_rotate_xor_hash MAL exception
  • 2712 [183]: un-rolled-back savepoints cause crash upon scenario exit
  • 2774 [184]: COPY INTO of double (~1.0e-310) cannot be imported.
  • 2890 [185]: Slow execution of COUNT in combination with DISTINCT
  • 2894 [186]: No consistent behavior of implicit cast of integer to double
  • 2897 [152]: Mapi.pm: short reads not handled when reading blocks from server
  • 2898 [187]: order by for multiple (two) columns fails for large tables when second ordering is 'desc': "mat.refine_reverse' undefined"
  • 2900 [188]: rand(int) should not return NULL
  • 2901 [151]: Make keyword arguments in Python DB API connect() function more standard
  • 2904 [189]: TypeException after calling UDF in other than sys schema (when it is initialised)
  • 2906 [190]: "ALTER TABLE mytable SET READ ONLY" results in server segfault
  • 2908 [150]: SQL query produces error with MAL statements
  • 2909 [191]: query result is not displayed when history = true

 

Aug 2011-SP1 bugfix release
SQL
  • Fixed a bug when using default values for interval columns. Bug 2877 [139].
  • A bug was fixed where deleted rows weren't properly accounted for in all operations. This was bug 2882 [140].
  • A bug was fixed which caused an update to an internal table to happen too soon. The bug could be observed on a multicore system with a query INSERT INTO t (SELECT * FROM t) when the table t is "large enough". This was bug 2883 [141].
Client Package
  • Fixed Perl DBD rowcount for larger results, bug #2889 [153]
  • Perl: We now distinguish properly between TABLE and GLOBAL TEMPORARY (the latter are recognized by being in the "tmp" schema).
  • Perl: fixed a bunch of syntax errors. This fixes bug 2884 [154]. With thanks to Rémy Chibois.
  • Perl: Fixed DBD::monetdb table_info and tabletype_info. This fixes bug 2885 [155]. With thanks to Rémy Chibois.
  • mclient: fix display of varchar columns with only NULL values.
  • Fixed a bug in mclient/msqldump where an internal error occurred during dump when there are GLOBAL TEMPORARY tables.
MonetDB5 Server
  • Fixed a problem where MAL variables weren't properly cleared before reuse of the data strucutre. This problem could cause the data flow scheduler to generate dependencies between instructions that didn't actually exist, which in turn could cause circular dependencies among instructions with deadlock as a result. Bugs 2865 [156] and 2888 [157].
Bug Fixes
  • 2865 [156]: mserver5 --set gdk_nr_threads=2 --forcemito: deadlock during first SQL client connect on virgin (empty) DB
  • 2877 [139]: assertion failed when creating table with default interval
  • 2879 [192]: A SELECT query fails to produce some results
  • 2882 [140]: Delete of some records don't work on Monetdb
  • 2883 [141]: insert into t (select * from t) bogus in multithreaded execution
  • 2884 [154]: Mapi.pm and GetInfo.pm syntax fixes for array usage
  • 2885 [155]: DBD::monetdb table_info error
  • 2887 [193]: Division by zero error, when should not divide at 0
  • 2888 [157]: test/null/null_in_case.sql hangs server
  • 2889 [153]: Mapi::getBlock is erasing query count on second call

 

 

Aug 2011 feature release
Build Environment
  • Fixed Fedora 15 (and presumably later) configuration that uses a tmpfs file system for /var/run. This fixes bug 2850 [194].
  • The default OID size for 64-bit Windows is now 64 bits. Databases with 32 bit OIDs are converted automatically.
Java Module
  • Add so_timeout Driver property to specify a SO_TIMEOUT value for the socket in use to the database. Setting this property to a value in milliseconds defines the timeout for read calls, which may 'unlock' the driver if the server hangs, bug #2828 [195]
  • Added a naive implementation for PreparedStatement.setCharacterStream
  • The obsolete Java-based implementation for PreparedStatements (formerly activated using the java_prepared_statements property) has been dropped
Client Package
  • msqldump now also accepts the database name as last argument on the command line (i.e. without -d).
  • Made error messages from the server in mclient go to stderr, instead of stdout.
  • Implemented SQL_ATTR_METADATA_ID attribute. The attribute is used in SQLColumns, SQLProcedures, and SQLTablePrivileges.
  • Implemented SQLTablePrivileges in the ODBC driver.
  • mclient now automatically sets the SQL `TIME ZONE' variable to its (the client's) time zone.
  • Removed perl/Cimpl, MonetDB-CLI-MapiLib and MonetDB-CLI-MapiXS
  • Switched implementation of MonetDB::CLI::MapiPP to Mapi.pm, and made it the default MonetDB::CLI provider.
  • Made Mapi.pm usable with current versions of MonetDB again
  • When the first non-option argument of mclient does not refer to an exising file, it now is taken as database name. This allows to simply do `mclient mydb`.
MonetDB5 Server
  • Make crackers optional and disable by default, since it wasn't used normally
  • Authorisation no longer takes scenarios into account. Access for only sql or mal is no longer possible. Any credentials now mean access to all scenarios that the server has available.
Geom Module
  • Implemented NULL checks in the geom module. Now when given NULL as input, the module functions return NULL instead of an exception. This fixes bug 2814 [196].
Merovingian
  • Fixed confusing 'Success' error message for monetdb commands where an invalid hostname was given
  • The path to the mserver5 binary is no longer returned for the mserver property with monetdbd get for a dbfarm which is currently served by a monetdbd. Since the called monetdbd needs not to be the same as the running monetdbd, the reported mserver5 binary may be incorrect, and obviously lead to confusing situations. Refer to the running monetdbd's logfile to determine the mserver5 binary location instead.
  • Added -n option to monetdbd start command, which prevents monetdbd from forking into the background.
  • Fix incorrect (misleading) path for pidfile in pidfile error message, bug #2851 [197]
  • added status property to get command
MonetDB Common
  • Removed conversion code for databases that still used the (more than two year) old format of "string heaps".
  • On Windows and Linux/Unix we can now read databases built on the other O/S, as long as the hardware-related architecture (bit size, floating point format, etc.) is identical.
  • Implemented automatic conversion of a 64-bit database with 32-bit OIDs to one with 64-bit OIDs.
Bug Fixes
  • 2336 [198]: ANSI Client clobbered data ::SQLTables
  • 2579 [23]: SQL: incorrect cast from double to int
  • 2701 [199]: Divide by zero in OPTmitosis with select name from tables
  • 2814 [196]: Throws exception if a SQL/GIS takes argument null
  • 2817 [200]: Failed to find a table producing function
  • 2828 [195]: No Socket Timeout: MonetDB Java client hangs forever on reads and consume threads when the database is crashed
  • 2836 [201]: compilation fails when --with-valgrind in effect
  • 2838 [202]: Nested case results in message: "Predicate CASE 42: time to implement some more"
  • 2839 [203]: test monetdb5/optimizer/Tests/inlineFunction2 fails
  • 2840 [204]: Test monetdb5/optimizer/Tests/reorder01 fails
  • 2841 [205]: Tests monetdb5/optimizer/Tests/inline0{2,3,6} fail
  • 2842 [206]: Tests monetdb5/optimizer/Tests/dataflow{,3,4} lack stable output
  • 2845 [207]: INSERT INTO fails (hangs) when values are derived from SELECT, in combination with a FOREIGN KEY constraint
  • 2847 [208]: SELECT statement throws str_cast exception
  • 2848 [209]: Problem when reusing column name as CASE... WHEN alias in subquery
  • 2849 [210]: Multi-column constraints cause data corruption
  • 2850 [194]: monetdbd cannot open pidfile on Fedora 15
  • 2851 [197]: Incorrect error message when pidfile cannot be opened
  • 2852 [211]: Query with CASE and nested functions results in very long execution
  • 2855 [212]: test monetdb5/modules/mal/Tests/inspect00 fails
  • 2856 [213]: test monetdb5/optimizer/Tests/dataflow fails
  • 2857 [214]: obsolete conditional HAVE_LIBXML2 still used on Windows
  • 2859 [215]: prepare statement tests fail with number of arguments mismatch
  • 2860 [216]: /etc/init.d/monetdb5-sql uses incorrect PIDFILE to test if monetdb is running
  • 2862 [122]: foreign key self references cause violation upon alter table add constraint...
  • 2866 [217]: test monetdb5/optimizer/Tests/dataflow.mal fails on all Windows & Solaris (only)
  • 2868 [218]: tests tst285 & pithole fail on some Windows platforms
  • 2869 [219]: SQL tests with "LIKE" crash on Windows2008 & WindowsXP while working fine on Windows7

 

April 2011

This is the first release of the MonetDB suite that does not contain MonetDB4 and MonetDB/XQuery. As already mentioned in the Mar2011 release notes [220], work on the Pathfinder compiler, which forms the basis of MonetDB/XQuery, continues at The University of Tübingen [221].

An upgrade from the Mar2011 release should be smooth, even if MonetDB/XQuery is installed. The Windows installers for MonetDB/SQL and MonetDB/XQuery have always been independent, and the Fedora, Debian, and Ubuntu versions have been made independent in the Mar2011 release.

The Apr2011 release uses a slightly changed SQL catalog than previous releases. The catalog will be updated automatically the first time an SQL database is opened. Note that because of the incompatibility, once updated, the database cannot be read by older versions. Also note that a database from the Feb2010 release or before cannot be read by this release. First update the database with the Mar2011 release.

This is also the first release where we have created FreeBSD binaries. These binaries are highly experimental. The software should run as well as on any other system, but the packaging is obviously different from other systems. As before, we also provide binaries for Windows, Fedora (13 and 14), Debian (5.0 and 6.0), Ubuntu (9.10, 10.04, and 10.10), Mac OS X, and Solaris.

 

 

Apr 2011-SP2 bugfix release
Build Environment
  • We can now build RPMs on CentOS 6.0. Since there is no geos library on CentOS, we do not support the geom modules there.
  • We can now also build on Fedora 15. This required some very minor changes.
  • Changed configure check for OpenSSL so that we can also build on CentOS 5.6. We now no longer demand that OpenSSL is at least version 0.9.8f, but instead we require that the hash functions we need are supported.
Client Package
  • Fix stethoscope's mod.fcn filter when using multiple targets, bug #2827 [222]
  • The separate Python distribution now uses the same version number as the main package.
MonetDB Common
  • Fixed a problem where appending string BATs could cause enormous growth of the string heap. This fixes bug 2820 [223].
  • Fixes to memory detection on FreeBSD.
Java Module
  • Return false from Statement.getMoreResults() instead of a NullPointerException when no query has been performed on the Statement yet, bug #2833 [224]
  • Fixed read-only interpretation. Connection.isReadOnly now always returns false, setReadOnly now generates a warning when called with true. Partly from bug #2818 [225]
  • Allow readonly to be set when autocommit is disabled as well. Bug #2818 [225]
SQL
  • Fixed regression where the superuser password could no longer be changed, bug #2844 [226]
  • Fixed incorrect insert counts.
  • Fixed bug 2823 [227]: MAL exeption on SQL query with subquery in the where part.
  • Redirect error from create scripts back to the first client. This fixes bug 2813 [228].
  • Added joinidx based semijoin; push join through union (using joinidx).
  • Fixed pushing select down.
Bug Fixes
  • 2058 [229]: scoping issue of functions and with
  • 2734 [230]: MALException:mkey.bulk_rotate_xor_hash on complex SQL query
  • 2767 [231]: `make uninstall` fails in clients/rubyadapter
  • 2797 [232]: mserver crashes at initialization when the disk is full
  • 2811 [233]: mserver crashes
  • 2812 [234]: Subqueries in FROM clause not accepted
  • 2813 [228]: SQL: catalog is built even error has occurred
  • 2818 [225]: JDBC: exception on calling Connection.setReadOnly
  • 2819 [235]: make text readable
  • 2820 [223]: SIGSEGV in BATins_kdiff triggered by a DELETE statement
  • 2821 [236]: segmentation fault when a select query (with group by) is run on server with "--single-user" option
  • 2822 [237]: A query with CASE and division results in ArithmeticException
  • 2823 [227]: MAL exeption on SQL query with subquery in the where part
  • 2825 [238]: "Insert into ... select distinct ..." fails with MALException:mkey.bulk_rotate_xor_hash:operation
  • 2826 [239]: sql function iterates instead of bulk processing
  • 2827 [222]: Incorrect filter setting when listening to several servers
  • 2829 [240]: logging issue
  • 2830 [241]: select 30.0 - 0.025; conversion to decimal type gives error
  • 2831 [242]: Crash in query analysis
  • 2832 [243]: Server terminates upon UPDATE
  • 2833 [224]: NullPointerException when calling getMoreResults() on Statement without a query executed
  • 2835 [244]: UNIQUE Constraint is not enforced
  • 2837 [245]: Server terminates after CREATE FUNCTION in multi-user scenario
  • 2844 [226]: ALTER USER SET PASSWORD doesn't recognise username

 

Apr 2011-SP1 bugfix release
MonetDB Common Fixed a bug where large files (> 2GB) didn't always get deleted on Windows.
Java Module Insertion via PreparedStatement and retrieval via ResultSet of timestamp and time fields with and without timezones was improved to better respect timezones, as partly indicated in bug #2781 [246].
MonetDB5 Server
 Fixed a bug in conversion from string to the URL type.  The bug was an incorrect call to free().
Bug Fixes
  • 2784 [247]: configure does not (properly) check for yacc/bison
  • 2798 [248]: Query did not produce a result set
  • 2806 [249]: Wrong logic in predicates over views defined with an open-interval predicate
  • 2807 [250]: wrong results for group by queries in the presence of primary keys
  • 2808 [251]: wrong results for count()/count(distinct) on the same attribute
  • 2810 [252]: Copy into should rollback correctly (i.e. restore the files) on a failure

 

Apr 2011 feature release

Client Package

Plugged a small memory leak occurring upon redirects by the server (e.g. via monetdbd)
Remove XQuery related code from Ruby adapter, PHP driver and Perl Mapi library
Make SQL the default language for mclient, e.g. to use when --language= or -l is omitted
MonetDB Common Fixed bugs in antiselect which gave the incorrect result when upper and lower bound were equal. This bug could be triggered by the SQL query SELECT * FROM t WHERE x NOT BETWEEN y AND y
Java Module Clarify exception messages for unsupported methods
Removed XQuery related XRPC wrapper and XML:DB code, removed support for language=xquery and language=mil from JDBC.
MonetDB5 Server
 Mserver5 no longer reads monetdb5.conf upon startup by default. Use --config=file to have mserver5 read a configuration on startup
SQL Some names in the SQL catalog were changed. This means that the database in the Apr2011 release is not compatible with pre-Apr2011 databases. The database is converted automatically when opened the first time. This database can then no longer be read by an older release.
Merovingian Fix monetdb return code upon failure to start/stop a database.
The forward property for databases has been removed. Instead, only a global proxy or redirect mode can be set using monetdbd.
Monetdbd can no longer log error and normal messages to separate logfiles, logging to stdout and stderr is no longer possible either.
The .merovingian_pass file is no longer in use, and replaced by the .merovingian_properties file. Use monetdbd (get|set) passphrase to view/edit the control passphrase. Existing .merovingian_pass files will automatically be migrated upon startup of monetdbd.
Monetdbd now understands commands that allow to create, start, stop, get and set properties on a given dbfarm. This behaviour is intended as primary way to start a MonetDB Database Server, on a given location of choice. monetdbd get and set are the replacement of editing the monetdb5.conf file (which is no longer in use as of the Apr2011 release). See monetdbd(1).
Bug Fixes
  • 1956 [253]: Geom: relation and spatial tests fail
  • 2547 [254]: create schema leads to sql_stmt symbol(x)->token = CREATE_SCHEMA
  • 2588 [255]: test sql/src/test/BugTracker/Tests/authorization.SF-1430616.SQL.sh shows different error message
  • 2590 [256]: test sql/src/test/BugTracker/Tests/mdb_starts_with_sql_debug_64.SF-1999354.py fails with "too many digits"
  • 2639 [257]: Readline auto-complete reads from 'tables' instead of 'sys.tables'
  • 2766 [258]: SQL: Compute modulo of rand() in an UPDATE stmt crashes mserver5
  • 2779 [259]: Make building GDK optional
  • 2793 [260]: EXTRACT(SECOND FROM timestamp) always returns 0 integer seconds part.
  • 2795 [261]: SQL: ALTER TABLE with incorrect syntax crashes mserver5
  • 2796 [262]: latest antiselect changes cause ~20 SQL tests to fail
  • 2800 [263]: Test_PStypes fails due to ambiguous identifier
  • 2801 [264]: Nested SELECT expression with JOIN causes outer WHERE to be disregarded sometimes

 

 

 

March 2011

MonetDB Mar2011 release

This is the last planned release for MonetDB4 and MonetDB/XQuery. MonetDB4 has served us well, but we need to move on. MonetDB/XQuery, being implemented on top of MonetDB4, is a casualty of this decision. The work on the Pathfinder compiler, which forms the basis of MonetDB/XQuery, continues at The University of Tübingen [221].

As of this "Mar2011" release (version 11.1.1), all components of MonetDB have been gathered into one single package, mainly to simplify the compilation process. If you're using the Mercurial clone [265], use the Mar2011 branch and the Mar2011_1 tag.

Contents

Mar 2011 feature release


Packaging

The packaging of MonetDB has been completely overhauled. In the past, there were various subpackages that had to be configured and built individually, but that had to be done in the correct order. From this release onward, the process of building MonetDB has been greatly simplified. There is only a single configuration/build/install cycle for the whole suite. On Linux/Unix systems this means one call to configure (when building from the Mercurial sources this has to be preceded by one call to the bootstrap script), one call to make, and one call to make install. On Windows, the bootstrap and configure stages can be skipped (and you need to use nmake instead of make).

The directory structure of the source tree was also completely overhauled. The MonetDB4 and MonetDB5 directories were renamed monetdb4 and monetdb5 respectively, the MonetDB directory was renamed gdk, src subdirectories were removed, and several subsystems were moved to the new common directory.

The version numbers have changed since we now have a single package. The version number of this release is 11.1.1.

Linux

We now package for Fedora 13 and 14, for Debian 5.0 (Lenny) and 6.0 (Squeeze), and for Ubuntu 9.10 (Karmic Koala), 10.04 (Lucid Lynx), and 10.10 (Maverick Meerkat).

On Debian and Ubuntu, updating to this release is a bit more involved than simply using apt-get upgrade or the "Mark All Upgrades" button in the Synaptic Package Manager. When using these commands, they will "keep back" the monetdb packages, so in order to actually update them you will need to give the apt-get install command and specify the relevant monetdb packages, or in Synaptic first select the relevant monetdb packages and then select Package->Mark for Upgrade.

Windows

Binaries are available.

Mac OS X

Binaries are available.

Solaris

Binaries are available.


MonetDB Common

  • Fixed memory detection on Darwin (Mac OS X) systems not to return bogus values
  • MonetDB/src/gdk was moved to gdk


MonetDB Server 4

MonetDB Server 5

  • sphinx module: update, adding limit/max_results support
  • Improve the performance of remote.put for BAT arguments. The put speed is now roughly equal to the speed of get on a BAT.


SQL

  • Fix bug #2648 [278], do not allow restarting a sequence with NULL via the result of a sub-query.
  • Allow clients to release prepared handles using Xrelease commands


XQuery

Client Packages

  • Fixed bug 2677 [279]: SQL_DESC_OCTET_LENGTH should give the size in bytes required to copy the data.
  • Disable timer functionality for non-XQuery languages since it is incorrect, bug #2705 [280]
  • Added mapi_get_uri function to retrieve mapi URI for the connection
  • Allow to dump table data using INSERT INTO statements, rather than COPY INTO + CSV data using the -N/--inserts flag of mclient and msqldump. Bug #2727 [281]
  • Added support for \dn to list schemas or describe a specific one
  • Added support for \df to list functions or describe a specific one
  • Added support for \ds to list sequences or describe a specific one
  • Added support for wildcards * and ? in object names given to \d commands, such that pattern matching is possible, e.g. \d my*
  • Added support for \dS that lists also system tables
  • object names given to \d are now lowercased, unless quoted by either single or double quotes
  • Strip any trailing whitespace with the \d command


Geom Module

  • Set endianness for wkb en/decoding.


Java Module

  • PreparedStatements now free the server-side resources attached to them when closed. This implements bug #2720 [282]


Bug fixes

The following bug tracker items were fixed since the previous "Oct 2010 [283]" release:

  • 104 [284]: SQL: extremely long and complex query causes SEGFAULT
  • 2351 [285]: SQL: transaction aborts on prepared statements in autocommit
  • 2358 [286]: Unrelevant error message when copying bad length field
  • 2586 [287]: times.sql: procedure times() crashes the server
  • 2593 [288]: test sql/src/test/Tests/load_dec_as_int.sql.src fails a file sql/src/test/load_dec_as_int.sql is missing
  • 2609 [289]: insert into large number of columns
  • 2634 [290]: !ERROR: CTrefine: both BATs must have the same cardinality ...
  • 2640 [291]: VOC test fails multi-threaded
  • 2647 [292]: M5 breaks with huge SQL query.
  • 2648 [278]: ALTER SEQUENCE ... RESTART WITH (SELECT NULL) NO CYCLE is accepted
  • 2673 [293]: select a string column from a view over a table with primary key crashes
  • 2677 [279]: SQLColAttribute using SQL_DESC_OCTET_LENGTH doesn't return the maximum length
  • 2679 [294]: timestamp selection based on now() takes too long
  • 2680 [295]: make clean too eager
  • 2698 [296]: test sql/src/test/bugs/Tests/except-union-intersect-bug-sf-1146079.SQL.sh produces wrong result with multi-threaded execution
  • 2704 [297]: temporary tables on readonly mode
  • 2705 [280]: Timer information problems
  • 2708 [298]: function with different value for arguments crashes mserver
  • 2710 [299]: implement \d{t,v,s,S,...} to describe other (or only) objects in the database
  • 2713 [300]: wrong error message when dropping an un-existing view with the same as an existent table.
  • 2715 [301]: Join with 2 AND-ed conditions where one is LIKE/ILIKE returns too many results
  • 2717 [302]: function call in readonly mode.
  • 2720 [282]: Allow a client to close a prepared handle
  • 2725 [303]: mclient/mapilib aborts during connect
  • 2726 [304]: join on two literal tables leads to
  • 2727 [281]: allow dumping database with INSERT INTO statements
  • 2729 [305]: mserver5: ../../../src/server/rel_bin.mx:537: exp_bin: Assertion `0' failed.
  • 2730 [306]: select * from sequence crashes mserver5
  • 2733 [307]: MAL parser accepts 1 + 1 + 1, MAL interpreter only computes 1 + 1
  • 2735 [308]: complex SQL query leads to multiple TypeExceptions
  • 2737 [309]: Sphinx module update, adding limit/max_results support
  • 2738 [310]: MonetDB 'allows' binding and connecting to ports above 65535 (has patch)
  • 2743 [311]: mapi_next_result skips over results (losing output)
  • 2744 [312]: UNLOCKED causes mserver to work indefinitely
  • 2745 [313]: foreign key type mismatch causes MAL stacktrace
  • 2748 [314]: COPY INTO does not work correctly with backslashes in VARCHARs
  • 2750 [315]: Function arguments of type string described not completely in SQL catalog
  • 2752 [316]: CALL function without warning
  • 2753 [317]: check existence of external names for SQL functions/procedures at creation time
  • 2757 [318]: ORDER BY DESC + LIMIT incorrectly yields empty result
  • 2768 [319]: "no such column 'u.name'" in users system view
  • 2773 [320]: starting database 'test'... FAILED
  • 2786 [321]: crash when first connection is with another user than monetdb
  • 2788 [322]: stop -a complains about already stopped databases
  • 2790 [323]: View returns erroneous result for squared average function as compared to query

The detailed list of changes can be found in the source code Mercurial logs.

June 2010

MonetDB Jun2010 release

Release "Jun2010" consists of the following updated components. If you're using the Mercurial clone [265], use the Jun2010 branch and the Jun2010_1 tag.

Name Version Subdirectory Note/Comment
MonetDB Common 1.38.1 MonetDB
MonetDB Clients 1.38.1 clients
MonetDB4 Server 4.38.1 MonetDB4
MonetDB4/XQuery 0.38.1 pathfinder
MonetDB5 Server 5.20.1 MonetDB5
MonetDB5/SQL 2.38.1 sql
MonetDB5/SQL/GIS 0.18.1 geom
MonetDB Java 1.38.1 java
MonetDB Testing 1.38.1 testing (developers only)

NOTE:
Before upgrading to the latest "Jun2010" release from a release of MonetDB older than the Nov2008 release, please make a dump (SQL [324], XQuery [325]) of your database, and rename/move/backup the database directories (dbfarm, sql_logs, xquery_logs). After the upgrade, restore (SQL [324], XQuery [326]) your database, again. Once you have checked and confirmed the correctness of the restore, you can delete the above renamed/moved/backed-up old database directories.

When upgrading from a release of MonetDB older than the May2009 release, you need to move your write-ahead log following the "recommended" instructions for SQL [327] and/or XQuery [328].

Contents

Jun 2010 feature release


Packaging

Linux

We now package for Fedora 11, 12, and 13, for Debian 5.0 (Lenny), and for Ubuntu 9.10 (Karmic Koala) and 10.04 (Lucid Lynx).


MonetDB Common

  • Updated Vendor information.
  • Made compilation of "testing" (and "java") independent of MonetDB. This is mainly for Windows, but also on other systems, "testing" can now be built independently of (and hence before) "MonetDB". Files that mimic configure functionality on Windows were moved from "MonetDB" to "buildtools"; hence, this affects all packages on Windows, requiring a complete rebuild from scratch on Windows. getopt() support in testing has changed; hence, (most probably) requiring a rebuild from scratch of testing on other systems.
  • Implemented build directory support for Windows, i.e., like on Unix/Linux also on Windows we can now build in a separate build directory as alternative to ...<package>NT, and thus keep the latter clean from files generated during the build. On Windows, the build directory must be a sibling of ...<package>NT .


MonetDB Server 4

  • Updated Vendor information.
  • Made compilation of "testing" (and "java") independent of MonetDB. This is mainly for Windows, but also on other systems, "testing" can now be built independently of (and hence before) "MonetDB". Files that mimic configure functionality on Windows were moved from "MonetDB" to "buildtools"; hence, this affects all packages on Windows, requiring a complete rebuild from scratch on Windows. getopt() support in testing has changed; hence, (most probably) requiring a rebuild from scratch of testing on other systems.
  • Implemented build directory support for Windows, i.e., like on Unix/Linux also on Windows we can now build in a separate build directory as alternative to ...<package>NT, and thus keep the latter clean from files generated during the build. On Windows, the build directory must be a sibling of ...<package>NT .


MonetDB Server 5

  • When libxml2 is available, the XML module is automatically loaded.
  • Updated Vendor information.
  • Created a new RPM MonetDB5-server-rdf for the optional MonetDB/RDF module.
  • Added the Z-order module to simplify future manipulation of arrays.
  • fixed BUG #2994521 [329] "mat.slice unable to cope with only empty BAT arguments" by making MATpackSliceInternal() handle empty input BATs correctly
  • Select <col> from <t> limit <n> has been improved by introducing mat.slice().
  • Made compilation of "testing" (and "java") independent of MonetDB. This is mainly for Windows, but also on other systems, "testing" can now be built independently of (and hence before) "MonetDB". Files that mimic configure functionality on Windows were moved from "MonetDB" to "buildtools"; hence, this affects all packages on Windows, requiring a complete rebuild from scratch on Windows. getopt() support in testing has changed; hence, (most probably) requiring a rebuild from scratch of testing on other systems.
  • Implemented build directory support for Windows, i.e., like on Unix/Linux also on Windows we can now build in a separate build directory as alternative to ...<package>NT, and thus keep the latter clean from files generated during the build. On Windows, the build directory must be a sibling of ...<package>NT .
  • The MAL debugger list command has been extended with an optional hash '#', which produces line numbers for each of reference and analysis of variable span.
  • The dataflow scheduler has been revamped to allow for more parallelism to be exploited.
  • The garbage collection administration has been changed. Every variable record now comes with an end-of-life field (eolife), which denotes the instruction after which the BAT variable reference counter can be decremented. The garbage collector is never called automagically on MAL functions, because there may be other pressing needs to retain them. This means that MAL functions defined and used in the context of SQL, and which are not inlined, may cause a leakage. The garbage collection has become part of each interpreter step. The new approach makes the SQL/MAL plans half the size as before.
  • Removed stethoscope from MonetDB5 sources. New location is in the clients repository.
  • Added the compression optimizer as an example of how to gain access to foreign file formats deep down in the kernel and transfer them just in time into a temporary BAT.
  • Renamed configure argument --with-console to --enable-console. Default remains console being enabled.
  • Fixed cleaning the user module context upon session end. Fixes bug #2956664 [330]
  • The config variable mapi_usock can now be used to instruct the server to listen for connections on a local UNIX domain socket on UNIX-like systems.
  • The MAL interpreter has been extended with an operation admission policy to control the memory claims of all concurrent running interpreters. Instructions are hold up unto there is sufficient resource or the query plan can not avoid its execution anymore.
  • The join path optimizer has been extended with searching for join*, semijoin*, and leftjoin* paths. Furthermore, it avoids duplicate work by factoring out all common simple join paths.


SQL

  • make it possible to use '?' in offset and limit
  • Added include and .lib files to Windows installers that are needed to compile client programs.
  • Updated Vendor information.
  • Made compilation of "testing" (and "java") independent of MonetDB. This is mainly for Windows, but also on other systems, "testing" can now be built independently of (and hence before) "MonetDB". Files that mimic configure functionality on Windows were moved from "MonetDB" to "buildtools"; hence, this affects all packages on Windows, requiring a complete rebuild from scratch on Windows. getopt() support in testing has changed; hence, (most probably) requiring a rebuild from scratch of testing on other systems.
  • Implemented build directory support for Windows, i.e., like on Unix/Linux also on Windows we can now build in a separate build directory as alternative to ...<package>NT, and thus keep the latter clean from files generated during the build. On Windows, the build directory must be a sibling of ...<package>NT .


XQuery

  • Updated Vendor information.
  • Made compilation of "testing" (and "java") independent of MonetDB. This is mainly for Windows, but also on other systems, "testing" can now be built independently of (and hence before) "MonetDB". Files that mimic configure functionality on Windows were moved from "MonetDB" to "buildtools"; hence, this affects all packages on Windows, requiring a complete rebuild from scratch on Windows. getopt() support in testing has changed; hence, (most probably) requiring a rebuild from scratch of testing on other systems.
  • Implemented build directory support for Windows, i.e., like on Unix/Linux also on Windows we can now build in a separate build directory as alternative to ...<package>NT, and thus keep the latter clean from files generated during the build. On Windows, the build directory must be a sibling of ...<package>NT .


Client Packages

  • Updated Vendor information.
  • The MonetDB ODBC driver now no longer depends on its own copy of the standard ODBC include files but instead depends on system include files (unixODBC on Linux and Microsoft SDK on Windows).
  • Made compilation of "testing" (and "java") independent of MonetDB. This is mainly for Windows, but also on other systems, "testing" can now be built independently of (and hence before) "MonetDB". Files that mimic configure functionality on Windows were moved from "MonetDB" to "buildtools"; hence, this affects all packages on Windows, requiring a complete rebuild from scratch on Windows. getopt() support in testing has changed; hence, (most probably) requiring a rebuild from scratch of testing on other systems.
  • Implemented build directory support for Windows, i.e., like on Unix/Linux also on Windows we can now build in a separate build directory as alternative to ...<package>NT, and thus keep the latter clean from files generated during the build. On Windows, the build directory must be a sibling of ...<package>NT .
  • Changed d output of mclient no longer to list (internal use only) system tables. Administrator users who like to inspect these tables can use SELECT * FROM tables; instead
  • Implemented the changes required for properly running ODBC on a 64-bit platform.
  • Added stethoscope, an utility to profile MonetDB5 instances, originally from the MonetDB5 repository.


Geom Module

  • Updated Vendor information.
  • Made compilation of "testing" (and "java") independent of MonetDB. This is mainly for Windows, but also on other systems, "testing" can now be built independently of (and hence before) "MonetDB". Files that mimic configure functionality on Windows were moved from "MonetDB" to "buildtools"; hence, this affects all packages on Windows, requiring a complete rebuild from scratch on Windows. getopt() support in testing has changed; hence, (most probably) requiring a rebuild from scratch of testing on other systems.
  • Implemented build directory support for Windows, i.e., like on Unix/Linux also on Windows we can now build in a separate build directory as alternative to ...<package>NT, and thus keep the latter clean from files generated during the build. On Windows, the build directory must be a sibling of ...<package>NT .

February 2010

Release "Feb2010" consists of the following updated components

Name Version CVS module CVS branch Note/Comment
MonetDB Common 1.36.1 MonetDB Feb2010
MonetDB Clients 1.36.1 clients Feb2010
MonetDB4 Server 4.36.1 MonetDB4 Feb2010
MonetDB4/XQuery 0.36.1 pathfinder Feb2010
MonetDB5 Server 5.18.1 MonetDB5 Feb2010
MonetDB5/SQL 2.36.1 sql Feb2010
MonetDB5/SQL/GIS 0.16.1 geom Feb2010
MonetDB Java 1.36.1 java Feb2010
MonetDB Testing 1.36.1 testing Feb2010 (developers only)

NOTE:
Before upgrading to the latest "Feb2010" release from a release of MonetDB older than the Nov2008 release, please make a dump (SQL [331], XQuery [332]) of your database, and rename/move/backup the database directories (dbfarm, sql_logs, xquery_logs). After the upgrade, restore (SQL [331], XQuery [333]) your database, again. Once you have checked and confirmed the correctness of the restore, you can delete the above renamed/moved/backed-up old database directories.

When upgrading from a release of MonetDB older than the May2009 release, you need to move your write-ahead log following the "recommended" instructions for SQL [334] and/or XQuery [335].

Contents

Feb 2010 feature release


Packaging

We now package for Fedora 11 and 12.


MonetDB Common

  • Various concurrency bugs were fixed.
  • Various changes were made to run better on systems that don't have enough memory to keep everything in core that was touched during query processing. This is done by having the higher layers giving hints to the database kernel about future use, and the database kernel giving hings to the operating system kernel about how (virtual) memory is going to be used.
  • Fixed bug in mergejoin implementation. This fixes bug #2952191.
  • Added support for compiling on Windows using the Cygwin-provided version of flex.
  • Fix compilation issue when configured with --with-curl. This fixes bug #2924999.
  • Added implementation of MT_getrss() for Solaris. This yields in the kernel knowing about its (approximate) memory usage to try and help the operating system to free that memory that is best to free, instead of a random page, e.g. the work of the vmtrim thread.
  • Implemented a "fast" string BAT append: Under certain conditions, instead of inserting values one-by-one, we now concatenate the string heap wholesale and just manipulate the offsets. This works both for BATins and BATappend.
  • Changed the string heap implementation to also contain the hashes of strings.
  • Changed the implementation of the string offset columns to be dynamically sized.


MonetDB Server 5

  • The MonetDB 5 code base underwent a series of small/medium changes:
    • Solving name conflicts with external libraries in mal_box.
    • Dependency graph generation in MAL debugger fixed.
    • Code hardening against out of memory errors and based on Coverity checks.
    • Recycler improved to deal with SQL plans from different sessions.
    • Profiler extended to report the argument types, user, and thread id.
    • MAL interpreter reports an event before and after the instruction.
    • Dataflow also allowed for updates on temporary BATs.
    • Reorder optimizer better respects the dataflow.
    • All update instructions return their target to mark the dataflow.
    • Clean up of (bat)calc module.
    • Packing pieces together simplified and sped up.
  • The MAL interpreter has been extended with an operator admission scheme, which is active during parallel execution. It blocks threads if the total amount of memory needed for the operator can not be claimed. Only if there is one operation left to execute, it won't block. The admission level is controlled by a threshold, which is set to 90% of the physical memory.
  • The ilike[u]select operations has been included in the repertoire recognized by the mergetable optimizer for push through of selections.
  • The joinpath optimizer has been extended with recognition of the leftjoin. This way, series of leftjoin operations can be optimized by looking at the smallest starting point.
  • Introduced a dictionary encoding option to the optimizer pipeline. Encoding is initiated with a SQL call compress(tablename).
  • Removed the ability to redirect to other running databases in the same dbfarm. This functionality has been taken over completely by merovingian, and only results in confusement these days, bug #2891191.
  • Added --enable-console configure argument, defaulting to 'yes' for now. Disabling the server console increases security by avoiding local access exploits. This is not the default since our Testing setup cannot deal with a console-less server yet.
  • Report detected amount of main memory and cpu cores in output of `mserver5 --version`.


SQL

Nothing to report.

XQuery

Nothing to report

Client Packages

  • Fixed a bug in the Mapi libary when sending very large queries to the server.
  • Implemented BLOB and CLOB support in ODBC driver.
  • Fixed crash upon connect to a server that is under maintenance but has the requested language (scenario) not loaded.
  • Fixed bug in the ODBC driver where a non-ASCII character at the end of a string caused an error when the string was converted to wide characters.
  • mclient timer information (-t option) is now written to standard error instead of standard output. This makes getting timer information easier when there is a large amout of regular output (which can be redirected to a file or /dev/null).
  • Implemented new function mapi_fetch_field_len() which returns the length (excluding trailing NULL byte) of the field returned by mapi_fetch_field().


Bug fixes

The following bug tracker items were fixed since the previous "Nov 2009 [344]" release: The detailed list of changes can be found in the source code CVS logs.

November 2009

MonetDB Nov2009-SP2 release

Release "Nov2009-SP2" consists of the following updated components

Name Version CVS module CVS branch Note/Comment
MonetDB Common 1.34.4 MonetDB Nov2009
MonetDB Clients 1.34.4 clients Nov2009
MonetDB4 Server 4.34.4 MonetDB4 Nov2009
MonetDB4/XQuery 0.34.4 pathfinder Nov2009
MonetDB5 Server 5.16.4 MonetDB5 Nov2009
MonetDB5/SQL 2.34.4 sql Nov2009
MonetDB5/SQL/GIS 0.14.4 geom Nov2009
MonetDB Java 1.34.4 java Nov2009
MonetDB Testing 1.34.4 testing Nov2009 (developers only)

NOTE:
Before upgrading to the latest "Nov2009-SP2" release from a release of MonetDB older than the Nov2008 release, please make a dump (SQL [331], XQuery [332]) of your database, and rename/move/backup the database directories (dbfarm, sql_logs, xquery_logs). After the upgrade, restore (SQL [331], XQuery [333]) your database, again. Once you have checked and confirmed the correctness of the restore, you can delete the above renamed/moved/backed-up old database directories.

When upgrading from a release of MonetDB older than the May2009 release, you need to move your write-ahead log following the "recommended" instructions for SQL [334] and/or XQuery [335].

Contents


Nov 2009 SP2 bug-fix release

This is the second bug fix release of the Nov2009 version. This version can be installed "on top of" the Nov2009 release. The database format is fully compatible, so there is no need to do a dump and restore. (Although it is *always* a good idea to make backups.)

Some of the bugs that were fixed:

  • No need to require bash for running the *-config scripts. (SF bug #2914563)
  • MonetDB5 development package now contains all development include files. (SF RFE #2929558)
  • Added support for compiling on OpenBSD (contributed, i.e. not tested by us).
  • General code improvements, partly inspired by running Coverity Scan.
  • mclient -D will exit with a non-zero exit code if dumping failed. (SF bug #2925674)
  • The Python interface now uses the same socket options as the C version, giving a boost in performance. (SF bug #2925750)
  • Fixed a bug in the Python interface to deal with fields with embedded newline characters. (SF bug #2917219)
  • Added initial support to the Python client interface for specifying the encoding.
  • Fixed a bug in execute argument parsing in Python interface.
  • Added missing .sql files to the Windows installer.
  • In MonetDB/SQL, fixed bug in not(invalidtable). (SF bug #2927174)

The following bug tracker items were fixed since the Nov2009-SP1 release:

  • [ 2911008 ] Cannot start because lib_replication.dll was not found
  • [ 2914558 ] Modify MonetDB.m4 to support OpenBSD
  • [ 2917219 ] Python client doesn't handle fields with newline characters
  • [ 2925674 ] mclient dump fails on UTF-8 char and gives no error
  • [ 2925750 ] setsockopt in Python client greatly improves performace
  • [ 2927174 ] SQL: not(invalidtable.bla) segfaults


Nov 2009 SP1 bug-fix release

This is a bug fix release of the Nov2009 version. This version can be installed "on top of" the Nov2009 release. The database format is fully compatible, so there is no need to do a dump and restore. (Although it is *always* a good idea to make backups.)

Some of the bugs that were fixed:

  • The Windows installers for MonetDB/SQL in the Nov2009 release were incomplete. The missing files have been added.
  • Fixed performance issue with loading complex schemas in readonly mode in MonetDB/SQL.
  • Fixed performance issue with grouping in MonetDB/SQL when one of the columns is sorted.
  • Fixed problems with LIKE joins in MonetDB/SQL.
  • Load all files in a multi-file COPY INTO query.
  • Fix error reporting of unexpected input in XQuery queries to be Unicode aware.
  • Correctly handle circular dependencies in FD property check. (SF bug #2908615.)
  • Fixes throughout the code base to better cope with unexpected situations (such as memory shortage).

The following bug tracker items were fixed since the Nov2009 release:

  • [ 1681244 ] Mapi ports and patfhinder/pftijah
  • [ 1893448 ] PF: xrpcConvert2 fails on Windows
  • [ 2686008 ] SQL: multi-attribute GROUP BY may ignore sortedness
  • [ 2879008 ] SQL: WHERE A.x LIKE A.y misses results
  • [ 2879011 ] SQL: leftjoin undefined
  • [ 2884050 ] Broken link to VOC tutorial
  • [ 2902320 ] SQL: COPY multiple files
  • [ 2903111 ] Wrong error / no check for new database dir
  • [ 2908615 ] Not clause breaks result


Nov 2009 feature release

Ubuntu

While the Ubuntu version of the Aug2009 release was built for Ubuntu 9.04, the Jaunty Jackalope, the Ubuntu version of the Nov2009 release has now been built for Ubuntu 9.10, the Karmic Koala.


Packaging


Fedora

In addition to Fedora 10 & 11, we now also provide RPMs for Fedora 12.

Python

Packaging of some Python support files which are only used for testing was changed. The MonetDB-python RPM has been removed, the files are now packaged in the MonetDB-testing RPM. Equally, the python-monetdb Debian/Ubuntu package has been removed.

Windows

On Windows, the bzip2 library is now also used. The server (e.g., using SQL's COPY INTO) can read files that are compressed using gzip and bzip2.


MonetDB Common

The Coverity [355] service has been used to uncover potential errors, such as dereference through NULL pointers and leakage. The warnings produced in the first run have been dealt with for >90%.


MonetDB Server 5

 

  • The M5 code underwent code hardening to better deal with rare events that would materialize mostly on 32bit systems. Mostly minor changes, e.g. display of debugging output has been improved to be more informative.


SQL

 

  • There is a new implementation of the SQL COPY INTO statement which uses multiple CPUs/cores when available. This version is a bit stricter when parsing the CSV file than the old version. E.g., when strings are quoted in the CSV file, the string delimiter must be specified in the COPY INTO statement, and to let empty field be treated as NULL values, the COPY INTO statement must specify NULL AS ''.
  • The TRACE statement modifier has been changed to collect the events in BATs first and returning them after the result set. The support library has been adjusted accordingly. See the documentation: SQL runtime features [356].
  • Several experimental optimizer pipelines are researched and being worked upon. They are not (yet) meant as the default pipeline for production applications. See the documentation: SQL runtime features [356].
  • Changing usernames is now possible from SQL, including the monetdb adminstrator user. Feature Request #2830257 [357]. The SQL syntax is: ALTER USER "monetdb" RENAME TO "newname";.
Merovingian

The following features were implemented in the merovingian and monetdb programs:

  • monetdb: Created databases are now always put under maintenance and as such they have to be explicitly released. This allows the DBA to initialise the new database without any client interference.
  • merovingian: Neighbour discovery now no longer announces databases under maintenance, such that they are not shown as being available to peers. This allows to pull a database out of the cluster easily.
  • monetdb: No operations can be performed with monetdb any more without a merovingian instance running.
  • merovingian: Implemented Feature Request #2830754 [358]. monetdb can now control a remote merovingian using the -h -p and -P global options to monetdb. The password used is stored at the merovingian site in the .merovingian_pass file in the controlled dbfarm. An example of how this can be used can be found in the Assets [359] section under the VOC loader script.
  • merovingian: Remote controllable merovingians now announce their controlport to peers, which are marked with an asterisk in the output of monetdb discover.
  • monetdb: Implemented Feature Request #2855846 [360]. The global -q option to monetdb makes it suppress any status messages, causing only error messages to be printed when they occurr.
  • merovingian: The outside URI that Merovingian is available on is now announced to each mserver5 started by merovingian via the merovingian_uri GDK environment variable.
  • monetdb: For each database now the property optpipe can be set which allows per database control of the SQL optimiser pipeline. Valid values can be found in the monetdb5.conf configuration file under the sql_optimizer section.
  • monetdb: Fixed bug where the uptime of databases with an uptime longer than 7 days was reported incorrectly.


XQuery

 

  • Use of value indices on comparisions between a value and an XPath ending in text() or attribute access (e.g., doc(x)//x/y[z/text() = "foo"] and for $x in doc//x/y where $x/@z = "bar" return $x); such queries do not need scanning data, but start at the index and navigate backwards from there.

    NOTE: This required a change in the physical database storage. While the Nov2009 release of MonetDB/XQuery can still read older (pre-Nov2009) databases, it is recommended to re-shred all documents to benefit from the new indices and achieve optimal performance.

  • A session expression cache, where you can run multiple queries in the same session and within that session keep intermediate results present (somewhat like a database cursor). One use is quickly paging in a GUI through a large result, another is acceleration of series of subsequent queries that overlap (e.g. zoom in to a result set). See: Session Expression Cache [361]


Client Packages


mclient

 

  • Result rendering has been changed to try harder to find an optimal rendering of results wider than the terminal allows. Unlike previous releases, mclient now emits results wider than the terminal width if it is impossible or unreasonable to fit the results, thereby aiming to avoid wrapping columns.
  • EXPLAIN, TRACE and PROFILE output are now correctly rendered.
  • Timer support for single statements was implemented, Bug #2872884.
  • Added support for .monetdb file with defaults for mclient. The DOTMONETDBFILE environment file can be used to point to a different location or file name. The file layout of a .monetdb file is a key=value file, where keys include user, password and language.
  • Data fed into mclient using a pipe is no longer read line by line, but per block, like files given as arguments are. This improves performance considerably, but changes the behaviour regarding error messages and understood commands. The internal \x commands of mclient are no longer respected when used with a pipe. To obtain the previous behaviour, add the -i option to mclient when using data fed from a pipe.
  • The default username (the name of the currently logged in user) is no longer used without asking. The -u option now always requires an argument, when this option is absent, mclient will ask for a username, unless a default was set in the .monetdb file. Note: mclient no longer logs in as monetdb/monetdb by default.
  • The start_with_help option was removed. --help now always gives the general help of mclient and exits afterwards. Instead, mclient now gives a welcome message with some diagnostic information and a hint how to get (language specific) help.
  • The \d command now describes all tables by reporting whether it is a TABLE or VIEW. Rendering mode of database object describe mode has been improved.
  • Autocommit status and changes therein are now reported by mclient. Empty responses are no longer reported as "0 tuples", instead proper status messages are returned, giving e.g. information about a generated identity key.
  • Error messages no longer include the leading !.
Python & Ruby

 

  • The native implementations have been matured further.
PHP

 

  • The SWIG-based PHP interface (leveraging the C MAPI library) has been replaced by a native PHP implementation. For the time being, the old SWIG-based interface is still in the code base, but only the new native interface is activated and distributed in the installers.
JDBC

 

  • JDBC received a workaround for a longstanding problem when running the driver using Java 6.
  • A bug was fixed which caused Connection warning messages to disappear when a connection to Merovingian was made.


Bug fixes

The following bug tracker items were fixed since the previous "Aug 2009 SP2 [362]" release:

  • ID: 2903111 [363]: Wrong error / no check for new database dir
  • ID: 2902320 [364]: SQL: COPY multiple files
  • ID: 2898944 [365]: pf takes 'forever' (> 1 hour) to compile particular query
  • ID: 2898378 [366]: M4: tests/by_Peter fail
  • ID: 2897126 [367]: pf fatal error: duplicate column in equi-join
  • ID: 2896012 [368]: Filter using same field in OR and AND throws typeexception
  • ID: 2895791 [369]: ORDER BY with Row_number() return wrong results
  • ID: 2895290 [370]: M5: segfault for tst019 when run with mclient
  • ID: 2894927 [371]: Mx: @include does not write any "#line" directives
  • ID: 2894150 [372]: test fail due to error message format/protocol changes
  • ID: 2893878 [373]: SQL: more tests fail with Nov2009 that worked with Aug2009
  • ID: 2893875 [374]: PF: 82 tests fail on Auf2009_NFI/Nov2009 but work on Aug2009
  • ID: 2893859 [375]: SQL: various VIA-M tests fail
  • ID: 2893846 [376]: SQL: set_history_and_drop_table.SF-2607045 fails
  • ID: 2893823 [377]: SQL: src/backends/monet5/Tests/optimizers.sql fails
  • ID: 2893821 [378]: SQL: src/test//Tests/median_stdev.sql fails
  • ID: 2893815 [379]: PF: xrpcConvert tests segfault since 2009/10/08
  • ID: 2893484 [380]: All CASE branches are evaluated
  • ID: 2893247 [381]: SQL: QOTupdateStatistics segfaults
  • ID: 2891718 [382]: M5/MAL: io.print() yields no output
  • ID: 2891191 [383]: unconscious redirect
  • ID: 2890914 [384]: StartOfHeaderParser.java#getNextAsString() Err: IndexOutOfB
  • ID: 2890702 [385]: M5: pma.mx fails to compile on various platforms
  • ID: 2890035 [386]: SQL: optimizer confused by concurrency
  • ID: 2887282 [387]: union bug
  • ID: 2879011 [388]: SQL: leftjoin undefined
  • ID: 2879008 [389]: SQL: WHERE A.x LIKE A.y misses results
  • ID: 2879005 [390]: SQL: duplicate column name
  • ID: 2878994 [391]: SQL: WHERE A.x LIKE B.x does not work
  • ID: 2877044 [392]: PF: binding sequence in a for clause has static type `empty'
  • ID: 2876948 [393]: PF: error "We do not support the value NaN"
  • ID: 2874045 [394]: SQL: segv for unknown identifier in subquery
  • ID: 2873564 [395]: Cannot create unique constraint on char column
  • ID: 2872884 [396]: mclient -t -s... does not show "Timer" info
  • ID: 2864179 [397]: function call not handled
  • ID: 2863458 [398]: PF/TIJAH: tokenize in a for-loop
  • ID: 2837050 [399]: SQL: src/test/xquery/Tests/q01.sql fails
  • ID: 2834654 [400]: PHP+XQUERY: multiple results not returned correctly
  • ID: 2817414 [401]: XQ: MonetDB allocates >5GB mem while querying small document
  • ID: 2791361 [402]: SQL: Division by zero error
  • ID: 2791356 [403]: SQL: Floating point exception
  • ID: 2771052 [404]: M5/SQL: parallel bulk-load (copy into) incorrect
  • ID: 2716723 [405]: PF: child-steps + replace() fails to compile
  • ID: 2686008 [406]: SQL: multi-attribute GROUP BY may ignore sortedness
  • ID: 2627137 [407]: PF: tests fail with BATpropcheck error
  • ID: 2211565 [408]: PF: type disjointness incorrect
  • ID: 2015135 [409]: PF: pf:mil() can't deal with nil values in MIL results
  • ID: 1991738 [410]: XQ: compilation or runtime choice
  • ID: 1981735 [411]: PF/alg: some error messages differ from PF/mps
  • ID: 1981735 [411]: PF/alg: some error messages differ from PF/mps
  • ID: 1976341 [412]: XQ: leftovers after deleting document
  • ID: 1964365 [413]: PF/Alg: insert..into inserts in wrong order
  • ID: 1959269 [414]: XQ+JDBC: protocol violation
  • ID: 1911209 [415]: XQ: MonetDB 'hangs' after requesting non-existing doc
  • ID: 1893448 [416]: PF: xrpcConvert2 fails on Windows
  • ID: 1809586 [417]: PF: busy port crashes Mserver
  • ID: 1747068 [418]: XQ: windows de-install should warn and empty bat/ dirs
  • ID: 1739353 [419]: EXPLAIN support from JDBC
  • ID: 1730547 [420]: XQ: Mserver crashes on concatenated query
  • ID: 1726599 [421]: XQuery: collection management broken
  • ID: 1681244 [422]: Mapi ports and patfhinder/pftijah
  • ID: 1637867 [423]: XQ: seqcast int? to numeric unsupported
  • ID: 1607210 [424]: XQ: server-side compilation crash (member benchmark)
  • ID: 1595129 [425]: XQ: property error on 64-bit Darwin - big-endian related?
  • ID: 987304 [426]: SQL: div by zero results in ascii art
  • ID: 921173 [427]: SQL: extremely long and complex query causes SEGFAULT

The detailed list of changes can be found in the source code CVS logs.

  
[428]
© 1994-2009 CWI [429] Contact us [430] Legal [431] SourceForge.net LogoSourceForge [432] Bugs [433] Tracker [434] TestWeb [435] PermaStore [436]

Augustus 2009

MonetDB Aug2009-SP2 release

Release "Aug2009-SP2" consists of the following updated components

Name Version CVS module CVS branch Note/Comment
MonetDB Common 1.32.4 MonetDB Aug2009
MonetDB Clients 1.32.4 clients Aug2009
MonetDB4 Server 4.32.4 MonetDB4 Aug2009
MonetDB4/XQuery 0.32.4 pathfinder Aug2009
MonetDB5 Server 5.14.4 MonetDB5 Aug2009
MonetDB5/SQL 2.32.4 sql Aug2009
MonetDB5/SQL/GIS 0.12.4 geom Aug2009
MonetDB Java 1.32.4 java Aug2009
MonetDB Testing 1.32.4 testing Aug2009 (developers only)

NOTE:
Before upgrading to the latest "Aug2009-SP2" release from a release of MonetDB older than the Nov2008 release, please make a dump (SQL [331], XQuery [332]) of your database, and rename/move/backup the database directories (dbfarm, sql_logs, xquery_logs). After the upgrade, restore (SQL [331], XQuery [333]) your database, again. Once you have checked and confirmed the correctness of the restore, you can delete the above renamed/moved/backed-up old database directories.

When upgrading from a release of MonetDB older than the May2009 release, you need to move your write-ahead log following the "recommended" instructions for SQL [334] and/or XQuery [335].

Contents

Aug 2009 SP2 bug-fix release

This is the second bug fix release of the Aug2009 version. This version can be installed "on top of" the Aug2009 release. The database format is fully compatible, so there is no need to do a dump and restore. (Although it is *always* a good idea to make backups.)

Some of the bugs that were fixed:

  • Changes in the memory mapping code to less zealously allocate space.
  • Changes to the use of the way the operating system is informed about how memory is used, resulting in a better fit between the predictions the operating system makes and the reality of what MonetDB actually does.
  • Fix to exit code.
  • Performance fix releated to anonymous virtual memory.
  • Performance fix for when a single item is joined to a table. (MonetDB4/XQuery)
  • When extracting fails, don't leave a partial output file (Mx, SF bug 2840225).
  • Added support for GCC version 4.4.* by disabling the strict-aiasing optimization in the compiler.
  • Various improvements to the Ruby interface.
  • Fix warning messages to disappear when a connection to merovingian is made.
  • Fix physical plan for merge-adjacent-text-nodes. (MonetDB/XQuery)
  • For some plans with aggregates, lax checking resulted in incorrect rewrites. (MonetDB/XQuery)
  • Fix bug in order lists merging. In case the replaced column is sorted in descending order we need to reverse the order of all replacement order criteria. (MonetDB/XQuery)
  • Fixes so that larger plans can be handled. (MonetDB/XQuery, SF bug 2860037)
  • Fixes so that less stack space is used during processing, and so that running out of stack space is recognized in more places. (MonetDB/SQL)
  • Fixed bug in IN handling. (MonetDB/SQL, SF bug 2869818)
  • Fixed crash when attempting to create a function in read-only mode by disallowing this. (MonetDB/SQL, SF bug 2872421)
  • Added partial support for compiling with bison++. Bison++ is not supported and can't be used out of the box, but only minor tweaks are needed to use it. (SF bug 2872565)
  • Fix warning messages to disappear when a connection to merovingian is made. (JDBC driver)

The following bug tracker items were fixed since the Aug2009-SP1 release:

  • [ 2812963 ] JDBC connection hangs if result is too long
  • [ 2839063 ] M5: property error in src/modules/mal/Tests/chopper03.mal
  • [ 2840225 ] Mx creates output files even on errors
  • [ 2847738 ] SuperBall build fails with PHP 5.3.0
  • [ 2860037 ] PF: fatal error: aborted too deep recursion on linux
  • [ 2860574 ] PF: fatal error: column not found / BATappend access denied
  • [ 2865631 ] SQL: incref assertion
  • [ 2869818 ] SQL: Program contains errors
  • [ 2871426 ] SQL: mclient-lsql-D fails everywhere but on 64-bit Fedora
  • [ 2872421 ] SQL: Crash when trying to create function to read only db
  • [ 2872880 ] mclient shows prompt with "-s" option without "-i&qu ot; option
  • [ 2872884 ] mclient -t -s... does not show "Timer" info
  • [ 2876948 ] PF: error "We do not support the value NaN"
  • [ 2879833 ] SQL: SIMILAR keyword does not exist

Aug 2009 SP1 bug-fix release

This is a bug fix release of the Aug2009 version. This version can be installed "on top of" the Aug2009 release. The database format is fully compatible, so there is no need to do a dump and restore. (Although it is *always* a good idea to make backups.)

Some of the bugs that were fixed:

  • On Windows, we switched to newer versions of the iconv and geos libraries (iconv 1.11.1 and geos 3.1.1, from iconv 1.11 and geos 3.0).
  • On Windows, MonetDB4/XQuery was compiled with a newer version of the Intel compiler (version 11.1.046 from version 10.1.022).
  • Better error handling if BATs are growing too large. (SF bug #2850513)
  • The MAL parser was made more robust against syntax errors.
  • A recursion depth check was added to the MAL interpreter. (SF bug #2852707)
  • The PHP client is not compatible with PHP 5.3.0 or later. We now check for that. (Note that in a future release, the PHP client code will be replaced by a pure PHP implementation where this restriction does not apply.) (SF bug #284773)
  • The SQL dump code (msqldump and mclient --dump) now specify all delimiters and the quote character on the generated COPY INTO query. This is needed to be compatible with the next feature release.
  • The SQL dump code (msqldump and mclient --dump) now also knows about CLOBs (Character Large Objects) and properly quotes the values.
  • The problem that occurred under certain circumstances that the JDBC driver would throw an SQLException when large result sets were retrieved is fixed. (SF bug #2812963)
  • Fixed a bug in MonetDB/XQuery where the comparison between values of types strings and untypedAtomic was not done correctly. (SF bug #2853441)
  • Fixed a bug in MonetDB/XQuery where the dead code elimination was a bit too eager. (SF bug #2852928)
  • Fixed a bug in MonetDB/XQuery where a warning was generated that a pin count was below zero. (SF bug #2863458)
  • Fixed a bug in PF/Tijah where OR queries were treated as AND queries. (SF bug #2836870)
  • Fixed a bug in PF/Tijah where the nexi scanner would split strings on '_'.
  • Fixed overflow checks in MonetDB/SQL. (SF bug #2853458)
  • Fixes to get the embedded SQL example working again.
  • Fixed startup of MonetDB/SQL server so that a client cannot connect before the server is ready. (SF bug #2855021)
  • Fixed a bug in MonetDB/SQL where old and dropped view definitions were sometimes incorrectly reused. (SF bug #2863804)
  • Fixed a bug in merovingian where the mero_port in the config file was not properly recognized. (SF bug #2844202)
  • Fixed a bug in MonetDB/SQL where it was not possible to add month intervals to a timestamp. (SF bug #2854949)
  • Fixed various bugs in the SQL engine. (SF bugs #2847106, #2862146)
  • Use bigint for RESTART value in MonetDB/SQL. (SF bug #2846238)
  • Fixed a bug in MonetDB/SQL with parsing hex values.
  • Fixed a bug in MonetDB/SQL when attempting to make changes to the schema when the database is in read-only mode. (SF bug #2830238)
  • Fixed a bug in MonetDB/SQL where adding a primary key constraint to a column containing a NULL value was allowed. (SF bug #2075241)
  • Fixed a bug in MonetDB/SQL where updating values in one column could affect the values in another column. (SF bug #2864313)

The following bug tracker items were fixed since the Aug2009 release:

  • [ 2808453 ] SQL: Trying a copy into in read only database segfault
  • [ 2830238 ] SQL: Crash when trying to write on read-only DB
  • [ 2831472 ] tijah:queryall crashes Mserver
  • [ 2836870 ] PFtijah: Use of 'or' in phrase search yelds incorrect result
  • [ 2840310 ] SQL: "Cannot access descriptor" on big endian
  • [ 2844202 ] Merovingian: mero_port configuration option ignored
  • [ 2846238 ] alter sequence restart on large integer
  • [ 2847106 ] SQL: crash upon join with advanced between
  • [ 2848525 ] mac10.6 and monetdb installation
  • [ 2850341 ] SQL: bit_and returns invalid values
  • [ 2852928 ] PF: updates broken using algebra front-end?
  • [ 2853441 ] PF: distinct values on two sequences
  • [ 2853458 ] SQL: cast exceeding maximum value, returns null
  • [ 2854949 ] SQL: + interval '1' year; Program contains errors
  • [ 2855021 ] SQL: mserver accepts sql connections before scenario load
  • [ 2855152 ] mac10.6 and monetdb installation
  • [ 2862146 ] Two COUNT(*) with LIMIT doesn't execute
  • [ 2863458 ] PF/TIJAH: tokenize in a for-loop
  • [ 2863804 ] SQL: old views incorrectly reused
  • [ 2864313 ] SQL: Update changes all columns with same type and default

Aug 2009 feature release


Packaging

Fedora

We now provide RPMs for Fedora 10 & 11, but no longer for Fedora 9. However, a first check indicates that the Fedora 10 RPMs also install and work fine on Fedora 9.


MonetDB Common

 

  • A bug-fix in the May2009 release had unnoticedly introduced a potential performance degradation, caused by non-optimal use of memory-mapped files, with workload that exceed physical memory, in particular with bulk-loading ("COPY INTO") large tables in MonetDB/SQL. This performance problem has been fixed.
  • Various problems related to sloppy type usage and type casting had surfaced with testing on big-endian hardware as well as on 64-bit systems when using 32-bit OIDs. These bugs have been fixed.
  • The compilation process has been improved by splitting the "infamous" large (generated) "gdk_scanselect_defs.c" into multiple files.


MonetDB Server 5

 

  • auto-sensing is no longer implicit based on the port number being 0, but can now be enabled and disabled via mapi_autosense in monetdb5.conf. When set to enabled, it will try to find a free port starting from the port given by mapi_port.


SQL

 

  • fix crash with select * from 'whatever' = bla, select * where bla = "whatever", Bug #2825974 [446]
  • Ignore escapes for non-escape sequences, instead of inserting a literal backslash. Bug #2830900 [447]
  • Decimals now are properly detected in PreparedStatements, allowing to use them. Bug #2831994 [448]
  • Fixed hang after invalid statement + COMMIT outside auto-commit mode. Bug #2814622 [449]
  • Fixed NUMERIC type detection. Bug #2818176 [450]
  • No longer allow SELECT in value lists, causing confusing error messages. Bug #2807609 [451]
  • Allow for NULL assignment and NULL as argument of a function. Bug #2812729 [452]
  • Added missing power (^) operator. Bug #2812721 [453]
  • DISTINCT was not properly recognised, causing incorrect count results. Bug #2826015 [454]
  • When encountering ambigious columns, fail instead of crashing. Bug #2807336 [455]
  • GROUP BY on a view with constants no longer yields in a crash. Bug #2837561 [456]
  • PLAN output is fixed to no longer violate the protocol and to be tuples instead of comments.
  • Optimisating pipelines can now be set in monetdb5.conf and changed without compiling SQL.
  • sql_logdir can no longer be set and defaults to a location close to the actual data. This approach was chosen since the logs are useless without the data.
  • The ILIKE operator was added. This operator is a case-insensitive LIKE operator, as also found in PostgreSQL.
Merovingian

The following features were implemented in the merovingian and monetdb programs:

  • merovingian: Implemented a configurable port for the neighbour discovery service. This also allows to entirely disable neighbour discovery by setting mero_discoveryport to 0. Feature request #2800964 [457]
  • merovingian: Implemented full per-database property support. monetdb now can manage the following per-database properties: forward (whether to proxy or redirect), shared (if it will be available in the network and how) and nthreads (number of working threads the mserver process uses). Feature Request #1837591 [458].
  • merovingian: Increased the default mero_exittimeout from 7 to 60 seconds. This makes merovingian by default wait longer for mservers to exit before forcefully terminating them.
  • merovingian: Implemented sharing of databases with tags. Using these tags, extra information is available on a remote merovingian to select a database during connect. For this, a client can now connect to a database using a glob expression, which is matched on the database plus tag. When multiple databases match such expression, a load-balancing technique is applied. See the merovingian man-page for details.
  • monetdb: Implemented dynamic width of monetdb status output. If a database name is wider than 14 chars, the name column is made wider as long as the terminal window allows. If a column cannot be made any wider this way, it is abbreviated as before.
  • monetdb: Upon monetdb destroy, avoid multiple questions by merging them into a single confirmation for the destroy command. This accidently also fixed a bug where a response such as 'yes' would abort the next database removal.
  • monetdb: Databases created by monetdb now can only have names restricted to the pattern [A-Za-z0-9-_]+.
  • monetdb: Allow setting the name of a database (a.k.a. renaming it).
  • monetdb: Allow an expression to be used to match against the list of discovered or local databases as argument in the commands discover and status.
  • monetdb: destroy now allows to remove a database under maintenance. This way an administrator can prevent users connecting to the database and hence starting it on demand, which in turn prevents removal of the database.


XQuery

 

  • xquery_logdir can no longer be set and defaults to a location close to the actual data. This approach was chosen since the logs are useless without the data.


Client Packages

mclient

The rendering of the mclient utility for SQL has been improved to better utilise the available space on the screen. Main improvement is that columns are no longer wrapped if they contain only NULLs, even though enough space on the terminal is availble.

Python

 

  • The python API now supports connections to Merovingian.
  • Transaction support was added.
  • Added compatibility with python3k.
  • Full Unicode/UTF-8 support.
  • autocommit, commit and revert functions were added.
Ruby

 

  • Fixed bugs that prevented the correct working of activerecords' migration; The activerecord connector now supports insertion, update and alter table operations; Type casting is working in activerecord; Added a rubygem and rakefile for activerecord-monetdb-adapter; Added a new usage example for activerecord to the README file; Added an example directory to the cvs tree; The driver now correctly works with Merovingian.
  • Support for ruby 1.9.
  • Both the standalone driver and activerecord now support transactions, nested transactions are simulated via savepoints in activerecord.
  • Type conversion in the standalone driver is now performed after data has been retrieved and can be executed on single fields.
JDBC

 

  • blockmode_blocksize option, and the -Xblksize option from JdbcClient were removed, as they are meaningless and inimplemented in the current JDBC driver.
  • Full redirection support was added to support Merovingian's tag-based connections for localisation and loadbalancing.

May 2009

MonetDB May2009-SP2 release

Release "May2009-SP2" consists of the following updated components

Name Version CVS module CVS branch Note/Comment
MonetDB Common 1.30.4 MonetDB May2009
MonetDB Clients 1.30.4 clients May2009
MonetDB4 Server 4.30.4 MonetDB4 May2009
MonetDB4/XQuery 0.30.4 pathfinder May2009
MonetDB5 Server 5.12.4 MonetDB5 May2009
MonetDB5/SQL 2.30.4 sql May2009
MonetDB5/SQL/GIS 0.10.4 geom May2009
MonetDB Java 1.30.4 java May2009
MonetDB Testing 1.30.4 testing May2009 (developers only)

IMPORTANT NOTE:
Before upgrading to the latest "May2009-SP2" release from a release of MonetDB older than the Nov2008 release, please make a dump (SQL [331], XQuery [332]) of your database, and rename/move/backup the database directories (dbfarm, sql_logs, xquery_logs). After the upgrade, restore (SQL [331], XQuery [333]) your database, again. Once you have checked and confirmed the correctness of the restore, you can delete the above renamed/moved/backed-up old database directories.

Contents

May 2009 SP2 bug-fix release

This is the second bug fix release of the May2009 version. This version can be installed "on top of" the May2009 release. The database format is fully compatible, so there is no need to do a dump and restore. (Although it is *always* a good idea to make backups.)

Some of the bugs that were fixed:

  • Fixed a problem where in certain condition you could get an error on Windows saying "GDKremovedir: rmdir(bat\DELETE_ME) failed".
  • Various improvements to the Python and Ruby clients.
  • Fix mserver5 crash when started on demand by merovingian (SF bug #2805340).
  • Check for write errors during dump and abort when they occur.
  • Some innocuous looking XQuery queries could produce large incorrect results and take a relatively long time to do so. SF bug #2815820.
  • Fixed some potential buffer overflow problems in the Mapi library.
  • Improved help message from mclient (SF bug 2818536).
  • Fixed certain SQL LIKE queries on some O/S.
  • Fixed SQL dump in mclient and msqldump to dump values of some less used types correctly.
  • The Debian/Ubuntu installation now also contains the MonetDB PHP client support files.
  • Fixed several performance issues (among others: SF bug 2821251).
  • The various client libraries now also recognize the wrd type which may be emitted by the SQL server.
  • In SQL, the char and varchar types now use Unicode code points (characters to you and me) as the unit for counting instead of bytes in the UTF-8 representation (SF bug 2822855).
  • Fixed some UI issues with the XQuery administrator console.
  • Fixed a bug in SQL when handling IN and OR lists with equal values.
  • Fixed a stack overflow condition in SQL with certain types of quite complex queries (SD bug 2827784).

The following bug tracker items were fixed since the May2009 SP1 release:

  • [ 1840239 ] BAT capacity checking needed
  • [ 2805340 ] Merovingian (?): mserver5 segfaults if started by a client
  • [ 2815820 ] PF: Endless loop in innocent xquery
  • [ 2818536 ] improve mclient -u / -p help
  • [ 2818931 ] MAL: segv on mortal soul's syntax error
  • [ 2819972 ] function problom ----return and elseif
  • [ 2821251 ] PF: Complex optimization overly complex
  • [ 2822855 ] utf8 real size upon insert

May 2009 SP1 bug-fix release

This is a bug fix release of the May2009 version. This version can be installed "on top of" the May2009 release. The database format is fully compatible, so there is no need to do a dump and restore. (Although it is *always* a good idea to make backups.)

Some of the bugs that were fixed:

  • mclient and msqldump would cache all results. This is fixed.
  • The Fedora RPM for the Ruby clients package is now included.
  • Some files were missing in the Python client directory. They are now included in the client source tar ball.
  • mclient now shows the results of all subqueries in a query batch that was passed on the command line using the -s option if there was an error in one of them.
  • mclient \d and \D interactive commands now obey the \| command.
  • Fixed naming of the document and collection when both the --input and --collection arguments are given to mclient (XQuery).
  • Various fixes to the low level memory mapped file support.
  • Fixes to the geom module.

The following bug tracker items were fixed since the May2009 release:

  • [ 2603876 ] XQ: fn:put() tests segfault with ALG back-end
  • [ 2687549 ] SQL: test type_dump_test.SF-989257.SQL.sh fails since Feb 28
  • [ 2737889 ] rand() is not executed for every row when using alias
  • [ 2782202 ] SQL: dump&restore fully usable only by admin
  • [ 2791356 ] Floating point exception
  • [ 2797767 ] mclient --dump caches dump
  • [ 2803477 ] alter table drop contraint error
  • [ 2804823 ] The Mapi Client Utility
  • [ 2805513 ] fn:trace() produces output _and_ error
  • [ 2807038 ] tab + ) + mal function = seg fault

May 2009 feature release


Packaging

Windows

The Windows binaries for SQL and ODBC were built using Microsoft Visual Studio 2008, unlike previous releases which were built using Microsoft Visual Studio 2005. The main implication of this change is that the ODBC driver should now also install on Windows Vista.

The Perl [466], PHP [467], and Python [468] clients were removed from the Windows installers. The clients are still available in the source distribution (although the Python client has changed significantly).

Fedora

There is no longer an RPM for MonetDB-client-ruby. In order to update to this release, you will first have to remove MonetDB-client-ruby after which you can update to this version. (A new version of the RPM will probably appear in a future release.)

Ubuntu

The Ubuntu version was built for Ubuntu 9.04, the Jaunty Jackalope. The Feb2009 release was built for Ubuntu 8.10, the Intrepid Ibex.


MonetDB Common

The way string data is stored in the database has changed for systems with a 64-bit word size and 32-bit OIDs. When the server first opens a database that was created using the old format, the database is updated automatically. This change affect only the 64-bit Windows build. The other 64-bit builds that are provided by us use 64-bit OIDs and are unaffected by the change.


MonetDB Server 5

Parallel loading has been disabled since there was an inherent problem with the implementation. This also affects SQL.


SQL

Write-ahead log

The default location of the write-ahead log has changed. At startup, when the server sees there is an SQL catalog but no write-ahead log, the server will exit immediately with an error message. You can do one of two things (not counting deleting your whole database):

  • move the log files to the new location (recommended): create the directory that is mentioned by the error message and move the log files (two files, log and log.n where n is some integer) there;
  • start the server with the --set sql_logdir=... option (where you replace ... with the appropriate path) to point the server to the old location.

In either case you need to know the old location of the write-ahead logs. This old location was $prefix/var/MonetDB5/sql_logs/<dbname>/sql/ where $prefix is the prefix under which MonetDB was installed and <dbname> the database name. The error message mentioned above gives the expected location of the write-ahead logs.

Note that when you use the Windows installer and you start the server using the Start menu, the write-ahead log is moved to the new location automatically.

Authentication

The authentication mechanism has changed somewhat so that it is now possible to dump and restore password information. Note that the passwords themselves cannot be retrieved: the dump/restore mechanism uses a hash of the password.

Merovingian

The following features were implemented in the merovingian and monetdb programs:

  • The logging of merovingian has changed. TME records are no longer issued, instead each line is prefixed with a timestamp, syslog style. Feature Request #2782776 [469]
  • merovingian now reopens its logfiles when being sent the SIGHUP signal. This allows utilities like e.g. logrotate to rotate the merovingian.log file. Feature Request #2782776 [470]
  • The merovingian default log file location has changed from /var/log/merovingian.log into /var/log/MonetDB/merovingian.log.


XQuery

Like the previous "Feb 2009 SP2 [471]" release, the default Algebra-based Pathfinder compiler ("ALG") still does not support a few XQuery features and extensions. A detailed list and temporary work-arounds by resorting to the deprecated "milprint_summer" ("MPS") version of the Pathfinder compiler can be found here [472].

Write-ahead log

The default location of the write-ahead log has changed. At startup, when the server sees there is an XQuery catalog but no write-ahead log, the server will exit immediately with an error message. You can do one of two things (not counting deleting your whole database):

  • move the log files to the new location (recommended): create the directory that is mentioned by the error message and move the log files (two files, log and log.n where n is some integer) there;
  • start the server with the --set xquery_logdir=... option (where you replace ... with the appropriate path) to point the server to the old location.

In either case you need to know the old location of the write-ahead logs. This old location was $prefix/var/MonetDB4/xquery_logs/<dbname>/xquery/ where $prefix is the prefix under which MonetDB was installed and <dbname> the database name. The error message mentioned above gives the expected location of the write-ahead logs.

Note that when you use the Windows installer and you start the server using the Start menu, the write-ahead log is moved to the new location automatically.


Client Packages

Mapi

Support for SSL-encrypted connections with the server has been discontinued.

The protocol between client and server has changed slightly. This means you cannot use an old client with a new server.

Python

The Python [473] client modules were rewritten and are now in pure Python and so do not depend on any other part of MonetDB. The new module has support for Python 2.5, 2.6 and 3.0, and it is Python DBAPI 2.0 compatible.

Ruby

The Ruby [474] client modules were rewritten and are now in pure Ruby and so do not depend on any other part of MonetDB.

February 2009

MonetDB Feb2009-SP2 release

Release "Feb2009-SP2" consists of the following updated components

Name Version CVS module CVS branch Note/Comment
MonetDB Common 1.28.4 MonetDB Feb2009
MonetDB Clients 1.28.4 clients Feb2009
MonetDB4 Server 4.28.4 MonetDB4 Feb2009
MonetDB4/XQuery 0.28.4 pathfinder Feb2009
MonetDB5 Server 5.10.4 MonetDB5 Feb2009
MonetDB5/SQL 2.28.4 sql Feb2009
MonetDB5/SQL/GIS 0.8.4 geom Feb2009
MonetDB Java 1.28.4 java Feb2009
MonetDB Testing 1.28.4 testing Feb2009 (developers, only)

IMPORTANT NOTE:
Before upgrading to the latest "Feb2009-SP2" release from a release of MonetDB older than the Nov2008 release, please make a dump (SQL [331], XQuery [332]) of your database, and rename/move/backup the database directories (dbfarm, sql_logs, xquery_logs). After the upgrade, restore (SQL [331], XQuery [333]) your database, again. Once you have checked and confirmed the correctness of the restore, you can delete the above renamed/moved/backed-up old database directories.

Contents


Feb 2009 SP2 bug-fix release

One set of fixes that affects all components is that stack overflow checking has been improved. It is now much less likely that the server crashes when queries need too much stack space. Instead, the server will return an error.

MonetDB/XQuery (tijah): fixes to UTF-8 handling; fixes towards being able to index more than 2**31 words on a 64 bit architecture.

MonetDB5: fixed a crash on exit of the server (this crash was not harmful, just annoying).

MonetDB: avoid integer overflow on very large tables.

mclient/msqldump: fixes to the order in which tables, views, and functions are dumped; also dump triggers; fixed the Ubuntu and Debian installers to include mclient and msqldump.

MonetDB/XQuery: fixed a bug where documents were not available when another document in the collection was loaded.

merovingian/monetdb: fixes to the way the server is stopped.

MonetDB/SQL: fixed a bad concurrency bug where updates were lost or data was incorrectly inserted.

The following bug tracker items were fixed since the Feb2009 SP1 release:

  • [ 1811229 ] [ADT] Adding large document, with update support
  • [ 1836980 ] XQ: Error with declared variable in module
  • [ 1894477 ] PF: "fatal error: aborted too deep recursion" on Windows
  • [ 2255453 ] mclient terminates connection unexpectedly
  • [ 2418722 ] pf:docname() problem
  • [ 2549582 ] M4: leftthetajoin estimate way to small
  • [ 2550429 ] nlthetajoin asks for 128 bytes too much memory
  • [ 2567316 ] Segfault after long period of use
  • [ 2636285 ] ancestor-or-self: illegal data in size column or context lis
  • [ 2637051 ] Wrong error message for invalid identifier in group by
  • [ 2664405 ] fn:collection("DOES-NOT-EXIST") does not fail
  • [ 2668437 ] PF runtime: parent step produces not/wrongly sorted result
  • [ 2683759 ] PF:memory leak when fireing multiple simultaneous clients
  • [ 2687705 ] Performance degradation due to memory management
  • [ 2693776 ] xquery fails: memory allocation
  • [ 2698327 ] XQ/GDK: strPut: Assertion failed
  • [ 2704016 ] 'null' from copy into gets wrong
  • [ 2710426 ] PF/TIJAH: dbat_extend fails
  • [ 2711258 ] Mtest.py fails with Python <=2.3.?
  • [ 2716585 ] Java fails to run make dist with latest autotools
  • [ 2722117 ] PF/TIJAH: dbat_finalize fails
  • [ 2722174 ] XQ: shredding 70GB XML fails.
  • [ 2726337 ] M4: Mserver startup errors after changes on 2009/04/01
  • [ 2747661 ] Merovingian: wait for stopping MonetDB
  • [ 2752158 ] M4: test output differs due to reduced error buffer size
  • [ 2756553 ] stethoscope -p parameter doesn't work
  • [ 2757113 ] crash in la_cse
  • [ 2757642 ] Inaccurate AVG of SQRT with 51M records
  • [ 2761480 ] compilation error
  • [ 2776100 ] SQL: "CREATE VIEW AS SELECT 1;" fails
  • [ 2777473 ] PF: incremental shredding broken


Feb 2009 SP1 bug-fix release

The main fix that was done in this release that affects all parts of the code, but most especially MonetDB/SQL is that we have plugged a large number of memory leaks. MonetDB/SQL can now handle larger databases and more demanding queries. This is especially noticeable on 32 bit Windows systems where the memory management system (the system malloc library) is not as sophisticated as the one in Linux.

mclient/msqldump: Fix to the order in which tables and views are dumped: tables must be dumped before views, since views may reference tables and the view creation will only work if the referenced tables already exist.

mclient: Fixed potential deadlock situation when there is a large input file and several queries early on produce large results.

MonetDB/XQuery: Fixed a bug where a replace operation could corrupt the document (bug #2642003).

The following bug tracker items were fixed since the Feb2009 release:

  • [ 2581675 ] mal variable type is persistent over sessions
  • [ 2627138 ] SQL: test fails with BATpropcheck error
  • [ 2642003 ] XQUF replace insert cycle corrupts database
  • [ 2651397 ] M5: manual.summary() produces empty result
  • [ 2654133 ] simple join fails on latest stable
  • [ 2654300 ] problem with views reusing aliases ?
  • [ 2654603 ] assertion failed in store.mx
  • [ 2656762 ] Views with select on top of join - attribute lookup problem
  • [ 2658243 ] SQL: INSERT INTO + ORDER BY
  • [ 2671445 ] Mapi read error when rows exceed maximum
  • [ 2678502 ] Scary: (not)equal evaluated wrong
  • [ 2686045 ] M5 fails to compile with optimization enabled (gcc & icc)
  • [ 2686121 ] XQ: oids in DocmgmTape
  • [ 2687040 ] M4: test src/modules/plain/Tests/bat.py fails
  • [ 2705666 ] Merovingian does not remove pid-file on clean shutdown

The following things were fixed or changed in merovingian and monetdb:

  • merovingian.mx: Remove the pid file upon shutdown, fixes bug #2705666
  • monetdb.mx: Small cosmetic fix not to overflow the uptime in monetdb status display
  • merovingian.mx: Fix hang on Linux when shutting down. select() calls are not interrupted on Linux, so make sure they time out. Fixes bug #2623838
  • merovingian.mx: Fixed crash of merovingian when a to be started database terminates immediately after startup (such as when it crashes)
  • merovingian.mx: Protect shared linked list structures from being accessed and/or modified by two or more threads concurrently. This resolves random crashes. Fixes bug #2605706
  • monetdb.mx: Fixed crash when using monetdb stop on a non-started database. Bug #2605105
  • merovingian.mx: Create logfiles when they do not exist. Bug #2556973

The following was changed in the JDBC interface:

  • Connection.setReadOnly() now always succeeds but sets a warning when called with true. This allows Mondrian to be used with MonetDB/SQL.
  • Holdability invocation methods were implemented, because the JDBC specs state that they need to be implemented and just be warned upon when unavailable (15.1.3.1).
  • Implemented retrieved keys, works only for the first auto-generated column, which is always of int-type.


Feb 2009 feature release


Packaging

We now provide Debian [485] (Lenny, i.e. 5.0) and Ubuntu [486] (Intrepid, i.e. 8.10) packages. These packages have only been very lightly tested (the packaged software is tested extensively, though). For now we provide 32 and 64 bit versions for Debian and only a 64 bit version for Ubuntu. This of course in addition to the Fedora [487] packages we already provided.


MonetDB Server 5

The code base has been tested for memory leaks reducing the change of running out of memory too quickly.

The Merovingian has undergone more thorough stress testing making it much more stable.


SQL

The optimizer of SQL has been overhauled. Rather than working on the binary plans, it performs optimization on the relational algebra tree before it produces the binary plans.


XQuery

Like the previous "Nov 2008 SP2 [488]" release, the default Algebra-based Pathfinder compiler ("ALG") still does not support a few XQuery features and extensions. A detailed list and temporary work-arounds by resorting to the deprecated "milprint_summer" ("MPS") version of the Pathfinder compiler can be found here [472].

Time and date related types and the comparison and extraction functions on top of them have been implemented in MonetDB/XQuery.


Mtest

Mainly for MonetDB developers:

The MonetDB testing tools (Mtest.py & friends) from MonetDB/src/testing/ have been moved into their own "testing" CVS module / MonetDB component.

IMPORTANT NOTE:

In case you plan to upgrade your existing CVS-based "MonetDB" installation, please run `make uninstall` in your "MonetDB" build directory, first, i.e., before running `cvs update` in your "MonetDB" source directory.

Once you update your "MonetDB" to the latest CVS development trunk version, and plan to continue (or start) using Mtest.py, you also need to check out the new "testing" CVS module (from SF) and compile and install it any time after you compiled and installed "MonetDB", and before you plan to use Mtest.py.

The new "testing" CVS module depends only on "MonetDB" (and hence requires that to be installed), but on no other MonetDB component (except from the "buildtools", in case your are compiling from CVS sources). No other MonetDB component depends on the new "testing" CVS module other than that you obviously need to have "testing" installed once you want/need to run tests using Mtest.py.

Details:

The MonetDB testing tools (Mtest.py & friends) are not part of the actual MonetDB functionality, but mainly for MonetDB development purposes (and hence only distributed in the source tarballs, but not in any (official) binary packages).

To make them less dependent on release cycles (and vice versa), as well as to prepare making them usable without MonetDB --- for now, they actually still depend on MonetDB; see below) --- we decided to separate the MonetDB testing tools from the "MonetDB" CVS module into their own "testing" CVS module.

For the time being (i.e., until further notice), "testing" depends on "MonetDB" --- more precisely only on some utils from MonetDB/src/common/, but not on the GDK kernel from MonetDB/src/gdk/, as

  1. Mtest.py (mainly for "convenience") still uses monet_options.py; and
  2. (on Windows only), Mdiff.c uses "our" getopt code.

Python modules "subprocess26.py" & "trace.py" have been moved from <PYTHON_LIBDIR>/MonetDB/ to <PYTHON_LIBDIR>/MonetDBtesting/ .


Bug fixes

A detailed list of the 65 bug reports that have been closed since the previous "Nov 2008 SP2 [488]" release can be found in the MonetDB Bug Tracker at SourceForge: page 1/2 [489], page 2/2 [490].
The detailed list of changes can be found in the source code CVS logs.


Known Issues in this Release

  • Parallel loading is known to be buggy. Actual workaround is to start the server using gdk_nr_threads=1 set in your monetdb5.conf file (or given via --set gdk_nr_threads=1 on your mserver5's command line) during the bulkload using COPY INTO FROM FILE with mclient.
    Details: (no known bug tracker reference)
  • ...

  
[428]
© 1994-2009 CWI [429] Contact us [430] Legal [431] SourceForge.net LogoSourceForge [432] Bugs [433] Tracker [434] TestWeb [435] PermaStore [436]

November 2008

MonetDB Nov2008-SP2 release

Release "Nov2008-SP2" consists of the following updated components

Name Version CVS module CVS branch
MonetDB Common 1.26.2 MonetDB Nov2008
MonetDB Clients 1.26.2 clients Nov2008
MonetDB4 Server 4.26.2 MonetDB4 Nov2008
MonetDB4/XQuery 0.26.2 pathfinder Nov2008
MonetDB5 Server 5.8.2 MonetDB5 Nov2008
MonetDB5/SQL 2.26.2 sql Nov2008
MonetDB5/SQL/GIS 0.6.2 geom Nov2008
MonetDB Java 20081118 java (none; CVS trunk)

IMPORTANT NOTE:
Before upgrading to the latest "Nov2008-SP2" release from a release of MonetDB older than the Nov2008 release, please make a dump (SQL [331], XQuery [332]) of your database, and rename/move/backup the database directories (dbfarm, sql_logs, xquery_logs). After the upgrade, restore (SQL [331], XQuery [333]) your database, again. Once you have checked and confirmed the correctness of the restore, you can delete the above renamed/moved/backed-up old database directories.

Contents

Nov 2008 SP2 bug-fix release

This is the second bug fix release of the Nov2008 version. This version can be installed "on top of" the Nov2008 and Nov2008-SP1 releases. The database format is fully compatible, so there is no need to do a dump and restore. (Although it is *always* a good idea to make backups.)

Some of the bugs that were fixed:

  • MonetDB/XQuery: There was a bug when shredding documents into an existing updatable collection when the existing collection happened to end at a page boundary. (Bug [ 2141012 ] XQ: unclear error).
  • General: Fix to configure to not complain when the raptor library isn't found, unless it was specifically asked for.
  • MonetDB/SQL: significantly speeded up loading of data when lines are extremely long (e.g. when loading BLOBs).
  • XRPC: Since the change of the XRPC protocol (by June 2008), the XRPC server also sends response messages (with empty nodes) for updating function calls. However, the XRPC serever failed to send response messages for calls to the Pathfinder document management functions, because they are handled differently than normal updating functions. This is now fixed.
  • XRPC: The function XRPCHTTPConnection.sendReceive() did not check if HttpURLConnection.getResponseCode() returns -1 or not. If the return value is indeed -1, subsequent use of the stream returned by HttpURLConnection.getInputStream() or HttpURLConnection.getErrorStream() might cause a NullPointerException. This is now fixed. The return value and the stream returned by those two functions are now checked and a proper error message is given.

The following bug tracker items were fixed since the Nov2008-SP1 release:

  • [ 1964658 ] Problem with names of reverse()'d BATs.
  • [ 2141012 ] XQ: unclear error
  • [ 2233581 ] MonetDB does not like '\0' in its input
  • [ 2388542 ] SQL bad blob insert causes crash
  • [ 2405952 ] Merovingian takes too much CPU resources
  • [ 2430201 ] Is distinct broken?
  • [ 2457195 ] Broken query results in OOM, working one has no output
  • [ 2460935 ] Pathfinder fails to compile with icc
  • [ 2460953 ] `make dist` (and hence `make rpm`) fails for pathfinder
  • [ 2466725 ] SubSelect with Outer Reference, result is wrong
  • [ 2468513 ] SQL: copy into does not detect syntax errors
  • [ 2471371 ] SQL: varchar bounds should be obeyed

Nov 2008 SP1 bug-fix release

This is the first bug fix release of the Nov2008 version. This version can be installed "on top of" the Nov2008 release. The database format is fully compatible, so there is no need to do a dump and restore. (Although it is *always* a good idea to make backups.)

Some of the bugs that were fixed:

  • Fixed formatted printing from MAL.
  • mclient now exits with an error code if one of the queries in a non-interactive session failed. [2388047]
  • Fixed a deadlock condition when multiple queries simultaneously update the same part of a document (XQuery).
  • When writing to a file, mclient now flushes the output when the query is done instead of waiting until the file is closed. [2344251]
  • Fixed a race condition in document management (XQuery). [2338542]

The following bug tracker items were fixed since the Nov2008 release:

  • [ 1911209 ] XQ: MonetDB 'hangs' after requesting non-existing doc
  • [ 1981762 ] PF/alg: "Mapping unique name to an original name failed."
  • [ 2049320 ] pf:collection doc count wrong (again?)
  • [ 2075180 ] the unique column can't insert more than 1 row with null
  • [ 2075241 ] When the NULL is in a column,it shouldn't add primary key
  • [ 2103348 ] unique problem with nulls
  • [ 2142522 ] XQ: Duplicates when not using intermediate result
  • [ 2144639 ] XQ: fn:collection() in algebra version
  • [ 2209871 ] most GEOM test fail after error handling changes
  • [ 2224233 ] M4 test ID.914935 fails after warning changes
  • [ 2233677 ] SQL Command kills DB
  • [ 2276427 ] CMDmulti_merged_union: operation failed
  • [ 2277120 ] XQ: following(-sibling)/preceding cause Mserver to crash
  • [ 2305746 ] merovingian crashes when starting a vaultkeyless database
  • [ 2338524 ] Deadlock on barrier
  • [ 2344251 ] Output isn't complete
  • [ 2388047 ] Mclient exit code
  • [ 2388424 ] ERROR: XML Generation: NULL BAT has a 24 head, but tail is N
  • [ 2420768 ] col from another schema inside function body

Nov 2008 feature release

Copyright & License

The copyright for the MonetDB suite was transferred to a newly formed company MonetDB B.V. One of the most important conditions for this transfer was that the software remains open source, and so the license under which this software is being distributed has not changed (apart from the copyright notice).

Windows Installers

The default installation location has been changed from C:\Program Files\CWI to C:\Program Files\MonetDB.

MonetDB Common (GDK)

The kernel administration for BATs has been changed to prepare for future enhancements. All descriptors are now initialized and kept in memory, which reduces the number of files and related IO accessing the descriptors.

Memory management for large database has been improved, with the scavenging thread becoming more aggressive in moving dirty pages to disk when we reach 70% of the physical memory.

Code cleaning and simplification continues to provide ease of compilation on many platforms.

MonetDB Server 5

The MonetDB 5 kernel underwent a code clean up too, e.g. datatypes have been more strictly applied and all kernel errors are propagated to MAL exceptions.

The number of experimental modules have been developed. The 'recycler' provides support for automatic re-use of intermediate results. It improves query throughput for long sequences with partially overlapping queries.

The system supports intra-query parallelism using the multi-core systems capabilities using the 'dataflow' optimizer.

The 'mergetable' optimizer has been extended to perform symbolic reasoning over oid-reasoning. Together with the MAT and dataflow optimizers it provides a basis to improve multi-core performance.

Several bugs were fixed in Merovingian, the MonetDB Database Server daemon, including a file descriptor leak, which caused Merovingian to terminate after a certain amount of connections using the proxy method. Merovingian has been extended with neighbour discovery features, allowing to redirect to discovered neighbours. The monetdb tool has been improved to give better feedback on starting and stopping databases. Most notably it now waits for completion of the start operation, and gives error feedback when starting failed for some reason.

SQL

This release comes with a few more optimizers enable by default. The most important one is the inter-query parallelism, which leads to better use of a multi-core platforms. The effects can be seen in our TPCH experiments [502].

The dataloading code has been improved during our effort to load a 2.6TB astronomical database. Upon request we can provide a detailed report on this experiment.

A query log can be captured with the history.sql script. It provides a basis for post-session analysis of execution times.

Several bugs reported have been solved and functionality in many other areas has been improved.

XQuery

Like the previous "Jun 2008 [503]" release, the default Algebra-based Pathfinder compiler ("ALG") still does not support a few XQuery features and extensions. A detailed list and temporary work-arounds by resorting to the deprecated "milprint_summer" ("MPS") version of the Pathfinder compiler can be found here [472].

Bug fixes

A detailed list of the 116 bug reports that have been closed since the previous "Jun 2008 [503]" release can be found in the MonetDB Bug Tracker at SourceForge: page 1/3 [504], page 2/3 [505], page 3/3 [506].
The detailed list of changes can be found in the source code CVS logs.

Known Issues in this Release

  • Merovingian crashes when starting a vaultkeyless database. Actual workaround is not to try and start databases not created using monetdb create.
    Details: SF bug tracker #2305746 [507]
  • Parallel loading is known to be buggy. Actual workaround is to start the server using gdk_nr_threads=1 set in your monetdb5.conf file (or given via --set gdk_nr_threads=1 on your mserver5's command line) during the bulkload using COPY INTO FROM FILE with mclient.
    Details: (no known bug tracker reference)

June 2008

Release "Jun2008" consists of the following updated components

Name Version CVS module CVS branch
MonetDB Common 1.24 MonetDB MonetDB_1-24
MonetDB Clients 1.24 clients Clients_1-24
MonetDB4 Server 4.24 MonetDB4 MonetDB_4-24
MonetDB4/XQuery 0.24 pathfinder XQuery_0-24
MonetDB5 Server 5.6 MonetDB5 MonetDB_5-6
MonetDB5/SQL 2.24 sql SQL_2-24
MonetDB5/SQL/GIS 0.4 geom Geom_0-4


and the jars for JDBC and XRPC.

IMPORTANT NOTE:
Before upgrading to the latest "Jun 2008 [503]" release from any previous release of MonetDB, please make a dump (SQL [331], XQuery [332]) of your database, and rename/move/backup the database directories (dbfarm, sql_logs, xquery_logs). After the upgrade, restore (SQL [331], XQuery [333]) your database, again. Once you have checked and confirmed the correctness of the restore, you can delete the above renamed/moved/backed-up old database directories.

Contents

 

Compilation from sources

For convenience of both developers and users as well as to comply even more with open source standards, we now set/use the following defaults for the configure options

	--enable-strict, --enable-assert, --enable-debug, --enable-optimize

When compiling from CVS sources (as mainly done by developers):

	strict=yes  assert=yes  debug=yes  optimize=no (*)

When compiling from packaged/distributed sources (i.e., tarballs) (as mainly done by users):

	strict=no   assert=no   debug=no   optimize=no (*)

For building binary distributions (RPMs, MSIs):

	strict=no   assert=no   debug=no   optimize=yes

With nightly testing for the "Stable" release branches:

	strict=yes  assert=no   debug=no   optimize=yes

With nightly testing for the "Current" development trunk:

	strict=yes  assert=yes  debug=no   optimize=yes

(*)

IMPORTANT NOTE:

 

Since

--enable-optimize=yes

is no longer the default for any case except binary packages, it is

strongly recommended

to (re)compile everything from scratch,

explicitly configured

with

	--enable-debug=no --enable-assert=no --enable-optimize=yes

in case you want/need to run any performance experiments with MonetDB!

Please note:
--enable-X=yes is equivalent to --enable-X, and
--enable-X=no is equivalent to --disable-X.

Windows Installers

There are some changes to the Windows installers since previous releases which are described in this section.

Backup and restore

It is recommended to dump the old database and remove it before installing the new version. After installation, the dump can be restored.

In order to dump the SQL database, start the MonetDB SQL Client program and type the commands

\>...\databasedump.sql
\D
\>

The path after

\>

should be an absolute path name (i.e. start with a drive letter) and be in a save location.

In order to dump the XQuery database, start the server and then connect to the administrative console http://127.0.0.1:50001/admin and use that to dump the database.

By default the database is located in %APPDATA%\MonetDB4 for the MonetDB4/XQuery server and %APPDATA%\MonetDB5 for the MonetDB5/SQL server. After having made a database dump, the latter folder can be removed, but the database dump for MonetDB4/XQuery is made inside the database folder, so remove anything except the folder backup and its contents. This folder is located inside the dbfarm\demo folder.

Restoring the SQL database can be done using the MonetDB SQL Client program with the following command

\<...\databasedump.sql

Restoring the XQuery database can be done from the administrative console.

Installation

The 32 bit Windows installers can be installed "over" the old installation. The installer will automatically remove the old version and install the new. The 64 bit Windows installer will not do this (this time), since the code identifying the package was changed. So on 64 bit Windows, first uninstall the previous version and then install the new. The reason for this change is so that the 32 bit version and 64 bit version can both be installed on a 64 bit platform.

The ODBC driver was removed from the MonetDB5-SQL installer. You now need to install the ODBC driver separately. On 64 bit Windows you should install both 32 and 64 bit versions of the ODBC driver.

Also, make sure that after you have uninstalled the old version or upgraded to the new, the files C:\WINDOWS\system32\lib*.dll are all gone. If not, remove them. The new version does not install any files in the system folder. (To be precise, the files are libstream.dll, libmutils.dll, libMonetODBC.dll, libMonetODBCs.dll, and libMapi.dll.)

The JDBC driver was removed from the Windows installers. The driver is now only available separately.

A new MonetDB5-Geom-Module installer is available which can get installed after the MonetDB5-SQL-Installer has been installed. This module provides GIS support to MonetDB/SQL.

MonetDB Common (GDK)

TODO: mmap, property checks, ...

SQL

TODO: ...

Geometry (SQL/GIS) [514]

This release includes (partial) support for Simple Feature Specification of OpenGIS. Follow the GEOM link above for a small tutorial on it.

XQuery

This release marks the beginning of a new era. The default for MonetDB/XQuery has been changed from using the original Pathfinder compiler (nick-named "milprint_summer" or simply "MPS") to using the new improved and enhanced Algebra-based Pathfinder compiler ("ALG").

Compared to the "preview" in the previous "Feb 2008 [515]" release, the functionality of the Algebra-based Pathfinder compiler has been significantly extended to (almost) match that of the original MPS-based compiler. Only the following XQuery features and extensions are not yet supported with the new Algebra-based Pathfinder compiler:

  • fn:put(), fn:collection()
    Please use the old MPS version as described below.
  • recursive functions
    If the recursion depth is known at compile time, adding the option
    	declare option pf:recursion-depth "<number>";

    to the query text will ensure that the recursive function is unfolded <number> times. If the recursion is deeper than <number> invocations a runtime error is triggered.
    For arbitrary recursive functions, please use the old MPS version as described below.

  • Arithmetic Functions [516]
    Please use the old MPS version as described below.
  • Prepared Queries [517]
    When using prepared queries, MonetDB/XQuery will automatically switch back to the old MPS version.
  • XRPC Extension [518]
    In case MonetDB/XQuery's XRPC Server [519] is contacted via XRPC (e.g., by the Administrative GUI [520]), the MonetDB/XQuery server will automatically switch back to the old MPS version to handle the XRPC request.
    To use the execute at XRPC Syntax [521], please use the old MPS version as described below.
  • StandOff Extension [522]
    Pathfinder automatically falls back to the old MPS version, in case StandOff extensions are enabled at runtime via
    Mserver --set standoff=enabled and/or pf --enable-standoff

While support for the old MPS version will not be continued, it remains available for backward compatibility without recompilation, at least for the time being.
For the stand-alone XQuery compiler, the MonetDB/XQuery server, and/or the MonetDB/XQuery client, simply use the following command line options to revert the default to the old MPS version:

  • pf -M
  • Mserver --dbinit='module(pathfinder);' --set xquery_backend=milprint_summer
  • mclient --language=xquery --no-algebra
  • mclient -lxquery -g

In an interactive `

mclient -lxquery

` session, commands "

\g

" / "

\G

" enable / disable the old

MPS

-based compiler in the MonetDB/XQuery server.

Bug fixes

A detailed list of the 139 bug reports that have been closed since the previous "Feb2008" release can be found in the MonetDB Bug Tracker at SourceForge: page 1/3 [523], page 2/3 [524], page 3/3 [525].
The detailed list of changes can be found in the source code CVS logs.

February 2008

Release "Feb2008" consists of the following updated components MonetDB Common 1.22, MonetDB Clients 1.22, MonetDB4 Server 4.22, MonetDB4/XQuery 0.22, MonetDB5 Server 5.4, MonetDB5/SQL 2.22, and the jars for JDBC and XRPC.

IMPORTANT NOTE:
The database format of this release is not compatible with that of any previous release. Before installing this release, make a dump of your database, and do a restore afterwards.

MonetDB Server This release marks the roll forward of the binary storage layout to a single column storage layout. This involved a major overhaul of the MonetDB Server code. Memory leaks have been addressed.

JDBC & XRPCwrapper The Java code (JDBC & XRPCwrapper) has been moved from the clients [526] and pathfinder [527] source trees to a dedicated java [528] source tree in CVS, which can simply be compiled using "ant". Pre-compiled ".jar" are available for Download.

SQL The functional extensions to SQL are limited. The mostly concern minor bugs and some of the optimizers have been improved to better exploit re-occurring joins. The core cracker code has been cleaned up for better inspection and sharing in the community.

XQuery Next to the original Pathfinder compiler (nick-named "milprint_summer" or simply "MPS"), the development of the new improved and enhanced Algebra-based Pathfinder compiler ("ALG") is progressing quickly. While the original MPS version remains the default in this release of MonetDB/XQuery, a preview of the new ALG version is now available without recompilation. For the stand-alone XQuery compiler, the MonetDB/XQuery server, and/or the MonetDB/XQuery client, simply use the following command line options to select the Algebra version as default:

  • pf -A
  • Mserver --dbinit='module(pathfinder);' --set xquery_backend=algebra
  • mclient --language=xquery --algebra
  • mclient -lxquery -G

In an interactive `

mclient -lxquery

` session, commands "

\G

" / "

\g

" enable / disable the

Algebra

-based compiler in the MonetDB/XQuery server.

 

The

Algebra

-based version of Pathfinder will become the default in the next release of MonetDB/XQuery.

Additionally, the following features have been added:

Algebra & MPS:
  • New Feature: (Restricted) optimistic type checking for functions.
    For every function argument whose type is not promotable to the type in the function definition we check whether the type would be promotable if we would apply fn:zero-or-one to the function argument. If that makes the types compatible we introduce an fn:zero-or-one function call for the argument.
    The effect is that a query writer now can quite often ignore adding fn:zero-or-one function calls (just to make the compiler happy :)).
Algebra:
  • XML import/export for logical algebra plans (pf -I / pf -X). There is however no official description of the used dialect.
  • New Feature: (Restricted) support for recursive functions in the algebra.
    If at query compile time the recursion depth is known the query can be extended with the following option:

    declare option pf:recursion-depth "<number>";

    which triggers the compiler to unfold recursive functions <number> times -- thus supporting recursive functions until depth <number>. If the recursion is deeper than <number> invocations a runtime error is triggered.

  • Implementations of the following functions have been added:
    op:to (e.g., "1 to 10"), fn:concat, fn:string-join, fn:substring, fn:string-length, fn:normalize-space, fn:upper-case, fn:lower-case, fn:translate, fn:contains, fn:starts-with, fn:ends-with, fn:substring-before, fn:substring-after, fn:matches, fn:replace, fn:name, fn:local-name, fn:namespace-uri, fn:error, pf:add-doc, pf:del-doc.

Bug fixes

A summary with the highlights is given below. A detailed list of

closed bug reports [529]

can be found in the MonetDB Bug Tracker at SourceForge. The detailed list of changes can be found in the source code CVS change logs.

SQL

  • 1884382 table dependency inside union...
  • 1865953 local temp table
  • 1855673 bug in xml attribute parsing
  • 1850809 create the same temp table after its drop...
  • 1850789 add and then drop the same constraint...
  • 1850781 alter column to set the same constraint
  • 1850187 length function is not working properly
  • 1848541 select sign...
  • 1848511 select insert....
  • 1848500 octet_length...
  • 1848499 difference function crashes
  • 1842550 Better error reports on bulk load (include line num)...
  • 1841759 insert text with repeat...
  • 1841754 select with full join...
  • 1841749 creation of exception for select crash...
  • 1841734 select with coalesce
  • 1841727 pkey in temp table
  • 1838816 SQL: can't use variable name staring with _digit in SQL func
  • 1838813 SQL: select 5 % 3 crashes
  • 1838785 column with date type
  • 1765919 API:Incomplete block read from stream (Mserver still alive?)

XQuery

  • 1883569 XQ: Algebra
  • 1881018 PF: value clash in compiler/include/algebra.h
  • 1877300 XQ: reduce: Assertion `rule' failed
  • 1872806 PF: weird results of fn:id()
  • 1871790 XQ: uncompilable comparison
  • 1871789 XQ: substitution fails
  • 1866944 Failed to compile current pathfinder

MonetDB Server

  • 1856038 Memory leaks when querying
  • 1856034 Memory leaks when bulk loading
  • 1852518 GDK/M4/PF: GDK assertion in PF/XQ tests since
  • 1842019 BULK load failing for NUMERIC columns

Miscellaneous

  • 1845266 Python converters don't quote Unicode strings...

November 2007

Release "2007-11-22" (released on Nov 22 2007) consists of the following updatedparts MonetDB Clients 1.20.2, MonetDB5 Server 5.2.2, MonetDB5/SQL 2.20.2.

This release is a bug fix release, covering the following items:

  1. Clients: The APIs for python, perl and php have been synchronized by the latest state of the Mapi protocol and implementation. JDBC contains a non-blocking IO fix for slow network connections.
  2. A performance fix of the group module. Improved grouping functionality significantly through a better estimate of the intermediate sizes.
  3. The MonetDB/SQL embedded version has been made to work again.
  4. Memory leakage in both the Server and SQL module have been addressed.

Bug fixes

List of fixed bugs

 

  • 1834941 cursor.executemany (python) fails
  • 1834442 Compiling clients (Stable) failed on GNU.32.32.d-Gentoo1.12.
  • 1833800 Python errors not getting passed through MAPI
  • 1831630 Clients: Debian gcj/ant doesn't work.
  • 1828314 make failed on MonetDB5
  • 1827627 --enable-bits doesn't work with icc
  • 1826708 Merovingian: cannot start out of the box, as dbfarm doesn't
  • 1823387 Cascade operations in different sessions...
  • 1823378 Incorrect return type of "with - seeded by - recurse"
  • 1819168 sql/src/test/mapi/python_dbapi test fails
  • 1814768 Python Client problems

Release "2007-10-17" (released on Oct 22 2007) consists of MonetDB Common 1.20, MonetDB Clients 1.20, MonetDB4 Server 4.20, MonetDB4/XQuery 0.20, MonetDB5 Server 5.2, MonetDB5/SQL 2.20.

This release marks a long summer of mostly core software development and maintenance. The restructuring of the source tree was partly successful. Semantic dependencies amongst packages, however, forced us to release more components together than originally planned. Close interaction with some large users provided valuable input for bug fixes and performance improvements.

SQL

We added a way to bulk request sequence numbers. When loading (copying) data into tables which have an auto_increment (or alike) column lots of time was spend in the many calls to get all the sequence numbers (observed a performance in crease of about 10x when loading pieces of 100K rows) TINYINT was added to the standard repertoire. We changed how selections over multiple columns of a single table are handled. Furthermore, several bugs stemming from large-scale deployment have been resolved.

XQuery In the XQuery area many issues have been dealt related to the transaction safe update functionality. Furthermore, a value based indexing scheme has been added to speedup the search.

Clients

The mclient utility has undergone a major overhaul. It moved from a Spartan rendering of SQL results towards a more user friendly tabular rendering on ascii terminals. In passing, the option structure and command options has been stratified and extended with a few use full functions, e.g. piping the tabular result as a CSV to an external program. Use the --help for an overview.

MonetDB 5

Merovingian has been added as the gatekeeper of all SQL database services in a particular farm. It redirect clients to the proper connection. The accompanying Monetdb toolkit provides a textual frontend for assessing the status of a collection of MonetDB servers.

Bug fixes

A summary with the highlights is given below. The detailed list of changes can be found in the source code CVS change logs.

SQL

  • 1806869 SQL: prepare output has unqualified table_name header
  • 1790101 SQL: parser warnings
  • 1786908 SQL+PF: loading both "sql_server" and "pathfinder" fails
  • 1775550 SQL: implied conversion from numeric to varchar disconnect
  • 1774312 SQL: conversion from decimal to varchar error
  • 1774217 SQL: order by timestamps column with limit crash
  • 1757923 SQL: Prepared statement type mismatch
  • 1737777 select id not in another select...
  • 1737668 Alter table to add Fkey....
  • 1734143 Update fkey with a values not in the pkey...
  • 1733712 Update pKey with the same value for different rows...

XQuery

  • 1786397 XQ: SAX serialization not working properly
  • 1771532 MIL-optimizer fails sometimes for big queries
  • 1766414 XQ: third pf-add into collection fails
  • 1763575 XQ: insert attribute gives ERROR in merged_union
  • 1763495 XQ: illegal offsets in swizzle (2)
  • 1760811 XQ: illegal offsets in swizzle
  • 1758902 XQUF replace function seems to corrupt the database
  • 1757327 XQ: attributes in 2nd doc don't show
  • 1755466 XQ: printing special characters in comment nodes
  • 1751684 XQ: merged_union error with fn:sum()
  • 1747017 XQ: admin console images not installed
  • 1743442 XQ: illegal XML syntax in serialized attributes
  • 1743433 XQ: problem with ampersand in attribute value
  • 1766173 Double for / position / sub-sequence bug
  • 1760445 PF: test attribute_access_mposjoin_error.SF-1726928 fails
  • 1760168 PF: wrong use of leftfetchjoin causes segfault in bat_cast
  • 1741459 PF: ws_create inconsistently adds a collection node
  • 1740502 PF: live-variable-limit-handling in mil_opt.c buggy
  • 1737735 128MB RAM needed to compile '1'?

Miscellaneous

  • 1774575 Single page manual damaged
  • 1754452 NullPointerException in MonetConnection
  • 1739356 TRACE,PROFILE,DEBUG should be impossible for JDBC and ODBC
  • 1739353 EXPLAIN support from JDBC
  • 1733744 Cannot connect via ODBC (from Powerbuilder 10)

June 2007

MonetDB 4.18, MonetDB/XQuery 0.18, MonetDB 5.0, MonetDB 5.0/SQL 2.18. The Mars release marks the official release of the MonetDB server 5 line as the basis for future investments in application development. This affects first and foremost the maintenance of SQL code base, where the MIL version is not being maintained actively anymore by the MonetDB team. A synopsis of the major improvements

  1. implementation of delete/update with cascading
  2. PSM procedure call implementation
  3. support for polymorphic functions
  4. autoloading of sql scripts upon server restart

WarningThe SQL catalog structure has been changed, which requires a database dump before you install the new version.WarningThe old Window installers do not always remove MonetDB DLLs. You may have to manually remove them from system32 before (re-)installing the new version. It concerns:

  • libstream.dll
  • libmutils.dll
  • libMonetODBCs.dll
  • libMonetODBC.dll
  • libMapi.dll

MonetDB XQuery XQuery Update Facility has undergone a large series of stress tests to improve correctness, performance and robustness under concurrent updates. In many areas the functionality has been improved:

  1. a completely revised and enhanced type checking code
  2. the administrative console for XQuery
  3. automatic use of value indexing
  4. improved and extended XRpc functionality (javascript library)
  5. all XPath axes looplifted (siblings will come soon)
  6. Intel-based native Windows build (i.e., 64-bit)

This release containsPF/Tijah, [530], a text search extension for XQuery. It provides a flexible environment for setting up search systems by supporting out-of-the-box solutions for common tasks, like index creation and result ranking of XML elements. Work continuous on the algebra version compiler and its SQL backend code generator.Release model The months passed since the bug-fix release of February 2007 showed that the MonetDB product family has grown to a size that more frequent releases are only possible if we continue our decomposition process. The SQL and XQuery release dependencies have now been cut. Source distribution The Super Source Tarball has become a stable means to access and install the packages on Linux platforms. Bug fixes A detailed list of changes can be found in the source code CVS change logs. A summary with the highlights is given below.MonetDB/SQLBug fixes SQL/Core:

  1. 1734143 Update fkey with a values not in the pkey...
  2. 1733712 Update pKey with the same value for different rows...
  3. 1730987 cast of expression hangs
  4. 1725349 large COPY INTO with primary key
  5. 1724151 SQL: mixing auto-commit with explicit transactions
  6. 1723863 SQL Query Crashes Server - CRITICAL
  7. 1723791 rand() is not executed for every row
  8. 1720934 Bind Variable with CAST function Crashes Server
  9. 1720293 WITH queries works only once!
  10. 1718253 SQL: decimal handling errors
  11. 1715679 CREATE TABLE WITH DATA...
  12. 1715672 LOCAL AND GLOBAL temporary table
  13. 1715669 Temporary tables....
  14. 1715659 LIKE for time values...
  15. 1715553 Set default value.....
  16. 1714829 SQL: CHECK CONSTRAINT...
  17. 1714814 SQL: View creation using a function....
  18. 1714809 Infinite loop....
  19. 1711947 src/test/Update_Delete_action: missing stable output files
  20. 1711251 M5: mserver5 segfaults after the second query
  21. 1711242 M5/SQL: wrong result for range predicate
  22. 1710678 CREATE PROCEDURE...
  23. 1710677 DROP PROCEDURE.....
  24. 1709182 SQL: M4 and M5 crash on Solaris/x86 when SQL module is loaded
  25. 1703497 The mserve crash due the null control...
  26. 1701823 copy into from 'filename' not closed
  27. 1695290 SQL changing column default gives error
  28. 1691646 REVOKE authorization crash
  29. 1691344 SQL UNION with ORDER BY crash
  30. 1691098 Mserver/SQL crashes on empty window_specification
  31. 1688915 SQL: crash on nested query
  32. 1684798 LIKE clause
  33. 1670739 SQL: Corrupt database after restart
  34. 1659889 SQL: sequence not restored properly

Bug fixes XQuery:

  1. 1731850 Error in preceding axis
  2. 1730556 XQ/JDBC: 250 results
  3. 1730547 XQ: Mserver crashes on concatenated query
  4. 1726954 XQuery: inserting (large) doc into other doc
  5. 1726928 XQuery: Attribute access causes mposjoin error
  6. 1726599 XQuery: collection management broken
  7. 1726548 XQuery: Administrative Console
  8. 1725352 PFtijah: failing alloc() causes assertion failure (segfault)
  9. 1724118 PF: coreopt optimization eats all available memory and swap
  10. 1723931 XQuery compiler warning treated as error
  11. 1723217 PF: compiler/sql/lalg2sql.brg fails to compile w/o assertion
  12. 1722078 PF: WebSite tests books[123] fail since 2007.05.03
  13. 1721701 PF: "!ERROR: Division by zero" after latest checkins
  14. 1719164 PF: new loop-lifted following step still buggy?
  15. 1718709 PF: select deleted node results in error
  16. 1718622 PF: strange behaviour of "rename"
  17. 1718066 PF: put() to /tmp crashes Mserver
  18. 1716158 PF: compiler/algebra/opt/opt_thetajoin.c icc-compil. fails
  19. 1712578 Mix-up in nested element construction
  20. 1706520 XQuery: boolean every
  21. 1705898 "forgot to cope with type 'double' in cast to integer"
  22. 1682059 XQ: internal integer representation
  23. 1698498 id-function fails to locate some kinds of nodes
  24. 1684416 Pathfinder does not compile due to undefined LLFMT
  25. 1718709 PF: select deleted node results in error
  26. 1718635 PF: node's sibling not accessable after deletion of the node
  27. 1718622 PF: strange behaviour of "rename"
  28. 1718066 PF: put() to /tmp crashes Mserver
  29. 1714603 PF: XRpc tests fail on MacOS (Darwin) since
  30. 1714399 XQuery/XRPC: refresh admin-page causes segmentation fault
  31. 1714360 PF/runtime: find_qn_bulk() adds new qnames twice
  32. 1712595 Error mposjoin: missing matches
  33. 1711754 shredder msync() bug
  34. 1706438 PF: property error when adding a doc. to an existing coll.
  35. 1683730 XQuery: mserver (0.16.2) sometimes crashes on complex query
  36. 1681921 XQ: Character escaping is bogus
  37. 1678593 XQ: document added/deleted with error occured
  38. 1676443 XQ: Interleave of put and doc via mod. func. crashes
  39. 1719164 PF: new loop-lifted following step still buggy?
  40. 1712463 PF: various tests fail since 2007.05.04
  41. 1712461 PF: several tests fail since 2007.05.03
  42. 1712459 PF: several tests fail since 2007.04.28
  43. 1712458 PF: prop. err. in update tests since 2007.04.26 (64-bit OIDs
  44. 1712456 PF: various tests fail since 2007.04.04
  45. 1712455 PF: test fails since 2007.04.03
  46. 1712454 PF: tests/W3C_use_cases/XQ/XMP/Q04alg fails since 2007.04.03
  47. 1712449 PF: "bogus_text_step" test fails since 2007.03.08
  48. 1714133 PFtijah: "/tmp/TIJAH_LOGGER" never cleaned => fills /tmp/ !
  49. 1714129 PFtijah: hard-wired "/tmp/TIJAH_LOGGER" not portable

Bug fixes APIs:

  1. 1733744 Cannot connect via ODBC (from Powerbuilder 10)
  2. 1675795 Float to decimal conversion still broken in v5.0.0_beta1_2
  3. 1671737 JDBC: exception in getChallengeResponse
  4. 1670732 XQ/JDBC: escaping backslash
  5. 1693694 ODBC driver not honoring host, username, or password
  6. 1473288 SQLTables fails if previous statement failed
  7. 1702698 Mapi < problem

Bug fixes Kernel:

  • 1713584 delete on committed dense persistent BAT causes prop. error
  • 1709519 M5: Linux process killed
  • 1706713 Database corruption after out-of-memory condition
  • 1705726 crash in leftfetchjoin due to lack of range checking
  • 1704972 GDK change causes FATAL BATpropcheck with XQuery
  • 1686756 Implicit declaration of function .fdatasync.
  • 1679800 Mserver segfaults during recursive module loading
  • 1673410 STREAM_DEBUG causes compilation error in stream.mx

January 2007

MonetDB 4.16, MonetDB/SQL 2.16, MonetDB/XQuery 0.16, MonetDB 5.0.0_beta1, MonetDB 5.0.0_beta1/SQL 2.16.

A bug fix release was made available on Feb 26, 2007. See the notes at the end of this paper.

The Venus release represents a major shift in functionality. First, we released the long awaited for XQuery Update Facility. As far as we know, it is the first W3C compliant XUF implementation available and provides full transaction control.

The next step was the

big bang

change in the packaging of the server layer. The successor, MonetDB Version 5 is moved to SourceForge. The MonetDB/SQL software line is available with both back-ends, but for Version 4 it is the final release. Finally, redesign over the website has lead to a better navigational structure and much more information on the product lines. A detailed list of changes can be found in the source code CVS change logs. A summary with the highlights is given below.

Source distribution MonetDB Version 5.0.0_beta1 was prepared and shipped to SourceForge.

SQL front-end The SQL functionality has been extended to support SQL triggers and fully functional Persistent Stored Modules. This extension was inspired by our ongoing port of the Skyserver [531] application. Dependencies between SQL objects are stored now and dropping objectsrespect the 'strict' object dependency rule. Loading large tables have been significantly improved and the heuristic query optimizer covers a much larger space.

XQuery This release contains an implementation of XUF (XQuery Update Facility) based on the W3C working draft [532]. In addition, the release adds a number of significant other features to XQuery:

  • "multi-document collections" allow much more efficient management of XML databases that consist of *many* (small) documents, a weakpoint of the previous releases.
  • XRPC: distributed query processing between multiple MonetDB servers; also useful as SOAP API.
  • document management interface: the XML document database can now be controlled fully from XQuery (no MIL needed anymore).
  • improvements in query performance.

Kernel improvements The memory mapped file functionality has been signficantly changed to accomodate efficient processing of XQuery/XUF.

Documentation The revamped website became alive.

Bugfix release Feb 26,2007

New versions of all components of the MonetDB suite are released.

  1. MonetDB/XQuery
    The handling of updates was incomplete. Several memory mapping and concurrency problems have been fixed. Moreover, empty pages that resulted from deletion of nodes are now properly handed back to the system for reuse.
  2. MonetDB/XRPC
    Security and concurrency handling has been improved.
  3. MonetDB/SQL
    Foreign key constraint checking fixed; View in view dependencies bug fixed; Resolved crashes caused by large views; Prepared statements with more than 30 arguments now work.
  4. MonetDB/API
    Haskell and TCL libraries were broken and have been removed; Perl and Python libraries were broken and have been fixed. The current version of the protocol is not compatible with the previous version.
  5. Bugs fixed:
    [ 1579510 ] XQ: update error for "(1,2)/."
    [ 1590523 ] XQ: substring-after
    [ 1636828 ] fork() behaves strangely
    [ 1637867 ] XQ: seqcast int? to numeric unsupported
    [ 1639108 ] mserver crashes on delete query
    [ 1640248 ] PF: ~500 tests randomly fail on Solaris x86
    [ 1644245 ] M5 & SQL5: various tests hang/timeout on Darwin
    [ 1648303 ] database dump does not save/restore auto-increment sequences
    [ 1650469 ] SQL: Armada sinks MonetDB ships
    [ 1651567 ] yet another delete bug, foreign key related
    [ 1651599 ] caching of inserts causes extreme memory consumption
    [ 1655818 ] SQL:Error inserting float in numeric column using JDBC
    [ 1655857 ] XQ: call to builtin function crashes XRPC client
    [ 1659889 ] SQL: sequence not restored properly
    [ 1644610 ] Conflict between normal query and XRPC query
    (and many more that hadn't been reported)

November 2006

MonetDB 4.14, MonetDB/SQL 2.14, MonetDB/XQuery 0.14, MonetDB 5.0_beta1

This release was created in November 2006, but abandoned a few weeks later as the code base was too unstable to warrant a public release.

June 2006

MonetDB 4.12, MonetDB/SQL 2.12, MonetDB/XQuery 0.12, and MonetDB/Five Alpha Release 1 were released on June 23, 2006.

This MonetDB release marks the end of a period where much attention was given to extend the XQuery/Xupdate functionality and efficiency. Although not finished, the many code improvements are valuable to the extended client base. Furthermore, this release marks a feature freeze for XUpdate functionality, claiming enough time to asses its stability and performance before an official release is announced. You may encounter some loose ends and rough edges. A detailed list of changes can be found in the change log of the CVS repository. A summary with the highlights is given below.

Source distribution The MonetDB Version 5 Alpha Release was prepared. It shares the SQL front-end with version 4.

SQL front-end Preliminary support for triggers was added. Now local and global temporary tables are supported. They caused a large rewrite of the session/transaction management. Next to these improvements, many small and large bugs have been fixed. Most notably are fixes to key constraint handling and concurrent transactions.

XQuery All open bugs of XQuery have been addressed, significantly improving its functionality. Serialization, a critical component in any XQuery implementation, has been revamped to obtain a 5-10x speedup in this area. Special attention has been given to parent/ancestor steps. Now they work many times faster inside for-loops (have been loop-lifted) Furthermore, we added XPath extensions for XML standoff annotations (--set standoff=enabled as runtime option to Mserver)

The development of XUpdate was set back by the necessary changes to the underlying MonetDB engine to support BATs with holes in the void sequences. More testing is needed to remove the rough edges.

APIs
The PHP library on Linux has been revamped. It is now based on the PostgreSQL PHP library code.

Kernel improvements The BAT recycle cache has been completed. This greatly improve the performance for applications that create/destroy an abundance of small BATs.

The transaction semantics of the kernel has been extended to support a sub-commit of BATs. This prepares the road to combine SQL and XQuery in one Mserver in the future.

Windows port Cross- and native- compilation for 64-bit Windows platforms versions has somewhat be improved, but is not complete yet. The distribution section will shortly contain a 64-bit version for download.

Documentation improvements Some minor changes, but still an area with a potential great return-on-investment.

January 2006

MonetDB 4.10, MonetDB/SQL 2.10 and MonetDB/XQuery 0.10 were released on Jan 24, 2006. This MonetDB release marks the end of a period where much attention was given to support simple SQL and XQuery applications efficiently. These improvements came from careful analysis of student programs and participating in the c't Magazine DVDstore benchmark.
A detailed list of changes can be found in the change log of the CVS repository. A summary with the highlights is given below.

Source distribution The MonetDB software base has been split to ease source distribution. The tools to prepare the source compilation (Mx), source code generators (MEL, Burg) and makefile generators (autogen) are assembled in the buildtools package. They are not needed anymore to recompile the source code distributed. It involved major changes to the configuarion scripts. A complete clean rebuild is advised.

SQL front end The SQL catalog management has been completely overhauled. It provides much better performance for lightweight queries and provides a cleaner interface for transaction management. All changes to a base table are assembled in insertion/deletion BATs, which can be merged at transaction commit with the base tables.

Sequences support has been added along the specification of SQL 2003. This allows for identity columns, serials (PostgreSQL) or auto_increments (MySQL) in MonetDB. Several minor changes to the build-in operators and syntax recognition, e.g. uchr has been dropped and temporal support has been improved.

SQL server's default transaction mode is now AUTO COMMIT. Support for SET AUTO_COMMIT=true/false has been removed and has been replaced with the lower level Xauto_commit statement issued from the communication libraries. AUTO COMMIT can be switched on and off using the SQL statements START TRANSACTION and COMMIT or ROLLBACK.

There were many small changes to the SQL back-end, such as private name space for cached queries and a large cleanup of the SQL result set code. The protocol for sending results set has been optimized to reduce the communication overhead.

Most BAT related code has been moved to their own directory to prepare for future different storage back-ends. This directory also contains the transaction and logger code. The transaction management has been inproved to allow for reusing the structures which store the transactions catalog information.

XQuery MonetDB/XQuery performance has been boosted using an index node and 'prepared' query/result caching. It now also supports XQuery module definitions. The performance of small XQueries has noticeably improved.

APIs
The experiments with the c't Magazine benchmark led to a re-design of parts of the MAPI protocol, aiming for less overhead in header analysis and falling back to a single exchange mode (blocked) for JDBC. It proofed to increase performance with at least a factor two.

JDBC has been largely revamped for additional speed improvements. Support for retrieval of more metadata has been added as well. The stage has been prepared to also support server-side redirections for future use.

The Perl DBI library has greatly be improved. It uses the MAPI functionality directly now. The module has been registered at the official Perl archive (CPAN)

ODBC has been upgraded to align with the protocol changes and also supports native (server side) prepared statements.

PHP, Python: No changes.

MIL The MIL internal representation (YYTREE) has been overhauled to improve execution of small XQuery operations. Minor language changes, e.g. the module statement is limited to a single argument.

Kernel improvements The BAT buffer pool re-uses the file names of BAT locations and the default temporary names. This improves handling of small BAT creations. The code has been prepared for re-use-garbage in the near future.

A constant module has been added. It supports BAT operations with a column designated by a scalar value.

Windows port A new windows installer preparation scheme under Linux has been deployed.

Documentation improvements Some minor changes, but still an area with a potential great return-on-investment.

May 2005

MonetDB 4.8 & MonetDB/SQL 2.8 were released on May 29, 2005. They were accompanied by the first MonetDB/XQuery 0.8 release (the Mercury release). A detailed list of changes can be found in the change log of the CVS repository. A summary with the highlights is given below.

SQL front end Added SQL statement modifiers EXPLAIN, PROFILE and PREPARE/EXEC. Explain returns the query execution plan as a MIL program and the query is not executed. PROFILE executes the query and stores profiling information in the 'history' table. The prepare and execute work together and can be used by the client programs to implement PREPARE and EXECUTE functionality. The SQL module functionality has been included . They can be used to introduce a module written in C or a plain SQL module. Modules can add new types, functions and tables to the system. [TODO] See for example the 'url' example module.

JDBC driver

  • JdbcClient
    • support for reading (gzipped) files from http web locations
    • better dumping support for time zones on temporal types
    • improved warning feedback system
  • Connection
    • client time zone support for support of INSERT INTO queries which deal with time zones
    • update to intermediate Mapi authorisation scheme with support for multiple catalogs if the server supports it
    • sliding window fetch size which improves speed on large result sets
    • various small changes to improve overall processing speed (not necessarily simplifying the code)
  • DatabaseMetaData
    • updated actual limits of the database measured in maximum column lengths
  • Driver
    • added support for multiple catalogs
  • PreparedStatement
    • fixed bug with Float processing (returned null)
  • ResultSet
    • various small fixes to improve performance
    • preliminary support for retrieving Clobs
    • fixed bug with Float processing
    • fixed time zone behaviour (also made it more efficient)
  • MonetSocketBlockMode
    • made small optimalisation in internal buffer management
  • Statement
    • if fetchSize is not set, a sliding window fetch size is now being used
  • Clob
    • preliminary implementation added

Application interfaces The PHP interface has been finalized. The Perl DBI library interface has reached a mature level.

Kernel improvements The kernel underwent some renovation to improve its usefulness in future releases. The BAT storage structures have been trimmed to consume just 1/3th of the previous space, which can be used to reduce the overhead of short queries by caching all descriptors in main memory. The BAT structure also underwent a revision to accomodate a more flexible property maintenance list. The changes have be shown to improve processing of 'compiled' small MIL queries to run up to 30\% faster.

Windows port The windows port can now be build both using Microsofts Compiler suites and using MinGW.

Documentation improvements The documentation for PHP5 installation and interaction has been added.

February 2005

MonetDB 4.6.0 & MonetDB/SQL 2.6.0 was released on February 14th, 2005, (the Valentijn release). Since the previous release, many improvements have been made. A detailed list of changes can be found in the change log of the CVS repository. A summary with the highlights is given below.

SQL front end MonetDB/SQL now survives the famous 'crashme [533]' program used by MySQL to analyze the features of a DBMS implementation.

Several improvements with regard to the SQL integrity and compliance with the standard were made, as well as new supporting features:

  • The LIKE operator has been overhauled for better support for search expressions
  • Closer conformance to the SQL standard for types
    • INT(x) is no longer allowed/supported
    • SMALLINT, INT and BIGINT are now supported 'native' types
    • convenience aliases for TINYINT (mapped on SMALLINT) and MEDIUMINT & INTEGER (mapped on INT)
    • CHARACTER VARYING and VARCHAR now need a mandatory length specification, ie. VARCHAR(x)
    • added support for CHARACTER LARGE OBJECT next to CLOB
  • Constraints that are not given a name explicitly now get an explanatory name, ie. mytable_mycolumn_pkey instead of c102773
  • Improved dumping support for MapiClient: now constraints and indexes are also dumped
  • Various fixes and optimizations in JDBC and JdbcClient
    • resolved a bug where the JDBC driver would hang if the server would stop sending results prematurely
    • improved support for types, ie. better and more consistent mapping from server to JDBC types
    • Newly implemented java.sql.ResultSet methods: getCatalogName(), isCaseSensitive(), isSearchable(), isCurrency(), isSigned() and getColumnClassName()
    • JdbcClient now correctly escapes identifier names in database dumps and describe actions
    • added -e of --echo flag to JdbcClient to echo input while reading from a file: results in a clear dialog, instead of answers only
    • small optimizations in the socket code, amongst others: avoidance of unnecessary memory copy, skipping of empty blocks.
    • SQLWarnings are now added to the warning lists of the Connection, Statement, and ResultSet objects for various problems or inconsistencies encountered by the driver, that are not fatal.
  • INTERVAL works with TIME and TIMESTAMP
  • Improvement of SQL parser errors: can lead to exact specification of what is expected in certain cases

Mbedded The code base has been reduced and the configuration parameterized to allow MonetDB to be linked with your application. The smallest system we have been able to run MonetDB was a single-board Linux computer, called the Gumstixs [535].

Application interfaces The MapiClient command line tool has been extended to enable automatic timing. Reading scripts as part of the interaction is now supported.

Steffen Goeldner joined our team and completely revamped the Perl DBI library interface.

JDBC: see above (section "SQL front end").

MIL cleanup The MIL syntax cleanup, started in version 4.4, was finished. The parse now denies all "legacy" (i.e., pre 4.4.0_rc0) syntax alternatives.

Kernel improvements MonetDB was known to crash on for example large cross-product calculations. This has been prevented by better boundary checking code at several places.

Documentation improvements The MonetDB source code has been largely moved to texi as a preparation step to improve documentation generation.

Code base improvements

Source code indentation has been normalized throughout the code base.

September 2004

Version 4.4.2 was released on October 26, 2004. It is primarily a bug-fix release.

SQL front end Several bugs affecting the SQL integrity were fixed:

  • Unique constraints were not checked in auto-commit mode
  • The LIKE operator returned wrong matches on patterns starting with %
  • Deleted rows reappeared on conditional selections
  • Removal of legacy variables in system tables
  • ALTER TABLE in auto-commit mode generated error "Error in result"
  • Leakage of BATs caused crashes after heavy load with updates

ODBC and JDBC JDBC suffered from a bug when non-ASCII characters were used in queries. Due to an incorrect charset handling the driver locked up. Another bug returned the incorrect username when being asked for the current logged in user using JDBC's DatabaseMetaData.getUser().

Synopsis of features available in Version 4.4.0

Version 4.4 was released on September 30, 2004 and the first official release. It was preceded by version 3.14, 3.16 and 3.18 delivered in 2004, which were all focused on reaching the point where researchers and casual users would benefit from the MonetDB solution. The feature list below summarizes the efforts spent.

SQL front end A fairly complete SQL-99 front end has been constructed. Amongst others, it provides schema management, user authorization, view definitions, referential constraint enforcement, time zone support, sub-queries, and query caching to speedup processing.

ODBC and JDBC The predominant application programming languages are supported using the MonetDB Application Programming Interface (MAPI) library. Amongst others, it is used to construct a functional rich ODBC implementation, which allows you to retrieve data from a MonetDB server from inside an Excel spreadsheet. The JDBC code is a class 4 implementation. It has been used to link MonetDB with other database application development environments.

MIL cleanup The MonetDB Interface Language (MIL) was originally designed as an intermediate and debugging language. Its definition has been brought more in line of a real programming language, without loosing its flexibility to write concise scripts.

Kernel improvements Heavy internal use revealed bugs and performance issues. In combination with the lengthly bug list compiled by our commercial partner over the years, we have reached the point that serious crashes have become a rarity.

Documentation improvements Documentation in a research setting often comes last, but we are confident that the information provided is sufficient for both a casual end-user, building an application like the VOC, and the demanding researcher, interested in our research on cache-conscious code.

Testing MonetDB is tested daily on 17 different platforms, configured with widely different hardware profiles, but also a variety of compilers. The MonetDB code base was cleaned up to the point that all compiler warnings were dealt with. Currently, it compiles with the strongest compilation options available. Testing has reached the point that a day with 24 hours is not enough to run and analyze the test suite.

Packaging Distribution of a software package as rich and complex as MonetDB calls for major (human) investments in software engineering tasks. The software heavily relies on open-source packages such as automake, auto-config. For the Windows platform the necessary files are included as well.

Copyright (c) 2008-2013 MonetDB BV All rights reserved
Source URL: http://www.monetdb.org/Documentation/UserGuide

Links:
[1] http://dev.monetdb.org/Assets/VOC/voc_dump.sql.gz
[2] http://dev.monetdb.org/Assets/VOC/voc_dump.sql.bz2
[3] http://www.monetdb.org/Documentation/Tutorial
[4] http://www.monetdb.org/Documentation/Manuals/SQLreference/CopyInto
[5] http://www.monetdb.org/Documentation/msqldump
[6] http://www.monetdb.org/Documentation/monetdbd-man-page
[7] http://www.monetdb.org/Documentation/mserver5-man-page
[8] http://www.monetdb.org/Documentation/monetdb-man-page
[9] http://www.monetdb.org/Documentation/mclient-man-page
[10] http://www.monetdb.org/Documentation/UserGuide/Compression
[11] http://www.monetdb.org/Documentation/Cookbooks/SQLrecipies/storage-model
[12] http://www.monetdb.org/Documentation/SQLreference/Schema
[13] http://www.monetdb.org/Downloads/ReleaseNotes
[14] http://bugs.monetdb.org/3158
[15] http://bugs.monetdb.org/3104
[16] http://bugs.monetdb.org/3155
[17] http://bugs.monetdb.org/3084
[18] http://bugs.monetdb.org/3125
[19] http://bugs.monetdb.org/3161
[20] http://bugs.monetdb.org/3182
[21] http://bugs.monetdb.org/3168
[22] http://bugs.monetdb.org/3192
[23] http://bugs.monetdb.org/2579
[24] http://bugs.monetdb.org/2659
[25] http://bugs.monetdb.org/2805
[26] http://bugs.monetdb.org/2886
[27] http://bugs.monetdb.org/2977
[28] http://bugs.monetdb.org/2978
[29] http://bugs.monetdb.org/2999
[30] http://bugs.monetdb.org/3048
[31] http://bugs.monetdb.org/3124
[32] http://bugs.monetdb.org/3136
[33] http://bugs.monetdb.org/3143
[34] http://bugs.monetdb.org/3163
[35] http://bugs.monetdb.org/3169
[36] http://bugs.monetdb.org/3173
[37] http://bugs.monetdb.org/3181
[38] http://bugs.monetdb.org/3185
[39] http://bugs.monetdb.org/3186
[40] http://bugs.monetdb.org/3188
[41] http://bugs.monetdb.org/3189
[42] http://bugs.monetdb.org/3190
[43] http://bugs.monetdb.org/3191
[44] http://bugs.monetdb.org/3199
[45] http://bugs.monetdb.org/3205
[46] http://bugs.monetdb.org/3176
[47] http://bugs.monetdb.org/3208
[48] http://bugs.monetdb.org/3209
[49] http://bugs.monetdb.org/3212
[50] http://bugs.monetdb.org/3016
[51] http://bugs.monetdb.org/3091
[52] http://bugs.monetdb.org/3099
[53] http://bugs.monetdb.org/3100
[54] http://bugs.monetdb.org/3103
[55] http://bugs.monetdb.org/3105
[56] http://bugs.monetdb.org/3109
[57] http://bugs.monetdb.org/3111
[58] http://bugs.monetdb.org/3112
[59] http://bugs.monetdb.org/3113
[60] http://bugs.monetdb.org/3114
[61] http://bugs.monetdb.org/3116
[62] http://bugs.monetdb.org/3119
[63] http://bugs.monetdb.org/3134
[64] http://bugs.monetdb.org/3135
[65] http://bugs.monetdb.org/3107
[66] http://bugs.monetdb.org/3075
[67] http://bugs.monetdb.org/3090
[68] http://bugs.monetdb.org/3093
[69] http://bugs.monetdb.org/3132
[70] http://bugs.monetdb.org/3139
[71] http://bugs.monetdb.org/3117
[72] http://bugs.monetdb.org/3144
[73] http://bugs.monetdb.org/3145
[74] http://bugs.monetdb.org/3150
[75] http://bugs.monetdb.org/2594
[76] http://bugs.monetdb.org/3085
[77] http://bugs.monetdb.org/3086
[78] http://bugs.monetdb.org/3087
[79] http://bugs.monetdb.org/3089
[80] http://bugs.monetdb.org/3096
[81] http://bugs.monetdb.org/2454
[82] http://bugs.monetdb.org/2916
[83] http://bugs.monetdb.org/2949
[84] http://bugs.monetdb.org/2958
[85] http://bugs.monetdb.org/2960
[86] http://bugs.monetdb.org/2972
[87] http://bugs.monetdb.org/2987
[88] http://bugs.monetdb.org/2992
[89] http://bugs.monetdb.org/3000
[90] http://bugs.monetdb.org/3028
[91] http://bugs.monetdb.org/3046
[92] http://bugs.monetdb.org/3063
[93] http://bugs.monetdb.org/3072
[94] http://bugs.monetdb.org/3073
[95] http://bugs.monetdb.org/3074
[96] http://bugs.monetdb.org/3077
[97] http://bugs.monetdb.org/3079
[98] http://bugs.monetdb.org/2997
[99] http://bugs.monetdb.org/2982
[100] http://bugs.monetdb.org/2983
[101] http://bugs.monetdb.org/2984
[102] http://bugs.monetdb.org/2994
[103] http://bugs.monetdb.org/3023
[104] http://bugs.monetdb.org/3030
[105] http://bugs.monetdb.org/3032
[106] http://bugs.monetdb.org/2915
[107] http://bugs.monetdb.org/2947
[108] http://bugs.monetdb.org/2968
[109] http://bugs.monetdb.org/2974
[110] http://bugs.monetdb.org/2988
[111] http://bugs.monetdb.org/2998
[112] http://bugs.monetdb.org/3002
[113] http://bugs.monetdb.org/3011
[114] http://bugs.monetdb.org/3012
[115] http://bugs.monetdb.org/3029
[116] http://bugs.monetdb.org/3031
[117] http://bugs.monetdb.org/3036
[118] http://bugs.monetdb.org/3038
[119] http://bugs.monetdb.org/3043
[120] http://bugs.monetdb.org/3045
[121] http://bugs.monetdb.org/3052
[122] http://bugs.monetdb.org/2862
[123] http://bugs.monetdb.org/2933
[124] http://bugs.monetdb.org/2946
[125] http://bugs.monetdb.org/2964
[126] http://bugs.monetdb.org/2969
[127] http://bugs.monetdb.org/2973
[128] http://bugs.monetdb.org/2979
[129] http://bugs.monetdb.org/2995
[130] http://bugs.monetdb.org/3004
[131] http://bugs.monetdb.org/3009
[132] http://bugs.monetdb.org/3013
[133] http://bugs.monetdb.org/3014
[134] http://bugs.monetdb.org/3017
[135] http://bugs.monetdb.org/3020
[136] http://bugs.monetdb.org/3034
[137] http://bugs.monetdb.org/3041
[138] http://bugs.monetdb.org/2834
[139] http://bugs.monetdb.org/2877
[140] http://bugs.monetdb.org/2882
[141] http://bugs.monetdb.org/2883
[142] http://bugs.monetdb.org/2952
[143] http://bugs.monetdb.org/2950
[144] http://bugs.monetdb.org/2943
[145] http://bugs.monetdb.org/2944
[146] http://bugs.monetdb.org/2936
[147] http://bugs.monetdb.org/2937
[148] http://bugs.monetdb.org/2938
[149] http://bugs.monetdb.org/2846
[150] http://bugs.monetdb.org/2908
[151] http://bugs.monetdb.org/2901
[152] http://bugs.monetdb.org/2897
[153] http://bugs.monetdb.org/2889
[154] http://bugs.monetdb.org/2884
[155] http://bugs.monetdb.org/2885
[156] http://bugs.monetdb.org/2865
[157] http://bugs.monetdb.org/2888
[158] http://bugs.monetdb.org/2953
[159] http://bugs.monetdb.org/2935
[160] http://bugs.monetdb.org/2740
[161] http://bugs.monetdb.org/2802
[162] http://bugs.monetdb.org/2873
[163] http://bugs.monetdb.org/2891
[164] http://bugs.monetdb.org/2910
[165] http://bugs.monetdb.org/2914
[166] http://bugs.monetdb.org/2934
[167] http://bugs.monetdb.org/2939
[168] http://bugs.monetdb.org/2940
[169] http://bugs.monetdb.org/2959
[170] http://bugs.monetdb.org/2963
[171] http://bugs.monetdb.org/2965
[172] http://bugs.monetdb.org/2975
[173] http://bugs.monetdb.org/2981
[174] http://bugs.monetdb.org/2875
[175] http://bugs.monetdb.org/2912
[176] http://bugs.monetdb.org/2919
[177] http://bugs.monetdb.org/2920
[178] http://bugs.monetdb.org/2921
[179] http://bugs.monetdb.org/2923
[180] http://bugs.monetdb.org/2924
[181] http://bugs.monetdb.org/2928
[182] http://bugs.monetdb.org/2346
[183] http://bugs.monetdb.org/2712
[184] http://bugs.monetdb.org/2774
[185] http://bugs.monetdb.org/2890
[186] http://bugs.monetdb.org/2894
[187] http://bugs.monetdb.org/2898
[188] http://bugs.monetdb.org/2900
[189] http://bugs.monetdb.org/2904
[190] http://bugs.monetdb.org/2906
[191] http://bugs.monetdb.org/2909
[192] http://bugs.monetdb.org/2879
[193] http://bugs.monetdb.org/2887
[194] http://bugs.monetdb.org/2850
[195] http://bugs.monetdb.org/2828
[196] http://bugs.monetdb.org/2814
[197] http://bugs.monetdb.org/2851
[198] http://bugs.monetdb.org/2336
[199] http://bugs.monetdb.org/2701
[200] http://bugs.monetdb.org/2817
[201] http://bugs.monetdb.org/2836
[202] http://bugs.monetdb.org/2838
[203] http://bugs.monetdb.org/2839
[204] http://bugs.monetdb.org/2840
[205] http://bugs.monetdb.org/2841
[206] http://bugs.monetdb.org/2842
[207] http://bugs.monetdb.org/2845
[208] http://bugs.monetdb.org/2847
[209] http://bugs.monetdb.org/2848
[210] http://bugs.monetdb.org/2849
[211] http://bugs.monetdb.org/2852
[212] http://bugs.monetdb.org/2855
[213] http://bugs.monetdb.org/2856
[214] http://bugs.monetdb.org/2857
[215] http://bugs.monetdb.org/2859
[216] http://bugs.monetdb.org/2860
[217] http://bugs.monetdb.org/2866
[218] http://bugs.monetdb.org/2868
[219] http://bugs.monetdb.org/2869
[220] http://www.monetdb.org/OldReleaseNotes/Mar2011
[221] http://www.pathfinder-xquery.org/
[222] http://bugs.monetdb.org/2827
[223] http://bugs.monetdb.org/2820
[224] http://bugs.monetdb.org/2833
[225] http://bugs.monetdb.org/2818
[226] http://bugs.monetdb.org/2844
[227] http://bugs.monetdb.org/2823
[228] http://bugs.monetdb.org/2813
[229] http://bugs.monetdb.org/2058
[230] http://bugs.monetdb.org/2734
[231] http://bugs.monetdb.org/2767
[232] http://bugs.monetdb.org/2797
[233] http://bugs.monetdb.org/2811
[234] http://bugs.monetdb.org/2812
[235] http://bugs.monetdb.org/2819
[236] http://bugs.monetdb.org/2821
[237] http://bugs.monetdb.org/2822
[238] http://bugs.monetdb.org/2825
[239] http://bugs.monetdb.org/2826
[240] http://bugs.monetdb.org/2829
[241] http://bugs.monetdb.org/2830
[242] http://bugs.monetdb.org/2831
[243] http://bugs.monetdb.org/2832
[244] http://bugs.monetdb.org/2835
[245] http://bugs.monetdb.org/2837
[246] http://bugs.monetdb.org/2781
[247] http://bugs.monetdb.org/2784
[248] http://bugs.monetdb.org/2798
[249] http://bugs.monetdb.org/2806
[250] http://bugs.monetdb.org/2807
[251] http://bugs.monetdb.org/2808
[252] http://bugs.monetdb.org/2810
[253] http://bugs.monetdb.org/1956
[254] http://bugs.monetdb.org/2547
[255] http://bugs.monetdb.org/2588
[256] http://bugs.monetdb.org/2590
[257] http://bugs.monetdb.org/2639
[258] http://bugs.monetdb.org/2766
[259] http://bugs.monetdb.org/2779
[260] http://bugs.monetdb.org/2793
[261] http://bugs.monetdb.org/2795
[262] http://bugs.monetdb.org/2796
[263] http://bugs.monetdb.org/2800
[264] http://bugs.monetdb.org/2801
[265] http://dev.monetdb.org/hg/MonetDB/
[266] http://monetdb.cwi.nl/Development/Releases/Mar2011/index.html#Mar2011
[267] http://monetdb.cwi.nl/Development/Releases/Mar2011/index.html#Packaging
[268] http://monetdb.cwi.nl/Development/Releases/Mar2011/index.html#GDK
[269] http://monetdb.cwi.nl/Development/Releases/Mar2011/index.html#M4
[270] http://monetdb.cwi.nl/Development/Releases/Mar2011/index.html#M5
[271] http://monetdb.cwi.nl/Development/Releases/Mar2011/index.html#SQL
[272] http://monetdb.cwi.nl/Development/Releases/Mar2011/index.html#XQuery
[273] http://monetdb.cwi.nl/Development/Releases/Mar2011/index.html#Merovingian
[274] http://monetdb.cwi.nl/Development/Releases/Mar2011/index.html#Clients
[275] http://monetdb.cwi.nl/Development/Releases/Mar2011/index.html#Geom
[276] http://monetdb.cwi.nl/Development/Releases/Mar2011/index.html#Java
[277] http://monetdb.cwi.nl/Development/Releases/Mar2011/index.html#Bugs
[278] http://bugs.monetdb.org/2648
[279] http://bugs.monetdb.org/2677
[280] http://bugs.monetdb.org/2705
[281] http://bugs.monetdb.org/2727
[282] http://bugs.monetdb.org/2720
[283] http://monetdb.cwi.nl/Development/Releases/Oct2010/index.html
[284] http://bugs.monetdb.org/104
[285] http://bugs.monetdb.org/2351
[286] http://bugs.monetdb.org/2358
[287] http://bugs.monetdb.org/2586
[288] http://bugs.monetdb.org/2593
[289] http://bugs.monetdb.org/2609
[290] http://bugs.monetdb.org/2634
[291] http://bugs.monetdb.org/2640
[292] http://bugs.monetdb.org/2647
[293] http://bugs.monetdb.org/2673
[294] http://bugs.monetdb.org/2679
[295] http://bugs.monetdb.org/2680
[296] http://bugs.monetdb.org/2698
[297] http://bugs.monetdb.org/2704
[298] http://bugs.monetdb.org/2708
[299] http://bugs.monetdb.org/2710
[300] http://bugs.monetdb.org/2713
[301] http://bugs.monetdb.org/2715
[302] http://bugs.monetdb.org/2717
[303] http://bugs.monetdb.org/2725
[304] http://bugs.monetdb.org/2726
[305] http://bugs.monetdb.org/2729
[306] http://bugs.monetdb.org/2730
[307] http://bugs.monetdb.org/2733
[308] http://bugs.monetdb.org/2735
[309] http://bugs.monetdb.org/2737
[310] http://bugs.monetdb.org/2738
[311] http://bugs.monetdb.org/2743
[312] http://bugs.monetdb.org/2744
[313] http://bugs.monetdb.org/2745
[314] http://bugs.monetdb.org/2748
[315] http://bugs.monetdb.org/2750
[316] http://bugs.monetdb.org/2752
[317] http://bugs.monetdb.org/2753
[318] http://bugs.monetdb.org/2757
[319] http://bugs.monetdb.org/2768
[320] http://bugs.monetdb.org/2773
[321] http://bugs.monetdb.org/2786
[322] http://bugs.monetdb.org/2788
[323] http://bugs.monetdb.org/2790
[324] http://www.monetdb.org/SQL/Documentation/Database-Dumps.html
[325] http://www.monetdb.org/XQuery/QuickTour/adminGUI/index.html#backup
[326] http://www.monetdb.org/XQuery/QuickTour/adminGUI/index.html#restore
[327] http://www.monetdb.org/Development/Releases/May2009/index.html#SQL
[328] http://www.monetdb.org/Development/Releases/May2009/index.html#XQuery
[329] http://bugs.monetdb/org/2994521
[330] http://bugs.monetdb.org/2956664
[331] http://monetdb.cwi.nl/SQL/Documentation/Database-Dumps.html
[332] http://monetdb.cwi.nl/XQuery/QuickTour/adminGUI/index.html#backup
[333] http://monetdb.cwi.nl/XQuery/QuickTour/adminGUI/index.html#restore
[334] http://monetdb.cwi.nl/Development/Releases/May2009/index.html#SQL
[335] http://monetdb.cwi.nl/Development/Releases/May2009/index.html#XQuery
[336] http://monetdb.cwi.nl/Development/Releases/Feb2010/#Feb2010
[337] http://monetdb.cwi.nl/Development/Releases/Feb2010/#Packaging
[338] http://monetdb.cwi.nl/Development/Releases/Feb2010/#GDK
[339] http://monetdb.cwi.nl/Development/Releases/Feb2010/#M5
[340] http://monetdb.cwi.nl/Development/Releases/Feb2010/#SQL
[341] http://monetdb.cwi.nl/Development/Releases/Feb2010/#XQuery
[342] http://monetdb.cwi.nl/Development/Releases/Feb2010/#Clients
[343] http://monetdb.cwi.nl/Development/Releases/Feb2010/#Bugs
[344] http://monetdb.cwi.nl/Development/Releases/Nov2009/index.html
[345] http://monetdb.cwi.nl/Development/Releases/Nov2009/index.html#Nov2009_SP2
[346] http://monetdb.cwi.nl/Development/Releases/Nov2009/index.html#Nov2009_SP1
[347] http://monetdb.cwi.nl/Development/Releases/Nov2009/index.html#Nov2009
[348] http://monetdb.cwi.nl/Development/Releases/Nov2009/index.html#Packaging
[349] http://monetdb.cwi.nl/Development/Releases/Nov2009/index.html#GDK
[350] http://monetdb.cwi.nl/Development/Releases/Nov2009/index.html#M5
[351] http://monetdb.cwi.nl/Development/Releases/Nov2009/index.html#SQL
[352] http://monetdb.cwi.nl/Development/Releases/Nov2009/index.html#XQuery
[353] http://monetdb.cwi.nl/Development/Releases/Nov2009/index.html#Clients
[354] http://monetdb.cwi.nl/Development/Releases/Nov2009/index.html#Bugs
[355] http://scan.coverity.com/index.html
[356] http://monetdb.cwi.nl/SQL/Documentation/SQL-Runtime-Features.html
[357] https://sourceforge.net/tracker/index.php?func=detail&amp;aid=2830257&amp;group_id=56967&amp;atid=482471
[358] http://sourceforge.net/tracker/?func=detail&amp;aid=2830754&amp;group_id=56967&amp;atid=482471
[359] http://monetdb.cwi.nl/Development/Assets
[360] https://sourceforge.net/tracker/?func=detail&amp;aid=2855846&amp;group_id=56967&amp;atid=482471
[361] http://monetdb.cwi.nl/XQuery/Documentation/Session-Expression-Cache.html
[362] http://monetdb.cwi.nl/Development/Releases/Aug2009/index.html
[363] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2903111&amp;group_id=56967
[364] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2902320&amp;group_id=56967
[365] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2898944&amp;group_id=56967
[366] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2898378&amp;group_id=56967
[367] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2897126&amp;group_id=56967
[368] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2896012&amp;group_id=56967
[369] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2895791&amp;group_id=56967
[370] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2895290&amp;group_id=56967
[371] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2894927&amp;group_id=56967
[372] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2894150&amp;group_id=56967
[373] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2893878&amp;group_id=56967
[374] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2893875&amp;group_id=56967
[375] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2893859&amp;group_id=56967
[376] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2893846&amp;group_id=56967
[377] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2893823&amp;group_id=56967
[378] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2893821&amp;group_id=56967
[379] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2893815&amp;group_id=56967
[380] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2893484&amp;group_id=56967
[381] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2893247&amp;group_id=56967
[382] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2891718&amp;group_id=56967
[383] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2891191&amp;group_id=56967
[384] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2890914&amp;group_id=56967
[385] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2890702&amp;group_id=56967
[386] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2890035&amp;group_id=56967
[387] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2887282&amp;group_id=56967
[388] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2879011&amp;group_id=56967
[389] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2879008&amp;group_id=56967
[390] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2879005&amp;group_id=56967
[391] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2878994&amp;group_id=56967
[392] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2877044&amp;group_id=56967
[393] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2876948&amp;group_id=56967
[394] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2874045&amp;group_id=56967
[395] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2873564&amp;group_id=56967
[396] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2872884&amp;group_id=56967
[397] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2864179&amp;group_id=56967
[398] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2863458&amp;group_id=56967
[399] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2837050&amp;group_id=56967
[400] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2834654&amp;group_id=56967
[401] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2817414&amp;group_id=56967
[402] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2791361&amp;group_id=56967
[403] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2791356&amp;group_id=56967
[404] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2771052&amp;group_id=56967
[405] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2716723&amp;group_id=56967
[406] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2686008&amp;group_id=56967
[407] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2627137&amp;group_id=56967
[408] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2211565&amp;group_id=56967
[409] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=2015135&amp;group_id=56967
[410] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=1991738&amp;group_id=56967
[411] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=1981735&amp;group_id=56967
[412] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=1976341&amp;group_id=56967
[413] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=1964365&amp;group_id=56967
[414] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=1959269&amp;group_id=56967
[415] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=1911209&amp;group_id=56967
[416] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=1893448&amp;group_id=56967
[417] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=1809586&amp;group_id=56967
[418] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=1747068&amp;group_id=56967
[419] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=1739353&amp;group_id=56967
[420] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=1730547&amp;group_id=56967
[421] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=1726599&amp;group_id=56967
[422] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=1681244&amp;group_id=56967
[423] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=1637867&amp;group_id=56967
[424] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=1607210&amp;group_id=56967
[425] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=1595129&amp;group_id=56967
[426] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=987304&amp;group_id=56967
[427] https://sourceforge.net/tracker/?func=detail&amp;atid=482468&amp;aid=921173&amp;group_id=56967
[428] http://sourceforge.net/
[429] http://www.cwi.nl/
[430] mailto:monet@cwi.nl
[431] http://monetdb.cwi.nl/Development/Legal/index.html
[432] http://sourceforge.net/projects/monetdb/
[433] http://sourceforge.net/tracker/?atid=482468&amp;group_id=56967&amp;func=browse
[434] http://sourceforge.net/tracker/?group_id=56967
[435] http://monetdb.cwi.nl/Development/TestWeb/index.html
[436] http://monetdb.cwi.nl/testing/projects/monetdb/permastore/
[437] http://monetdb.cwi.nl/Development/Releases/Aug2009/index.html#Aug2009_SP2
[438] http://monetdb.cwi.nl/Development/Releases/Aug2009/index.html#Aug2009_SP1
[439] http://monetdb.cwi.nl/Development/Releases/Aug2009/index.html#Aug2009
[440] http://monetdb.cwi.nl/Development/Releases/Aug2009/index.html#Packaging
[441] http://monetdb.cwi.nl/Development/Releases/Aug2009/index.html#GDK
[442] http://monetdb.cwi.nl/Development/Releases/Aug2009/index.html#M5
[443] http://monetdb.cwi.nl/Development/Releases/Aug2009/index.html#SQL
[444] http://monetdb.cwi.nl/Development/Releases/Aug2009/index.html#XQuery
[445] http://monetdb.cwi.nl/Development/Releases/Aug2009/index.html#Clients
[446] http://sourceforge.net/tracker/index.php?func=detail&amp;aid=2825974&amp;group_id=56967&amp;atid=482468
[447] http://sourceforge.net/tracker/index.php?func=detail&amp;aid=2830900&amp;group_id=56967&amp;atid=482468
[448] http://sourceforge.net/tracker/index.php?func=detail&amp;aid=2831994&amp;group_id=56967&amp;atid=482468
[449] https://sourceforge.net/tracker/?func=detail&amp;aid=2814622&amp;group_id=56967&amp;atid=482468
[450] https://sourceforge.net/tracker/?func=detail&amp;aid=2818176&amp;group_id=56967&amp;atid=482468
[451] https://sourceforge.net/tracker/?func=detail&amp;aid=2807609&amp;group_id=56967&amp;atid=482468
[452] https://sourceforge.net/tracker/?func=detail&amp;aid=2812729&amp;group_id=56967&amp;atid=482468
[453] https://sourceforge.net/tracker/?func=detail&amp;aid=2812721&amp;group_id=56967&amp;atid=482468
[454] https://sourceforge.net/tracker/?func=detail&amp;aid=2826015&amp;group_id=56967&amp;atid=482468
[455] https://sourceforge.net/tracker/?func=detail&amp;aid=2807336&amp;group_id=56967&amp;atid=482468
[456] https://sourceforge.net/tracker/?func=detail&amp;aid=2837561&amp;group_id=56967&amp;atid=482468
[457] https://sourceforge.net/tracker/?func=detail&amp;aid=2800964&amp;group_id=56967&amp;atid=482471
[458] https://sourceforge.net/tracker/?func=detail&amp;aid=1837591&amp;group_id=56967&amp;atid=482471
[459] http://monetdb.cwi.nl/Development/Releases/May2009/index.html#May2009_SP2
[460] http://monetdb.cwi.nl/Development/Releases/May2009/index.html#May2009_SP1
[461] http://monetdb.cwi.nl/Development/Releases/May2009/index.html#May2009
[462] http://monetdb.cwi.nl/Development/Releases/May2009/index.html#Packaging
[463] http://monetdb.cwi.nl/Development/Releases/May2009/index.html#GDK
[464] http://monetdb.cwi.nl/Development/Releases/May2009/index.html#M5
[465] http://monetdb.cwi.nl/Development/Releases/May2009/index.html#Clients
[466] http://www.perl.org/
[467] http://www.php.net/
[468] http://www.python.org/
[469] https://sourceforge.net/tracker/?func=detail&amp;aid=2782776&amp;group_id=56967&amp;atid=482471
[470] https://sourceforge.net/tracker/?func=detail&amp;aid=2705670&amp;group_id=56967&amp;atid=482471
[471] http://monetdb.cwi.nl/Development/Releases/Feb2009/index.html
[472] http://monetdb.cwi.nl/Development/Releases/Version4.24/index.html#XQuery
[473] http://python.org/
[474] http://www.ruby-lang.org/
[475] http://monetdb.cwi.nl/Development/Releases/Feb2009/index.html#Feb2009_SP2
[476] http://monetdb.cwi.nl/Development/Releases/Feb2009/index.html#Feb2009_SP1
[477] http://monetdb.cwi.nl/Development/Releases/Feb2009/index.html#Feb2009
[478] http://monetdb.cwi.nl/Development/Releases/Feb2009/index.html#Packaging
[479] http://monetdb.cwi.nl/Development/Releases/Feb2009/index.html#M5
[480] http://monetdb.cwi.nl/Development/Releases/Feb2009/index.html#SQL
[481] http://monetdb.cwi.nl/Development/Releases/Feb2009/index.html#XQuery
[482] http://monetdb.cwi.nl/Development/Releases/Feb2009/index.html#Mtest
[483] http://monetdb.cwi.nl/Development/Releases/Feb2009/index.html#Bugs
[484] http://monetdb.cwi.nl/Development/Releases/Feb2009/index.html#Issues
[485] http://www.debian.org/
[486] http://www.ubuntu.com/
[487] http://fedoraproject.org/
[488] http://monetdb.cwi.nl/Development/Releases/Nov2008/index.html
[489] http://sourceforge.net/tracker/index.php?func=browse&amp;group_id=56967&amp;atid=482468&amp;set=custom&amp;_assigned_to=0&amp;_status=2&amp;_category=0&amp;_group=0&amp;order=close_date&amp;sort=ASC&amp;offset=1796
[490] http://sourceforge.net/tracker/index.php?func=browse&amp;group_id=56967&amp;atid=482468&amp;set=custom&amp;_assigned_to=0&amp;_status=2&amp;_category=0&amp;_group=0&amp;order=close_date&amp;sort=ASC&amp;offset=1846
[491] http://monetdb.cwi.nl/Development/Releases/Nov2008/index.html#Nov2008_SP2
[492] http://monetdb.cwi.nl/Development/Releases/Nov2008/index.html#Nov2008_SP1
[493] http://monetdb.cwi.nl/Development/Releases/Nov2008/index.html#Nov2008
[494] http://monetdb.cwi.nl/Development/Releases/Nov2008/index.html#Copyright
[495] http://monetdb.cwi.nl/Development/Releases/Nov2008/index.html#Windows
[496] http://monetdb.cwi.nl/Development/Releases/Nov2008/index.html#GDK
[497] http://monetdb.cwi.nl/Development/Releases/Nov2008/index.html#M5
[498] http://monetdb.cwi.nl/Development/Releases/Nov2008/index.html#SQL
[499] http://monetdb.cwi.nl/Development/Releases/Nov2008/index.html#XQuery
[500] http://monetdb.cwi.nl/Development/Releases/Nov2008/index.html#Bugs
[501] http://monetdb.cwi.nl/Development/Releases/Nov2008/index.html#Issues
[502] http://monetdb.cwi.nl/SQL/Benchmark/TPCH/
[503] http://monetdb.cwi.nl/Development/Releases/Version4.24/index.html
[504] http://sourceforge.net/tracker/index.php?func=browse&amp;group_id=56967&amp;atid=482468&amp;set=custom&amp;_assigned_to=0&amp;_status=2&amp;_category=0&amp;_group=0&amp;order=close_date&amp;sort=ASC&amp;offset=1651
[505] http://sourceforge.net/tracker/index.php?func=browse&amp;group_id=56967&amp;atid=482468&amp;set=custom&amp;_assigned_to=0&amp;_status=2&amp;_category=0&amp;_group=0&amp;order=close_date&amp;sort=ASC&amp;offset=1701
[506] http://sourceforge.net/tracker/index.php?func=browse&amp;group_id=56967&amp;atid=482468&amp;set=custom&amp;_assigned_to=0&amp;_status=2&amp;_category=0&amp;_group=0&amp;order=close_date&amp;sort=ASC&amp;offset=1751
[507] https://sourceforge.net/tracker/index.php?func=detail&amp;aid=2305746&amp;group_id=56967&amp;atid=482468
[508] http://monetdb.cwi.nl/Development/Releases/Version4.24/index.html#Compilation
[509] http://monetdb.cwi.nl/Development/Releases/Version4.24/index.html#Windows
[510] http://monetdb.cwi.nl/Development/Releases/Version4.24/index.html#GDK
[511] http://monetdb.cwi.nl/Development/Releases/Version4.24/index.html#SQL
[512] http://monetdb.cwi.nl/Development/Releases/Version4.24/index.html#Geometry
[513] http://monetdb.cwi.nl/Development/Releases/Version4.24/index.html#Bugs
[514] http://monetdb.cwi.nl/SQL/Documentation/SQL_002fGIS.html
[515] http://monetdb.cwi.nl/Development/Releases/Version4.22/index.html
[516] http://monetdb.cwi.nl/XQuery/Documentation/Arithmetic-Functions.html
[517] http://monetdb.cwi.nl/XQuery/Documentation/Prepared-Queries.html
[518] http://monetdb.cwi.nl/XQuery/Documentation/XRPC-Extension.html
[519] http://monetdb.cwi.nl/XQuery/Documentation/XRPC-Server.html
[520] http://monetdb.cwi.nl/XQuery/QuickTour/adminGUI/index.html
[521] http://monetdb.cwi.nl/XQuery/Documentation/XRPC-Syntax.html
[522] http://monetdb.cwi.nl/XQuery/Documentation/StandOff-Extension.html
[523] http://sourceforge.net/tracker/index.php?func=browse&amp;group_id=56967&amp;atid=482468&amp;set=custom&amp;_assigned_to=0&amp;_status=2&amp;_category=0&amp;_group=0&amp;order=close_date&amp;sort=ASC&amp;offset=1512
[524] http://sourceforge.net/tracker/index.php?func=browse&amp;group_id=56967&amp;atid=482468&amp;set=custom&amp;_assigned_to=0&amp;_status=2&amp;_category=0&amp;_group=0&amp;order=close_date&amp;sort=ASC&amp;offset=1557
[525] http://sourceforge.net/tracker/index.php?func=browse&amp;group_id=56967&amp;atid=482468&amp;set=custom&amp;_assigned_to=0&amp;_status=2&amp;_category=0&amp;_group=0&amp;order=close_date&amp;sort=ASC&amp;offset=1601
[526] http://monetdb.cvs.sourceforge.net/monetdb/clients/?pathrev=Clients_1-22
[527] http://monetdb.cvs.sourceforge.net/monetdb/pathfinder/?pathrev=XQuery_0-22
[528] http://monetdb.cvs.sourceforge.net/monetdb/java/
[529] https://sourceforge.net/tracker/index.php?func=browse&amp;group_id=56967&amp;atid=482468&amp;set=custom&amp;_assigned_to=0&amp;_status=2&amp;_category=100&amp;_group=100&amp;order=close_date&amp;sort=DESC&amp;offset=0
[530] http://dbappl.cs.utwente.nl/pftijah
[531] http://cas.sdss.org/dr5/en/
[532] http://www.w3.org/TR/2006/WD-xqupdate-20060711/
[533] http://dev.mysql.com/doc/mysql/en/mysql-benchmarks.html
[534] http://www.monetdb.org/todo
[535] http://www.gumstix.org/