December 2016 - developers-list

python3 client bug+fix
by Bas Bloemsaat 23 Dec '16

23 Dec '16

Hi, Working with MonetDB I encountered a problem with queries containing utf-8 characters. Some queries errored. I traced the error back to mapi.py module for python3. The problem was that encode() was applied after splicing the packet. With encode, any utf-8 characters are expanded to multiple octets. For a given block that exceeded MAX_PACKAGE_LENGTH this resulted in the last few octets falling off a package. The patch below remedies this, please apply. Regards, Bas Bloemsaat --- /srv/src/monetdb/MonetDB-11.23.13/clients/python3/monetdb/mapi.py 2016-10-07 11:00:50.000000000 +0200 +++ /home/bas/.pyenv/versions/ent_venv1/lib/python3.5/site-packages/monetdb/mapi.py 2016-12-23 12:02:53.431357999 +0100 @@ -275,8 +275,9 @@ class Connection(object): def _putblock_inet(self, block): pos = 0 last = 0 + block = block.encode() while not last: - data = block[pos:pos + MAX_PACKAGE_LENGTH].encode() + data = block[pos:pos + MAX_PACKAGE_LENGTH] length = len(data) if length < MAX_PACKAGE_LENGTH: last = 1

2 2

Announcement: New Dec2016 Feature release of MonetDB suite
by Panagiotis Koutsourakis 21 Dec '16

21 Dec '16

The MonetDB team at CWI/MonetDB BV is pleased to announce the Dec2016 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Dec 2016 feature release MonetDB5 Server * Removed the zorder module with functions zorder.encode, zorder.decode_x and zorder.decode_y. * Removed command bbp.getHeat(). * Removed bat.setColumn with two arguments and bat.setRole. Use bat.setColumn with one argument instead. * Removed function BKCappend_reverse_val_wrap: it was unused. * The "wrd" type has been removed from GDK and MAL. The type was defined to be a 32 bit integer on 32 bit architectures and a 64 bit integer on 64 bit architectures. We now generally use "lng" (always 64 bits) where "wrd" was used. * Keep a collection of full traces. Each time the SQL user applies the TRACE option, the full json trace is retained within the <dbpath>/<dbname>/sql_traces Build Environment * New packages MonetDB-python2 (Fedora) and monetdb-python2 (Debian/Ubuntu) have been created for Python 2 integration into MonetDB. * With OID size equal to ABI/word size, mserver5 does not need to print the OID size, anymore. * Removed obsolete code associated with long gone static linking option. * Removed configure option --enable-oid32 to compile with 32 bit OIDs on a 64 bit architecture. . * The Perl, PHP, and Python clients, and the JDBC driver each now have their own repositories and release cycles. The Python client is maintained by Gijs Molenaar on Github (https://github.com/gijzelaerr/pymonetdb), the other clients are maintained by CWI/MonetDB on our own server (https://dev.monetdb.org/hg/monetdb-java, https://dev.monetdb.org/hg/monetdb-perl, https://dev.monetdb.org/hg/monetdb-php). MonetDB Common * The tnokey values must now be 0 if it is not known whether all values in a column are distinct. * The 2-bit tkey field in the bat descriptor has been split into two single bit fields: tkey and tunique. The old tkey&BOUND2BTRUE value is now stored in tunique. * Implemented conversion to str from any type (not just the internal types). * VALcopy and VALinit both return their first argument on success or (and that's new) NULL on (allocation) failure. * BATattach now can also create a str BAT from a file consisting of null-terminated strings. The input file must be encoded using UTF-8. * BATattach now copies the input file instead of "stealing" it. * Removed the lastused "timestamp" from the BBP. * Removed batStamp field from BAT descriptor, and removed the BBPcurstamp function. * Removed unused functions BBPhot and BBPcold. * Removed BATderiveTailProps and BATderiveProps. Just set the properties you know about, or use BATsettrivprop. * Removed the macro BUNfirst. It can be replaced by 0. * Changed BATroles by removing the argument to set the name of the head column. * The head column is now completely gone. MonetDB is completely "headless". * The format of the BBP.dir file was simplified. Since the head column is VOID, the only value that needs to be stored is the head seqbase. * BATs now only have a single (logical) name. * The function BATmirror is gone. The HEAD column is always VOID (with a non-nil seqbase) and the TAIL column carries the data. All functions that deal with data work on the TAIL column. * BATkey now works on the TAIL column instead of the HEAD column. * Replaced BATseqbase with BAThseqbase and BATtseqbase, the former for setting the seqbase on the HEAD, the latter for setting the seqbase on the TAIL. * Replaced function BATnew with COLnew with slightly different arguments: the first argument of COLnew is the SEQBASE of the head column (which is always VOID). * The "wrd" type has been removed from GDK and MAL. The type was defined to be a 32 bit integer on 32 bit architectures and a 64 bit integer on 64 bit architectures. We now generally use "lng" (always 64 bits) where "wrd" was used. SQL * Removed functions sys.zorder_encode, sys.zorder_decode_x, and sys.zorder_decode_y. * The experimental recycler code is moved to the attic. * The syntax of bat.new(:oid,:any) has been changed by dropping the superflous :oid. All BATs are now binary associations between a void column and a materialized value column. (except for the internal :bat[:void,:void] representation of simple oid ranged tails.) Bug Fixes * 3357: Implement setQueryTimeout() * 3445: Add support for database name to dotmonetdb file * 3597: SQL to MAL listing looses types * 3973: JDBC hangs * 3976: Performance enhancement to LIKE without wildcards * 4005: Correlated update causes incorrect null constraint violation * 4016: merge table only optimises for point query * 4040: sys.storage call can take a long time * 4047: Segfault when updating a dropped table * 4050: Database corruption when running low on inode * 4057: missing bulk operations between constant and bat * 4061: SIGSEGV in candscan_lng * 4066: Deadlocked monetdbd * 6068: Error message about incompatible BBP version should be clearer * 6069: query with union all silently crashes * 6070: setting negative session query timeout should not be possible/allowed * 6071: where clause with cast and floor fails to sigsegv * 6072: Bind to UPD delta column does not get/show type information in EXPLAIN * 6073: Missing type information for constants in MAL explain * 6074: SET ROLE command does not work * 6075: gdk_calc.c:13113: BATcalcifthenelse_intern: Assertion `col2 != NULL' failed. * 6076: rel_optimizer.c:5426: rel_push_project_up: Assertion `e' failed. * 6077: mserver5: rel_optimizer.c:5444: rel_push_project_up: Assertion `e' failed. * 6078: rel_bin.c:2402: rel2bin_project: Assertion `0' failed. * 6084: Merge table point to wrong columns if columns in partition tables are deleted * 6108: monetdb5-sql sysv init script does not wait for shutdown * 6114: segfault raised in the query rewriter due to a null pointer * 6115: Assertion hit in the codegen * 6116: Codegen does not support certain kind of selects on scalar subqueries * 6117: Assertion hit in the query rewriting stage during the push down phase * 6118: SIGSEGV in strPut due to shared heap * 6119: Assertion hit in the MAL optimiser on a complex query * 6120: QUANTILE() treats NULL as if it is zero * 6121: SELECT a.col IN ( b.col FROM b ) FROM a statements with no error but no result * 6123: Assertion hit in the codegen #2 * 6124: CASE <column> WHEN NULL THEN 0 ELSE 1 END returns wrong result * 6125: Stack overflow in the query rewriter with a query having an OR condition and a nested SELECT subexpression * 6126: batcalc.== can't handle void BATs * 6139: Debian libmonetdb13 conflicts with libmonetdb5-server-geom

1 0

RE: Need Help on enabling GIS (MonetDB/GIS) module in MonetDB > (Joweria Nakibuuka) RE: Need Help on enabling GIS (MonetDB/GIS) module in MonetDB > (Joweria Nakibuuka)
by Joweria Nakibuuka 01 Dec '16

01 Dec '16

joweria@ubuntu:~$ mserver5 --dbpath=/tmp/MyDBfarm/MyDB </dev/null # MonetDB 5 server v11.24.0 # This is an unreleased version # Serving database 'MyDB', using 2 threads # Compiled for x86_64-unknown-linux-gnu/64bit with 128bit integers # Found 1.470 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2016 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ # MonetDB/GIS module loaded # MonetDB/SQL module loaded >joweria@ubuntu:~$ joweria@ubuntu:~$ mserver5 --dbpath=/tmp/MyDBfarm/MyDB </dev/null # MonetDB 5 server v11.24.0 # This is an unreleased version # Serving database 'MyDB', using 2 threads # Compiled for x86_64-unknown-linux-gnu/64bit with 128bit integers # Found 1.470 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2016 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ # MonetDB/GIS module loaded # MonetDB/SQL module loaded But when i try to reach another database as shown below, the module is not loaded joweria@ubuntu:~$ mserver5 --dbpath=/home/joweria/Spartial/mydbfarm/Gisbase !FATAL: GDKlockHome: Database lock '.gdk_lock' denied joweria@ubuntu:~$ sudo su [sudo] password for joweria: root@ubuntu:/home/joweria# mserver5 --dbpath=/home/joweria/ Spartial/mydbfarm/Gisbase # MonetDB 5 server v11.24.0 # This is an unreleased version # Serving database 'Gisbase', using 2 threads # Compiled for x86_64-unknown-linux-gnu/64bit with 128bit integers # Found 1.470 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2016 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ # MonetDB/SQL module loaded > Also when i run with my user priviledges

1 0

Re: developers-list Digest, Vol 52, Issue 1
by Joweria Nakibuuka 01 Dec '16

01 Dec '16

joweria@ubuntu:~$ mserver5 --dbpath=/tmp/MyDBfarm/MyDB # MonetDB 5 server v11.24.0 # This is an unreleased version # Serving database 'MyDB', using 2 threads # Compiled for x86_64-unknown-linux-gnu/64bit with 128bit integers # Found 1.470 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2016 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ # MonetDB/GIS module loaded # MonetDB/SQL module loaded 2016-12-01 13:00 GMT+02:00 <developers-list-request(a)monetdb.org>: > Send developers-list mailing list submissions to > developers-list(a)monetdb.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://www.monetdb.org/mailman/listinfo/developers-list > or, via email, send a message with subject or body 'help' to > developers-list-request(a)monetdb.org > > You can reach the person managing the list at > developers-list-owner(a)monetdb.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of developers-list digest..." > > > Today's Topics: > > 1. RE: Need Help on enabling GIS (MonetDB/GIS) module in MonetDB > (Joweria Nakibuuka) > 2. Re: Need Help on enabling GIS (MonetDB/GIS) module in MonetDB > (Stefan Manegold) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 1 Dec 2016 12:11:49 +0200 > From: Joweria Nakibuuka <joweria.nakibuuka(a)unitn.it> > To: developers-list(a)monetdb.org > Subject: RE: Need Help on enabling GIS (MonetDB/GIS) module in MonetDB > Message-ID: > <CANbNUqbw+N3=tb929XArQ8q+AiCG-o5DYNR9Ndk2POChHpdi1g@ > mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Dear Admin, > I am running Ubuntu linux 14.04 LTS, x86_64-Ubuntu-linux-gnu/64bit with > 128bit integers. i recently installed MonetDB5 default branch from the > mercurial repository. > > I tried to rebuild and enable geom module but to no avail, it shows no > error, after compilation it shows that the module is enabled,However when i > run the mserver the monetDB/GIS module is not loaded as expected. I am > stuck and i don't seem to know how to proceed forward in order to run my > spatial queries > > Are there any setting i need to do explicitly to load this module? > > the command i used: ./configure --prefix=/home/joweria/MonetDB-Nov2016-SP4 > --enable-geom=yes --enable-pyintegration=yes --with-geos=yes > > and then after compilation under the LDFLAGS-enabled componets > geom is one of the enabled componet but when i run the mserver5 the module > is not loaded as expected. > > I rebuild monetDB again and for first time i run the mserver5 commans > and the GIS module was successfully loaded ie (MonetDB/SQL/GIS module is > loaded) but when i closed the session (\q) and ran the same command again > the module was not loaded. i tried exporting paths to no avail. what could > be the problem? please i need your kind help. > > Thank you > > Regards, > Joweria nakibuuka >

2 2

RE: Need Help on enabling GIS (MonetDB/GIS) module in MonetDB
by Joweria Nakibuuka 01 Dec '16

01 Dec '16

Dear Admin, I am running Ubuntu linux 14.04 LTS, x86_64-Ubuntu-linux-gnu/64bit with 128bit integers. i recently installed MonetDB5 default branch from the mercurial repository. I tried to rebuild and enable geom module but to no avail, it shows no error, after compilation it shows that the module is enabled,However when i run the mserver the monetDB/GIS module is not loaded as expected. I am stuck and i don't seem to know how to proceed forward in order to run my spatial queries Are there any setting i need to do explicitly to load this module? the command i used: ./configure --prefix=/home/joweria/MonetDB-Nov2016-SP4 --enable-geom=yes --enable-pyintegration=yes --with-geos=yes and then after compilation under the LDFLAGS-enabled componets geom is one of the enabled componet but when i run the mserver5 the module is not loaded as expected. I rebuild monetDB again and for first time i run the mserver5 commans and the GIS module was successfully loaded ie (MonetDB/SQL/GIS module is loaded) but when i closed the session (\q) and ran the same command again the module was not loaded. i tried exporting paths to no avail. what could be the problem? please i need your kind help. Thank you Regards, Joweria nakibuuka

2 1