[MonetDB-users] MonetSQLdb python API
Hi,
I finally have time to work on porting our app to monetdb again and am immediately running into a problem: I can't get MonetSQLdb to work properly :-(
I've started an Mserver process:
$ Mserver --dbinit="module(sql_server); sql_server_start();"
This works correcty, I can connect to it using the MapiClient binary and do SQL queries. However, this python code fails:
import MonetSQLdb c = MonetSQLdb.connect()
It says:
!ERROR mapi_reconnect: connect: Connection refused Traceback (most recent call last): File "t.py", line 2, in ? c = MonetSQLdb.connect() File "/opt/monet-4.10/opt/python2.4/lib/python2.4/site-packages/ MonetSQLdb/__init__.py", line 51, in Connect return Connection(**kwargs) File "/opt/monet-4.10/opt/python2.4/lib/python2.4/site-packages/ MonetSQLdb/__init__.py", line 117, in __init__ kwargs['user'], kwargs['password'], kwargs['lang']) File "/opt/monet-4.10/opt/python2.4/lib/python2.4/site-packages/ CMapi.py", line 26, in __init__ raise IOError(self.error_str()) IOError: Setup connection failed
Looking into the source code for the python client it should try to connect to port 45123, which the server is listening to. What it actually seems to do is connect to port 42312, but I have no idea why it does so (and hunting this down through several layers of generated code, one of which is swig, is no fun at all).
What am I doing wrong?
Ronald
On 03-10-2006 10:27:29 +0200, Ronald Oussoren wrote:
Hi,
I finally have time to work on porting our app to monetdb again and am immediately running into a problem: I can't get MonetSQLdb to work properly :-(
I've started an Mserver process:
$ Mserver --dbinit="module(sql_server); sql_server_start();"
This works correcty, I can connect to it using the MapiClient binary and do SQL queries. However, this python code fails:
import MonetSQLdb c = MonetSQLdb.connect()
It says:
!ERROR mapi_reconnect: connect: Connection refused
What version of MonetDB are you using? As far as I know, MonetDB should listen to port 50000 for all languages these days. One reason for your connection refuse can be that you connect to the hostname of your computer, instead of localhost (127.0.0.1). By default the server only listens on the local loopback interface.
Looking into the source code for the python client it should try to connect to port 45123, which the server is listening to. What it actually seems to do is connect to port 42312, but I have no idea why it does so (and hunting this down through several layers of generated code, one of which is swig, is no fun at all).
Weird.
On Oct 3, 2006, at 10:32 AM, Fabian Groffen wrote:
On 03-10-2006 10:27:29 +0200, Ronald Oussoren wrote:
Hi,
I finally have time to work on porting our app to monetdb again and am immediately running into a problem: I can't get MonetSQLdb to work properly :-(
I've started an Mserver process:
$ Mserver --dbinit="module(sql_server); sql_server_start();"
This works correcty, I can connect to it using the MapiClient binary and do SQL queries. However, this python code fails:
import MonetSQLdb c = MonetSQLdb.connect()
It says:
!ERROR mapi_reconnect: connect: Connection refused
What version of MonetDB are you using? As far as I know, MonetDB should listen to port 50000 for all languages these days. One reason for your connection refuse can be that you connect to the hostname of your computer, instead of localhost (127.0.0.1). By default the server only listens on the local loopback interface.
The machine I was testing on is running 4.10.3 and is definitely listening on port 45123 and only that. I'll upgrade to the latest version (daily snapshot of the release branch) to check if that changes things.
Ronald
On 03-10-2006 10:42:24 +0200, Ronald Oussoren wrote:
The machine I was testing on is running 4.10.3 and is definitely listening on port 45123 and only that. I'll upgrade to the latest
Correct, that version is quite old.
version (daily snapshot of the release branch) to check if that changes things.
Ok.
On Oct 3, 2006, at 10:45 AM, Fabian Groffen wrote:
On 03-10-2006 10:42:24 +0200, Ronald Oussoren wrote:
The machine I was testing on is running 4.10.3 and is definitely listening on port 45123 and only that. I'll upgrade to the latest
Correct, that version is quite old.
'make check' for MonetDB-4.12.1 (downloaded today) complains a lot about:
!ERROR: interpret: no matching MIL operator to 'quit()'. quit();
This is on a linux system (SLES9). I aborted 'make check' because it didn't seem to make any progress except printing this error message a lot.
BTW. I also had to suply both '--enable-debug' and '--disable- optimize' to the configure script, without the latter configure claimed I couldn't have both --enable-optimze and --enable-debug.
I'm building from source because I want to install into /opt/ monet-4.12.1 and link to a custom build of python.
BTW2. If monetdb is installed in a different prefix than python the python bindings won't end up on the default search path of the python interpreter. I've added a .pth file to site-packages to fix that, maybe 'make install' should do that automaticly. Building/installing using distutils would be even better :-)
BTW3. The version info for the shared libraries is incorrect, which means you cannot have two versions of monetdb installed at the same time, I got a Mserver 4.12.1 that claimed it was 4.10 and didn't work at all. I guess this is "don't do that" territory ;-). Removing the older installation solved this.
version (daily snapshot of the release branch) to check if that changes things.
Ok.
Upgrading has solved the problem I was having. I can now perform SQL queries from Python.
The MonetSQLdb module doesn't contain a callable named 'Binary'. Is that intentional? This callable is required according to PEP 249. The same is true for the value BINARY which isn't present as well (although it is listed in __all__). SQLObject needs the Binary callable, I'll send a patch when I have a working version of this.
Ronald
On 03-10-2006 11:55:52 +0200, Ronald Oussoren wrote:
On Oct 3, 2006, at 10:45 AM, Fabian Groffen wrote:
On 03-10-2006 10:42:24 +0200, Ronald Oussoren wrote:
The machine I was testing on is running 4.10.3 and is definitely listening on port 45123 and only that. I'll upgrade to the latest
Correct, that version is quite old.
'make check' for MonetDB-4.12.1 (downloaded today) complains a lot about:
!ERROR: interpret: no matching MIL operator to 'quit()'. quit();
That doesn't feel right.
This is on a linux system (SLES9). I aborted 'make check' because it didn't seem to make any progress except printing this error message a lot.
BTW. I also had to suply both '--enable-debug' and '--disable- optimize' to the configure script, without the latter configure claimed I couldn't have both --enable-optimze and --enable-debug.
There is a default involved here. In order to have monetdb's configure respect your CFLAGS, you have to axe out CFLAGS hackery from the configure script, e.g.:
sed -i \ -e 's|CFLAGS="$CFLAGS \$(X_CFLAGS)"||' \ configure || echo "Fixing configure failed!"
Then configure with --disable-debug --disable-optimize
I'm building from source because I want to install into /opt/ monet-4.12.1 and link to a custom build of python.
That's fine and should work as expected (--prefix=/opt/monetdb-${PV})
BTW2. If monetdb is installed in a different prefix than python the python bindings won't end up on the default search path of the python interpreter. I've added a .pth file to site-packages to fix that, maybe 'make install' should do that automaticly. Building/installing using distutils would be even better :-)
I think that's what PYTHONPATH is for, but I leave that to the python experts. If your solution is to write in the non-prefix to make some sort of alias, then it won't work, as you don't have to be root. Maybe you don't even want to do that. (Using another version of python, for example.)
BTW3. The version info for the shared libraries is incorrect, which means you cannot have two versions of monetdb installed at the same time, I got a Mserver 4.12.1 that claimed it was 4.10 and didn't work at all. I guess this is "don't do that" territory ;-). Removing the older installation solved this.
I think slotting is not supported, and that the best way out of there is to just install into different prefixes.
Upgrading has solved the problem I was having. I can now perform SQL queries from Python.
That's great.
The MonetSQLdb module doesn't contain a callable named 'Binary'. Is that intentional? This callable is required according to PEP 249. The same is true for the value BINARY which isn't present as well (although it is listed in __all__). SQLObject needs the Binary callable, I'll send a patch when I have a working version of this.
I have no clue on this. Maybe some Python expert has.
On Oct 3, 2006, at 12:38 PM, Fabian Groffen wrote:
On 03-10-2006 11:55:52 +0200, Ronald Oussoren wrote:
On Oct 3, 2006, at 10:45 AM, Fabian Groffen wrote:
On 03-10-2006 10:42:24 +0200, Ronald Oussoren wrote:
The machine I was testing on is running 4.10.3 and is definitely listening on port 45123 and only that. I'll upgrade to the latest
Correct, that version is quite old.
'make check' for MonetDB-4.12.1 (downloaded today) complains a lot about:
!ERROR: interpret: no matching MIL operator to 'quit()'. quit();
That doesn't feel right.
That's what I thought as well ;-)
This is on a linux system (SLES9). I aborted 'make check' because it didn't seem to make any progress except printing this error message a lot.
BTW. I also had to suply both '--enable-debug' and '--disable- optimize' to the configure script, without the latter configure claimed I couldn't have both --enable-optimze and --enable-debug.
There is a default involved here. In order to have monetdb's configure respect your CFLAGS, you have to axe out CFLAGS hackery from the configure script, e.g.:
sed -i \ -e 's|CFLAGS="$CFLAGS \$(X_CFLAGS)"||' \ configure || echo "Fixing configure failed!"
Then configure with --disable-debug --disable-optimize
Is this needed at all when I just want to compile with '-g'? -- disable-optimize --enable-debug seems to work correctly for this. I mentioned the need for --disable-optimize because most other programs with a configure script seem to have no problems with having both optimization and debugging enabled (GCC can compile code like that, even if debugging optimized code can be quite confusing).
I'm building from source because I want to install into /opt/ monet-4.12.1 and link to a custom build of python.
That's fine and should work as expected (--prefix=/opt/monetdb-${PV})
It does, I was trying to avoid questions about why I wasn't using the RPMs.
BTW2. If monetdb is installed in a different prefix than python the python bindings won't end up on the default search path of the python interpreter. I've added a .pth file to site-packages to fix that, maybe 'make install' should do that automaticly. Building/installing using distutils would be even better :-)
I think that's what PYTHONPATH is for, but I leave that to the python experts. If your solution is to write in the non-prefix to make some sort of alias, then it won't work, as you don't have to be root. Maybe you don't even want to do that. (Using another version of python, for example.)
PYTHONPATH works as well, but I prefer .pth files unless I want to use multiple versions. Anyway, this seems to be a bug in the install target, I didn't notice this before but the python files are installed in ${monetPrefix}${pythonPrefix} instead of just $ {pythonPrefix}, MonetSQLdb.__file__ is '/opt/monet-4.12.1/opt/ python2.4/lib/python2.4/site-packages/MonetSQLdb/__init__.pyc' on my system.
Just to be anal: you cannot share extensions between multiple versions of python. It happens to work in most cases, but python 2.x and 2.y (x != y) aren't binary compatible.
BTW3. The version info for the shared libraries is incorrect, which means you cannot have two versions of monetdb installed at the same time, I got a Mserver 4.12.1 that claimed it was 4.10 and didn't work at all. I guess this is "don't do that" territory ;-). Removing the older installation solved this.
I think slotting is not supported, and that the best way out of there is to just install into different prefixes.
I did, but had both prefixes on LD_LIBRARY_PATH (that is, I wasn't paying attentention)
Thanks for your help, Ronald
On Tue, Oct 03, 2006 at 12:53:37PM +0200, Ronald Oussoren wrote:
On Oct 3, 2006, at 12:38 PM, Fabian Groffen wrote:
On 03-10-2006 11:55:52 +0200, Ronald Oussoren wrote:
On Oct 3, 2006, at 10:45 AM, Fabian Groffen wrote:
On 03-10-2006 10:42:24 +0200, Ronald Oussoren wrote:
The machine I was testing on is running 4.10.3 and is definitely listening on port 45123 and only that. I'll upgrade to the latest
Correct, that version is quite old.
'make check' for MonetDB-4.12.1 (downloaded today) complains a lot about:
!ERROR: interpret: no matching MIL operator to 'quit()'. quit();
does this happen with only MonetDB-4.12.1 present on your system, or with a "legacy" MonetDB-4.10.3 installed as well (cf. BTW3 below)?
That doesn't feel right.
That's what I thought as well ;-)
This is on a linux system (SLES9). I aborted 'make check' because it didn't seem to make any progress except printing this error message a lot.
BTW. I also had to suply both '--enable-debug' and '--disable- optimize' to the configure script, without the latter configure claimed I couldn't have both --enable-optimze and --enable-debug.
that's absolutely correct and IMHO desired behavior: the default for the release is "--enable-optimze", and --enable-optimze & --enable-debug do conflict; hence, if you want/need --enable-debug, you also must switch off optimization (--disable-optimize).
There is a default involved here. In order to have monetdb's configure respect your CFLAGS, you have to axe out CFLAGS hackery from the configure script, e.g.:
sed -i \ -e 's|CFLAGS="$CFLAGS \$(X_CFLAGS)"||' \ configure || echo "Fixing configure failed!"
this is IMHO only required if you really need to explicitly proved CFLAGS with configure; --disable-optimize, --enable-debug do (should) not require such hack(s)...
Then configure with --disable-debug --disable-optimize
Is this needed at all when I just want to compile with '-g'? -- disable-optimize --enable-debug seems to work correctly for this. I mentioned the need for --disable-optimize because most other programs with a configure script seem to have no problems with having both optimization and debugging enabled (GCC can compile code like that, even if debugging optimized code can be quite confusing).
gcc can (to a certain extend) compile with both debugging and optimization (at least -O2) enabled (I'm not sure whether it stil works with our "aggressive" optimization options in case of --enable-optimize, i.e., "-O6 -fomit-frame-pointer -finline-functions -malign-loops=4 -malign-jumps=4 -malign-functions=4 -fexpensive-optimizations -funroll-all-loops -funroll-loops -frerun-cse-after-loop -frerun-loop-opt"...)
However, when using gdb on gcc compiled code with -g -Ox, (x>0), practice shows that things do not work as you expect them to work (e.g., source tracing not necessarily aligned with actual exection, etc.); hence, we consider only "-g -O0" as "save" option for (indeed) debugable code...
I'm building from source because I want to install into /opt/ monet-4.12.1 and link to a custom build of python.
That's fine and should work as expected (--prefix=/opt/monetdb-${PV})
It does, I was trying to avoid questions about why I wasn't using the RPMs.
BTW2. If monetdb is installed in a different prefix than python the python bindings won't end up on the default search path of the python interpreter. I've added a .pth file to site-packages to fix that, maybe 'make install' should do that automaticly. Building/installing using distutils would be even better :-)
I think that's what PYTHONPATH is for, but I leave that to the python experts. If your solution is to write in the non-prefix to make some sort of alias, then it won't work, as you don't have to be root. Maybe you don't even want to do that. (Using another version of python, for example.)
PYTHONPATH works as well, but I prefer .pth files unless I want to use multiple versions. Anyway, this seems to be a bug in the install target, I didn't notice this before but the python files are installed in ${monetPrefix}${pythonPrefix} instead of just $ {pythonPrefix}, MonetSQLdb.__file__ is '/opt/monet-4.12.1/opt/ python2.4/lib/python2.4/site-packages/MonetSQLdb/__init__.pyc' on my system.
Just to be anal: you cannot share extensions between multiple versions of python. It happens to work in most cases, but python 2.x and 2.y (x != y) aren't binary compatible.
BTW3. The version info for the shared libraries is incorrect, which means you cannot have two versions of monetdb installed at the same time, I got a Mserver 4.12.1 that claimed it was 4.10 and didn't work at all. I guess this is "don't do that" territory ;-). Removing the older installation solved this.
If Mserver claims to be 4.10, then it indeed is 4.10 --- to be more precise, it at least loads the 4.10 libmonet.so for the "known" location; in case you install both 4.12.1 & 4.10, you need to make sure that they are installed in different locations (prefixes), and your PATH (& LD_LIBRARY_PATH) point indeed to the version you intend to use.
I think slotting is not supported, and that the best way out of there is to just install into different prefixes.
I did, but had both prefixes on LD_LIBRARY_PATH (that is, I wasn't paying attentention)
Ok, then probably when calling the 4.12.1 Mserver, your 4.10 libdir was first in your LD_LIBRARY_PATH --- amaising that it did load the wrong libmonet at all --- this might also explain the make check problems you mentioned above ...
Stefan
Thanks for your help, Ronald
On Oct 3, 2006, at 1:22 PM, Stefan Manegold wrote:
On Tue, Oct 03, 2006 at 12:53:37PM +0200, Ronald Oussoren wrote:
On Oct 3, 2006, at 12:38 PM, Fabian Groffen wrote:
On 03-10-2006 11:55:52 +0200, Ronald Oussoren wrote:
On Oct 3, 2006, at 10:45 AM, Fabian Groffen wrote:
On 03-10-2006 10:42:24 +0200, Ronald Oussoren wrote:
The machine I was testing on is running 4.10.3 and is definitely listening on port 45123 and only that. I'll upgrade to the latest
Correct, that version is quite old.
'make check' for MonetDB-4.12.1 (downloaded today) complains a lot about:
!ERROR: interpret: no matching MIL operator to 'quit()'. quit();
does this happen with only MonetDB-4.12.1 present on your system, or with a "legacy" MonetDB-4.10.3 installed as well (cf. BTW3 below)?
This was with the wrong monetdb present, but the problem also exists with a correct installation.
I managed to catch the head of the failure as well:
Mtest.py: ERROR: 'Mtimeout -timeout 60 Mserver "--config=/home/ xsupport/src/monet/build/conf/MonetDB.conf" --debug=10 --set "monet_mod_path=/home/xsupport/src/monet/build/src/modules/ plain/.libs:/home/xsupport/src/monet/build/src/modules/plain:/home/ xsupport/src/monet/build/src/modules/contrib/.libs:/home/xsupport/src/ monet/build/src/modules/contrib:/home/xsupport/src/monet/build/src/ modules/calibrator/.libs:/home/xsupport/src/monet/build/src/modules/ calibrator:/home/xsupport/src/monet/build/src/modules/mnetcdf/.libs:/ home/xsupport/src/monet/build/src/modules/mnetcdf:/home/xsupport/src/ monet/build/src/mapi/.libs:/home/xsupport/src/monet/build/src/mapi" -- set "gdk_dbfarm=/home/xsupport/src/monet/build/dbfarm" --set "sql_logdir=/home/xsupport/src/monet/build/log" --set mapi_port=37305 --set monet_prompt= --trace --dbname=mTests` failed:
!WARNING: GDKlockHome: ignoring empty or invalid .gdk_lock. !WARNING: BBPdir: initializing BBP. # Monet Database Server V4.12.1 # Copyright (c) 1993-2006, CWI. All rights reserved. # Compiled for i686-pc-linux-gnu/32bit with 32bit OIDs; dynamically linked. # Visit http://monetdb.cwi.nl/ for further information. !ERROR: moduleClient: module(builtin) load error. !ERROR: moduleClient: module(arith) load error. !ERROR: moduleClient: module(str) load error. !ERROR: moduleClient: module(constant) load error. !ERROR: moduleClient: module(bat) load error. !ERROR: moduleClient: module(algebra) load error. !ERROR: moduleClient: module(sys) load error. !ERROR: moduleClient: module(trans) load error. quit(); !ERROR: interpret: no matching MIL operator to 'quit()'. quit(); !ERROR: interpret: no matching MIL operator to 'quit()'. quit(); !ERROR: interpret: no matching MIL operator to 'quit()'. quit(); !ERROR: interpret: no matching MIL operator to 'quit()'. quit(); !ERROR: interpret: no matching MIL operator to 'quit()'. quit(); !ERROR: interpret: no matching MIL operator to 'quit()'.
Something completely different: how do I coax the python DB-API implementation out of autocommit mode?
import MonetSQLdb conn = MonetSQLdb.connect() cursor = conn.cursor() conn.commit() # -> raises RuntimeError
Any updates you do through 'conn' are also instantationious, e.g. no transactions. I can use the 'START TRANSACTION' query to explictly start a transaction, but I then have to keep track of whether I've used 'START TRANSACTION' on one of the active cursors because executing START TRANSACTION inside a transaction seems to break that transaction.
The effect I'm looking after is that of the interactive MapiClient when autocommit is disabled: you don't have to start transactions explicitly but are always in one.
Ronald
On 03-10-2006 13:59:16 +0200, Ronald Oussoren wrote:
Something completely different: how do I coax the python DB-API implementation out of autocommit mode?
import MonetSQLdb conn = MonetSQLdb.connect() cursor = conn.cursor() conn.commit() # -> raises RuntimeError
Any updates you do through 'conn' are also instantationious, e.g. no transactions. I can use the 'START TRANSACTION' query to explictly start a transaction, but I then have to keep track of whether I've used 'START TRANSACTION' on one of the active cursors because executing START TRANSACTION inside a transaction seems to break that transaction.
The effect I'm looking after is that of the interactive MapiClient when autocommit is disabled: you don't have to start transactions explicitly but are always in one.
Isn't there a conn.setAutoCommit() or something? Switching off auto-commit, is done through a control command Xautocommit 0 (or 1 to enable it again). It is different from START TRANSACTION in that sense that upon a COMMIT the auto-commit is retained (switched off).
On Oct 3, 2006, at 2:03 PM, Fabian Groffen wrote:
On 03-10-2006 13:59:16 +0200, Ronald Oussoren wrote:
Something completely different: how do I coax the python DB-API implementation out of autocommit mode?
import MonetSQLdb conn = MonetSQLdb.connect() cursor = conn.cursor() conn.commit() # -> raises RuntimeError
Any updates you do through 'conn' are also instantationious, e.g. no transactions. I can use the 'START TRANSACTION' query to explictly start a transaction, but I then have to keep track of whether I've used 'START TRANSACTION' on one of the active cursors because executing START TRANSACTION inside a transaction seems to break that transaction.
The effect I'm looking after is that of the interactive MapiClient when autocommit is disabled: you don't have to start transactions explicitly but are always in one.
Isn't there a conn.setAutoCommit() or something? Switching off auto-commit, is done through a control command Xautocommit 0 (or 1 to enable it again). It is different from START TRANSACTION in that sense that upon a COMMIT the auto-commit is retained (switched off).
I did find a mapi_setAutocommit function in MapiLib that isn't exposed through the DB-API interface. I've added a setAutocommit method to MonetSQLdb.Connection that calls the mapi method of the same name and that does work.
Thanks again, Ronald
Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash http://www.techsay.com/default.php? page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
This was with the wrong monetdb present, but the problem also exists with a correct installation.
I managed to catch the head of the failure as well:
Mtest.py: ERROR: 'Mtimeout -timeout 60 Mserver "--config=/home/ xsupport/src/monet/build/conf/MonetDB.conf" --debug=10 --set "monet_mod_path=/home/xsupport/src/monet/build/src/modules/ plain/.libs:/home/xsupport/src/monet/build/src/modules/plain:/home/ xsupport/src/monet/build/src/modules/contrib/.libs:/home/xsupport/src/ monet/build/src/modules/contrib:/home/xsupport/src/monet/build/src/ modules/calibrator/.libs:/home/xsupport/src/monet/build/src/modules/ calibrator:/home/xsupport/src/monet/build/src/modules/mnetcdf/.libs:/ home/xsupport/src/monet/build/src/modules/mnetcdf:/home/xsupport/src/ monet/build/src/mapi/.libs:/home/xsupport/src/monet/build/src/mapi" -- set "gdk_dbfarm=/home/xsupport/src/monet/build/dbfarm" --set "sql_logdir=/home/xsupport/src/monet/build/log" --set mapi_port=37305 --set monet_prompt= --trace --dbname=mTests` failed:
!WARNING: GDKlockHome: ignoring empty or invalid .gdk_lock. !WARNING: BBPdir: initializing BBP. # Monet Database Server V4.12.1 # Copyright (c) 1993-2006, CWI. All rights reserved. # Compiled for i686-pc-linux-gnu/32bit with 32bit OIDs; dynamically linked. # Visit http://monetdb.cwi.nl/ for further information. !ERROR: moduleClient: module(builtin) load error. !ERROR: moduleClient: module(arith) load error. !ERROR: moduleClient: module(str) load error. !ERROR: moduleClient: module(constant) load error. !ERROR: moduleClient: module(bat) load error. !ERROR: moduleClient: module(algebra) load error. !ERROR: moduleClient: module(sys) load error. !ERROR: moduleClient: module(trans) load error. quit(); !ERROR: interpret: no matching MIL operator to 'quit()'. quit(); !ERROR: interpret: no matching MIL operator to 'quit()'. quit(); !ERROR: interpret: no matching MIL operator to 'quit()'. quit(); !ERROR: interpret: no matching MIL operator to 'quit()'. quit(); !ERROR: interpret: no matching MIL operator to 'quit()'. quit(); !ERROR: interpret: no matching MIL operator to 'quit()'.
And this happens on a "clean" machine with a from scratch installation of MonetDB 4.12.1 from CVS, i.e.,
cd <monetdb-sourcedir> ./bootstrap mkdir <monetdb-builddir> cd <monetdb-builddir> <monetdb-sourcedir>/configure <configure_options> make make check
with <monetdb-builddir>=/home/xsupport/src/monet/build/ in your case ?
To me it looks as if there are no modules yet/any more(?) in /home/xsupport/src/monet/build/src/modules/plain/ and/or /home/xsupport/src/monet/build/src/modules/plain/.libs/ ... ??
Could you please check that?
Stefan
On Oct 3, 2006, at 2:13 PM, Stefan Manegold wrote:
This was with the wrong monetdb present, but the problem also exists with a correct installation.
I managed to catch the head of the failure as well:
Mtest.py: ERROR: 'Mtimeout -timeout 60 Mserver "--config=/home/ xsupport/src/monet/build/conf/MonetDB.conf" --debug=10 --set "monet_mod_path=/home/xsupport/src/monet/build/src/modules/ plain/.libs:/home/xsupport/src/monet/build/src/modules/plain:/home/ xsupport/src/monet/build/src/modules/contrib/.libs:/home/xsupport/ src/ monet/build/src/modules/contrib:/home/xsupport/src/monet/build/src/ modules/calibrator/.libs:/home/xsupport/src/monet/build/src/modules/ calibrator:/home/xsupport/src/monet/build/src/modules/mnetcdf/.libs:/ home/xsupport/src/monet/build/src/modules/mnetcdf:/home/xsupport/src/ monet/build/src/mapi/.libs:/home/xsupport/src/monet/build/src/ mapi" -- set "gdk_dbfarm=/home/xsupport/src/monet/build/dbfarm" --set "sql_logdir=/home/xsupport/src/monet/build/log" --set mapi_port=37305 --set monet_prompt= --trace --dbname=mTests` failed:
!WARNING: GDKlockHome: ignoring empty or invalid .gdk_lock. !WARNING: BBPdir: initializing BBP. # Monet Database Server V4.12.1 # Copyright (c) 1993-2006, CWI. All rights reserved. # Compiled for i686-pc-linux-gnu/32bit with 32bit OIDs; dynamically linked. # Visit http://monetdb.cwi.nl/ for further information. !ERROR: moduleClient: module(builtin) load error. !ERROR: moduleClient: module(arith) load error. !ERROR: moduleClient: module(str) load error. !ERROR: moduleClient: module(constant) load error. !ERROR: moduleClient: module(bat) load error. !ERROR: moduleClient: module(algebra) load error. !ERROR: moduleClient: module(sys) load error. !ERROR: moduleClient: module(trans) load error. quit(); !ERROR: interpret: no matching MIL operator to 'quit()'. quit(); !ERROR: interpret: no matching MIL operator to 'quit()'. quit(); !ERROR: interpret: no matching MIL operator to 'quit()'. quit(); !ERROR: interpret: no matching MIL operator to 'quit()'. quit(); !ERROR: interpret: no matching MIL operator to 'quit()'. quit(); !ERROR: interpret: no matching MIL operator to 'quit()'.
And this happens on a "clean" machine with a from scratch installation of MonetDB 4.12.1 from CVS, i.e.,
cd <monetdb-sourcedir> ./bootstrap mkdir <monetdb-builddir> cd <monetdb-builddir> <monetdb-sourcedir>/configure <configure_options> make make check
with <monetdb-builddir>=/home/xsupport/src/monet/build/ in your case ?
To me it looks as if there are no modules yet/any more(?) in /home/xsupport/src/monet/build/src/modules/plain/ and/or /home/xsupport/src/monet/build/src/modules/plain/.libs/ ... ??
Could you please check that?
For what it's worth, the installed copy seems to work correctly in this regard:
$ /opt/monet-4.12.1/bin/Mserver --dbinit="module(sql_server); sql_server_start();" # Monet Database Server V4.12.1 # Copyright (c) 1993-2006, CWI. All rights reserved. # Compiled for i686-pc-linux-gnu/32bit with 32bit OIDs; dynamically linked. # Visit http://monetdb.cwi.nl/ for further information. MonetDB>quit();
I'll have a closer look at the end of the day.
Ronald
Stefan
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
On Tuesday, October 03, 2006, at 01:22PM, Stefan Manegold Stefan.Manegold@cwi.nl wrote:
'make check' for MonetDB-4.12.1 (downloaded today) complains a lot about:
!ERROR: interpret: no matching MIL operator to 'quit()'. quit();
does this happen with only MonetDB-4.12.1 present on your system, or with a "legacy" MonetDB-4.10.3 installed as well (cf. BTW3 below)?
I've tested again with a clean build of the current snapshot (downloaded half an hour ago), and this gives the same error. Oddly enough running 'make install' and then 'Mtest.py -r' does result in a finished test run. I do get a number of failures:
!ERROR: Testing FAILED SIGNIFICANTLY !
4 out of 313 tests could not be executed 1 out of 313 tests produced slightly different output 6 out of 313 tests produced SIGNIFICANTLY different output
I can send you the test results, I haven't attached them to this message because the zipfile with the results is about 6 MBytes.
What I'm doing (on a SuSE Enterprise Linux 9 system on x86):
$ tar zxf MonetDB-4.12.1.tar.gz $ mkdir build $ cd build $ ../MonetDB-4.12.1/configure '--with-php=no' '--with-perl=no' '--with-python=/opt/python2.4/bin/python' '--with-python-incdir=/opt/python2.4/include/python2.4' '-with-python-library=/opt/python2.4/lib' '--with-python-libdir=/opt/python2.4/lib/python2.4/site-packages' '--prefix=/opt/monet-4.12.1' '--enable-debug' '--disable-optimize' $ make $ make check # -> this results in the 'quit' error mentioned earlier $ make install $ Mtest.py -r # -> works correctly
I'll rebuild without debug and with optimization once I have my application running (which mostly requires some further tweaking of the SQLObject ORM for Python).
On Thu, Oct 05, 2006 at 10:31:17AM +0200, Ronald Oussoren wrote:
On Tuesday, October 03, 2006, at 01:22PM, Stefan Manegold Stefan.Manegold@cwi.nl wrote:
'make check' for MonetDB-4.12.1 (downloaded today) complains a lot about:
!ERROR: interpret: no matching MIL operator to 'quit()'. quit();
does this happen with only MonetDB-4.12.1 present on your system, or with a "legacy" MonetDB-4.10.3 installed as well (cf. BTW3 below)?
I've tested again with a clean build of the current snapshot (downloaded half an hour ago), and this gives the same error. Oddly enough running 'make install' and then 'Mtest.py -r' does result in a finished test run. I do get a number of failures:
!ERROR: Testing FAILED SIGNIFICANTLY !
4 out of 313 tests could not be executed 1 out of 313 tests produced slightly different output 6 out of 313 tests produced SIGNIFICANTLY different output
that looks quite normal some tests are known to fail on some architectures; just compare your result to those of our nightly testing; for "Stable" (MonetDB 4.12.1) http://monetdb.cwi.nl/Development/TestWeb/Stable/index.html http://monetdb.cwi.nl/testing/projects/monetdb/Stable/MonetDB/.mTests103/ind... for "Current" (MonetDB 4.13.1) http://monetdb.cwi.nl/Development/TestWeb/Current/index.html http://monetdb.cwi.nl/testing/projects/monetdb/Current/MonetDB/.mTests103/in...
If the same tests fail, that's fine (i.e., known to us, and considered "non-crucial at least for the "Stable" release). If different tests fail, please let us know, preferably with the differences/errors produced by those test.
I can send you the test results, I haven't attached them to this message because the zipfile with the results is about 6 MBytes.
As said, I don't expect the differences to be "crucial", but if you want a detailed analysis, please feel free to send me your results directly.
What I'm doing (on a SuSE Enterprise Linux 9 system on x86):
$ tar zxf MonetDB-4.12.1.tar.gz $ mkdir build $ cd build $ ../MonetDB-4.12.1/configure '--with-php=no' '--with-perl=no' '--with-python=/opt/python2.4/bin/python' '--with-python-incdir=/opt/python2.4/include/python2.4' '-with-python-library=/opt/python2.4/lib' '--with-python-libdir=/opt/python2.4/lib/python2.4/site-packages' '--prefix=/opt/monet-4.12.1' '--enable-debug' '--disable-optimize' $ make $ make check # -> this results in the 'quit' error mentioned earlier $ make install $ Mtest.py -r # -> works correctly
looks perfectly fine to me --- except from the make check problem --- I have to admit, though, that I never tried make check with the tar ball (only from CVS source so far) --- I'll later try it with the tar ball (from our "Daily Builds"), too...
I'll rebuild without debug and with optimization once I have my application running (which mostly requires some further tweaking of the SQLObject ORM for Python).
That fine for a "production environment" (i.e., better performacne ;-)) but will not have any impact on the failing "make check"...
Stefan
On Thu, Oct 05, 2006 at 11:12:52PM +0200, Stefan Manegold wrote:
On Thu, Oct 05, 2006 at 10:31:17AM +0200, Ronald Oussoren wrote:
On Tuesday, October 03, 2006, at 01:22PM, Stefan Manegold Stefan.Manegold@cwi.nl wrote:
'make check' for MonetDB-4.12.1 (downloaded today) complains a lot about:
!ERROR: interpret: no matching MIL operator to 'quit()'. quit();
[...]
What I'm doing (on a SuSE Enterprise Linux 9 system on x86):
$ tar zxf MonetDB-4.12.1.tar.gz $ mkdir build $ cd build $ ../MonetDB-4.12.1/configure '--with-php=no' '--with-perl=no' '--with-python=/opt/python2.4/bin/python' '--with-python-incdir=/opt/python2.4/include/python2.4' '-with-python-library=/opt/python2.4/lib' '--with-python-libdir=/opt/python2.4/lib/python2.4/site-packages' '--prefix=/opt/monet-4.12.1' '--enable-debug' '--disable-optimize' $ make $ make check # -> this results in the 'quit' error mentioned earlier $ make install $ Mtest.py -r # -> works correctly
looks perfectly fine to me --- except from the make check problem --- I have to admit, though, that I never tried make check with the tar ball (only from CVS source so far) --- I'll later try it with the tar ball (from our "Daily Builds"), too...
Found it!
[For the "insiders":] The problem is that with the source tarballs all .mil parts of the modules have already been extrated from the .mx files and reside readily in source tree. During `make`, they are hence(?) not copied/links to the build tree. RunMtest, however, that is used to run Mtest.py when `make check` is executed, assumes that the modules' .mil files are in the build tree (as they are in case we build from the CVS sources) and sets the monet_mod_path accordingly. Thus, Mserver cannot find the modules' .mil files, and hence no modules.
There are three potential solutions:
1) We need to change the Makefiles to ensure that also with the source tarballs, `make` copies the modules' .mil files from the source tree to the build tree (and least in case they are not identical)
2) We patch configure and RumMtest.in so that RunMtest "knows" whether it use in a "from-CVS" context or a "from-tarball" context, and hence uses either builddir or srcdir, respectively, for the directories in the modpath that contain the .mil files of the modules.
3) We replicate the directories in RunMtest.in's modpath to point to both the source tree and the build tree.
To be honest, I don't know right now how to do 1) or 2) quickly (and correctly), hence, I'll just try 3) --- see my checkins later tonight ...
Stefan
Ronald,
new source tarballs (and RPMs) for the daily builds of "Stable" release (4.12.1) are available at http://monetdb.cwi.nl/testing/projects/monetdb/Stable/.DailyBuilds./
With these, `make check` should work fine --- at least it does for me.
Please let us know, in case you in case you still have problems or encounter new ones.
Stefan
On Thu, Oct 05, 2006 at 11:12:52PM +0200, Stefan Manegold wrote:
On Thu, Oct 05, 2006 at 10:31:17AM +0200, Ronald Oussoren wrote:
On Tuesday, October 03, 2006, at 01:22PM, Stefan Manegold Stefan.Manegold@cwi.nl wrote:
>'make check' for MonetDB-4.12.1 (downloaded today) complains a lot >about: > >!ERROR: interpret: no matching MIL operator to 'quit()'. >quit();
[...]
What I'm doing (on a SuSE Enterprise Linux 9 system on x86):
$ tar zxf MonetDB-4.12.1.tar.gz $ mkdir build $ cd build $ ../MonetDB-4.12.1/configure '--with-php=no' '--with-perl=no' '--with-python=/opt/python2.4/bin/python' '--with-python-incdir=/opt/python2.4/include/python2.4' '-with-python-library=/opt/python2.4/lib' '--with-python-libdir=/opt/python2.4/lib/python2.4/site-packages' '--prefix=/opt/monet-4.12.1' '--enable-debug' '--disable-optimize' $ make $ make check # -> this results in the 'quit' error mentioned earlier $ make install $ Mtest.py -r # -> works correctly
On Oct 6, 2006, at 11:44 AM, Stefan Manegold wrote:
Ronald,
new source tarballs (and RPMs) for the daily builds of "Stable" release (4.12.1) are available at http://monetdb.cwi.nl/testing/projects/monetdb/Stable/.DailyBuilds./
With these, `make check` should work fine --- at least it does for me.
It works for me too.
Thanks, Ronald
On Oct 5, 2006, at 11:12 PM, Stefan Manegold wrote:
On Thu, Oct 05, 2006 at 10:31:17AM +0200, Ronald Oussoren wrote:
On Tuesday, October 03, 2006, at 01:22PM, Stefan Manegold Stefan.Manegold@cwi.nl wrote:
'make check' for MonetDB-4.12.1 (downloaded today) complains a lot about:
!ERROR: interpret: no matching MIL operator to 'quit()'. quit();
does this happen with only MonetDB-4.12.1 present on your system, or with a "legacy" MonetDB-4.10.3 installed as well (cf. BTW3 below)?
I've tested again with a clean build of the current snapshot (downloaded half an hour ago), and this gives the same error. Oddly enough running 'make install' and then 'Mtest.py -r' does result in a finished test run. I do get a number of failures:
!ERROR: Testing FAILED SIGNIFICANTLY !
4 out of 313 tests could not be executed 1 out of 313 tests produced slightly different output 6 out of 313 tests produced SIGNIFICANTLY different outputthat looks quite normal some tests are known to fail on some architectures;
Hmm, wouldn't it be more useful to add some more logic to the testing infrastructure to deal with this? Your testing infrastructure seems to suffer from the same problem as a lot of other test harnasses: tests don't have two possible outcomes (pass and fail) but four: tests can not only pass or fail, but can also be expected to fail and do that or not.
Adding support for testcases that are supposed to fail should be fairly easy and would reduce confusion when someone that isn't part the development team runs the testsuite ;-)
Ronald
Hi Ronald,
indeed, make our testing system more robust would be desirable, however, we lack some man power to do so right now --- it's open source, hence, any contribution is more than welcome ;-)
In fact, the test in question are "known to fail" in some cases/on some platform, but not really "expected to fail", i.e., they are indeed bug --- though minor ones that do not have any severe impact on "normal operation".
Tests (actually queries) that are "supposed to fail" are handled in our testing system by providing the correct/expected error message as stable reference output.
All tests that are "known to fail" actually do so because some system/situation yields different output than others, i.e., the tests themselves are not robust enough to produce identical or at least determenistic output in all cases. Hence, it's actually the tests that need to be changed, not the testing system.
In the stable release, there should not be any tests that "known to fail", i.e., tests for open bugs; those tests should only be in the development version.
To analyse/assess your test results in detail, I'd need to know which tests fail, and what the output resp. the difference to the correct/expected output is.
Stefan
On Tue, Oct 10, 2006 at 10:29:44AM +0200, Ronald Oussoren wrote:
On Oct 5, 2006, at 11:12 PM, Stefan Manegold wrote:
On Thu, Oct 05, 2006 at 10:31:17AM +0200, Ronald Oussoren wrote:
I've tested again with a clean build of the current snapshot (downloaded half an hour ago), and this gives the same error. Oddly enough running 'make install' and then 'Mtest.py -r' does result in a finished test run. I do get a number of failures:
!ERROR: Testing FAILED SIGNIFICANTLY !
4 out of 313 tests could not be executed 1 out of 313 tests produced slightly different output 6 out of 313 tests produced SIGNIFICANTLY different output
that looks quite normal some tests are known to fail on some architectures;
Hmm, wouldn't it be more useful to add some more logic to the testing infrastructure to deal with this? Your testing infrastructure seems to suffer from the same problem as a lot of other test harnasses: tests don't have two possible outcomes (pass and fail) but four: tests can not only pass or fail, but can also be expected to fail and do that or not.
Adding support for testcases that are supposed to fail should be fairly easy and would reduce confusion when someone that isn't part the development team runs the testsuite ;-)
Ronald
On Oct 10, 2006, at 10:59 AM, Stefan Manegold wrote:
Hi Ronald,
indeed, make our testing system more robust would be desirable, however, we lack some man power to do so right now --- it's open source, hence, any contribution is more than welcome ;-)
I know the feeling, I have barely enough time to work on OSS projects that I do know more about (maintance of the Mac port of Python and development on PyObjC).
I'm working on a patch that brings MonetSQLdb more in line with the DB-API 2.0 specification, but I'm afraid that's probably it for the forseeable future.
In fact, the test in question are "known to fail" in some cases/on some platform, but not really "expected to fail", i.e., they are indeed bug --- though minor ones that do not have any severe impact on "normal operation".
That's indeed what I meant, bad english on my part. Sometimes the software doesn't do what you'd like it to do (aka contains bugs), but actually fixing the bug isn't high on the TODO list. Lots test harnasses don't have a way to deal with that, which can be annoying when you want to anything that's related to test driven development.
Tests (actually queries) that are "supposed to fail" are handled in our testing system by providing the correct/expected error message as stable reference output.
All tests that are "known to fail" actually do so because some system/situation yields different output than others, i.e., the tests themselves are not robust enough to produce identical or at least determenistic output in all cases. Hence, it's actually the tests that need to be changed, not the testing system.
In the stable release, there should not be any tests that "known to fail", i.e., tests for open bugs; those tests should only be in the development version.
Thanks for the explanations, I assumed that the stable release contained some tests for open bugs.
To analyse/assess your test results in detail, I'd need to know which tests fail, and what the output resp. the difference to the correct/expected output is.
I haven't looked at that yet, the size of the test output directory scared me :-0.
Ronald
On Oct 10, 2006, at 11:17 AM, Ronald Oussoren wrote:
I'm working on a patch that brings MonetSQLdb more in line with the DB-API 2.0 specification, but I'm afraid that's probably it for the forseeable future.
Speaking of which, what are the contraints w.r.t. supported Python versions? I noticed some code in MonetSQLdb.converters that refers to python 1.5. Is that still needed and if not, what is the minimal version of python that the python bindings should work with?
Ronald
On Tue, Oct 10, 2006 at 11:36:31AM +0200, Ronald Oussoren wrote:
On Oct 10, 2006, at 11:17 AM, Ronald Oussoren wrote:
I'm working on a patch that brings MonetSQLdb more in line with the DB-API 2.0 specification, but I'm afraid that's probably it for the forseeable future.
Speaking of which, what are the contraints w.r.t. supported Python versions? I noticed some code in MonetSQLdb.converters that refers to python 1.5. Is that still needed and if not, what is the minimal version of python that the python bindings should work with?
Indeed 1.5 is very old. We could go up to 2.3.5 (our debian systems have that).
Niels
Ronald
Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=D... _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
On Tue, Oct 10, 2006 at 11:17:13AM +0200, Ronald Oussoren wrote:
On Oct 10, 2006, at 10:59 AM, Stefan Manegold wrote:
[...]
To analyse/assess your test results in detail, I'd need to know which tests fail, and what the output resp. the difference to the correct/expected output is.
I haven't looked at that yet, the size of the test output directory scared me :-0.
Ronald
well, there is a single .html file as entry point that should "quickly" guide you to only the "interesting" results, i.e., those that differ from the expected ones ;-)
Stefan
participants (4)
-
Fabian Groffen -
Niels Nes -
Ronald Oussoren -
Stefan Manegold