December 2007 - users-list

[MonetDB-users] Is there a Front End for SPARQL to MonetDB Backend
by sateesh 21 Dec '07

21 Dec '07

HI All, I am wondering whether any effort is made to build a frontend which accepts SPARQL Queries and convers them into MIL for the Monet Back-end ?? Or how difficult to use MIL Directly in a Programming language like Java?? (Which is basically developing the front-end) If such front-end accepts either SQL or XQuery etc then it has to be a Query Parser and Optimizer . How ever if it uses MIL Directly then i guess we don't need a Query Parser and Optimizer..!! Any comments or info reg SPARQL Related front-ends will be of great help Thanks Sateesh -- View this message in context: http://www.nabble.com/Is-there-a-Front-End-for-SPARQL-to-MonetDB-Backend-tp… Sent from the monetdb-users mailing list archive at Nabble.com.

3 2

[MonetDB-users] Problem with running TPC-H query No.5
by Konstantinos Krikellas 18 Dec '07

18 Dec '07

I am using MonetDB for benchmarking on the TPC-H workload. I have successfully run queries 1, 3 and 10. However, when I try to run query 5: select n_name, sum(l_extendedprice * (1 - l_discount)) as revenue from customer, orders, lineitem, supplier, nation, region where c_custkey = o_custkey and l_orderkey = o_orderkey and l_suppkey = s_suppkey and c_nationkey = s_nationkey and s_nationkey = n_nationkey and n_regionkey = r_regionkey and r_name = 'ASIA' and o_orderdate >= date '1994-01-01' and o_orderdate < date '1994-01-01' + interval '1' year group by n_name order by revenue desc; it takes about two (2) hours to evaluate it. It is obvious that the query cannot be executed inside the main memory, so the system thrashes through paging. Still, this behaviour does not appear for the rest of the queries, as they are computed within one or two seconds. In all cases, the results are correct. The specifications of my system are: Processor: Intel Core 2 Duo @ 1.86 Mhz RAM: 2 Gb Operating System: Linux Ubuntu Gutsy 7.10 MonetDB version: Mserver5 (recently downloaded from your site). Can you give me some indications about why this query takes too long and how can I bypass this response delay? Thanks.

2 4

[MonetDB-users] install script (monetdb-install.sh) doesn't work for current branch
by Ying Zhang 18 Dec '07

18 Dec '07

Hello, I try to install the nightly build of current branch using the install script "monetdb-install.sh" (just downloaded from http://monetdb.cwi.nl/projects/monetdb/Assets/monetdb-install.sh) , but it stops when trying to rename the directory for MonetDB- client-1.21.0 I run the script with the following options: monetdb-install.sh --prefix=$HOME/monetdb --build=$HOME/tmp/ monetdb_build --enable-xquery --nightly=current --enable-debug The log is attached below. A request: would the owner of the script please add the new module "java" in to the script? Thanks a lot! Jennie ============================== $ cat monetdb_build/log Fri Dec 7 12:37:45 CET 2007 called with arguments: --prefix=/export/scratch0/streettivo/monetdb -- build=/export/scratch0/streettivo/tmp/monetdb_build --enable-xquery -- nightly=current --enable-debug --12:37:46-- http://www.ins.cwi.nl/~monet/tars/latest/current/MonetDB.tar.gz Resolving www.ins.cwi.nl... 192.16.196.21 Connecting to www.ins.cwi.nl|192.16.196.21|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 1198570 (1.1M) [application/x-gzip] Saving to: `/export/scratch0/streettivo/tmp/monetdb_build/MonetDB/ MonetDB.tar.gz' 0K ........ ........ .. 100% 4.85M=0.2s 12:37:46 (4.85 MB/s) - `/export/scratch0/streettivo/tmp/monetdb_build/ MonetDB/MonetDB.tar.gz' saved [1198570/1198570] unpacking MonetDB.tar.gz --12:37:46-- http://www.ins.cwi.nl/~monet/tars/latest/current/MonetDB-client.tar.gz Resolving www.ins.cwi.nl... 192.16.196.21 Connecting to www.ins.cwi.nl|192.16.196.21|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 920374 (899K) [application/x-gzip] Saving to: `/export/scratch0/streettivo/tmp/monetdb_build/MonetDB- client/MonetDB-client.tar.gz' 0K ........ ...... 100% 32.9M=0.03s 12:37:46 (32.9 MB/s) - `/export/scratch0/streettivo/tmp/monetdb_build/ MonetDB-client/MonetDB-client.tar.gz' saved [920374/920374] unpacking MonetDB-client.tar.gz mv: cannot stat `clients-[0-9]*': No such file or directory failed to rename sources to the right name Please refer to the full log at /export/scratch0/streettivo/tmp/ monetdb_build/log If you believe this is an error in the script or the software, file a bug and attach the logfile. Fri Dec 7 12:37:46 CET 2007 The following tools might be relevant to debug your problem: : 1.15.4.6 $ /usr/bin/autoconf autoconf (GNU Autoconf) 2.59 Written by David J. MacKenzie and Akim Demaille. Copyright (C) 2003 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. /usr/bin/automake automake (GNU automake) 1.9.6 Written by Tom Tromey <tromey(a)redhat.com>. Copyright 2005 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. /usr/bin/libtool ltmain.sh (GNU libtool) 1.5.22 (1.1220.2.365 2005/12/18 22:14:06) Copyright (C) 2005 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. glibtool cannot be found in the path /usr/lib64/ccache/gcc gcc (GCC) 4.1.2 20070626 (Red Hat 4.1.2-13) Copyright (C) 2006 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. /usr/bin/ld GNU ld version 2.17.50.0.6-5.fc6 20061020 Copyright 2005 Free Software Foundation, Inc. This program is free software; you may redistribute it under the terms of the GNU General Public License. This program has absolutely no warranty. /usr/bin/make GNU Make 3.81 Copyright (C) 2006 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. This program built for x86_64-redhat-linux-gnu /usr/bin/gmake GNU Make 3.81 Copyright (C) 2006 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. This program built for x86_64-redhat-linux-gnu /usr/bin/python 2.4.4 (#1, Oct 23 2006, 13:58:18) [GCC 4.1.1 20061011 (Red Hat 4.1.1-30)] /bin/gawk GNU Awk 3.1.5 Copyright (C) 1989, 1991-2005 Free Software Foundation. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. /bin/sed GNU sed version 4.1.5 Copyright (C) 2003 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE, to the extent permitted by law. gsed cannot be found in the path

4 17

Re: [MonetDB-users] MonetDB size limitation questions.
by Martin Kersten 17 Dec '07

17 Dec '07

Thomas Backlund wrote: > > Hi, > Hi Thomas. Thanks for your interest in MonetDB. > > I stumbled over MonetDB a few days ago and it really cought my > interest. But it is hard to find information about it, so I hope You > could answer a few questions. > > I need a database engine that could handle many rows, about 350*10^9 > rows of about 1000TB data. > This is really large, do you really mean 1 PetaByte? For MonetDB the first interesting part is that it will automatically split the table into 350 tables of 1000.000.000 rows, each would then be be 3 TB.. which means you are using a special column type? I think we need a snippet of the relation declaration to understand your schema We are currently working on automatic horizontal fragmentation with distribution to indeed handle them as single tables. To understand if MonetDB is the solution to your problem, it is also relevant to know what kind of queries you intend to run. We know there are users moving MonetDB into this direction (sizewise) and we would certainly be interested in learning the problems encountered. Given, of course, you have the time to help us on this track and you know of any database system that can already solve the problem out of the box. regards, Martin Kersten > > > Well I dont really need it right now, but I want to be prepared. > > My option up til now has been to use MySQL and to do some kind of > sharding using alot of databases. > But that complicates stuff alot since I still want to do selects in > this gigantic table and that will be a bit of a problem doing selects > if the data is split up in 8000 different MySQL databases. > > Do You think MonetDB could handle this in one single table? > Even if MonetDB could handle this I still will need a hard disk of > 1000TB, do You know of any technique merging many computers together > into one virtiualized hard drive that MonetDB would accept? > > Thank You very much! > > Best regards > Thomas Backlund, > Sweden. >

1 0

Re: [MonetDB-users] Shell script to start MonetDB as service on Linux
by McKennirey.Matthew 13 Dec '07

13 Dec '07

Fabian, thanks for your quick reply, and pointing me in the right direction. We are running on OpenSuse, and for the moment have settled on the following script. I'm sure it could use a lot of improvement, but we are up and running ********** !/bin/sh # Portions Copyright (c) 1995-2002 SuSE Linux AG Nuernberg, Germany. # Author: Lenz Grimmer <feedback(a)suse.de> # # /etc/init.d/monetdb ### BEGIN INIT INFO # Provides: monetdb # Required-Start: $network $remote_fs # Required-Stop: # Default-Start: 2 3 5 # Default-Stop: # Description: Start the MonetDB database server ### END INIT INFO # Shell functions sourced from /etc/rc.status: # rc_check check and set local and overall rc status # rc_status check and set local and overall rc status # rc_status -v ditto but be verbose in local rc status # rc_status -v -r ditto and clear the local rc status # rc_failed set local and overall rc status to failed # rc_failed <num> set local and overall rc status to <num> # rc_reset clear local rc status (overall remains) # rc_exit exit appropriate to overall rc status /etc/rc.status # First reset status of this service rc_reset # Return values acc. to LSB for all commands but status: # 0 - success # 1 - generic or unspecified error # 2 - invalid or excess argument(s) # 3 - unimplemented feature (e.g. "reload") # 4 - insufficient privilege # 5 - program is not installed # 6 - program is not configured # 7 - program is not running # # update these as appropriate for your server MDB_USER='root' MDB_GROUP='root' MDB_HOME=/srv/monetdb MDB_BINDIR=/srv/monetdb/bin echo "" echo "Using configuration at " $MDB_HOME/etc/monetdb5.conf echo "" # Test, if path to merovingian is valid unset MEROVINGIAN if test -x $MDB_BINDIR/merovingian then MEROVINGIAN=$MDB_BINDIR/merovingian fi test "$MEROVINGIAN" || { echo "MonetDB daemon (merovingian) is not available at indicated path:" $MDB_BINDIR "verify path settings in /etc/monetdb"; rc_failed 5; rc_status -v; rc_exit; } depend() { use net } start(){ echo -n "Starting MonetDB/SQL" startproc -s -u $MDB_USER -g $MDB_GROUP $MDB_BINDIR/merovingian # Rmember status and be verbose rc_status -v } stop() { echo -n "Stopping MonetDB/SQL" # shut down mserver if it is running kill -INT $(pidof mserver5) # stop merovingian daemon kill -INT $(pidof merovingian) # Remember status and be verbose rc_status -v } status(){ echo -n "Checking for MonetDb service: " # NOTE: checkproc returns LSB compliant status values. checkproc $MEROVINGIAN rc_status -v } case "$1" in start) start ;; stop) stop ;; status) status ;; restart) stop start ;; *) echo "Usage: {start|stop|status|restart}" exit 1 ;; esac rc_exit

2 1

Re: [MonetDB-users] connecting to monetDB from a Java program
by Ying Zhang 13 Dec '07

13 Dec '07

Hello Gerben, On Dec 13, 2007, at 15:36 , <e.g.broenink(a)student.utwente.nl> <e.g.broenink(a)student.utwente.nl > wrote: > Hello Jennie, > > I got the system working, I can write SOAP messages and execute them > on the server. However there is a new problem. > I'm using the system to execute a few thousand queries, and the > problem is that MonetDB is caching a lot of (intermediate) results, > therefore, after a few honderd queries the server has cached over > 800MB, and the system becomes very slow. Mmm, this is annoying. Do you mean that MonetDB is using more memory ("over 800MB") or is the size of you dbfarm growing? If the memory usage is growing, it sounds like a leak, but I don't know if XRPC is eating up your memory, or is MonetDB doing that (so I cc this e-mail to the monetdb-users mailing-list). It's an interesting problem to investigate. Would you please provide us more information about your problem? - the xml data you are working on - the xquery functions you are using, or are causing your problems - the request messages you use - which version of MonetDB4 are you using? - which OS (and it version) are you using? > I would like to restart the server after a few honderd queries, do > you know how I can do that? (can I send a SOAP message to restart?) > Or is there another way of solving this problem? No, it is not possible to send SOAP messages to restart an Mserver. You might be able to send MIL commands to stop Mserver to stop it, and use a script to check regularly if your Mserver is still running, and (re)start Mserver if it is not running (any more). Jennie

1 0

[MonetDB-users] Shell script to start MonetDB as service on Linux
by Matthew McKennirey 13 Dec '07

13 Dec '07

Is anyone aware of any UNIX shell scripts (bash, bourne, C) that exist to start the MonetDB server as a service on Linux (/etc/init.d). I have successfully (and, well done to the team, painlessly) installed MonetDB from the MonetDB-Mars-SuperBall-SR4.tar.bz2 and tried out the VOC demo. All is well. However I am struggling with merovingian and monetdb trying to figure out how to get things to start correctly in a scripted manner so that the database is ready when the server finishes booting. I have searched the users list, and others and read presentations on the issue of painlessly starting MonetDB (/etc/init.d/monetdb start) but am no further ahead. If anyone has a script I would be grateful. Thanks

2 1

[MonetDB-users] Shell script to start MonetDB as service on Linux
by McKennirey.Matthew 13 Dec '07

13 Dec '07

1 0

[MonetDB-users] Info reg Column Oriented DB
by sateesh 13 Dec '07

13 Dec '07

Hi All, I am evaluating MonetDB for my project and we are planning to Store Semantic Web related info which is basically in RDF Format and wants to use a Colum Oriented DB for this purpose.. Now when i looked at the Sample DB VOC that comes with DB most of the tables have more than two columns. So is this sample (voc) not meant for semantic storage .. Thanks Sateesh -- View this message in context: http://www.nabble.com/Info-reg-Column-Oriented-DB-tp14303148p14303148.html Sent from the monetdb-users mailing list archive at Nabble.com.

2 1

[MonetDB-users] Persistent Stored Modules ?
by Venks 08 Dec '07

08 Dec '07

I need to perform some ETL processing for which I need CURSOR support in a PSAM. I read the meta-data in a CURSOR and dynamically create SQL statements for execution. According to documentation CURSORS are not supported because of the underlying data model in a PSM. This means I can't use PSM for my ETL processing. Let me know if there is anything I could use a PSM in my ETL process. I only wish you had a driver for Ruby. I see that you have PERL, Python and PHP. It would be great if we have a Ruby Library (:- Regards,

3 2