March 2009 - users-list

[MonetDB-users] indexing
by Yue Sheng 20 Mar '09

20 Mar '09

In light of the MonetDB SQL indexing: "....MonetDB/SQL interprets these statements as an advice and often freely neglects it, relying on its own decision to create and maintain indexes for fast access...." Does it mean there's no need for the user to explicitly create any index?

1 0

[MonetDB-users] Open source Column based DB
by Yue Sheng 20 Mar '09

20 Mar '09

I was wondering if anyone know of other column-based DBs suited to vase amount of time series data? Tried Googling. It'd be nice to hear from someone who has any insight on this. Thanks.

6 12

Re: [MonetDB-users] Open source Column based DB ... for timeseries
by Jens Oehlschlägel 20 Mar '09

20 Mar '09

> There is one other open source column based DB that might be worth > looking at: Infobright Community Edition (http://www.infobright.org/) > But if you want the moon on a stick, have a look at Esper, a realtime > streaming database that supports the kinds of queries you mentioned > in the UDF examples If you want to look at this from a statistical point of view: the statistical interpreter R (www.r-project.org) has now on its package repository (cran.r-project.org) package 'ff', which allows fast memory-mapped access to binary vector and matrix files. Amongst the many time-series packages R has, one ('kza') started exploiting package 'ff' for large time-series. And when it comes to quickly combining filtering conditions on large vectors, package 'bit' might interest you. Jens Oehlschlägel ( author of 'ff' and 'bit' :-)

2 2

[MonetDB-users] Windows 64 bit / PHP
by dariuszs 19 Mar '09

19 Mar '09

Hi, How do I get PHP library on Windows 64 bit? Thanks. Dariusz.

2 5

Re: [MonetDB-users] Windows 64 bit / PHP
by dariuszs 19 Mar '09

19 Mar '09

Hi, This is the error that I'm getting: PHP Warning: PHP Startup: Unable to load dynamic library 'C:\\xampp\\php\\ext\\php_monetdb.dll' - The specified module could not be found.\r\n in Unknown on line 0 Thanks for your help. Dariusz.

1 1

[MonetDB-users] Status DateTime support in MonetDB XQuery
by Hans van Rijswijk (DT) 19 Mar '09

19 Mar '09

Hi Monet development team, Can anyone (e.g. lsidir) tell me what the status is of the development on the date/time support in xquery. If the development has been finished: - In which version of Mserver can I find a working implementation, e.g. source tarball or Fedora 10 RPMs (URL, version, date, stable, daily build)? - How does an xml representation of a date time object look like? - What XQuery extensions can I use to address / compare date objects? - Do you know of any performance improvement / degradation with respect to using date objects? If the development has not been finished: - Same questions, and when is the release planned? Regards, Hans van Rijswijk.

3 4

[MonetDB-users] parallel loading
by Yue Sheng 18 Mar '09

18 Mar '09

under 2009 release note, I see "Known issues in this release" pertaining to buggy parallel loading. I was wondering (1) what is the manifestation of this error? (2) does this bug also happen in the "nov 2008 sp2"? thx

1 0

Re: [MonetDB-users] 64bit MonetDB, JDBC Insert via RJDBC, >300 million rows
by Martin Kersten 18 Mar '09

18 Mar '09

Yue Sheng wrote: > Martin, > > It almost worked... > > This is what I did and what have happened: > > I have 322 files to insert into data base, totaling 650 million rows > > I divided the file list into two, then for each sub list > > (a) I insert first file in the list with N set to 650milllion rows, > (b) all subsequent files have N set to the number of lines in *that* file > > once first list done, then > > (c) I insert first file in the second list with N set to 650million rows, > (d) all subsequent files have N set to the number of lines in *that* file > > Then the same problem happened: it stucked at file number 316. ok. using the 650M enables MonetDB to allocate enough space and does not have to fall back on guessing. Guessing is painful, because when a file of N records has been created and it needs more, it makes a file of size 1.3xN. This leads to memory fragmentation. in your case i would have been a little mode spacious and used 700M as a start, because miscalculation of 1 gives a lot of pain. Such advice is only needed in (a) > > Note: This is farther then previous tries, which all stopped in the > region of file number 280 +/- a few. > > My observation: > (i) at (a), the VSIZE went up to around 46GB, then after first insert, > it drops to around 36GB ok fits > (ii) at (c), the VSIZE went up to around 130GB, then after first insert, > it drops to around 45GB you tell the system to extend existing BATs prepare for another 650 M, which means it allocates 2*36 G, plus room for the old one gives 108GB then during processings some temporary BATs may be needed,e.g. to check integrity constraints after each file,. Then it runs out of swapspace. > (iii) the "Free Memory", as reported by Activity Monitor, just before it > failed at file number 316, dipped to as low as 7.5MB! yes, you are running out of swapspace on your system. This should not have happened, because the system uses mmapped files and may be an issue with the MacOS or relate to a problem we fixed recently > > My question: > (1) why we need to ramp N up to total number of lines (it takes along > time to do that), then only have it drop down to 30GB-40GB right after this might indicate that on MacOS, just like Windows, mmaped files need to be written to disk. With a disk bandwidth of 50MB/sec it still takes several minutes > the first insert and stay roughly there? Does it mean we're giving back > all the pre-allocation space back to the OS? Then should we set N always > to total number of lines? If so, it would take much much longer to > process all the files... > (2) How come RSIZE never goes above 4GB? > (3) Does sql log file size have some limit, that we need to tweak? no limit > (4) Has anyone successfully implemented the 64bit version of MondeDB and > successfully inserted more than 1billion rows? you platform may be the first, but Amherst has worked with Macs for years > (5) when you say you "...The VSIZE of 44G is not too problematic, i > am looking at queries letting it tumble between 20-80 GB....," What does > it mean? Mine went up to as high as 135GB... explained above. regards, Martin > > Thanks, as always. > > > >

2 1

[MonetDB-users] 64bit MonetDB, JDBC Insert via RJDBC, >300 million rows
by Yue Sheng 17 Mar '09

17 Mar '09

I'm been trying to insert from file for over 322 files, totally 12GB with over 600 million rows. The RJDBC connection hangs after 350 million row. I noticed the VSIZE in Activity Monitor is 30.85GB, closer to 32GB system memory. Is this the reason that I cannot connect to the database? What should I do? Error message I get from MonetDB server: # MonetDB server v5.8.4, based on kernel v1.26.4 # Serving database 'bigdata', using 8 threads # Compiled for i386-apple-darwin9.6.0/64bit with 64bit OIDs dynamically linked # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008- MonetDB B.V., all rights reserved # Visit http://monetdb.cwi.nl/ for further information #warning: please don't forget to set your vault key! #(see /Users/someuser/MonetDB64/etc/monetdb5.conf) # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ # MonetDB/SQL module v2.26.4 loaded >!ERROR: GDKsave: failed name=24/2460, ext=tail, mode 0 !ERROR: GDKsave: failed name=23/2317, ext=tail, mode 0 !ERROR: GDKsave: failed name=25/2503, ext=tail, mode 0 !ERROR: GDKsave: failed name=26/2657, ext=tail, mode 0 !ERROR: GDKsave: failed name=22/2274, ext=tail, mode 0 !ERROR: GDKsave: failed name=22/2224, ext=tail, mode 0 !ERROR: GDKsave: failed name=01/112, ext=head, mode 0 !ERROR: logger_exit: could not open /Users/someuser/MonetDB64/var/MonetDB5/sql_logs/bigdata/sql/log System:MacPro Dual Quad Xenon Leopard OS X 32GB RAM R 2.8.1 32Bit MonetDB v5.8.4 complied for i386-apple-darmin9.6.0/64bit Many thanks. Yuesheng

4 28

[MonetDB-users] Apples and oranges
by Nils Grimsmo 17 Mar '09

17 Mar '09

Hi! I have some questions as to how MonetDB/XQuery should be compared fairly to other systems. If I re-run a query multiple times in a single call to `mclient`, is any calculation re-used? How about if I run multiple similar queries in a single call? Example: $ cat www.xq count(doc("dblp")//www) $ cat www_s10.xq (count(doc("dblp")//www), count(doc("dblp")//www), count(doc("dblp")//www), count(doc("dblp")//www), count(doc("dblp")//www), count(doc("dblp")//www), count(doc("dblp")//www), count(doc("dblp")//www), count(doc("dblp")//www), count(doc("dblp")//www)) $ cat www_s100.xq (count(doc("dblp")//www), ... count(doc("dblp")//www)) $ mclient --language=xquery --time < www.xq 11760 Timer 22.552 msec (Assert we are running hot.) $ mclient --language=xquery --time < www.xq 11760 Timer 21.661 msec $ mclient --language=xquery --time < www_s10.xq 11760, [snip] 11760 Timer 33.063 msec $ mclient --language=xquery --time < www_s100.xq 11760, [snip] 11760 Timer 252.414 msec So the average execution times are 22, 3.3 and 2.5 milliseconds. Is the extra cost for the first query just starting up the client program, or is some calculation re-used? If we now look at more expensive queries: $ cat dblp_authors.xq count(doc("dblp")/dblp//author[text()="Grzegorz Rozenberg"]) Just repeating the same: $ cat dblp_authors_s10.xq (count(doc("dblp")/dblp//author[text()="Grzegorz Rozenberg"]), count(doc("dblp")/dblp//author[text()="Grzegorz Rozenberg"]), count(doc("dblp")/dblp//author[text()="Grzegorz Rozenberg"]), count(doc("dblp")/dblp//author[text()="Grzegorz Rozenberg"]), count(doc("dblp")/dblp//author[text()="Grzegorz Rozenberg"]), count(doc("dblp")/dblp//author[text()="Grzegorz Rozenberg"]), count(doc("dblp")/dblp//author[text()="Grzegorz Rozenberg"]), count(doc("dblp")/dblp//author[text()="Grzegorz Rozenberg"]), count(doc("dblp")/dblp//author[text()="Grzegorz Rozenberg"]), count(doc("dblp")/dblp//author[text()="Grzegorz Rozenberg"]), count(doc("dblp")/dblp//author[text()="Grzegorz Rozenberg"])) Different but related queries: $ cat dblp_authors_x10.xq (count(doc("dblp")/dblp//author[text()="Grzegorz Rozenberg"]), count(doc("dblp")/dblp//author[text()="Wen Gao"]), count(doc("dblp")/dblp//author[text()="Irith Pomeranz"]), count(doc("dblp")/dblp//author[text()="Hector Garcia-Molina"]), count(doc("dblp")/dblp//author[text()="Moshe Y. Vardi"]), count(doc("dblp")/dblp//author[text()="Joseph Y. Halpern"]), count(doc("dblp")/dblp//author[text()="Noga Alon"]), count(doc("dblp")/dblp//author[text()="Wei Li"]), count(doc("dblp")/dblp//author[text()="Ming Li"]), count(doc("dblp")/dblp//author[text()="Donald F. Towsley"]) ) $ mclient --language=xquery --time < dblp_authors.xq 351 Timer 1238.436 msec $ mclient --language=xquery --time < dblp_authors.xq 351 Timer 1253.927 msec $ mclient --language=xquery --time < dblp_authors_s10.xq 351, ... 351 Timer 1284.191 msec $ mclient --language=xquery --time < dblp_authors_x10.xq 351, 347, 346, 341, 334, 334, 330, 320, 320, 317 Timer 2610.589 msec Here the average times are 1238, 128 and 261 milliseconds. Here the difference is clearly not just startup of the client. If this was not a client-server architecture, I would guess the difference came from opening files, getting stuff into cache, etc.. Are there similar reasons here? What parts of the calculations are actually done inside the client, if any? If the answer is none, why is this behavior seen? In conclusion: When running multiple queries, what would be the most fair way to compare MonetDB/XQuery to other client/server architectures in your view? Concatenating the queries in a single call to `mclient`, or multiple calls? When timing a single query, can it be repeated multiple times in a single call, and the average taken, without being unfair? If I use for example MS SQL Server 2008, there is no gain from a single invocation of the client, whether I have multiple SQL statements SELECT x.query('$q) FROM t; ..., SELECT x.query('$q) FROM t; Or a single SQL statement with a list of XPath queries SELECT x.query('($q, ..., $q)') FROM t; Klem fra Nils -- http://www.idi.ntnu.no/~nilsgri/ Why is this thus? What is the reason of this thusness? - Artemus Ward

4 4