Hello,
I am interested in evaluating MonetDB/XQuery. I have some XML collections
for a project I work on that are of 1-2 million individual XML documents
for which I use XQuery to access for OLAP style reporting operations. This
is a Java-based project, and while working on this project I developed a
thin XQuery-centric API so that I could evaluate/use many different XQuery
implementations, which my company released as an open source project
called Xaj (http://sourceforge.net/projects/xaj). Currently I am using a
Berkeley DB XML implementation, but am interested in exploring other
options.
I'd like to implement a MonetDB version of this API now, but I am having
some difficulty finding out how to effectively access MonetDB through
Java. So far I was able to use the MonetDB JDBC driver to construct XQuery
statements and get XML results, and also I was able to construct MonetDB
pf:add-doc() statements to add documents, but I am wondering if this is
the best approach (and if so, how to add documents to a remote MonetDB
server using the pf:add-doc() mechanism).
The API I am trying to implement you can view here, to give you an idea
what I'm trying to do:
http://xaj.svn.sourceforge.net/viewvc/xaj/xaj/src/net/sf/xaj/XmlDb.java?rev…
It was somewhat modeled after the defunct XML:DB API, but focused just on
add/store/XQuery operations. Any help/advise would be greatly appreciated!
-- m@
Is there any method, besides restarting the database,
of shrinking the sql log file?
When I am performing a large set of large inserts (via
COPY INTO) the log file gets very large. I wish there
was a way to cycle the log file.
Also during these large batches of inserts the
database server becomes less and less responsive until
I am forced to restart the server. Upon restart and
the normal recovery monet returns to a very responsive state.
Hi all,
I am new to monetDB and ran into the following problem (I'll try to be as
verbose as I can, perhaps it helps helping me):
I've installed MonetDB server 5.0.0, clients and commons 1.18.2 and sql
2.18.2 on Ubuntu Feisty. Then I've taken the JDBC example which is available
online and modified it to be able to insert data into a test database. More
specifically I'm using prepared statements and executeBatch() together with
the monetdb-1.6 JDBC driver.
Which is quite successful until about 5 million records are inserted -
afterwards the speed of the inserts drops dramatically.
I've experimented with different batch sizes (10000 records seems to work
ok), and I've tried to understand the chapter about memory tuning for
MonetDB (not sure yet about how to use mem_cursize() though).
In any case at first inserting 2 million records only takes about 5 minutes
(measured with Javas System.timemillis), later on this time increases and
the server gets rather unresponsive.
I found only two posts concerning the speed of JDBC inserts (from January
2007 I think) and I would be interested if a solution or explanation has
been found off-list.
Thank you
--
View this message in context: http://www.nabble.com/JDBC-insert-performance-tf4589411.html#a13100422
Sent from the monetdb-users mailing list archive at Nabble.com.
Recent installation of MonetDB showed the following text at the end of
the installation.
====================================
You can start using MonetDB/SQL by starting
/root/countdb/bin/merovingian
in the background. Read the merovingian(1), monetdb(1) and
mclient(1) manpages to learn how to create a database and
afterwards connect to it.
====================================
Is "merovingian" ready for usage?
When I type "merovingian" I am getting the following error:
===============================================================================
unable to create .merovingian_lock file in
/root/countdb/var/MonetDB5/dbfarm: No such file or directory
===============================================================================
That's because there is no MonetDB5 directory yet. This may be of
importance to us as I need to do the following:
- Run MonetDB in background
- Create the database directory, LOG, CheckPoint and any other files
related to any database in a user specified directory.
Right now I am using mserver5 executable in a script to start MonetDB
in the background. This script was provided by you (Fabian)
Please let me know if I need to start using "merovingian" for my needs.
Thanks,
-Venks
I thought this might be of interest to some of you.
New Java releases are normally upwards binary-compatible with prior
releases with only few minor/niche exceptions.
This implies that one should normally be able to use recent JVM versions
to run MonetDB's (JDBC) class/jar files, even if these were compiled
with an older JDK version.
See: http://java.sun.com/j2se/1.5.0/compatibility.html
and: http://java.sun.com/javase/6/webnotes/compatibility.html#binary
Note that the opposite does not hold: class/jar files cannot be run with
a JVM older than that from the JDK used to compile them. (Every newer
JDK marks its generated class files with a different class file version
number, which is checked by the JVM.)
I hope this helps.
Sander Kruseman
Hi,
Using server v5.0.0 on Windows with the JDBC terminal v1.18.0, SQL DELETE
statements appear to fail. Working in autocommit mode, I issue the
following commands:
create table test (id int unique not null, name varchar(10) not null);
insert into test (id, name) values (1, 'jan');
delete from test where id = 1;
select * from test;
I get no rows, as was to be expected. However, using DbVisualizer (also
using JDBC) I can still see the one inserted table row (after Reload
View).
After restarting both server and JDBC terminal, I also get the row when
using the JDBC terminal. Bug or ignorance?
Note that I am using my own schema having been setup with its own user,
similar to the VOC tutorial.
kind regards,
Sander Kruseman
I am trying to create a large database almost 10GB (TPC-H).
When I try to bulk load a 7GB file the server does not work anymore. First,
the server quits in the middle of the bulk load, then when I try to re-start
the server it does not start showing a message about the OS not having
enough space.
Then I split the 7GB file into individuals 1GB files, after I upload around
4GB individually, the server crashes. Then when I re-start the server it
does not start stating that the BACKUP folder cannot be accessed and not
enough permissions!
I am using MonetDB 5 on Windows XP 1GB RAM.
Any ideas?
Thank You.
alan
--
View this message in context: http://www.nabble.com/Large-Databases-tf4710020.html#a13462531
Sent from the monetdb-users mailing list archive at Nabble.com.
I am using MonetDB 5.0.0 and 1GB Ram on Windows XP. I am trying to use the
TPC-H benchmark.
With 1GB of test data everything worked fine, but if I try to load a table
with more than 1GB of data the server crashes. Does this happen due to the
1GB of RAM?
Also, when I use the number of records with the COPY command it copies more
records.
Any pointers?
Thanks
--
View this message in context: http://www.nabble.com/Bulk-Load-Problem-tf4702911.html#a13442730
Sent from the monetdb-users mailing list archive at Nabble.com.
Hi list,
I was just doing some basic queries on MonetDB5/SQL to see if it is
suitable for my application, I'm doing lots of aggregates on some
logfile-abstractions. Basically they all boil down to 'how many unique
visitors and total pageviews where there in period X-Y in section Z'.
I have this table:
pageviews (
timestamp timestamp not null,
clientip varchar(15) not null,
sectionid smallint not null,
itemid integer not null,
channelid smallint default 0
)
Currently it only contains data for last september, with about 2M
records/day, and 5.6M in total.
There are no additional indexes in this case.
When doing a query like this, monetdb very fast. Once the data is in the
memory cache, it returns (according to trace) in about half a second.
select count(*) from pageviews
where timestamp between '2007-09-21 00:00:00' and '2007-09-22 00:00:00';
The result is 1916813
This one is also pretty fast, taking about 1.7 second
select count(distinct clientip) from pageviews
where timestamp between '2007-09-21 00:00:00' and '2007-09-22 00:00:00';
The result is 165700
And the third which is pretty fast:
select channelid, count(*) from pageviews
where timestamp between '2007-09-21 00:00:00' and '2007-09-22 00:00:00'
group by channelid;
Here's the distribution, and its returned in about 0.6 second
[ 0, 538187 ]
[ 1, 1108478 ]
[ 4, 42867 ]
[ 3, 145565 ]
[ 2, 81716 ]
But when I combine those last two queries, the result isn't returned in
a reasonable amount of time, I waited for more than half an hour and it
still hadn't returned the results.
select channelid, count(distinct clientip), count(*)
from pageviews
where timestamp between '2007-09-21 00:00:00' and '2007-09-22 00:00:00'
group by channelid;
I'm not very good at reading your explain output yet, so I've attached
the resulting explain for that query.
Is there a way to speed up this type of query? It seems a bit odd that
it's taking more than half an hour (postgresql does it in about 20
seconds) while the other queries return much faster (postgresql does
them in about 14 seconds).
Best regards,
Arjen
function user.s5_1():void;
_1:bat[:oid,:timestamp]{rows=55914459:lng,bid=1968} := sql.bind("sys","pageviews","timestamp",0);
_10:bat[:oid,:timestamp]{rows=0:lng,bid=2323} := sql.bind("sys","pageviews","timestamp",1);
constraints.emptySet(_10);
_10:bat[:oid,:timestamp]{rows=0:lng,bid=2323} := nil;
_12:bat[:oid,:timestamp]{rows=0:lng,bid=2324} := sql.bind("sys","pageviews","timestamp",2);
constraints.emptySet(_12);
_12:bat[:oid,:timestamp]{rows=0:lng,bid=2324} := nil;
_17:bat[:oid,:sht]{notnil=true,rows=55914459:lng,bid=1969} := sql.bind("sys","pageviews","channelid",0);
_24:bat[:oid,:str]{rows=55914459:lng,bid=1961} := sql.bind("sys","pageviews","clientip",0);
_8{rows=27957230:lng} := algebra.uselect(_1,2007-09-21 00:00:00.000,2007-09-22 00:00:00.000,true,true);
_1:bat[:oid,:timestamp]{rows=55914459:lng,bid=1968} := nil;
_15{rows=27957230:lng} := algebra.markT(_8,0@0);
_8{rows=27957230:lng} := nil;
_16{rows=27957230:lng} := bat.reverse(_15);
_15{rows=27957230:lng} := nil;
_19{rows=27957230:lng} := algebra.join(_16,_17);
(ext40,grp38):= group.new(_19);
_19{rows=27957230:lng} := nil;
_22 := bat.mirror(ext40);
ext40 := nil;
_23 := algebra.joinPath(_22,_16,_17);
_17:bat[:oid,:sht]{notnil=true,rows=55914459:lng,bid=1969} := nil;
_26{rows=27957230:lng} := algebra.join(_16,_24);
_16{rows=27957230:lng} := nil;
_24:bat[:oid,:str]{rows=55914459:lng,bid=1961} := nil;
(_27,grp55):= group.new(grp38);
(_29,grp57):= group.derive(_27,grp55,_26);
_27 := nil;
grp55 := nil;
_31 := bat.mirror(_29);
_29 := nil;
_32 := algebra.semijoin(_26,_31);
_26{rows=27957230:lng} := nil;
_31 := nil;
_33:bat[:oid,:int] := aggr.count_no_nil(_32,grp38,_22);
_32 := nil;
_34:bat[:oid,:int] := aggr.count(grp38,grp38,_22);
grp38 := nil;
_22 := nil;
_35 := sql.resultSet(3,1,_23);
sql.rsColumn(_35,"sys.pageviews","channelid","smallint",16,0,_23);
_23 := nil;
sql.rsColumn(_35,"sys.pageviews","count_no_nil_clientip","int",15,0,_33);
_33:bat[:oid,:int] := nil;
sql.rsColumn(_35,"sys.pageviews","count_channelid","int",32,0,_34);
_34:bat[:oid,:int] := nil;
sql.exportResult(_35,"");
end s5_1;
Hello, I have installed Monet and created the TPC-H schema and populated it
with data. Once all data was copied, I restarted the computer to try the
queries. However, now the server starts, but when I open the client
everything closes immediately. I uninstalled it, and re-installed it and the
same happens.
Any help?
Thank You.
Regards,
Alan
--
View this message in context: http://www.nabble.com/Server-Crashes-when-Client-is-Opened-tf4657643.html#a…
Sent from the monetdb-users mailing list archive at Nabble.com.
