Hi,
I was wondering if you can give me some ideas on how to retrieve data from
MonetDB at very high throughput. The query will be very simple, issued
against a single table. This table contains billions of records and I'd
like to retrieve the data at a rate of at least one million records per
second. From my naive tests with mclient running on the same 16-core
machine as mserver5, I am only able to extract the data at about 20,000
records per second with the Feb2010 release.
As a baseline case, with the same data in compressed (gzip'd) ASCII form
stored in a regular file on the same (SAN) file system as MonetDB, I am able
to read at the desired speed of one million records per second.
I understand that there is communication overhead between mserver5 and
mclient (or whatever client I use). Therefore, one possibility is to embed
my application within mserver5. The embedded application basically just
needs to be able to issue a SQL (or even MAL) query against the enclosing
mserver5 and process the result set. If this is a viable approach, I'd like
some guidance on where the hooks are.
Thanks.
Hering Cheng
Hi,
Sometimes I get !MALException:setScenario:Scenario not initialized 'sql'
from the server when I try to start it in scenarios that usually work
without any problems. I can't reproduce it consistently, but sometimes I get
it. What does this exception mean?
Thanks.
--
View this message in context: http://www.nabble.com/Meaning-of%3A-Scenario-not-initialized-%27sql%27-tp26…
Sent from the monetdb-users mailing list archive at Nabble.com.
Resent, with some of thread details removed due to limits on message length.
On Thu, May 27, 2010 at 2:49 PM, Hering Cheng <hering.cheng(a)computer.org>wrote:
> Hi,
>
> MonetDB core dumped after I had 10 Java threads retrieving a total of 23
> million records from a single table via JDBC. The threads actually all
> completed successfully. The crash seems to occur when another process tried
> to open a connection to the same database. I can reproduce the crash
> consistently.
>
> After the core file is produced, processes can connect to MonetDB
> successfully, with Merovingian starting up the dead mserver5 automatically.
>
> $ tail
> ~/gaal/rdcuxsrv220-local-disk/chenher/monetdb/feb2010/var/log/MonetDB/merovingian.log
> 2010-05-27 14:27:39 MSG merovingian[19478]: client rdcuxsrv220:61292 has
> disconnected from proxy
> 2010-05-27 14:27:42 MSG merovingian[19478]: client rdcuxsrv220:61296 has
> disconnected from proxy
> 2010-05-27 14:27:42 MSG merovingian[19478]: client rdcuxsrv220:61294 has
> disconnected from proxy
> 2010-05-27 14:27:42 MSG merovingian[19478]: client rdcuxsrv220:61298 has
> disconnected from proxy
> 2010-05-27 14:27:42 MSG merovingian[19478]: client rdcuxsrv220:61288 has
> disconnected from proxy
> 2010-05-27 14:27:42 MSG merovingian[19478]: client rdcuxsrv220:61304 has
> disconnected from proxy
> 2010-05-27 14:27:42 MSG merovingian[19478]: client rdcuxsrv220:61302 has
> disconnected from proxy
> 2010-05-27 14:32:33 MSG merovingian[19478]: proxying client
> rdcuxsrv220:61784 for database 'taq' to mapi:monetdb://127.0.0.1:50001/taq
> 2010-05-27 14:33:01 MSG merovingian[19478]: client rdcuxsrv220:61784 has
> disconnected from proxy
> 2010-05-27 14:34:06 MSG merovingian[19478]: database 'taq' (17289) has
> crashed (dumped core)
>
> This is a Solaris SPARC server and MonetDB was built from the Feb2010
> source code using Sun Studio 12.1:
>
> $ ( ulimit -d $[32*1024*1024] && ulimit -n $[10*1024]; export
> LD_PRELOAD_64=/usr/lib/64/libumem.so:${LD_PRELOAD_64}; export
> LD_PRELOAD=/usr/lib/libumem.so:${LD_PRELOAD}; export
> MONETDB5CONF=/GAAL/chenher/rdcuxsrv220-local-disk/chenher/monetdb/feb2010/etc/monetdb5.conf;
> /GAAL/chenher/share/monetdb/distro-sparc-feb2010-64bit-debug/bin/mserver5
> --version; )
> MonetDB server v5.18.1 (64-bit), based on kernel v1.36.1 (64-bit oids)
> Copyright (c) 1993-July 2008 CWI
> Copyright (c) August 2008-2010 MonetDB B.V., all rights reserved
> Visit http://monetdb.cwi.nl/ for further information
> Found 32.0GiB available memory, 16 available cpu cores
> Configured for prefix:
> /GAAL/chenher/share/monetdb/distro-sparc-feb2010-64bit-debug
> Libraries:
> libpcre: 8.01 2010-01-19 (compiled with 8.01)
> openssl: OpenSSL 0.9.8k 25 Mar 2009 (compiled with )
> libxml2: 2.6.23 (compiled with 2.6.23)
> Compiled by: chenher@rdcuxsrv220 (sparc-sun-solaris2.10)
> Compilation: cc -m64 -xcode=pic32
> -I/GAAL/chenher/share/hans_boehm_gc/distro-sparc-6.8-64bit/include/ -g
> Linking : /usr/ccs/bin/ld -m64 -L/GAAL/chenher/share/openssl-64bit/lib
> -L/GAAL/chenher/share/pcre-64bit/lib
> -L/GAAL/chenher/share/hans_boehm_gc/distro-sparc-6.8-64bit/lib/
>
> $ dbx ~/gaal/share/monetdb/distro-sparc-feb2010-64bit-debug/bin/mserver5
> ~/gaal/rdcuxsrv220-local-disk/chenher/monetdb/feb2010/var/MonetDB5/dbfarm/taq/core
> For information about new features see `help changes'
> To remove this message, put `dbxenv suppress_startup_message 7.7' in your
> .dbxrc
> Reading mserver5
> ...
> Reading lib_logger.so.5.18.1
> Reading libuuid.so.1
> t@4 (l@4) terminated by signal SEGV (no mapping at the fault address)
> Current function is putName
> 174 for(l= nme[0]; l && namespace.nme[l]; l=
> namespace.link[l]){
> (dbx) where
> current thread: t@4
> =>[1] putName(nme = 0xffffffff6aec3928 "exportValue", len = 11U), line 174
> in "mal_namespace.c"
> [2] initSQLreferences(), line 49 in "sql_gencode.c"
> [3] SQLinitClient(c = 0xffffffff7f352628), line 379 in "sql_scenario.c"
> [4] runPhase(c = 0xffffffff7f352628, phase = 5), line 363 in
> "mal_scenario.c"
> [5] runScenarioBody(c = 0xffffffff7f352628), line 392 in "mal_scenario.c"
> [6] runScenario(c = 0xffffffff7f352628), line 438 in "mal_scenario.c"
> [7] MSserveClient(dummy = 0xffffffff7f352628), line 368 in
> "mal_session.c"
> (dbx) threads
> t@1 a l@1 ?() LWP suspended in __pollsys()
> t@2 a l@2 SERVERlistenThread() LWP suspended in __pollsys()
> t@3 a l@3 mvc_logmanager() LWP suspended in __pollsys()
> o> t@4 a l@4 ?() signal SIGSEGV in putName()
> t@5 a l@5 runDFLOWworker() sleep on 0x100c14b08 in
> __lwp_park()
> t@6 a l@6 runDFLOWworker() sleep on 0x100c14b08 in
> __lwp_park()
> t@7 a l@7 runDFLOWworker() sleep on 0x100c14b08 in
> __lwp_park()
> t@8 a l@8 runDFLOWworker() sleep on 0x100c14b08 in
> __lwp_park()
> t@9 a l@9 runDFLOWworker() sleep on 0x100c14b08 in
> __lwp_park()
> t@10 a l@10 runDFLOWworker() sleep on 0x100c14b08 in
> __lwp_park()
> t@11 a l@11 runDFLOWworker() sleep on 0x100c14b08 in
> __lwp_park()
> t@12 a l@12 runDFLOWworker() sleep on 0x100c14b08 in
> __lwp_park()
> t@13 a l@13 runDFLOWworker() sleep on 0x100c14b08 in
> __lwp_park()
> t@14 a l@14 runDFLOWworker() sleep on 0x100c14b08 in
> __lwp_park()
> t@15 a l@15 runDFLOWworker() sleep on 0x100c14b08 in
> __lwp_park()
> t@16 a l@16 runDFLOWworker() sleep on 0x100c14b08 in
> __lwp_park()
> t@17 a l@17 runDFLOWworker() sleep on 0x100c14b08 in
> __lwp_park()
> t@18 a l@18 runDFLOWworker() sleep on 0x100c14b08 in
> __lwp_park()
> t@19 a l@19 runDFLOWworker() sleep on 0x100c14b08 in
> __lwp_park()
> t@20 a l@20 runDFLOWworker() sleep on 0x100c14b08 in
> __lwp_park()
> t@21 a l@21 ?() sleep on 0xffffffff7f352a98 in __lwp_park()
> t@22 a l@22 ?() sleep on 0xffffffff7f352d58 in __lwp_park()
> t@23 a l@23 ?() sleep on 0xffffffff7f353018 in __lwp_park()
> t@24 a l@24 ?() sleep on 0xffffffff7f3532d8 in __lwp_park()
> t@25 a l@25 ?() sleep on 0xffffffff7f353598 in __lwp_park()
> t@26 a l@26 ?() sleep on 0xffffffff7f353858 in __lwp_park()
> t@27 a l@27 ?() sleep on 0xffffffff7f353b18 in __lwp_park()
> t@28 a l@28 ?() sleep on 0xffffffff7f353dd8 in __lwp_park()
> t@29 a l@29 ?() sleep on 0xffffffff7f354098 in __lwp_park()
> t@30 a l@30 ?() sleep on 0xffffffff7f354358 in __lwp_park()
> t@31 a l@31 runDFLOWworker() sleep on 0x1012f8988 in
> __lwp_park()
> t@32 a l@32 runDFLOWworker() sleep on 0x1012f8988 in
> __lwp_park()
> t@33 a l@33 runDFLOWworker() sleep on 0x1012f8988 in
> __lwp_park()
>
...
> t@174 a l@174 runDFLOWworker() sleep on 0x102c9f908 in
> __lwp_park()
> t@175 a l@175 runDFLOWworker() sleep on 0x102c9f908 in
> __lwp_park()
>
> Thanks.
> Hering
>
Since the FEB2010+SP1 release we're encountering loading issues.
With the FEB2010 release we were able to load records until the HDD was full
(500+ million records) on PCs with 2,3 or 24 GB ram memory without going to
the SWAP. This was done by loading CSV-files with 1 million records each
through the COPY INTO command (in the mclient and through a c-script).
Since the FEB2010+SP1 release the system starts to SWAP after loading the
first couple of million records. This results in increased loadtimes per
loaded CSV-file:
1th million: 14 sec
2nd million: 26 sec
3rd million: 44 sec
4th million: 50 sec
5th million: 69 sec
etc.
We have tested this on several PCs and all PCs encounter the same problem.
Test files used are the same for both releases as is the OS (Ubuntu 9.10).
The only variable that has changed is MonetDB.
--
View this message in context: http://old.nabble.com/Loading-problems-since-FEB2010%2BSP1-release-tp282876…
Sent from the monetdb-users mailing list archive at Nabble.com.
Hello,
I would like to get the oid of a stored uri using the Mapi library.
The mal commands that need to be executed are:
tokenizer.open("rdf");
id := tokenizer.locate("http://uri");
io.print();
tokenizer.close();
I tried to run the following program
...tokenizer.open("rdf");...
...
if ((hdl = mapi_query(dbh, "tokenizer.locate(\"http://uri\")")) ==
NULL || mapi_error(dbh) != MOK)
die(dbh, hdl);
if (mapi_close_handle(hdl) != MOK)
die(dbh, hdl);
while (mapi_fetch_row(hdl))
result=mapi_fetch_field(hdl, 0);
...
...tokenizer.close();...
.. but I get the following error
Mapi.mx:5141: mapi_fetch_row: Assertion `(hdl)->mid' failed.
Could you tell me how can I solve this problem?
Kind regards,
Petros
Loading and querying simultaneously seems to slowdown the database
significantly.
I can understand if the queries became slower during loading, but it seems
that only the first query before loading is finished is slow and all queries
coming after.
I can only fix the slowdown by using "killall mserver5" in a terminal, which
is not an acceptable solution.
I start 1 "copy into" action with one connection and at the same time a
series of simple queries on another connection.
The Simple query looks something like this: SELECT x,y FROM table WHERE x =
'519';
Used
OS: Ubuntu 9.10 and 10.04
MonetDB: Feb2010 release (SP! and SP2 are no option due to them giving
loading problems)
Results
Query: 1 --> 0.000191 SECONDS
:
:
:
Query: 10501 --> 0.000170 SECONDS
Query: 10502 --> 0.000169 SECONDS
Query: 10503 --> 0.000169 SECONDS
Query: 10504 --> 0.000172 SECONDS
Query: 10505 --> 0.000172 SECONDS
Query: 10506 --> 0.000174 SECONDS
Query: 10507 --> 0.000178 SECONDS
Query: 10508 --> 1.199149 SECONDS
Load : 1 --> 3.238532 SECONDS
Query: 1-10509 --> 1.238804 SECONDS
Query: 1-10510 --> 1.175408 SECONDS
Query: 1-10511 --> 1.141172 SECONDS
Query: 1-10512 --> 1.132525 SECONDS
Query: 1-10513 --> 1.159554 SECONDS
Query: 1-10514 --> 1.158303 SECONDS
Query: 1-10515 --> 1.160997 SECONDS
How do I solve this or is this an error in MonetDB?
Thanks in advance.
--
View this message in context: http://old.nabble.com/Loading-and-querying-on-the-same-database.-tp28605728…
Sent from the monetdb-users mailing list archive at Nabble.com.
Hi,
I tried to update a column of a record more than once within a
transaction. But it seemed that only the first update took effect.
For example, for a table "CREATE TABLE people (id TINYINT PRIMARY KEY,
name VARCHAR(128) NOT NULL)",
sql>select * from people;
+------+-----------------------------+
| id | name |
+======+=============+
| 0 | Phil Ivey |
| 1 | Michael Jordan |
| 2 | Lionel Messi |
+------+-----------------------------+
sql>start transaction;
auto commit mode: off
sql>update people set id = -1 where name='Phil Ivey';
1 affected row
sql>select * from people;
+------+-----------------------------+
| id | name |
+======+=============+
| 1 | Michael Jordan |
| 2 | Lionel Messi |
| -1 | Phil Ivey |
+------+-----------------------------+
sql>update people set id = -2 where name='Phil Ivey';
1 affected row
sql>select * from people;
+------+-----------------------------+
| id | name |
+======+=============+
| 1 | Michael Jordan |
| 2 | Lionel Messi |
| -1 | Phil Ivey |
+------+-----------------------------+
sql>commit;
auto commit mode: on
sql>select * from people;
+------+-----------------------------+
| id | name |
+======+=============+
| -1 | Phil Ivey |
| 1 | Michael Jordan |
| 2 | Lionel Messi |
+------+-----------------------------+
It is not what we would expect, isn't it? My MonetDB version is v5.18.3
Min
Hi,
I've just pulled a largish dataset into monetdb (419851473 rows, 8
columns) and it's crashing when I try to do the following:
select count(distinct a) from calldata;
"a" is an integer column and monetdb gets the right answer of 886693
when I run either:
select count(a) from (select a from calldata group by a) x;
or
select count(a) from (select distinct a from calldata) x;
I'm running the latest 64bit .deb packages under ubuntu 10.4. The
following is what I get back from mclient:
sql>select count(distinct a) from calldata;
MALException:!ERROR: HEAPalloc: Insufficient space for HEAP of 3358812160 bytes.
ERROR: HEAPalloc: Insufficient space for HEAP of 3358812160 bytes.
ERROR: HEAPalloc: Insufficient space for HEAP of 3358812160 bytes.
Timer 93948.763 msec 1 rows
I've not got much swap space, does it just happen to need it for the
first version but not the other versions?
--
Sam http://samason.me.uk/
Hi,
So what happens (performance wise) when you have 50GB of memory, 300GB
of data and on average 100 concurrent users using all the tables and
columns in your database? Thanks. Dariusz.
