Hello,
Running the same query on the same data leeds to different results
between postgresql and monetdb.
I have not been able to get a simple example to reproduce the problem.
So here is what I observe:
the query:
select temps_mois.rfoperdmo as c1,
sum((case when dwhinv.dwhinv___rfodomide = 'RH' and
dwhinv.dwhinv___rfoindide = 'tech_rh_effectif_rem_reel' then
dwhinv.dwhinvqte else 0 end)) as m0
from rfoper_temps_mois as temps_mois,
dwhinv as dwhinv,
rfovsn as rfovsn_0,
rrhamv as rrhamv_1,
rrhcov as rrhcov_2,
rfoadv as rfoadv_3
where temps_mois.rfoper___rforefide = 'HPLUS'
and dwhinv.dwhinv___rforefide = 'HPLUS'
and dwhinv.dwhinv___rfodomide = 'RH'
and dwhinv.dwhinv___rfoindide = 'tech_rh_effectif_rem_reel'
and dwhinv.dwhinvdtd = temps_mois.rfoperdtd
and temps_mois.rfoperyea = '2011'
and rfovsn___rforefide = 'HPLUS'
and dwhinv.dwhinv___rfovsnide = rfovsn_0.rfovsnide
and rfovsn_0.rfovsnide = '201111_reel'
and rrhamv_1.rrhamv___rrhvemide='GRACOR'
AND rrhamv_1.rrhamvrvs=1
AND rrhamv_1.rrhamv___rforefide= 'HPLUS'
and dwhinv.dwhinv___rrhempide = rrhamv_1.rrhamvinf
and rrhamv_1.rrhamvsup = 'CEMP'
and rrhcov_2.rrhcov___rrhvcoide='CONTRATS'
AND rrhcov_2.rrhcovrvs=1
AND rrhcov_2.rrhcov___rforefide= 'HPLUS'
and dwhinv.dwhinv___rrhcntide = rrhcov_2.rrhcovinf
and rrhcov_2.rrhcovsup = 'CONTRATS'
and rfoadv_3.rfoadv___rfovdeide='STRC'
AND rfoadv_3.rfoadvrvs=1
AND rfoadv_3.rfoadv___rforefide= 'HPLUS'
and dwhinv.dwhinv_p2rfodstide = rfoadv_3.rfoadvinf
and rfoadv_3.rfoadvsup = 'HPLUS'
group by c1
order by c1
Postgresql result:
c1 | m0
----+-----------------
01 | 7111.5376967750
02 | 7100.9108821426
03 | 7150.2597967742
04 | 7151.4283666667
05 | 7109.1641451610
06 | 6976.2108421239
07 | 6329.7404193564
08 | 6302.9823032247
09 | 6426.9459633351
10 | 6519.6889580648
11 | 6549.5235033402
12 | 6492.5477161292
(12 lignes)
Monetdb result:
+------+----------------------+
| c1 | m0 |
+======+======================+
| 12 | 1.000000 |
| 11 | 1.000000 |
| 10 | 2.000000 |
| 09 | 0.366667 |
| 08 | 1.000000 |
| 07 | 1.000000 |
| 06 | 1.000000 |
| 05 | 3.000000 |
| 04 | 4.000000 |
| 03 | 15.000000 |
| 02 | 9.964286 |
| 01 | 1.000000 |
+------+----------------------+
12 tuples (212.346ms)
As you see, all "m0" values are very different.
I have no idea of what is causing this but if I modify the query in
order to simplify the CASE...WHEN...ELSE part of the select, it produce
the expected result:
replacing sum((case when dwhinv.dwhinv___rfodomide = 'RH' and
dwhinv.dwhinv___rfoindide = 'tech_rh_effectif_rem_reel' then
dwhinv.dwhinvqte else 0 end)) as m0
by sum((case when dwhinv.dwhinv___rfoindide =
'tech_rh_effectif_rem_reel' then dwhinv.dwhinvqte else 0 end)) as m0
Its not exactly the same query but it must leeds to the same result due
to my test datas (its the case in postgres)
I join to this email two files with the TRACES, respectively for the
request with bad/correct result (ko.txt/ok.txt)
Config:
Ubuntu Server 11.04 x64
MonetDB v11.7.9 (Dec2011-SP2), MonetDB Database Server v1.6
(Dec2011-SP2), MonetDB Database Server Toolkit v1.0 (Dec2011-SP2)
Thank you very much for your help!!
--
*Matthieu Guamis*
*Logo Axège <http://www.axege.com/>* /Axège//
23,rue Saint-Simon
63000 Clermont-Ferrand/
Tél: +33 (0)4 63 05 95 40
Fax: +33 (0)4.73.70.65.29
Email: matthieu.guamis(a)axege.com <mailto:matthieu.guamis@axege.com>
Hello,
I need to create temporary tables, but using the statement "create temporary table AS select XXX WITH DATA" results in an empty table;
====
sql>CREATE TABLE table1 (a INTEGER);
operation successful (1.320ms)
sql>INSERT INTO table1 VALUES(1);
1 affected row (0.693ms)
sql>INSERT INTO table1 VALUES(2);
1 affected row (0.511ms)
sql>
sql>CREATE TEMPORARY TABLE temp1 AS SELECT * FROM table1 WITH DATA ;
operation successful (0.735ms)
sql>SELECT * FROM temp1;
+---+
| a |
+===+
+---+
0 tuples (1.032ms)
===
Whereas if I don't use "temporary" for the table creation, it works fine:
===
sql>CREATE TABLE temp2 AS SELECT * FROM table1 WITH DATA ;
operation successful (1.842ms)
sql>SELECT * FROM temp2;
+------+
| a |
+======+
| 1 |
| 2 |
+------+
2 tuples (1.898ms)
===
The example above is contrived, but I need to create two temporary tables to facilitate a join.
Does the "temporary" keyword have different meaning than in other databases ?
I couldn't find a good explanation in the on-line documentation...
I'm using MonetDB v11.14.0, compiled from source with revision 45564:14983b3e103e .
Thanks,
-gordon
_______________________________________________
users-list mailing list
users-list(a)monetdb.org
http://mail.monetdb.org/mailman/listinfo/users-list
As list administrator, your authorization is requested for the
following mailing list posting:
List: users-list(a)monetdb.org
From: ajdamico(a)gmail.com
Subject: invalid tuple received from server, got 138 columns, expected 476, ignoring
Reason: Post by non-member to a members-only list
At your convenience, visit:
http://mail.monetdb.org/mailman/admindb/users-list
to approve or deny the request.
Hello,
I'm seeing this error when starting a database. Not sure if anyone has
had the same issue and whether anyone had a fix?
The following is what I did
% monetdbd start ~/dbfarm
% monetdbd start ~/dbfarm
% monetdb create voc
% monetdb start voc
starting database 'voc'... FAILED
start: starting 'voc' failed: database 'voc' appears to shut itself
down after starting, check monetdbd's logfile for possible hints
The log says the following
2012-09-27 16:56:04 MSG merovingian[14761]: database 'voc' (14801) has
exited with exit status 1
2012-09-27 16:56:04 ERR control[14761]: (local): failed to fork
mserver: database 'voc' appears to shut itself down after starting,
check monetdbd's logfile for possible hints
Any comments / suggestions?
Thank you,
Peter
_______________________________________________
users-list mailing list
users-list(a)monetdb.org
http://mail.monetdb.org/mailman/listinfo/users-list
Hi All,
I just tried monetdb with large tables (600,000 rows by 50,000 columns).
There are two issues:
1. performance was low, it took about 4 seconds to insert a row of record
when the number of columns was large (50,000). However it was pretty fast
(about 100 rows per second) when the column number was small (for example,
10).
2. serious errors occurred very often. For example, after inserting random
number of records (50, for example), monetdb started to report errors, like:
monetdbd: internal error while starting mserver, please refer to the
logs
So I opened the log file, and the error was:
2012-09-14 08:26:36 ERR merovingian[13730]: client error: unknown or
impossible state: 4
The Makefile (it is also contained in the attachments) listed below will
start database server, create a database farm, create a database, create a
big table and then insert records to the big table. Here it is:
#=====start of Makefile=====================
ROW_NUM=600000
COLUMN_NUM=50000
COLUMN_NUM_MINUS_ONE=$(shell echo $(COLUMN_NUM)-1 | bc)
all: clean
monetdbd create ./mydbfarm
monetdbd start ./mydbfarm
monetdb create test_big_db
monetdb release test_big_db
make create_table_sql -s > create_big_table.sql
make insert_data_sql -s > insert_data.sql
mclient -d test_big_db create_big_table.sql
for i in $$(seq 1 $(ROW_NUM)); do \
mclient -d test_big_db insert_data.sql ;\
echo 'done inserting ' $$i ' rows'; \
done;
mclient -d test_big_db -s "select * from test_big_table" >
test_big_table.txt
create_table_sql:
@echo 'START TRANSACTION;'
@echo 'CREATE TABLE "test_big_table" ('
@for i in $$(seq 1 $(COLUMN_NUM_MINUS_ONE)); do \
echo ' "gen'$$i'" DOUBLE,'; \
done
@echo ' "gen$(COLUMN_NUM)" DOUBLE'
@echo ');'
@echo 'COMMIT;'
insert_data_sql:
@echo -n 'INSERT INTO "test_big_table" VALUES (' ;
@for j in $$(seq 1 $(COLUMN_NUM_MINUS_ONE)) ; do \
echo -n '0.323, '; \
done;
@echo '0.324);';
insert_all_data_sql:
@echo 'START TRANSACTION;'
@for i in $$(seq 1 $(ROW_NUM)); do\
echo -n 'INSERT INTO "test_big_table" VALUES (' ;\
for j in $$(seq 1 $(COLUMN_NUM_MINUS_ONE)) ; do \
echo -n '0.323, '; \
done; \
echo '0.324);'; \
done
@echo 'COMMIT;'
clean:
-killall mserver5
-killall monetdbd
rm -rf ./mydbfarm
rm -rf create_big_table.sql test_big_table.txt insert_data.sql
#=====end of Makefile======================
Thank you very much !
Best wishes,
Joshua Shuai Yuan
On Thu, Sep 13, 2012 at 12:43 PM, Joshua Shuai Yuan <
shuaiyuan.emory(a)gmail.com> wrote:
> Sure, I will see what I can do.
>
> Best wishes,
> Joshua Shuai Yuan
>
>
> On Wed, Sep 12, 2012 at 11:40 PM, Wang, Fusheng <fusheng.wang(a)emory.edu>wrote:
>
>> Hi Joshua,****
>>
>> ** **
>>
>> The table we are managing is in the scale of 30K x 5K: 150M cells. If
>> each table cell (double type) needs 8 bytes to represent (ideally), the
>> space needed will be a couple of GB, or at most, in the scale of tens of
>> GB. If that is the case, distributed setup may not be needed, as the data
>> can be mostly cached by the database. MonetDB does support multi-cores
>> and multi-disks, but setup across multiple machines is not supported based
>> on my knowledge. ****
>>
>> ** **
>>
>> Even though MonetDB claims unlimited number of columns, we should still
>> be cautious on what performance we can achieve for the queries we want to
>> provide. A pilot study on this could provide us some guideline. Do you
>> think you can setup MonetDB, and create a benchmark table we can do some
>> performance study?****
>>
>> ** **
>>
>> Interestingly, the large column issue is also discussed in a famous
>> database blog:****
>>
>>
>> http://www.dbms2.com/2011/03/13/so-how-many-columns-can-a-single-table-have…
>> ****
>>
>> ** **
>>
>> One guy commented:****
>>
>> ** **
>>
>> “Genomics models were the primary driver. these folks typically have
>> short but wide datasets of the order of 300,000 columns.”****
>>
>> ** **
>>
>> So we are not alone. And we have only a subset of the columns (30K
>> versus 300K)? Then we are lucky!****
>>
>> ** **
>>
>> ** **
>>
>> Fusheng****
>>
>> ** **
>>
>> ** **
>>
>> ** **
>>
>> ** **
>>
>> *From:* Joshua Shuai Yuan [mailto:shuaiyuan.emory@gmail.com]
>> *Sent:* Wednesday, September 12, 2012 11:14 PM
>> *To:* Wang, Fusheng
>> *Cc:* Qin, Zhaohui
>> *Subject:* Re: Array database for large matrix****
>>
>> ** **
>>
>> Hi Dr. Wang,****
>>
>> That's really a good news. Does it support distributed database? Or do
>> we need distributed one?****
>>
>> ** **
>>
>>
>> Best wishes,
>> Joshua Shuai Yuan
>>
>> ****
>>
>> On Wed, Sep 12, 2012 at 3:54 PM, Wang, Fusheng <fusheng.wang(a)emory.edu>
>> wrote:****
>>
>> Hi guys,****
>>
>> ****
>>
>> It looks like the matrix structure can be nicely supported by array
>> databases, like MonetDB. It supports unlimited number of columns for a
>> table. It’s also open source. ****
>>
>> http://www.monetdb.org/Home/Features****
>>
>> ****
>>
>> I know the group quite well, and I will chat with them on the use case to
>> see if it’s a good fit. ****
>>
>> ****
>>
>> Thanks,****
>>
>> ****
>>
>> Fusheng****
>>
>> ****
>>
>> ****
>>
>> ** **
>> ------------------------------
>>
>>
>> This e-mail message (including any attachments) is for the sole use of
>> the intended recipient(s) and may contain confidential and privileged
>> information. If the reader of this message is not the intended
>> recipient, you are hereby notified that any dissemination, distribution
>> or copying of this message (including any attachments) is strictly
>> prohibited.
>>
>> If you have received this message in error, please contact
>> the sender by reply e-mail message and destroy all copies of the
>> original message (including attachments).****
>>
>> ** **
>>
>
>
We would like to inform you that as of Januari 2013, the monetdb project
on SourceForge will be removed. All services that SourceForge has
kindly provided us with in the past will be migrated to our monetdb.org
infrastructure.
Currently, the only remaining services hosted on SourceForge are the
three mailing lists monetdb-announce, monetdb-users and
monetdb-developers. As of this message, these lists are migrated to
their equivalents on monetdb.org. The new lists are as follows:
monetdb-announce(a)lists.sourceforge.net -> announce-list(a)monetdb.org
monetdb-users(a)lists.sourceforge.net -> users-list(a)monetdb.org
monetdb-developers(a)lists.sourceforge.net -> developers-list(a)monetdb.org
We invite you to subscribe to the new lists on monetdb.org at your
earliest convenience. Any posts to the old lists are rejected from now
on. Posts to the new lists at monetdb.org are forwarded in batches to
the old lists until the end of this year.
The MonetDB team would like to take this opportunity to thank
SourceForge for the services provided over the past years. We hope to
welcome all current subscribers on our new lists at monetdb.org, see:
http://mail.monetdb.org/mailman/listinfo
The MonetDB team
Hi,
I am trying to insert data into monetdb using sqoop, via the monet jdbc
interface. It keeps crashing 20 odd rows into the import, with a monetdb
ROLLBACK. The debug info from Sqoop is poor. Does anyone know if the monetdb
ROLLBACK can be disabled, so that I can see the last row imported.
Has anyone tried Sqoop to Monetdb, are there any debugging tips that you can
give ?
Regards,
Jeremy
sqoop export --connect jdbc:monetdb://cluster-03/ngpm -driver
nl.cwi.monetdb.jdbc.MonetDriver --num-mappers 1 --password monetdb
--username monetdb --table query_start --export-dir
/test/dash_test/QueryStart.Log --fields-terminated-by ','
12/09/24 17:27:49 INFO mapreduce.Job: Task Id :
attempt_1347971665109_0247_m_000000_1, Status : FAILED
Error: java.io.IOException: java.sql.SQLException: EXEC: wrong number of
arguments for prepared statement: 100, expected 102
at
org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.j
ava:192)
at
org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java
:610)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:730)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
va:1232)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147)
Caused by: java.sql.SQLException: EXEC: wrong number of arguments for
prepared statement: 100, expected 102
at
nl.cwi.monetdb.jdbc.MonetConnection$ResponseList.executeQuery(MonetConnectio
n.java:2524)
at
nl.cwi.monetdb.jdbc.MonetConnection$ResponseList.processQuery(MonetConnectio
n.java:2274)
at
nl.cwi.monetdb.jdbc.MonetStatement.internalExecute(MonetStatement.java:508)
at
nl.cwi.monetdb.jdbc.MonetStatement.execute(MonetStatement.java:349)
at
nl.cwi.monetdb.jdbc.MonetPreparedStatement.execute(MonetPreparedStatement.ja
va:247)
at
org.apache.sqoop.mapreduce.AsyncSqlOutputFormat$AsyncSqlExecThread.run(Async
SqlOutputFormat.java:230)
The MonetDB team at CWI/MonetDB BV is pleased to announce the
Jul2012-SP2 bugfix release of the MonetDB suite of programs.
More information about MonetDB can be found on our website at
<http://www.monetdb.org/>.
For details on this release, please see the release notes at
<http://www.monetdb.org/Downloads/ReleaseNotes>.
As usual, the download location is <http://dev.monetdb.org/downloads/>.
Jul 2012-SP2 bugfix release
Java Module * Fixed a bug where DatabaseMetaData.getURL() did return
null:0 for hostname:port.
MonetDB5 Server * Changed the way nclients maximum was calculated to
avoid 'out of client slots' errors way before the
maximum was reached.
Merovingian * Resolved a problem where monetdb could fail to start a
database with the message "database 'X' started up,
but failed to open up a communication channel". Bug
#3134, comment #7.
MonetDB Common * Fixed a bug in BATantijoin when either side is a
singleton BAT. This fixes bug 3139.
Bug Fixes * 3117: test
sql/test/BugTracker-2011/Tests/history.Bug-2909.sql
produces extra profiling output since changeset
817328b74e2f
* 3134: Database gets killed by timeout during startup
* 3139: Not equal evaluation does not work
* 3144: Memleak when connecting to a database (mclient
and mapi)
* 3145: mclient error when enabling history
* 3150: Multiple string equality tests fail to match
(Test_JdbcClient)