Hi Stefan,
Thanks for your reply!
We have run the query a few times with different size of data. There we
used 16G RAM(actually 13.5G was used), and find the size of 10G's data
is the critical point that can run the query. All of the data files'
size are listed below, each file name is a table name(there are only a
few tables are refered -- store_sales, date_dim, item, customer,
catalog_sales, web_sales):
7.4K call_center.dat
1.6M catalog_page.dat
212M catalog_returns.dat
2.9G catalog_sales.dat
27M customer_address.dat
64M customer.dat
77M customer_demographics.dat
9.9M date_dim.dat
77B dbgen_version.dat
149K household_demographics.dat
328B income_band.dat
2.6G inventory.dat
28M item.dat
61K promotion.dat
1.7K reason.dat
1.1K ship_mode.dat
27K store.dat
323M store_returns.dat
3.8G store_sales.dat
4.9M time_dim.dat
1.2K warehouse.dat
19K web_page.dat
98M web_returns.dat
1.5G web_sales.dat
12K web_site.dat
So we guess that the monetdb has no memory management?
For the output of `mserver5 --version` is:
MonetDB 5 server v11.27.13 "Jul2017-SP4" (64-bit, 128-bit integers)
Copyright (c) 1993 - July 2008 CWI
Copyright (c) August 2008 - 2018 MonetDB B.V., all rights reserved
Visit https://www.monetdb.org/ for further information
Found 17.0GiB available memory, 40 available cpu cores
Libraries:
libpcre: 8.38 2015-11-23 (compiled with 8.38)
openssl: OpenSSL 1.0.2g 1 Mar 2016 (compiled with OpenSSL 1.0.2g 1
Mar 2016)
libxml2: 2.9.3 (compiled with 2.9.3)
Compiled by: monetdb(a)MonetDB-0.0 (x86_64-pc-linux-gnu)
Compilation: gcc -g -O2
Linking : /usr/bin/ld -m elf_x86_64
And the size of processes is not limited.
To let you reproduce the problem conveniently, I'll provide more details
here:
you can get tpc-ds from its website(we use version 2.6.0).
Install the tpc-ds, access the directory v2.6.0/tools and run `./dsdgen
-scale 10 -dir /home/monetdb/tpc-ds_test_data10G` to generate the data.
When data has been generated, using the script /expe.sh/ to create
tables and load the data. The query script is 123.tpcds.23.sql.(The
syntaxs of other queries that tpc-ds generates is not suitable for
monetdb all, we don't modify them all when the problem occurred).
One more question, I can't get your reply email, so I don't know how to
reply you, for this case, I could only send a new mail echo time.
Thanks!
Regards,
Rancho
Hi all,
I relatively new to MonetDB, but have many year exposure to database
enignes.
I have 2 x vm's, almost identical with the exception of IP's and host
names. both are Ubuntu Server 18.04 LTS. I have installed MonetDB on both
VM's and created a db farm and a database on each.
When running 'monetdb discover' on any of the VM's it finds nothing.
Any help is welcome please.
Regards,
Frik
Outputs from 'monetdbd get all /data/monetdb/dbfarm/'
*VM 1:*
root@ubuntu-srv-001:~# monetdbd start /data/monetdb/dbfarm/
another monetdbd is already running
root@ubuntu-srv-001:~# monetdbd get all /data/monetdb/dbfarm/
property value
hostname ubuntu-srv-001
dbfarm /data/monetdb/dbfarm/
status monetdbd[1941] 1.7 (Aug2018) is serving this dbfarm
mserver /usr/bin/mserver5
logfile /data/monetdb/dbfarm//merovingian.log
pidfile /data/monetdb/dbfarm//merovingian.pid
sockdir /tmp
listenaddr 0.0.0.0
port 50000
exittimeout 60
forward proxy
discovery yes
discoveryttl 600
control yes
passphrase
{SHA512}a73f1d86383446438ac64f56e15ada38b41fbb18f029d2181723aeb2acac6a831f60e5fdbd64ac2c8c70e035dd44cbbe3b45565ef2d58feb2821a2078c7fad35
mapisock /tmp/.s.monetdb.50000
controlsock /tmp/.s.merovingian.50000
root@ubuntu-srv-001:~# monetdb discover
root@ubuntu-srv-001:~#
*VM 2:*
root@ubuntu-srv-002:~# monetdbd get all /data/monetdb/dbfarm/
property value
hostname ubuntu-srv-002
dbfarm /data/monetdb/dbfarm/
status monetdbd[2373] 1.7 (Aug2018) is serving this dbfarm
mserver /usr/bin/mserver5
logfile /data/monetdb/dbfarm//merovingian.log
pidfile /data/monetdb/dbfarm//merovingian.pid
sockdir /tmp
listenaddr 0.0.0.0
port 60001
exittimeout 60
forward proxy
discovery yes
discoveryttl 600
control yes
passphrase
{SHA512}a73f1d86383446438ac64f56e15ada38b41fbb18f029d2181723aeb2acac6a831f60e5fdbd64ac2c8c70e035dd44cbbe3b45565ef2d58feb2821a2078c7fad35
mapisock /tmp/.s.monetdb.60001
controlsock /tmp/.s.merovingian.60001
root@ubuntu-srv-002:~#
I'm trying to debug n issue I have with data disappearing after a database
restart.
Because part of the process for creating the table involves a mixed Mapi /
JDBC approach as described in monetdb-java/example/SQLcopyinto.java, I
started from this example.
When I run it though, I get a final count of 0 (should be 100).
Can anyone confirm?
Mar2018, JDBC 2.28
Roberto
Hello,
I am currently evaluating the performance of MonetDB on the Join Order Benchmark, proposed by Leis et al.(http://www.vldb.org/pvldb/vol9/p204-leis.pdf <http://www.vldb.org/pvldb/vol9/p204-leis.pdf>). There are a few queries where performance seems suboptmal, e.g., query 16b and 17e (https://github.com/gregrahn/join-order-benchmark <https://github.com/gregrahn/join-order-benchmark>): MonetDB takes around 400 seconds to evaluate query 17e with a single thread, more than 20 seconds with 48 threads, Postgres takes only around 9 seconds. I suspect that the query optimizer selects the wrong plan, this is the output of the plan command.
| project ( |
| | group by ( |
| | | join ( |
| | | | join ( |
| | | | | join ( |
| | | | | | join ( |
| | | | | | | join ( |
| | | | | | | | join ( |
| | | | | | | | | table(sys.movie_companies) [ "movie_companies"."movie_id" NOT NULL as "mc"."movie_id", "movie_companies"."company_id" NOT NULL as "mc"."company_id" ] COUNT , |
| | | | | | | | | select ( |
| | | | | | | | | | table(sys.company_name) [ "company_name"."id" NOT NULL HASHCOL as "cn"."id", "company_name"."country_code" as "cn"."country_code" ] COUNT |
| | | | | | | | | ) [ "cn"."country_code" = varchar(255) "[us]" ] |
| | | | | | | | ) [ "mc"."company_id" NOT NULL = "cn"."id" NOT NULL HASHCOL ], |
| | | | | | | | table(sys.title) [ "title"."id" NOT NULL HASHCOL as "t"."id" ] COUNT |
| | | | | | | ) [ "t"."id" NOT NULL HASHCOL = "mc"."movie_id" NOT NULL ], |
| | | | | | | table(sys.cast_info) [ "cast_info"."person_id" NOT NULL as "ci"."person_id", "cast_info"."movie_id" NOT NULL as "ci"."movie_id" ] COUNT |
| | | | | | ) [ "ci"."movie_id" NOT NULL = "t"."id" NOT NULL HASHCOL , "ci"."movie_id" NOT NULL = "mc"."movie_id" NOT NULL ], |
| | | | | | table(sys.name) [ "name"."id" NOT NULL HASHCOL as "n"."id", "name"."name" NOT NULL as "n"."name" ] COUNT |
| | | | | ) [ "n"."id" NOT NULL HASHCOL = "ci"."person_id" NOT NULL ], |
| | | | | table(sys.movie_keyword) [ "movie_keyword"."movie_id" NOT NULL as "mk"."movie_id", "movie_keyword"."keyword_id" NOT NULL as "mk"."keyword_id" ] COUNT |
| | | | ) [ "t"."id" NOT NULL HASHCOL = "mk"."movie_id" NOT NULL, "ci"."movie_id" NOT NULL = "mk"."movie_id" NOT NULL, "mc"."movie_id" NOT NULL = "mk"."movie_id" NOT NULL ], |
| | | | select ( |
| | | | | table(sys.keyword) [ "keyword"."id" NOT NULL HASHCOL as "k"."id", "keyword"."keyword" NOT NULL as "k"."keyword" ] COUNT |
| | | | ) [ "k"."keyword" NOT NULL = clob "character-name-in-title" ] |
| | | ) [ "mk"."keyword_id" NOT NULL = "k"."id" NOT NULL HASHCOL ] |
| | ) [ ] [ sys.min no nil ("n"."name" NOT NULL) NOT NULL as "L3"."L3" ] |
| ) [ "L3"."L3" NOT NULL as "L4"."member_in_charnamed_movie" ] |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I was wondering whether anyone might have a hint about what the problem could be and how one could improve performance for those specific queries?
Thanks a lot!
Hello,
Is there away to measure the physical or virtual memory consumption of Monetdb using an SQL query.
Something similar to select * from storage() where table_name = name;
I believe storage() helps with the disk consumption not main memory consumption, am I right?
Thank you a lot in advance,
Mahmoud Mohsen
Hi!
We are running some "exports" using the COPY INTO FILE, the problem we are
facing is that the file created by monetdb is Read Write ONLY by the user
running the database.
-rw------- 1 root monet 9477 Sep 7 15:08 exp.csv
Is there any way so the created file has also RW permission to the group
and NOT only the user.
In this case, we are using an NFS share so we are not able to use setfacl.
Does anybody has an idea how to solve this ?
Thanks a lot in advance!
Rgds
Ariel
Hello,
I am currently working on a project related to query optimization I have several questions, 1) Is it possible to force the query engine to use the specified join order? 2) When I change the table order in the “FROM” clause, the query time varies a lot (many orders of magnitude)? Does the table order in the “FROM” clause affect the query plan? 3) I want to know the details of MonetDB query plan. When I use “explain” command, I just get the MAL code. Are there any tools to visualize it, just like PostgreSQL “analyze" command?
Thanks a lot.
Hello,
I am currently working on a project related to query optimization I have several questions, 1) Is it possible to force the query engine to use the specified join order? 2) When I change the table order in the “FROM” clause, the query time varies a lot (many orders of magnitude)? Does the table order in the “FROM” clause affect the query plan? 3) I want to know the details of MonetDB query plan? But when I use “explain” command, I just get the MAL code. Are there any tools to visualize it, just like PostgreSQL “analyze" command?
Thanks a lot.
