Hi Stefan,
Thanks for your reply!
We have run the query a few times with different size of data. There we
used 16G RAM(actually 13.5G was used), and find the size of 10G's data
is the critical point that can run the query. All of the data files'
size are listed below, each file name is a table name(there are only a
few tables are refered -- store_sales, date_dim, item, customer,
catalog_sales, web_sales):
7.4K call_center.dat
1.6M catalog_page.dat
212M catalog_returns.dat
2.9G catalog_sales.dat
27M customer_address.dat
64M customer.dat
77M customer_demographics.dat
9.9M date_dim.dat
77B dbgen_version.dat
149K household_demographics.dat
328B income_band.dat
2.6G inventory.dat
28M item.dat
61K promotion.dat
1.7K reason.dat
1.1K ship_mode.dat
27K store.dat
323M store_returns.dat
3.8G store_sales.dat
4.9M time_dim.dat
1.2K warehouse.dat
19K web_page.dat
98M web_returns.dat
1.5G web_sales.dat
12K web_site.dat
So we guess that the monetdb has no memory management?
For the output of `mserver5 --version` is:
MonetDB 5 server v11.27.13 "Jul2017-SP4" (64-bit, 128-bit integers)
Copyright (c) 1993 - July 2008 CWI
Copyright (c) August 2008 - 2018 MonetDB B.V., all rights reserved
Visit https://www.monetdb.org/ for further information
Found 17.0GiB available memory, 40 available cpu cores
Libraries:
libpcre: 8.38 2015-11-23 (compiled with 8.38)
openssl: OpenSSL 1.0.2g 1 Mar 2016 (compiled with OpenSSL 1.0.2g 1
Mar 2016)
libxml2: 2.9.3 (compiled with 2.9.3)
Compiled by: monetdb(a)MonetDB-0.0 (x86_64-pc-linux-gnu)
Compilation: gcc -g -O2
Linking : /usr/bin/ld -m elf_x86_64
And the size of processes is not limited.
To let you reproduce the problem conveniently, I'll provide more details
here:
you can get tpc-ds from its website(we use version 2.6.0).
Install the tpc-ds, access the directory v2.6.0/tools and run `./dsdgen
-scale 10 -dir /home/monetdb/tpc-ds_test_data10G` to generate the data.
When data has been generated, using the script /expe.sh/ to create
tables and load the data. The query script is 123.tpcds.23.sql.(The
syntaxs of other queries that tpc-ds generates is not suitable for
monetdb all, we don't modify them all when the problem occurred).
One more question, I can't get your reply email, so I don't know how to
reply you, for this case, I could only send a new mail echo time.
Thanks!
Regards,
Rancho
Hi,
We are having a problem, for a while, that is difficult to isolate.
Our ETL includes the following steps:
1) Open JDBC connection, add data to a table, commit, close connection
2) Open JDBC connection, some ALTER TALBLE statements, commit, close
connection
Occasionally, we get the following during step 2) :
java.sql.SQLException: ALTER TABLE: set READ or INSERT ONLY not possible
with outstanding updates (wait until updates are flushed)
at
nl.cwi.monetdb.jdbc.MonetConnection$ResponseList.executeQuery(MonetConnection.java:2732)
What puzzles me is that step one is completed when this happens, the commit
is done and the connection is closed (we triple-checked that is actually is
closed).
Notice that inserting a sleep between the to steps makes it work correctly.
As I said, this is hard to isolate and reproduce. Still, can anyone guess
what is exactly happening?
Can it be that a background process is still flushing updates from step 1) ?
Even if the JDBC connection is closed ?
Can we force a blocking flush, so that it doesn't return until it's safe?
If not, how can we know when updates are flushed?
Thanks, Roberto
Dear all,
Not sure how long this has been going on, but I was just trying to refresh my MonetDB installation (compiled from Mercurial source code), but it failed with the following errors:
In file included from /Users/jennie/monet/Mar2018/sql/backends/monet5/UDF/pyapi3/pyapi3.c:13:
/Users/jennie/monet/Mar2018/sql/backends/monet5/UDF/pyapi3/../pyapi/pyapi.c:968:18: error:
assigning to 'char *' from 'const char *' discards qualifiers
[-Werror,-Wincompatible-pointer-types-discards-qualifiers]
retnames[i] = PyUnicode_AsUTF8(colname);
^ ~~~~~~~~~~~~~~~~~~~~~~~~~
/Users/jennie/monet/Mar2018/sql/backends/monet5/UDF/pyapi3/../pyapi/pyapi.c:1382:19: error:
assigning to 'char *' from 'const char *' discards qualifiers
[-Werror,-Wincompatible-pointer-types-discards-qualifiers]
py_error_string = PyString_AS_STRING(error);
^ ~~~~~~~~~~~~~~~~~~~~~~~~~
Mac OS 10.13.5
Darwin Kernel Version 17.6.0
Happens with both Mar2018 and default branch
Does anyone know how to fix these errors and get MonetDB compiled? Thanks in advance!
Regards,
Jennie
Compiling from Mar2018 branch, today's hg tip, with production settings:
--disable-debug --disable-developer --disable-assert --enable-optimize
Fails with this on Fedora 28:
In file included from /usr/include/string.h:494,
from
/home/roberto/MonetDBServer/MonetDB.Spinque_Mar2018/src/gdk/gdk.h:316,
from
/home/roberto/MonetDBServer/MonetDB.Spinque_Mar2018/src/gdk/gdk_calc.c:10:
In function ‘memcpy’,
inlined from ‘addstr_loop’ at
/home/roberto/MonetDBServer/MonetDB.Spinque_Mar2018/src/gdk/gdk_calc.c:3415:3:
/usr/include/bits/string_fortified.h:34:10: error: ‘__builtin_memcpy’
forming offset [3, 65535] is out of the bounds [0, 2] of object ‘str_nil’
with type ‘const char[2]’ [-Werror=array-bounds]
return __builtin___memcpy_chk (__dest, __src, __len, __bos0 (__dest));
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
(same error in several files)
Compiling with debug settings succeeds:
--enable-debug --enable-developer --enable-assert --disable-optimize
Is this reproducible or has it something to do with my machine?
Hello,
MonetDB release - MonetDB Mar2018
We are facing issues when trying to run queries concurrently on the newer version of MonetDB.
When 2 or more sessions (create/drop tables) are executed concurrently, sessions get terminated without any error messages.
Do we need to explicitly enable error feedback?
Does hitting concurrency abruptly terminates the session/connection ?
[monetadmin@lnx1490 ~]$ /monet_binaries/MonetDB-11.27.13_PY/bin/mclient mapi:monetdb://lnx1490.ch3.prod.i.com:50105/TSV_PROD_DB_A_C -u DM_POS_TSV_MRT_P6A
password:
Welcome to mclient, the MonetDB/SQL interactive terminal (Jul2017-SP4)
Database: MonetDB v11.29.7 (Mar2018-SP1), 'mapi:monetdb://lnx1490.ch3.prod.i.com:50105/TSV_PROD_DB_A_C'
Type \q to quit, \? for a list of available commands
auto commit mode: on
sql>\</factorydata/prddata02/prd/crosscut_output/create_desc_it_dsj_campbell_33423.sql
[monetadmin@lnx1490 ~]$
Thank You,
Gautham