users-list July 2018

users-list@monetdb.org

5 participants
6 discussions

GDK reported error. MT_mremap: mremap failed. !OS: Cannot allocate memory

by RanchoYuan

Hi Stefan, Thanks for your reply! We have run the query a few times with different size of data. There we used 16G RAM(actually 13.5G was used), and find the size of 10G's data is the critical point that can run the query. All of the data files' size are listed below, each file name is a table name(there are only a few tables are refered -- store_sales, date_dim, item, customer, catalog_sales, web_sales): 7.4K call_center.dat 1.6M catalog_page.dat 212M catalog_returns.dat 2.9G catalog_sales.dat 27M customer_address.dat 64M customer.dat 77M customer_demographics.dat 9.9M date_dim.dat 77B dbgen_version.dat 149K household_demographics.dat 328B income_band.dat 2.6G inventory.dat 28M item.dat 61K promotion.dat 1.7K reason.dat 1.1K ship_mode.dat 27K store.dat 323M store_returns.dat 3.8G store_sales.dat 4.9M time_dim.dat 1.2K warehouse.dat 19K web_page.dat 98M web_returns.dat 1.5G web_sales.dat 12K web_site.dat So we guess that the monetdb has no memory management? For the output of `mserver5 --version` is: MonetDB 5 server v11.27.13 "Jul2017-SP4" (64-bit, 128-bit integers) Copyright (c) 1993 - July 2008 CWI Copyright (c) August 2008 - 2018 MonetDB B.V., all rights reserved Visit https://www.monetdb.org/ for further information Found 17.0GiB available memory, 40 available cpu cores Libraries: libpcre: 8.38 2015-11-23 (compiled with 8.38) openssl: OpenSSL 1.0.2g 1 Mar 2016 (compiled with OpenSSL 1.0.2g 1 Mar 2016) libxml2: 2.9.3 (compiled with 2.9.3) Compiled by: monetdb(a)MonetDB-0.0 (x86_64-pc-linux-gnu) Compilation: gcc -g -O2 Linking : /usr/bin/ld -m elf_x86_64 And the size of processes is not limited. To let you reproduce the problem conveniently, I'll provide more details here: you can get tpc-ds from its website(we use version 2.6.0). Install the tpc-ds, access the directory v2.6.0/tools and run `./dsdgen -scale 10 -dir /home/monetdb/tpc-ds_test_data10G` to generate the data. When data has been generated, using the script /expe.sh/ to create tables and load the data. The query script is 123.tpcds.23.sql.(The syntaxs of other queries that tpc-ds generates is not suitable for monetdb all, we don't modify them all when the problem occurred). One more question, I can't get your reply email, so I don't know how to reply you, for this case, I could only send a new mail echo time. Thanks! Regards, Rancho

3 years

outstanding updates

by Roberto Cornacchia

Hi, We are having a problem, for a while, that is difficult to isolate. Our ETL includes the following steps: 1) Open JDBC connection, add data to a table, commit, close connection 2) Open JDBC connection, some ALTER TALBLE statements, commit, close connection Occasionally, we get the following during step 2) : java.sql.SQLException: ALTER TABLE: set READ or INSERT ONLY not possible with outstanding updates (wait until updates are flushed) at nl.cwi.monetdb.jdbc.MonetConnection$ResponseList.executeQuery(MonetConnection.java:2732) What puzzles me is that step one is completed when this happens, the commit is done and the connection is closed (we triple-checked that is actually is closed). Notice that inserting a sleep between the to steps makes it work correctly. As I said, this is hard to isolate and reproduce. Still, can anyone guess what is exactly happening? Can it be that a background process is still flushing updates from step 1) ? Even if the JDBC connection is closed ? Can we force a blocking flush, so that it doesn't return until it's safe? If not, how can we know when updates are flushed? Thanks, Roberto

3 years, 11 months

Mercurial source compilation fails on Mac OS 10.13.5

by Ying Zhang

Dear all, Not sure how long this has been going on, but I was just trying to refresh my MonetDB installation (compiled from Mercurial source code), but it failed with the following errors: In file included from /Users/jennie/monet/Mar2018/sql/backends/monet5/UDF/pyapi3/pyapi3.c:13: /Users/jennie/monet/Mar2018/sql/backends/monet5/UDF/pyapi3/../pyapi/pyapi.c:968:18: error: assigning to 'char *' from 'const char *' discards qualifiers [-Werror,-Wincompatible-pointer-types-discards-qualifiers] retnames[i] = PyUnicode_AsUTF8(colname); ^ ~~~~~~~~~~~~~~~~~~~~~~~~~ /Users/jennie/monet/Mar2018/sql/backends/monet5/UDF/pyapi3/../pyapi/pyapi.c:1382:19: error: assigning to 'char *' from 'const char *' discards qualifiers [-Werror,-Wincompatible-pointer-types-discards-qualifiers] py_error_string = PyString_AS_STRING(error); ^ ~~~~~~~~~~~~~~~~~~~~~~~~~ Mac OS 10.13.5 Darwin Kernel Version 17.6.0 Happens with both Mar2018 and default branch Does anyone know how to fix these errors and get MonetDB compiled? Thanks in advance! Regards, Jennie

4 years

Mar2018 production compilation fails on Fedora 28

by Roberto Cornacchia

Compiling from Mar2018 branch, today's hg tip, with production settings: --disable-debug --disable-developer --disable-assert --enable-optimize Fails with this on Fedora 28: In file included from /usr/include/string.h:494, from /home/roberto/MonetDBServer/MonetDB.Spinque_Mar2018/src/gdk/gdk.h:316, from /home/roberto/MonetDBServer/MonetDB.Spinque_Mar2018/src/gdk/gdk_calc.c:10: In function ‘memcpy’, inlined from ‘addstr_loop’ at /home/roberto/MonetDBServer/MonetDB.Spinque_Mar2018/src/gdk/gdk_calc.c:3415:3: /usr/include/bits/string_fortified.h:34:10: error: ‘__builtin_memcpy’ forming offset [3, 65535] is out of the bounds [0, 2] of object ‘str_nil’ with type ‘const char[2]’ [-Werror=array-bounds] return __builtin___memcpy_chk (__dest, __src, __len, __bos0 (__dest)); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ (same error in several files) Compiling with debug settings succeeds: --enable-debug --enable-developer --enable-assert --disable-optimize Is this reproducible or has it something to do with my machine?

4 years

MonetDB Mar2018 Release - Connection terminates abruptly with no error codes/messages

by Murthy, Gautham

Hello, MonetDB release - MonetDB Mar2018 We are facing issues when trying to run queries concurrently on the newer version of MonetDB. When 2 or more sessions (create/drop tables) are executed concurrently, sessions get terminated without any error messages. Do we need to explicitly enable error feedback? Does hitting concurrency abruptly terminates the session/connection ? [monetadmin@lnx1490 ~]$ /monet_binaries/MonetDB-11.27.13_PY/bin/mclient mapi:monetdb://lnx1490.ch3.prod.i.com:50105/TSV_PROD_DB_A_C -u DM_POS_TSV_MRT_P6A password: Welcome to mclient, the MonetDB/SQL interactive terminal (Jul2017-SP4) Database: MonetDB v11.29.7 (Mar2018-SP1), 'mapi:monetdb://lnx1490.ch3.prod.i.com:50105/TSV_PROD_DB_A_C' Type \q to quit, \? for a list of available commands auto commit mode: on sql>\</factorydata/prddata02/prd/crosscut_output/create_desc_it_dsj_campbell_33423.sql [monetadmin@lnx1490 ~]$ Thank You, Gautham

4 years

MonetDB Crashed - Version Jul2017-SP4 - GDKmmap(327680) fails, try to free up space memory in use=48376596752, virtual memory in use=102304729360

by Murthy, Gautham

Hello All, We had a crash reported on our MonetDB database, Below are the error messages. Can you share any insights on the below reported issue. There were no memory/cpu crunch during the crash, System had ample resource available. MonetDB Version - Jul2017-SP4 2018-07-03 06:33:40 ERR TSV_PROD_DB_A315293<https://monetdbsolutions.atlassian.net/wiki/display/SUPPORT/315293>: #GDKmmap(327680) fails, try to free up space memory in use=48376596752,virtual memory in use=102304729360<https://monetdbsolutions.atlassian.net/wiki/display/SUPPORT/memory+in+use%3…> 2018-07-03 06:33:40 ERR TSV_PROD_DB_A315293<https://monetdbsolutions.atlassian.net/wiki/display/SUPPORT/315293>: #GDKmmap(327680) fails, try to free up space memory in use=48368938424,virtual memory in use=102295694776<https://monetdbsolutions.atlassian.net/wiki/display/SUPPORT/memory+in+use%3…> 2018-07-03 06:33:40 ERR TSV_PROD_DB_A315293<https://monetdbsolutions.atlassian.net/wiki/display/SUPPORT/315293>: #GDKmmap(327680) fails, try to free up space memory in use=48367612864,virtual memory in use=102294369216<https://monetdbsolutions.atlassian.net/wiki/display/SUPPORT/memory+in+use%3…> 2018-07-03 06:33:40 ERR TSV_PROD_DB_A315293<https://monetdbsolutions.atlassian.net/wiki/display/SUPPORT/315293>: #GDKmmap(327680) fails, try to free up space memory in use=48374211792,virtual memory in use=102300968144<https://monetdbsolutions.atlassian.net/wiki/display/SUPPORT/memory+in+use%3…> 2018-07-03 06:33:47 ERR TSV_PROD_DB_A315293<https://monetdbsolutions.atlassian.net/wiki/display/SUPPORT/315293>: #GDKmmap(524288) fails, try to free up space memory in use=48312652864,virtual memory in use=103295521856<https://monetdbsolutions.atlassian.net/wiki/display/SUPPORT/memory+in+use%3…> 2018-07-03 06:34:33 MSG merovingian315279<https://monetdbsolutions.atlassian.net/wiki/display/SUPPORT/315279>: database 'TSV_PROD_DB_A' has crashed after start on 2018-07-02 04:52:12, attempting restart, up min/avg/max: 2h/6d/4w, crash average: 1.00 0.60 0.20 (14-8=6) 2018-07-03 06:34:38 ERR control315279<https://monetdbsolutions.atlassian.net/wiki/display/SUPPORT/315279>: !monetdbd: an internal error has occurred 'unknown or impossible state: 4' 2018-07-03 06:34:40 ERR merovingian315279<https://monetdbsolutions.atlassian.net/wiki/display/SUPPORT/315279>: client error: unknown or impossible state: 4 2018-07-03 06:34:40 ERR control315279<https://monetdbsolutions.atlassian.net/wiki/display/SUPPORT/315279>: !monetdbd: an internal error has occurred 'unknown or impossible state: 4' 2018-07-03 06:34:41 ERR merovingian315279<https://monetdbsolutions.atlassian.net/wiki/display/SUPPORT/315279>: client error: unknown or impossible state: 4 2018-07-03 06:34:41 ERR control315279<https://monetdbsolutions.atlassian.net/wiki/display/SUPPORT/315279>: !monetdbd: an internal error has occurred 'unknown or impossible state: 4' What was running during the crash ? : Users were dropping, recreating tables and loading data. monetadmin@lnx1535 ~<https://monetdbsolutions.atlassian.net/wiki/display/SUPPORT/monetadmin%40ln…>$ cat /etc/redhat-release CentOS Linux release 7.4.1708 (Core) monetadmin@lnx1535 ~<https://monetdbsolutions.atlassian.net/wiki/display/SUPPORT/monetadmin%40ln…>$ free -g total used free shared buff/cache available Mem: 1007 199 419 1 388 804 Swap: 31 0 31 monetadmin@lnx1535 ~<https://monetdbsolutions.atlassian.net/wiki/display/SUPPORT/monetadmin%40ln…>$ cat /proc/cpuinfo|grep -i processor|wc -l 128 Thank You, Gautham

4 years

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

users-list July 2018