users-list January 2019

users-list@monetdb.org

9 participants
8 discussions

GDK reported error. MT_mremap: mremap failed. !OS: Cannot allocate memory

by RanchoYuan

Hi Stefan, Thanks for your reply! We have run the query a few times with different size of data. There we used 16G RAM(actually 13.5G was used), and find the size of 10G's data is the critical point that can run the query. All of the data files' size are listed below, each file name is a table name(there are only a few tables are refered -- store_sales, date_dim, item, customer, catalog_sales, web_sales): 7.4K call_center.dat 1.6M catalog_page.dat 212M catalog_returns.dat 2.9G catalog_sales.dat 27M customer_address.dat 64M customer.dat 77M customer_demographics.dat 9.9M date_dim.dat 77B dbgen_version.dat 149K household_demographics.dat 328B income_band.dat 2.6G inventory.dat 28M item.dat 61K promotion.dat 1.7K reason.dat 1.1K ship_mode.dat 27K store.dat 323M store_returns.dat 3.8G store_sales.dat 4.9M time_dim.dat 1.2K warehouse.dat 19K web_page.dat 98M web_returns.dat 1.5G web_sales.dat 12K web_site.dat So we guess that the monetdb has no memory management? For the output of `mserver5 --version` is: MonetDB 5 server v11.27.13 "Jul2017-SP4" (64-bit, 128-bit integers) Copyright (c) 1993 - July 2008 CWI Copyright (c) August 2008 - 2018 MonetDB B.V., all rights reserved Visit https://www.monetdb.org/ for further information Found 17.0GiB available memory, 40 available cpu cores Libraries: libpcre: 8.38 2015-11-23 (compiled with 8.38) openssl: OpenSSL 1.0.2g 1 Mar 2016 (compiled with OpenSSL 1.0.2g 1 Mar 2016) libxml2: 2.9.3 (compiled with 2.9.3) Compiled by: monetdb(a)MonetDB-0.0 (x86_64-pc-linux-gnu) Compilation: gcc -g -O2 Linking : /usr/bin/ld -m elf_x86_64 And the size of processes is not limited. To let you reproduce the problem conveniently, I'll provide more details here: you can get tpc-ds from its website(we use version 2.6.0). Install the tpc-ds, access the directory v2.6.0/tools and run `./dsdgen -scale 10 -dir /home/monetdb/tpc-ds_test_data10G` to generate the data. When data has been generated, using the script /expe.sh/ to create tables and load the data. The query script is 123.tpcds.23.sql.(The syntaxs of other queries that tpc-ds generates is not suitable for monetdb all, we don't modify them all when the problem occurred). One more question, I can't get your reply email, so I don't know how to reply you, for this case, I could only send a new mail echo time. Thanks! Regards, Rancho

3 years

Program contains errors.:(NONE).multiplex

by Niklas B

Hi, I'm upgrading from an old MonetDB (11.26.0 source snapshot) to the latest proper release. However, my Python aggregate functions appear to have stopped working (Program contains errors.:(NONE).multiplex): Running mserver5 with: /usr/local/bin/mserver5 --daemon=yes --set embedded_py=true --dbpath=/data/monetdb-farm/ --set gdk_nr_threads=1 --set max_clients=6 Reproduce with: CREATE FUNCTION test_udf("conum" int) RETURNS STRING LANGUAGE PYTHON3 {{ }}; Create table test("id" uuid, "date" DATE, "number" INT); select test_udf("number") from test; Gives: Program contains errors.:(NONE).multiplex I'm running it via Docker (stock Debian) with only MonetDB installed via apt-get. Not sure if this is connected: https://www.monetdb.org/bugzilla/show_bug.cgi?id=6378 <https://www.monetdb.org/bugzilla/show_bug.cgi?id=6378> Anyone having any similar issues? Tried compiling from source (April 2019 bransch) as well, no luck Regards, Niklas

3 years, 4 months

Error when installing on Centos 7

by Trevor Andreas

Hello, I am getting an error trying to install MonetDB following the instructions here: https://dev.monetdb.org/downloads/epel/ The error I am getting is when running: yum install -y MonetDB-SQL-server5 MonetDB-client failure: repodata/repomd.xml from monetdb: [Errno 256] No more mirrors to try. https://dev.monetdb.org/downloads/epel/7/x86_64/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found Looks like the repodata is missing for Centos 7? It looks to be there for Centos 6 https://dev.monetdb.org/downloads/epel/6/x86_64/repodata/ Thanks! Trevor

3 years, 6 months

Invalid Credentials error

by AUBERT Simon

Good morning, I have used Monetdb for a few weeks now and this evening I cannot login after a full system restart (windows+monetdb). Environment : Windows 10 (well, it's my work computer, I don't have a lot of choice) Monetdb 5 server v11.31.11 I start monetdb by using C:\Program Files\MonetDB\MonetDB5\bin\mserver5.exe Then mclient on C:\Program Files\MonetDB\MonetDB5\mclient.bat (and usually it work just fine) I also tried with mclient.exe using the -x command to debug. Error message : C:\Program Files\MonetDB\MonetDB5\bin>mclient -X -d demo user(win32):monetdb password: MAPI = monetdb@localhost:50000 ACTION= mapi_reconnect ERROR = !InvalidCredentialsException:checkCredentials:invalid credentials for user 'monetdb' Of course, I use the default monetdb/monetdb credentials. I have no idea of what happens and moreover I have lost an entire schema (I see that if I connect through Tableau Desktop...). I have done an upgrade on the last version (11.31.13) and same result, impossible to connect. And same result. I spend the entire evening on that a Thank in advance. Simon AUBERT [cid:691B9F4089B9F34B8E0C22B39B7117EC@eurprd07.prod.outlook.com] Bourse Maritime - 1, place Lainé - 33 000 Bordeaux Mob : +33(0)6 66 28 52 04

3 years, 6 months

table-returning function with subselect in input - no mitosis?

by Roberto Cornacchia

Hi, * Background: I have a custom string tokenizer that I invoke this way: SELECT * FROM tokenize( (SELECT id, s FROM my_strings) ); The output has of course a 1-to-N rows relation with the input. The function has a bulk implementation, so it takes M rows and it outputs M rows. * The question: No matter how large the input table my_strings is, mitosis does not seem to kick in. Is it not allowed in this case because it cannot assume that my custom function allows to concatenate partial results? Do you see any chance for me to enable mitosis with this scenario? Thanks, Roberto

3 years, 6 months

Announcement: New Aug2018-SP2 Bugfix release of MonetDB suite

by Sjoerd Mullender

The MonetDB team at CWI/MonetDB BV is pleased to announce the Aug2018-SP2 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://dev.monetdb.org/downloads/>. Aug 2018-SP2 bugfix release (11.31.13) SQL Frontend * Disabled function sys.getcontent(url). Bug Fixes * 6643: schema name qualifier in create global temporary table json.table6643 ... is not honered * 6645: optimizer treats all the functions with no or constant parameters as constant * 6650: PREPARE SQL statement fails to compile user defined functions with parameter/s * 6651: Multi-column IN clause for subquery produces wrong results * 6653: CREATE TABLE accepts empty table/column name * 6654: Incorrect handling of 'TRUE' in compound select * 6657: Restart sequence with a non atomic sub-query cardinality gives wrong error message * 6660: GRANTing a ROLE is not idempotent * 6662: Concurrency Conflicts Exception string "!transaction is aborted because of concurrency conflicts, will ROLLBACK instead" not shown on prompt. * 6664: mserver5 crash: infinite recursive happens at rel_bin.c:489 * 6665: Creation of serial types does not accept negative numbers * 6666: COPY INTO from .. LOCKED doubles input data * 6668: The SAMPLE key word doesn't work in a subquery. * 6669: COPY [xxx RECORDS] INTO foo FROM STDIN ... doesn't work without specifying nr of to be copied records * 6672: SQLGetData with SQL_C_WCHAR string truncation and invalid StrLen_or_Ind value

3 years, 6 months

Re: Performance degradation

by Jeremy Norris

Here's some additional observations on this puzzling issue: The slow table is "cmdb", the normal table is "cmdb2". Both have the same number of rows according to "SELECT count(*)" (73553310). However, sys.statistics shows something a bit different: sql>SELECT x.name, count(*) AS columns_with, z.count FROM sys.tables AS x, sys.columns AS y, sys.statistics AS z WHERE x.id = y.table_id AND y.id = z.column_id GROUP BY x.name, z.count; +-------+--------------+----------+ | name | columns_with | count | +=======+==============+==========+ | cmdb | 8 | 73553310 | | cmdb | 145 | 73553311 | | cmdb2 | 153 | 73553310 | +-------+--------------+----------+ 3 tuples sql> For the slow table, "cmdb", there are 145/153 columns that have an extra "row". This row count mismatch causes it to skip parallelization optimizations in "monetdb5/optimizer/opt_mitosis.c", line 212. A few related questions: 1) In the columnar model, is is valid for conceptual tuples to exist which don't have values for each column like this? (This seems strange as now there's a difference between an explicit NULL and "not existing"). How might this situation have happened? (Only transactional full-row inserts were applied to this DB). Can this be repaired? 2) Is the conditional in "monetdb5/optimizer/opt_mitosis.c", line 212 sufficiently robust? I may be misunderstanding the full context of this line, but it seems like we could still benefit from splitting/parallelization when r < rowcnt, no? Thanks, -Jeremy Norris

3 years, 6 months

Performance degradation

by Herman Knief

Greetings, Running MonetDB v11.31.11 (Aug2018-SP1), we ran into an issue where queries that have been running quite well, suddenly slowed by about 5x. Looking at the explain, it seems we hit a condition that is preventing mitosis? Copying the tables related to the query seems to restore performance to the previous level, but would be very curious to understand what happened here. What metrics or stats should we be looking at? Changes we see in the plan are as follows: Bad example: sql.tid(X_4:int, "waltest":str, "cmdb":str); Good example: sql.tid(X_4:int, "waltest":str, "cmdb2":str, 0:int, 32:int); Looking at the storage info, the sizes (heap and column), type width and counts look the same between the two tables. Any thoughts or suggestions would be appreciated. - Herman -- Please excuse typos... I can't type.

3 years, 6 months

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

users-list January 2019