users-list April 2018

users-list@monetdb.org

13 participants
15 discussions

GDK reported error. MT_mremap: mremap failed. !OS: Cannot allocate memory

by RanchoYuan

Hi Stefan, Thanks for your reply! We have run the query a few times with different size of data. There we used 16G RAM(actually 13.5G was used), and find the size of 10G's data is the critical point that can run the query. All of the data files' size are listed below, each file name is a table name(there are only a few tables are refered -- store_sales, date_dim, item, customer, catalog_sales, web_sales): 7.4K call_center.dat 1.6M catalog_page.dat 212M catalog_returns.dat 2.9G catalog_sales.dat 27M customer_address.dat 64M customer.dat 77M customer_demographics.dat 9.9M date_dim.dat 77B dbgen_version.dat 149K household_demographics.dat 328B income_band.dat 2.6G inventory.dat 28M item.dat 61K promotion.dat 1.7K reason.dat 1.1K ship_mode.dat 27K store.dat 323M store_returns.dat 3.8G store_sales.dat 4.9M time_dim.dat 1.2K warehouse.dat 19K web_page.dat 98M web_returns.dat 1.5G web_sales.dat 12K web_site.dat So we guess that the monetdb has no memory management? For the output of `mserver5 --version` is: MonetDB 5 server v11.27.13 "Jul2017-SP4" (64-bit, 128-bit integers) Copyright (c) 1993 - July 2008 CWI Copyright (c) August 2008 - 2018 MonetDB B.V., all rights reserved Visit https://www.monetdb.org/ for further information Found 17.0GiB available memory, 40 available cpu cores Libraries: libpcre: 8.38 2015-11-23 (compiled with 8.38) openssl: OpenSSL 1.0.2g 1 Mar 2016 (compiled with OpenSSL 1.0.2g 1 Mar 2016) libxml2: 2.9.3 (compiled with 2.9.3) Compiled by: monetdb(a)MonetDB-0.0 (x86_64-pc-linux-gnu) Compilation: gcc -g -O2 Linking : /usr/bin/ld -m elf_x86_64 And the size of processes is not limited. To let you reproduce the problem conveniently, I'll provide more details here: you can get tpc-ds from its website(we use version 2.6.0). Install the tpc-ds, access the directory v2.6.0/tools and run `./dsdgen -scale 10 -dir /home/monetdb/tpc-ds_test_data10G` to generate the data. When data has been generated, using the script /expe.sh/ to create tables and load the data. The query script is 123.tpcds.23.sql.(The syntaxs of other queries that tpc-ds generates is not suitable for monetdb all, we don't modify them all when the problem occurred). One more question, I can't get your reply email, so I don't know how to reply you, for this case, I could only send a new mail echo time. Thanks! Regards, Rancho

3 years

Copy a set database files and register on another database

by Murthy, Gautham

Hello, On MonetDB, Can we list the files for a schema and copy the files and register to another database ? We would want to know if we can - List the files for a specific table or a schema(all tables under a schema)? Copy the files to target database and register them on the other database Thank You,

4 years, 3 months

MonetDB Read-Only without restarting the database

by Murthy, Gautham

Hello, This is regarding read-only mode in MonetDB. Can we mark the database read-only without restarting the database ? If Yes, How do we do it ? What are the options available to copy a database without restarting to another node or DB Farm? Msqldump the export and import is serialized, How do I increase the performance of exp/imp? Thank You, Gautham

4 years, 3 months

mclient backward compatibility

by Roberto Cornacchia

mclient Mar2018 is not really compatible with mserver5 Jul2017. Not a real issue for me, but I'm just curious: is mclient supposed to be compatible with an earlier release of mserver5 (at least the previous release)? $ mclient x -u monetdb password: Welcome to mclient, the MonetDB/SQL interactive terminal (Mar2018) Database: MonetDB v11.27.14 (unreleased), 'mapi:monetdb:// photon.hq.spinque.com:55500/x' Type \q to quit, \? for a list of available commands auto commit mode: on sql>-- \d silently fails (nothing dumped) sql>\d sql> sql>-- \d on a table doesn't find table comments (introduced in Mar2018) sql>\d tables SELECT: no such table 'comments' Shouldn't queries that rely on the system catalog be defined server-side?

4 years, 3 months

double precision IS NULL does not work ?

by Nicolas Paris

Hello I have those DOUBLE PRECISION fields, sometimes containing null values: sql>select count(1), range_low from omop.measurement group by range_low; +-----------+--------------------------+ | L2 | range_low | +===========+==========================+ | 304875924 | null | | 9448250 | 50 | | 16519843 | 0 | | 13198932 | 10 | | 14364002 | 20 | | 3964808 | 30 | | 2998528 | 3 | | 855829 | 2 | +-----------+--------------------------+ 8 tuples However, I am not able to filter based on is null: sql>select count(1) from omop.measurement where range_low is null; +------+ | L3 | +======+ | 0 | +------+ 1 tuple I am not able to understand if this behavior is normal regarding the documentation: "The types REAL, FLOAT and DOUBLE map to the underlying implementation system. No special attention is given to the value NaN." Thanks

4 years, 3 months

mserver5 process consuming resources while not busy

by Nicolas Paris

Hello Sorry, I gave up digging in the mailing list because didn't found search functionality there ( https://www.monetdb.org/pipermail/users-list/ ) Then I wonder why I get that result when running "top" command on my server: > PID UTIL. PR NI VIRT RES SHR S %CPU %MEM TEMPS+ COM. > 3297 monetdb 20 0 53,007g 9,028g 8,396g S 100,5 28,8 117:20.95 mserver5 There is no query running AFAIK (select * from sys.queue is empty). I loaded my database several hours before from scratch, and then ran some sql. How can I figure out what appends in the background ? Is it indexing based on queries I made ? Apparently the disk is stable, by mean there is no creation of new indexes, or files under the hood. Any insight welcome, Thanks,

4 years, 3 months

MonetDB Settings and Open Files

by Murthy, Gautham

Hello, We have MonetDB open files usage reaching 65k and beyond. How do we minimize the openfiles, We see most of them are mapped to memory. What is the storage(); function? What does the storage() function do when we execute it ? The usage of open files drastically reduced when the select *from storage() was executed on the database. What are the effects of running the select *from storage(); when users are accessing the database? What is the maximum size of database MonetDB can support ? Does running the storage() function when there is a large set of open files crash the database ? Do we need to set any system level parameters ? shmmax, file-max etc., Do we have any standard settings as prerequisites that needs to be considered if we are hosting databases ranging 1TB to 2TB? What should be the open files setting if we have 6-10 databases with size ranging from 100GB to 2TB ? Do we have options to export specific schemas rather that complete database or tables? Request you to please review and share the above information. Open files Test Case - DB size 630Gb Max Open files set to - 65536 [oracle@lnx1386 scripts]$ sh 10min_ofc.sh lsof -u oracle|wc -l Timestamp: 04-14-18 01:56:34 Open Files: 6559 lsof -u oracle|grep -i tcp|wc -l Timestamp: 04-14-18 01:56:34 Open Files Of TCP: 21 lsof -u oracle | grep -i /data/MYDBFARM_DIR/SV11DMP_DBFARM/TSV_PROD_DB | wc -l Timestamp: 04-14-18 01:56:35 Open Files - DB Files : 5864 lsof -u oracle | grep -i protocol: TCP | wc -l Timestamp: 04-14-18 01:56:35 Open Files Of - protocol: TCP : 2 lsof -u oracle | grep -i CLOSE_WAIT | wc -l Timestamp: 04-14-18 01:56:35 Open Files of TCP - CLOSE_WAIT: 0 lsof -u oracle | grep -i ESTABLISHED | wc -l Timestamp: 04-14-18 01:56:36 Open Files Of TCP - ESTABLISHED: 15 netstat -an |grep tcp|wc -l Timestamp: 04-14-18 01:56:36 netstat -an |grep tcp : 62 [oracle@lnx1386 scripts]$ sh /home/oracle/scripts/purge_op_fl.sh 50005 TSV_PROD_DB 1>/home/oracle/scripts/logs/purge_op_fl.stout 2>/home/oracle/scripts/logs/purge_op_fl.sterr [oracle@lnx1386 scripts]$ cat /home/oracle/scripts/purge_op_fl.sh /usr/bin/mclient -p ${1} -d ${2} < /home/oracle/scripts/purge_op_fl.sql [oracle@lnx1386 scripts]$ cat /home/oracle/scripts/purge_op_fl.sql select * from storage(); [oracle@lnx1386 scripts]$ [oracle@lnx1386 scripts]$ sh 10min_ofc.sh lsof -u oracle|wc -l Timestamp: 04-14-18 02:09:49 Open Files: 862 lsof -u oracle|grep -i tcp|wc -l Timestamp: 04-14-18 02:09:49 Open Files Of TCP: 25 lsof -u oracle | grep -i /data/MYDBFARM_DIR/SV11DMP_DBFARM/TSV_PROD_DB | wc -l Timestamp: 04-14-18 02:09:50 Open Files - DB Files : 163 lsof -u oracle | grep -i protocol: TCP | wc -l Timestamp: 04-14-18 02:09:50 Open Files Of - protocol: TCP : 2 lsof -u oracle | grep -i CLOSE_WAIT | wc -l Timestamp: 04-14-18 02:09:50 Open Files of TCP - CLOSE_WAIT: 0 lsof -u oracle | grep -i ESTABLISHED | wc -l Timestamp: 04-14-18 02:09:50 Open Files Of TCP - ESTABLISHED: 19 netstat -an |grep tcp|wc -l Timestamp: 04-14-18 02:09:50 netstat -an |grep tcp : 66 [oracle@lnx1386 scripts]$ Thank You, Gautham

4 years, 3 months

Window Functions

by Doherty, Adam

Hello: I am trying to build an event tree from a series timestamps, however for each event I want to be able to return the preceding and succeeding events +/- 2 seconds to capture a complete series of events. I believe this is called lead/lag in other databases. I'm not sure how to proceed in building a CTE, short of pulling all the events out an running a filter function on them in Python. Thanks kindly, Adam Doherty

4 years, 3 months

statistical MODE function

by Sharma, Sreejith

All, Is there any statistical MODE function available in MonetDB? If not, has anyone tried it for custom function? Need help here. Regards, Sreejith

4 years, 3 months

INSERT using multiple cores (inspired by COPY from CSV in LOCKED mode)

by Anton Kravchenko

Hi there, 1) It looks like INSERTing into tables is bounded by a single CPU. And it kills performance of a query that normally uses multiple CPUs... CREATE TEMP TABLE t1 AS (some 'select query' that uses multiple cores when run standalone) ON COMMIT PRESERVE ROWS; or INSERT into t1 (some 'select query' that uses multiple cores when run standalone); 2) Do you think you could implement INSERT with LOCKED option? (like it was done for COPY from CSV in LOCKED mode) Please let me know whether you think it's doable. Thank you, Anton

4 years, 3 months

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

users-list April 2018