April 2018 - users-list

[MonetDB-users] Proofreading your professional documents
by Richard Hounslow 24 Feb '24

24 Feb '24

Dear Professor/Lecturer Please excuse us for contacting you directly. We are offering an efficient service to help you ensure your academic and professional work is written in perfect English. We will check the grammar and style of your work and return it to you to meet your requirements and deadlines. Proof-Reading-Service.com can provide you with a professional proofreading service at a very reasonable rate. All our proofreaders are highly qualified native English speakers. Many work as leading academics in their fields and all have extensive experience of proofreading to the highest standards. If you are interested in our service, please take a look at our website: www.proof-reading-service.com All you have to do is send us your document as a word attachment with the deadline and we will guarantee delivery of a perfectly written document to give you complete confidence when you submit your work. The fee is worked out on a flat rate (£6.95 per thousand words or 0.695 pence per word), so you know exactly how much the proofreading will cost in advance. Once we have received your document(s) we will confirm the word count, the price and the deadline. If you have not heard from us within 2 hours during normal business hours after you have sent your work please resend it from a different email account or send the file to proofreadings(a)gmail.com. We look forward to hearing from you! Yours faithfully Proof-Reading-Service.com

2 2

GDK reported error. MT_mremap: mremap failed. !OS: Cannot allocate memory
by RanchoYuan 04 Jul '19

04 Jul '19

Hi Stefan, Thanks for your reply! We have run the query a few times with different size of data. There we used 16G RAM(actually 13.5G was used), and find the size of 10G's data is the critical point that can run the query. All of the data files' size are listed below, each file name is a table name(there are only a few tables are refered -- store_sales, date_dim, item, customer, catalog_sales, web_sales): 7.4K call_center.dat 1.6M catalog_page.dat 212M catalog_returns.dat 2.9G catalog_sales.dat 27M customer_address.dat 64M customer.dat 77M customer_demographics.dat 9.9M date_dim.dat 77B dbgen_version.dat 149K household_demographics.dat 328B income_band.dat 2.6G inventory.dat 28M item.dat 61K promotion.dat 1.7K reason.dat 1.1K ship_mode.dat 27K store.dat 323M store_returns.dat 3.8G store_sales.dat 4.9M time_dim.dat 1.2K warehouse.dat 19K web_page.dat 98M web_returns.dat 1.5G web_sales.dat 12K web_site.dat So we guess that the monetdb has no memory management? For the output of `mserver5 --version` is: MonetDB 5 server v11.27.13 "Jul2017-SP4" (64-bit, 128-bit integers) Copyright (c) 1993 - July 2008 CWI Copyright (c) August 2008 - 2018 MonetDB B.V., all rights reserved Visit https://www.monetdb.org/ for further information Found 17.0GiB available memory, 40 available cpu cores Libraries: libpcre: 8.38 2015-11-23 (compiled with 8.38) openssl: OpenSSL 1.0.2g 1 Mar 2016 (compiled with OpenSSL 1.0.2g 1 Mar 2016) libxml2: 2.9.3 (compiled with 2.9.3) Compiled by: monetdb(a)MonetDB-0.0 (x86_64-pc-linux-gnu) Compilation: gcc -g -O2 Linking : /usr/bin/ld -m elf_x86_64 And the size of processes is not limited. To let you reproduce the problem conveniently, I'll provide more details here: you can get tpc-ds from its website(we use version 2.6.0). Install the tpc-ds, access the directory v2.6.0/tools and run `./dsdgen -scale 10 -dir /home/monetdb/tpc-ds_test_data10G` to generate the data. When data has been generated, using the script /expe.sh/ to create tables and load the data. The query script is 123.tpcds.23.sql.(The syntaxs of other queries that tpc-ds generates is not suitable for monetdb all, we don't modify them all when the problem occurred). One more question, I can't get your reply email, so I don't know how to reply you, for this case, I could only send a new mail echo time. Thanks! Regards, Rancho

4 3

Copy a set database files and register on another database
by Murthy, Gautham 24 Apr '18

24 Apr '18

Hello, On MonetDB, Can we list the files for a schema and copy the files and register to another database ? We would want to know if we can - List the files for a specific table or a schema(all tables under a schema)? Copy the files to target database and register them on the other database Thank You,

1 0

MonetDB Read-Only without restarting the database
by Murthy, Gautham 24 Apr '18

24 Apr '18

Hello, This is regarding read-only mode in MonetDB. Can we mark the database read-only without restarting the database ? If Yes, How do we do it ? What are the options available to copy a database without restarting to another node or DB Farm? Msqldump the export and import is serialized, How do I increase the performance of exp/imp? Thank You, Gautham

1 0

mclient backward compatibility
by Roberto Cornacchia 18 Apr '18

18 Apr '18

mclient Mar2018 is not really compatible with mserver5 Jul2017. Not a real issue for me, but I'm just curious: is mclient supposed to be compatible with an earlier release of mserver5 (at least the previous release)? $ mclient x -u monetdb password: Welcome to mclient, the MonetDB/SQL interactive terminal (Mar2018) Database: MonetDB v11.27.14 (unreleased), 'mapi:monetdb:// photon.hq.spinque.com:55500/x' Type \q to quit, \? for a list of available commands auto commit mode: on sql>-- \d silently fails (nothing dumped) sql>\d sql> sql>-- \d on a table doesn't find table comments (introduced in Mar2018) sql>\d tables SELECT: no such table 'comments' Shouldn't queries that rely on the system catalog be defined server-side?

2 1

double precision IS NULL does not work ?
by Nicolas Paris 16 Apr '18

16 Apr '18

Hello I have those DOUBLE PRECISION fields, sometimes containing null values: sql>select count(1), range_low from omop.measurement group by range_low; +-----------+--------------------------+ | L2 | range_low | +===========+==========================+ | 304875924 | null | | 9448250 | 50 | | 16519843 | 0 | | 13198932 | 10 | | 14364002 | 20 | | 3964808 | 30 | | 2998528 | 3 | | 855829 | 2 | +-----------+--------------------------+ 8 tuples However, I am not able to filter based on is null: sql>select count(1) from omop.measurement where range_low is null; +------+ | L3 | +======+ | 0 | +------+ 1 tuple I am not able to understand if this behavior is normal regarding the documentation: "The types REAL, FLOAT and DOUBLE map to the underlying implementation system. No special attention is given to the value NaN." Thanks

2 1

mserver5 process consuming resources while not busy
by Nicolas Paris 15 Apr '18

15 Apr '18

Hello Sorry, I gave up digging in the mailing list because didn't found search functionality there ( https://www.monetdb.org/pipermail/users-list/ ) Then I wonder why I get that result when running "top" command on my server: > PID UTIL. PR NI VIRT RES SHR S %CPU %MEM TEMPS+ COM. > 3297 monetdb 20 0 53,007g 9,028g 8,396g S 100,5 28,8 117:20.95 mserver5 There is no query running AFAIK (select * from sys.queue is empty). I loaded my database several hours before from scratch, and then ran some sql. How can I figure out what appends in the background ? Is it indexing based on queries I made ? Apparently the disk is stable, by mean there is no creation of new indexes, or files under the hood. Any insight welcome, Thanks,

1 0

MonetDB Settings and Open Files
by Murthy, Gautham 14 Apr '18

14 Apr '18

Hello, We have MonetDB open files usage reaching 65k and beyond. How do we minimize the openfiles, We see most of them are mapped to memory. What is the storage(); function? What does the storage() function do when we execute it ? The usage of open files drastically reduced when the select *from storage() was executed on the database. What are the effects of running the select *from storage(); when users are accessing the database? What is the maximum size of database MonetDB can support ? Does running the storage() function when there is a large set of open files crash the database ? Do we need to set any system level parameters ? shmmax, file-max etc., Do we have any standard settings as prerequisites that needs to be considered if we are hosting databases ranging 1TB to 2TB? What should be the open files setting if we have 6-10 databases with size ranging from 100GB to 2TB ? Do we have options to export specific schemas rather that complete database or tables? Request you to please review and share the above information. Open files Test Case - DB size 630Gb Max Open files set to - 65536 [oracle@lnx1386 scripts]$ sh 10min_ofc.sh lsof -u oracle|wc -l Timestamp: 04-14-18 01:56:34 Open Files: 6559 lsof -u oracle|grep -i tcp|wc -l Timestamp: 04-14-18 01:56:34 Open Files Of TCP: 21 lsof -u oracle | grep -i /data/MYDBFARM_DIR/SV11DMP_DBFARM/TSV_PROD_DB | wc -l Timestamp: 04-14-18 01:56:35 Open Files - DB Files : 5864 lsof -u oracle | grep -i protocol: TCP | wc -l Timestamp: 04-14-18 01:56:35 Open Files Of - protocol: TCP : 2 lsof -u oracle | grep -i CLOSE_WAIT | wc -l Timestamp: 04-14-18 01:56:35 Open Files of TCP - CLOSE_WAIT: 0 lsof -u oracle | grep -i ESTABLISHED | wc -l Timestamp: 04-14-18 01:56:36 Open Files Of TCP - ESTABLISHED: 15 netstat -an |grep tcp|wc -l Timestamp: 04-14-18 01:56:36 netstat -an |grep tcp : 62 [oracle@lnx1386 scripts]$ sh /home/oracle/scripts/purge_op_fl.sh 50005 TSV_PROD_DB 1>/home/oracle/scripts/logs/purge_op_fl.stout 2>/home/oracle/scripts/logs/purge_op_fl.sterr [oracle@lnx1386 scripts]$ cat /home/oracle/scripts/purge_op_fl.sh /usr/bin/mclient -p ${1} -d ${2} < /home/oracle/scripts/purge_op_fl.sql [oracle@lnx1386 scripts]$ cat /home/oracle/scripts/purge_op_fl.sql select * from storage(); [oracle@lnx1386 scripts]$ [oracle@lnx1386 scripts]$ sh 10min_ofc.sh lsof -u oracle|wc -l Timestamp: 04-14-18 02:09:49 Open Files: 862 lsof -u oracle|grep -i tcp|wc -l Timestamp: 04-14-18 02:09:49 Open Files Of TCP: 25 lsof -u oracle | grep -i /data/MYDBFARM_DIR/SV11DMP_DBFARM/TSV_PROD_DB | wc -l Timestamp: 04-14-18 02:09:50 Open Files - DB Files : 163 lsof -u oracle | grep -i protocol: TCP | wc -l Timestamp: 04-14-18 02:09:50 Open Files Of - protocol: TCP : 2 lsof -u oracle | grep -i CLOSE_WAIT | wc -l Timestamp: 04-14-18 02:09:50 Open Files of TCP - CLOSE_WAIT: 0 lsof -u oracle | grep -i ESTABLISHED | wc -l Timestamp: 04-14-18 02:09:50 Open Files Of TCP - ESTABLISHED: 19 netstat -an |grep tcp|wc -l Timestamp: 04-14-18 02:09:50 netstat -an |grep tcp : 66 [oracle@lnx1386 scripts]$ Thank You, Gautham

2 1

Window Functions
by Doherty, Adam 13 Apr '18

13 Apr '18

Hello: I am trying to build an event tree from a series timestamps, however for each event I want to be able to return the preceding and succeeding events +/- 2 seconds to capture a complete series of events. I believe this is called lead/lag in other databases. I'm not sure how to proceed in building a CTE, short of pulling all the events out an running a filter function on them in Python. Thanks kindly, Adam Doherty

1 0

statistical MODE function
by Sharma, Sreejith 13 Apr '18

13 Apr '18

All, Is there any statistical MODE function available in MonetDB? If not, has anyone tried it for custom function? Need help here. Regards, Sreejith

2 1