users-list August 2017

users-list@monetdb.org

18 participants
18 discussions

DEC2016-SP2 and BINARY bulk load

by Lynn Carol Johnson

Hi all – I have always used the COPY BINARY INTO … commands to load my 2.0 Billion row genetic data into a monetdb table. With 135 columns, it has been blindingly fast. Last week I moved from the June2016-SP2 release to dec2016-SP2. My binary loads are taking WAY longer. I killed one after 3 hours (via “call sys.stop(pid)” so it could clean up properly). I then started the load again, thinking perhaps the problem was related to the new columns I was adding. I have since dropped the table and remade it using the same data and scripts that worked in just over 3 minutes in February on the jun2016-SP2 load. It is really chugging along – I’m up to 30 minutes and counting. I don’t have access to the sql log files, but the Merovingian.log shows nothing. I do notice that previously the binary files, once loaded, were removed from the loading directly. This does not happen now. Were these files previously “moved” and now they are copied? Has anyone see this performance issue with Dec2016-SP2 COPY BINARY INTO …. Commands? Thanks - Lynn

4 years, 6 months

MonetDB Blob Inserts

by Brian Hood

Hi Guys, I don't seem to be able to INSERT into a blob field is there any information how to achieve this. Regards, Brian Hood

4 years, 7 months

Setting Up Tools

by Sandy

Just getting back into trying MonetDB after many years. Got the environment set up pretty well with the test DB `VOC` loaded up and able to do all expected in the command line tools. So I figured I would give a couple of the mentioned GUI tools a shot and see how they work against the server I just set up on a virtual box. I picked up the JDBC drivers and installed on my Window's 10 install of SQuirrel SQL and DBeaver. I got both of the drivers works sort of... On DBeaver I can execute and see queries in the tool but can't do much more than that. Overall looks like an issue with accessing the system table `sys.environment`, not sure if anything related to set up or rights. Using the `voc` user as created with the example db in the getting started. Any help or thought on where to start? Thanks! Sandy -------- Errors * DBeaver 4.0.8 On inspection of some of the logs I see things like - `Error loading object metadata from database` `SQL Error [22000]: SELECT: access denied for voc to table 'sys.environment' java.sql.SQLException: SELECT: access denied for voc to table 'sys.environment'` * SQuirreL SQL Client 3.7.1 java.sql.SQLException: SELECT: access denied for voc to table 'sys.environment' -------- Driver and DB versions I'm using this JAR for the drivers - C:\Users\Squidward\Downloads\monetdb-jdbc-2.25.jar, 2.25 (Liberica 20170330 based on MCL v1.15) MonetDB version - MonetDB v11.25.23 (Dec2016-SP5), 'mapi:monetdb://sandy-VirtualBox-18:50000/voc'

4 years, 9 months

How to get Memory Dump. Is there any way to eliminate Memory leaks.

by Sharma, Sreejith

All, Is there any way to get the memory dump from MonetDB? The memory consumed by merver5 post concurrent query execution are not coming down. Need to investigate why memory leak is happening here. Regards, Sreejith

4 years, 10 months

Monetdb on windows with embedded python 3.6

by imad hajj chahine

Hi, I have been successfully using monetdb (11.25.3) on windows with embedded python 2.7, recently I needed a library that is only supported on python 3.6, is it possible somehow to point monetdb to the python 3.6 folder? I know this is possible on linux, but my dev environment is on windows. Thank you.

4 years, 10 months

Defining a table with a sorted column

by Cota, Luis

Is it possible to define a table such that the server maintains the table sorted with each new row? I'm dealing with observations on market data, and while market data for a specific stock symbol will be sorted, I will be adding multiple symbols to the same table in chunks, so that each symbol is contained within its own load file, and is sorted within that file. How can I enforce this constraint? - Luis ________________________________ IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.

4 years, 10 months

Function with table as argument that return a table

by imad hajj chahine

Hi, I want to create a Python function that accept a query as argument, do some processing and return another table, In addition to the table columns I want to have some additional scalar parameters and i am getting the following error: function requires a single sub query. CREATE function f1(col1 string, col2 int, col3 double, scalar1 int) RETURNS TABLE(col1 string, col2 double, col3 double, col4 double) LANGUAGE PYTHON Is this the normal behavior of Table function, as in aggregate function I can combine columns with scalar parameters, any workaround for this issue. Thank you.

4 years, 10 months

Possible bug with table generating functions

by Mark Clements

Is there a restriction on the use of the same table generating function in the same select statement? This relates to the following possible bug: -- reimplement generate_series for MonetDBLite CREATE FUNCTION my_generate_series(start INT, finish INT) RETURNS TABLE (value INT) BEGIN DECLARE TABLE tmp_generate_series(value INT); DECLARE i INT; SET i = start; WHILE (i <= finish) DO INSERT INTO tmp_generate_series VALUES(i); SET i = i + 1; END WHILE; RETURN tmp_generate_series; END; --- This seems to work quite nicely: select * from my_generate_series(1,10); -- and it's fast: select count(*) from my_generate_series(1,100000); -- however, I get odd and non-deterministic counts for the following: select count(*) from my_generate_series(1,100) as t1, my_generate_series(1,100) as t2; This is using MonetDB v11.23.3 (Jun2016) that was compiled with embedded R, but not run with --set embedded_r=true. I got similar errors with MonetDBLite. Kindly, Mark.

4 years, 11 months

Looking for Admin UI / Tools / Resources

by Cota, Luis

Can anyone recommend any tools (commercial or open source) which assist in administering a MonetDB instance? Things I'd like to do are: Create tables from existing data files (CSV, TXT, HDF5) Import or query Oracle databases and either create tables, or populate tables, with the results of those queries Query existing monetdb instances Thanks, Luis ________________________________ IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.

4 years, 11 months

Error after updating monetdb

by Mahmoud Mohsen

Hello, I have updated monetdb from Dec2016-SP5 to version v11.27.5 "Jul2017-SP1, but after this when trying to accessing the database via mclient -u monetdb -d tpch2 I am getting this error: internal error while starting mserver 'database 'tpch2' started up, but failed to open up a communication channel', please refer to the logs looking at the logs, 2017-08-18 22:42:02 ERR merovingian[10645]: client error: database 'voc' started up, but failed to open up a communication channel 2017-08-18 22:42:02 MSG merovingian[10645]: database 'voc' (10660) has exited with exit status 0 2017-08-18 22:42:02 MSG merovingian[10645]: starting database 'voc', up min/avg/max: 0s/0s/0s, crash average: 0.00 0.00 0.00 (1-1=0) 2017-08-18 22:42:02 MSG merovingian[10645]: sending process 10663 (database 'voc') the TERM signal 2017-08-18 22:42:02 MSG voc[10663]: arguments: /usr/bin/mserver5 --dbpath=/home/mohsen/mfarm/voc --set merovingian_uri=mapi:monetdb://u145-16:50000/voc --set mapi_open=false --set mapi_port=0 --set mapi_usock=/home/mohsen/mfarm/voc/.mapi.sock --set monet_vault_key=/home/mohsen/mfarm/voc/.vaultkey --set gdk_nr_threads=8 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2017-08-18 22:42:02 MSG voc[10663]: # MonetDB 5 server v11.27.5 "Jul2017-SP1" 2017-08-18 22:42:02 MSG voc[10663]: # Serving database 'voc', using 8 threads 2017-08-18 22:42:02 MSG voc[10663]: # Compiled for x86_64-pc-linux-gnu/64bit with 128bit integers 2017-08-18 22:42:02 MSG voc[10663]: # Found 251.889 GiB available main-memory. 2017-08-18 22:42:02 MSG voc[10663]: # Copyright (c) 1993-July 2008 CWI. 2017-08-18 22:42:02 MSG voc[10663]: # Copyright (c) August 2008-2017 MonetDB B.V., all rights reserved 2017-08-18 22:42:02 MSG voc[10663]: # Visit https://www.monetdb.org/ for further information 2017-08-18 22:42:02 MSG voc[10663]: !SyntaxException:parseError:include sqlcatalog; 2017-08-18 22:42:02 MSG voc[10663]: !SyntaxException:parseError:^LoaderException:loadLibrary:Loading error failed to open library sql (from within file '/usr/lib/monetdb5/lib_sql.so'): /usr/lib/monetdb5/lib_sql.so: undefined symbol: mnstr_writeFlt 2017-08-18 22:42:02 MSG voc[10663]: !SyntaxException:parseError:include sqlcatalog; 2017-08-18 22:42:02 MSG voc[10663]: !SyntaxException:parseError: ^';' expected 2017-08-18 22:42:03 MSG merovingian[10645]: database 'voc' has shut down 2017-08-18 22:42:03 ERR control[10645]: !monetdbd: an internal error has occurred 'database 'voc' started up, but failed to open up a communication channel' 2017-08-18 22:42:07 ERR merovingian[10645]: client error: database 'voc' started up, but failed to open up a communication channel 2017-08-18 22:42:07 MSG merovingian[10645]: database 'voc' (10663) has exited with exit status 0 Thank you a lot in advance, Best regards, Mahmoud Mohsen

4 years, 11 months

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

users-list August 2017