July 2020 - users-list

Issues with None values handling in Python UDFs returning TABLE
by Daniel Glöckner 20 Jul '20

20 Jul '20

Hi, when testing TABLE-returning UDFs, I noticed that there are some issues with handling of None values from Python. Is there any magic approach on how to return NULLS / None values from such a UDF? Please see my examples below which are all not working ;) Kind regards, Daniel # 1 Does not work - Henk is just repeated in the second row drop function if exists python_table; CREATE FUNCTION python_table() RETURNS TABLE(name STRING, country STRING, age INTEGER) LANGUAGE PYTHON { result = dict() result['name'] = ['Henk', None, 'Elizabeth'] result['country'] = ['NL', 'USA', 'UK'] result['age'] = [25, 30, 33] return result }; SELECT * FROM python_table() ; # 2 Also does not work - Henk is just repeated in the second row drop function if exists python_table; CREATE FUNCTION python_table() RETURNS TABLE(name STRING, country STRING, age INTEGER) LANGUAGE PYTHON { result = dict() names = numpy.empty(3, dtype=object) names[0] = 'Henk' names[2] = 'Elizabeth' result['name'] = names result['country'] = ['NL', 'USA', 'UK'] result['age'] = [25, 30, 33] return result }; SELECT * FROM python_table() ; # 3 Also does not work - Henk is just repeated in the second row drop function if exists python_table; CREATE FUNCTION python_table() RETURNS TABLE(name STRING, country STRING, age INTEGER) LANGUAGE PYTHON { result = dict() names = numpy.full_like(numpy.arange(3, dtype=object), None) #names[0] = 'Henk' names[2] = 'Elizabeth' result['name'] = names result['country'] = ['NL', 'USA', 'UK'] result['age'] = [25, 30, 33] return result }; SELECT * FROM python_table() ; # 4 breaks if all values for a column are None # "Error converting dict return value "name": An array of size 3 was returned, yet we expect a list of 1 columns. The result is invalid.. " drop function if exists python_table; CREATE FUNCTION python_table() RETURNS TABLE(name STRING, country STRING, age INTEGER) LANGUAGE PYTHON { result = dict() names = numpy.full_like(numpy.arange(3, dtype=object), None) result['name'] = names result['country'] = ['NL', 'USA', 'UK'] result['age'] = [25, 30, 33] return result }; SELECT * FROM python_table() ;

1 0

Implementing bulk export to Parquet
by Daniel Glöckner 19 Jul '20

19 Jul '20

Hi, the project which I'm involved in would require us to integrate MonetDB with "big data technologies" so exporting data from MonetDB to Parquet for further processing in the big data tools would be an obvious choice. Can anyone guide me what it would mean to extend MonetDB with a writer for Parquet? It would be awesome if COPY INTO would be able to produce Parquet efficiently, e.g. using Arrow library: https://arrow.apache.org/docs/cpp/parquet.html Kind regards, Daniel

3 2

MonetDB/e and R
by Mark Clements 17 Jul '20

17 Jul '20

It is exciting to see the release of MonetDB/e - congratulations. I have two questions to offer. First, has anyone begun work on an R package? As a follow-up question if not, which of MonetDB.R, MonetDBLite or DuckDB packages for R would be a better starting point? Second, could someone kindly explain the potential advantages and disadvantages of using either MonetDB/e or DuckDB, please? They both seem to come from CWI, so this is a little confusing. Sincerely, Mark. När du skickar e-post till Karolinska Institutet (KI) innebär detta att KI kommer att behandla dina personuppgifter. Här finns information om hur KI behandlar personuppgifter<https://ki.se/medarbetare/integritetsskyddspolicy>. Sending email to Karolinska Institutet (KI) will result in KI processing your personal data. You can read more about KI’s processing of personal data here<https://ki.se/en/staff/data-protection-policy>.

3 2

PHP insert performance
by - - 16 Jul '20

16 Jul '20

I have run the series of INSERT tests in PHP for monetdb and SAP SQLAnywhere. The tests show: performance of the monetdb on INSERT operations is almost twice slower against SQLA with the equal conditions (DDL, index, data, autocommit, 10K records), monetdb is in in-memory mode . What can be a reason? p.s. It is not benchmark, I just used the SQLA test server already running. -- - -

4 12

News on MonetDB
by Martin Kersten 14 Jul '20

14 Jul '20

Hi, In recent weeks we have issued a few blog posts on the MonetDB site and announced them on Twitter. The latest one is a demonstration of MonetDB as a drop in library in Python. see https://monetdb.org/blog/taking-MonetDBe-for-a-taxi-ride Feedback on the embedded version of MonetDB is always welcome. regards, Martin Kersten

1 0

Error with concurrent inserts via PymonetDB mapi
by Ioannis Foufoulas 13 Jul '20

13 Jul '20

Hi, I have a weird error when I am sending the following queries in parallel from 2 different python clients with MAPI: insert into table1 select numpy_count(c1) as c1 from view; insert into table2 select numpy_count(c1) as c1 from view2; where numpy_count is: CREATE or replace AGGREGATE numpy_count(val INTEGER) RETURNS INTEGER LANGUAGE PYTHON { import time time.sleep(3) return val.size }; The error is: pymonetdb.exceptions.OperationalError: GDK reported error: GDKreleasesem: semctl failed: Invalid argument However, if I run these 2 queries in parallel from 2 instances of mclient they work and AFTER doing this, they also work from python’s client with mapi. Is there any hint on this? Best, Yannis

2 2

mserver5 options
by - - 12 Jul '20

12 Jul '20

Hi, How to pass/set mserver5 options via monetdb(d) ? I’m trying to get started monetdb in in-memory mode. mserver5 has option --in-memory Run database in-memory only however all attempts to pass it were unsuccessful. -- - -

2 4

Many aggregated columns / using custom aggregation function for pivot
by Daniel Glöckner 10 Jul '20

10 Jul '20

Hi, today I checked out how to implement custom aggregation functions in Python. Works pretty neat and scales well when you have few columns which you want to aggregate, however I have many :) Imagine that the following aggregate functions has variants which perform a similar iteration but call a different function (not sum). The call to numpy.unique() and the iteration would have to be performed for each aggregated column which is bad in terms of performance. Background: I have records in a key / value like structure so there's a key column and a value column. I want to group by primary key and then create a column for each key. It's like pivot but without aggregation. Any ideas how to do this with SQL or custom functions in MonetDB? Kind regards, Daniel CREATE AGGREGATE python_aggregate(val INTEGER) RETURNS INTEGER LANGUAGE PYTHON { try: unique = numpy.unique(aggr_group) x = numpy.zeros(shape=(unique.size)) for i in range(0, unique.size): x[i] = numpy.sum(val[aggr_group==unique[i]]) except NameError: # aggr_group doesn't exist. no groups, aggregate on all data x = numpy.sum(val) return(x) };

1 0

Killing long running SQL queries
by Daniel Glöckner 09 Jul '20

09 Jul '20

Hi, sometimes things just go wrong ;) Is there a way to 1) list (long) running queries and 2) kill them? Kind regards, Daniel

2 1

Re: Re: Bit of documentation lacking (indexes, performance)
by gloeckner.daniel＠gmail.com 09 Jul '20

09 Jul '20

Hi, Thanks Martin Kersten, Alex Ossipov and Martin van Dinther for the valuable advice! It’s actually quite impressive to see how well MonetDB being a “knobless” system performs without any performance tuning. That’s what you do most of your day when using “big data” technologies from my experience 😉 but it’s of course not what you’re ultimately aiming to do… I think it could be considered as a unique selling point for MonetDB. Maybe the website should highlight this more and improve new user’s confidence level (indexes are not missing features but tuned automatically (how?)…., etc.). I’ll give PKs and FKs a shot to see how they affect performance. General advice, hire a working student and have her or him revamp the website 😃 There’s quite a bit of information but sometimes outdated and hard to navigate. My journey into MonetDB land continues, no blockers so far. Kind regards, Daniel

2 1