Hi all –
I have always used the COPY BINARY INTO … commands to load my 2.0 Billion row genetic data into a monetdb table. With 135 columns, it has been blindingly fast.
Last week I moved from the June2016-SP2 release to dec2016-SP2. My binary loads are taking WAY longer. I killed one after 3 hours (via “call sys.stop(pid)” so it could clean up properly). I then started the load again, thinking perhaps the problem was related to the new columns I was adding.
I have since dropped the table and remade it using the same data and scripts that worked in just over 3 minutes in February on the jun2016-SP2 load. It is really chugging along – I’m up to 30 minutes and counting. I don’t have access to the sql log files, but the Merovingian.log shows nothing.
I do notice that previously the binary files, once loaded, were removed from the loading directly. This does not happen now. Were these files previously “moved” and now they are copied?
Has anyone see this performance issue with Dec2016-SP2 COPY BINARY INTO …. Commands?
Thanks - Lynn
Hi, all,
Consider the following scenario. I have a bunch of user defined functions, f_i, that create, populate, and use a local temporary table with the same name "res":
create function f_i()
returns table( ... )
begin
# Result stats table.
create local temporary table res(
param1 int,
param2 int
);
/* Populate res */
/* Use res in some queries */
return table(
/* Query involving res */
);
end;
My first question is: Does "create local temporary table x" create a table that is disposed/dropped as soon as the function where it was declared finishes execution? I assumed this was the case, but I'm now skeptical as we are running into all sorts of concurrency issues that point to the "res" table not being cleared out properly from memory.
This takes me to the second question: How do I drop a temporary table if such table exists? The reason for this is to prevent "table create" exceptions (which we are currently running into). I'd like to achieve something along these lines:
create function f_i()
returns table( ... )
begin
# Drop local temporary table if such table exists.
drop local temporary table res; <-------- How do I do this in MonetDB?
# Result stats table.
create local temporary table res(
param1 int,
param2 int
);
/* Populate res */
/* Use res in some queries */
return table(
/* Query involving res */
);
end;
Where, as you can see, I want to explicitly drop the local temporary table "res" if such exists. If I leave the "local temporary" markers in the drop statement, MonetDB complains that it found unexpected "LOCAL" or "TEMPORARY". I wouldn't like to remove these markers because a statement like:
drop table res;
will drop any table in the default function schema with that name.
Any help and hints are well appreciated!
Thanks for the help in advance
~ Luis Angel
The information transmitted, including any attachments, is intended only for the individual or entity to which it is addressed, and may contain confidential and/or privileged information. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by individuals or entities other than the intended recipient is prohibited, and all liability arising therefrom is disclaimed. If you have received this communication in error, please delete the information from any computer and notify the sender.
Hi,
We’d like to know if there is a way to create and execute prepared statements within an SQL function. In particular, once that we use:
“prepare select * from table”
How can I retrieve the prepared statement ID to use in an exec()?
Thanks,
Luis Ángel.
The information transmitted, including any attachments, is intended only for the individual or entity to which it is addressed, and may contain confidential and/or privileged information. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by individuals or entities other than the intended recipient is prohibited, and all liability arising therefrom is disclaimed. If you have received this communication in error, please delete the information from any computer and notify the sender.
Hi,
We are having issues when executing embedded R in a MonetDB SQL function. We have compiled and configured MonetDB accordingly, so as to be able to run embedded R; however, the simple example in the documentation fails to execute.
This is the function we have:
create function rapi00()
returns table( d int ) language R
{
return seq(1,10);
};
Having just "seq(1,10)" doesn't make any difference either. When this function gets created, we don't get any error; however, when we do the following:
select d from rapi00() as r where d > 5;
we get the this error:
Query failed:
TypeException:user.s6_3[3]:'rapi.eval' undefined in: X_4:bat[:int] := rapi.eval(7fdb9830e210:ptr,"{\n\treturn seq(1,10);\n};":str);
We don't know how to tackle this issue. We have followed the process from https://www.monetdb.org/content/embedded-r-monetdb. Do you have any suggestion to make this work?
Thanks!
Luis Ángel.
The information transmitted, including any attachments, is intended only for the individual or entity to which it is addressed, and may contain confidential and/or privileged information. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by individuals or entities other than the intended recipient is prohibited, and all liability arising therefrom is disclaimed. If you have received this communication in error, please delete the information from any computer and notify the sender.
Hello Stefan and developers –
Has any more thought been given to performance issues on binary bulk loading of large tables? I’m hoping to hear the “steal” option will be available in an upcoming release.
Thanks - Lynn
Can someone describe how exactly replica tables are used to push joins to
worker nodes?
Assume a master node and two worker nodes. I assume my reference table
would go on my master node, and on my worker nodes I would have my large
data table and replicas of the reference tables from the master node.
Is that the general strategy?
Thanks,
-K
Hi there, hope you are doing good.
We were trying to implement Monetdb/python i.e. embedded python and tried
the following code but its giving an error. (We have tried it on Linux)
CREATE FUNCTION python_times_two(i INTEGER) RETURNS INTEGER LANGUAGE PYTHON
{
return i * 2
};
CREATE TABLE integers(i INTEGER);
INSERT INTO integers VALUES (1), (2), (3), (4), (5);
SELECT python_times_two(i) AS result FROM integers;
Error:
TypeException:user.s12_1[10]:'pyapi.eval' undefined in: X_18:int :=
pyapi.eval(0x7f051006a340:ptr,"{\n return i * 2\n};":str,X_17:int);
Please help us on what should be done in order to code to work.
Also need clarification on following points:
1. Does embedded R or embedded python work on windows?
2. Does it take data in memory when we use machine learning with
Numpy and Scikitlearn?
Awaiting for your response.
Warm Regards,
Ganesh Anbhule
Data Scientist
Hi,
I wonder if anyone knows how I can retrieve all schemata on which a user has been granted permissions/authorization. I have checked most of the hinting tables I get from querying
>> select id, name, schema_id, query from sys.tables where system = true;
but I have had no luck with that. I only get the default schema for every user when I perform the following:
>> select * from sys.users
Any ideas or suggestions are welcome.
Thanks!
Luis Ángel.
The information transmitted, including any attachments, is intended only for the individual or entity to which it is addressed, and may contain confidential and/or privileged information. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by individuals or entities other than the intended recipient is prohibited, and all liability arising therefrom is disclaimed. If you have received this communication in error, please delete the information from any computer and notify the sender.
Hi,
How much time it will take for new release ?
On Wednesday 03 May 2017 03:30 PM, users-list-request(a)monetdb.org wrote:
> Error while installing Monetdb
Hi,
New release of monetdb is not installing on ubuntu 16.04 LTS. I
think problem will be your new release which is happened on 1 May 2017.
Change
https://dev.monetdb.org/downloads/deb/dists/xenial/monetdb/binary-amd64/Rel…:
Valid-Until: Tue, 30 Nov 2038 00:00:00 JST this is not working. JST to some other proper format
