User defined aggregation function (SQL)
Hi all,
I am evaluating MonetDB in the context of genomic data. Therefore I'm trying to implement a custom aggregation function.
To get a feeling for the system I set up a UDF that just returns identical values. Now I'm trying to implement this simple function as aggregate function for groupings but always run into an error when executing this:
select grpident(base) from test.bases group by pos;
"TypeException:user.s1_1[18]:'udf.subident' undefined in: _30:bat[:any,:str] := udf.subident(_15:bat[:oid,:str], _27:bat[:oid,:oid], r1_27:bat[:oid,:oid], _1:bit) program contains errors"
Here is the MAL plan: +-----------------------------------------------------------------+ | mal | +=================================================================+ | function user.s1_1{autoCommit=true}():void; | | X_2 := sql.mvc(); | | X_3:bat[:oid,:oid] := sql.tid(X_2,"test","bases"); | | X_6 := sql.bind(X_2,"test","bases","base",0); | | (X_9,r1_9) := sql.bind(X_2,"test","bases","base",2); | | X_12 := sql.bind(X_2,"test","bases","base",1); | | X_14 := sql.delta(X_6,X_9,r1_9,X_12); | | X_15 := algebra.leftfetchjoin(X_3,X_14); | | calc.str("bases"); | | calc.str("base"); | | X_18 := sql.bind(X_2,"test","bases","pos",0); | | (X_20,r1_20) := sql.bind(X_2,"test","bases","pos",2); | | X_22 := sql.bind(X_2,"test","bases","pos",1); | | X_23 := sql.delta(X_18,X_20,r1_20,X_22); | | X_24 := algebra.leftfetchjoin(X_3,X_23); | | calc.str("bases"); | | calc.str("pos"); | | (X_27,r1_27,r2_27) := group.subgroupdone(X_24); | | X_30:bat[:any,:str] := udf.subident(X_15,X_27,r1_27,true); | | calc.str("bases"); | | calc.str("L1"); | | calc.str("bases"); | | calc.str("L1"); | | X_36 := sql.resultSet(1,1,X_30); | | sql.rsColumn(X_36,"test.bases","L1","clob",1,0,X_30); | | X_40 := io.stdout(); | | sql.exportResult(X_40,X_36); | | end s1_1; | +-----------------------------------------------------------------+
I'm using following setup: MonetDB 5 server v11.15.7 "Feb2013-SP2" (64-bit, 64-bit oids) Found 1.9GiB available memory, 2 available cpu cores Libraries: libpcre: 8.31 2012-07-06 (compiled with 8.31) openssl: OpenSSL 1.0.1c 10 May 2012 (compiled with OpenSSL 1.0.1c 10 May 2012) libxml2: 2.9.0 (compiled with 2.9.0)
Here's what my related code looks like:
udf.c ##### str GRPident(int *ret, int *bid, int *eid){ ... }
udf.h ##### udf_export str GRPident(int *ret, int *bid, int *eid);
udf.mal ####### module udf ... command ident(b:bat[:oid,:str], e:bat[:oid,:str]) :bat[:oid,:str] address GRPident comment "identity of groups";
udf.sql ####### create aggregate grpident(src string) returns string external name udf.ident;
I already renamed the MAL command to "subident" and changed the SQL expression accordingly without success. Furthermore I already had a look at "batxml.c" as suggested here: http://old.nabble.com/users-list-Digest,-Vol-3,-Issue-30-td34736774.html but don't see what I'm doing wrong.
What do I have to do to implement a custom aggregate function?
Thanks, Sebastian
Hi Sebastian,
I understand your pain, had to go through the same process. What version of Monet are you using? The aggr definition changed a bit between the two last versions I think. Also, one problem I hit was that the bat signatures have to be precise, otherwise even if the code compiles, it will throw an error.
On 06/17/2013 01:59 PM, Sebastian Dorok wrote:
Hi all,
I am evaluating MonetDB in the context of genomic data. Therefore I'm trying to implement a custom aggregation function.
To get a feeling for the system I set up a UDF that just returns identical values. Now I'm trying to implement this simple function as aggregate function for groupings but always run into an error when executing this:
select grpident(base) from test.bases group by pos;
"TypeException:user.s1_1[18]:'udf.subident' undefined in: _30:bat[:any,:str] := udf.subident(_15:bat[:oid,:str], _27:bat[:oid,:oid], r1_27:bat[:oid,:oid], _1:bit) program contains errors"
Here is the MAL plan: +-----------------------------------------------------------------+ | mal | +=================================================================+ | function user.s1_1{autoCommit=true}():void; | | X_2 := sql.mvc(); | | X_3:bat[:oid,:oid] := sql.tid(X_2,"test","bases"); | | X_6 := sql.bind(X_2,"test","bases","base",0); | | (X_9,r1_9) := sql.bind(X_2,"test","bases","base",2); | | X_12 := sql.bind(X_2,"test","bases","base",1); | | X_14 := sql.delta(X_6,X_9,r1_9,X_12); | | X_15 := algebra.leftfetchjoin(X_3,X_14); | | calc.str("bases"); | | calc.str("base"); | | X_18 := sql.bind(X_2,"test","bases","pos",0); | | (X_20,r1_20) := sql.bind(X_2,"test","bases","pos",2); | | X_22 := sql.bind(X_2,"test","bases","pos",1); | | X_23 := sql.delta(X_18,X_20,r1_20,X_22); | | X_24 := algebra.leftfetchjoin(X_3,X_23); | | calc.str("bases"); | | calc.str("pos"); | | (X_27,r1_27,r2_27) := group.subgroupdone(X_24); | | X_30:bat[:any,:str] := udf.subident(X_15,X_27,r1_27,true); | | calc.str("bases"); | | calc.str("L1"); | | calc.str("bases"); | | calc.str("L1"); | | X_36 := sql.resultSet(1,1,X_30); | | sql.rsColumn(X_36,"test.bases","L1","clob",1,0,X_30); | | X_40 := io.stdout(); | | sql.exportResult(X_40,X_36); | | end s1_1; | +-----------------------------------------------------------------+
I'm using following setup: MonetDB 5 server v11.15.7 "Feb2013-SP2" (64-bit, 64-bit oids) Found 1.9GiB available memory, 2 available cpu cores Libraries: libpcre: 8.31 2012-07-06 (compiled with 8.31) openssl: OpenSSL 1.0.1c 10 May 2012 (compiled with OpenSSL 1.0.1c 10 May 2012) libxml2: 2.9.0 (compiled with 2.9.0)
Here's what my related code looks like:
udf.c ##### str GRPident(int *ret, int *bid, int *eid){ ... }
udf.h ##### udf_export str GRPident(int *ret, int *bid, int *eid);
udf.mal ####### module udf ... command ident(b:bat[:oid,:str], e:bat[:oid,:str]) :bat[:oid,:str] address GRPident comment "identity of groups";
udf.sql ####### create aggregate grpident(src string) returns string external name udf.ident;
I already renamed the MAL command to "subident" and changed the SQL expression accordingly without success. Furthermore I already had a look at "batxml.c" as suggested here: http://old.nabble.com/users-list-Digest,-Vol-3,-Issue-30-td34736774.html but don't see what I'm doing wrong.
What do I have to do to implement a custom aggregate function?
Thanks, Sebastian
users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
Hi Miguel,
thanks for your reply and your hint at the signatures. A took a deeper look at the MAL execution plan and changed all signatures in MAL, .c and .h files accordingly. Now it works like a charm ;)
On 17.06.2013 18:09, Miguel Ping wrote:
Hi Sebastian,
I understand your pain, had to go through the same process. What version of Monet are you using? The aggr definition changed a bit between the two last versions I think. Also, one problem I hit was that the bat signatures have to be precise, otherwise even if the code compiles, it will throw an error.
On 06/17/2013 01:59 PM, Sebastian Dorok wrote:
Hi all,
I am evaluating MonetDB in the context of genomic data. Therefore I'm trying to implement a custom aggregation function.
To get a feeling for the system I set up a UDF that just returns identical values. Now I'm trying to implement this simple function as aggregate function for groupings but always run into an error when executing this:
select grpident(base) from test.bases group by pos;
"TypeException:user.s1_1[18]:'udf.subident' undefined in: _30:bat[:any,:str] := udf.subident(_15:bat[:oid,:str], _27:bat[:oid,:oid], r1_27:bat[:oid,:oid], _1:bit) program contains errors"
Here is the MAL plan: +-----------------------------------------------------------------+ | mal | +=================================================================+ | function user.s1_1{autoCommit=true}():void; | | X_2 := sql.mvc(); | | X_3:bat[:oid,:oid] := sql.tid(X_2,"test","bases"); | | X_6 := sql.bind(X_2,"test","bases","base",0); | | (X_9,r1_9) := sql.bind(X_2,"test","bases","base",2); | | X_12 := sql.bind(X_2,"test","bases","base",1); | | X_14 := sql.delta(X_6,X_9,r1_9,X_12); | | X_15 := algebra.leftfetchjoin(X_3,X_14); | | calc.str("bases"); | | calc.str("base"); | | X_18 := sql.bind(X_2,"test","bases","pos",0); | | (X_20,r1_20) := sql.bind(X_2,"test","bases","pos",2); | | X_22 := sql.bind(X_2,"test","bases","pos",1); | | X_23 := sql.delta(X_18,X_20,r1_20,X_22); | | X_24 := algebra.leftfetchjoin(X_3,X_23); | | calc.str("bases"); | | calc.str("pos"); | | (X_27,r1_27,r2_27) := group.subgroupdone(X_24); | | X_30:bat[:any,:str] := udf.subident(X_15,X_27,r1_27,true); | | calc.str("bases"); | | calc.str("L1"); | | calc.str("bases"); | | calc.str("L1"); | | X_36 := sql.resultSet(1,1,X_30); | | sql.rsColumn(X_36,"test.bases","L1","clob",1,0,X_30); | | X_40 := io.stdout(); | | sql.exportResult(X_40,X_36); | | end s1_1; | +-----------------------------------------------------------------+
I'm using following setup: MonetDB 5 server v11.15.7 "Feb2013-SP2" (64-bit, 64-bit oids) Found 1.9GiB available memory, 2 available cpu cores Libraries: libpcre: 8.31 2012-07-06 (compiled with 8.31) openssl: OpenSSL 1.0.1c 10 May 2012 (compiled with OpenSSL 1.0.1c 10 May 2012) libxml2: 2.9.0 (compiled with 2.9.0)
Here's what my related code looks like:
udf.c ##### str GRPident(int *ret, int *bid, int *eid){ ... }
udf.h ##### udf_export str GRPident(int *ret, int *bid, int *eid);
udf.mal ####### module udf ... command ident(b:bat[:oid,:str], e:bat[:oid,:str]) :bat[:oid,:str] address GRPident comment "identity of groups";
udf.sql ####### create aggregate grpident(src string) returns string external name udf.ident;
I already renamed the MAL command to "subident" and changed the SQL expression accordingly without success. Furthermore I already had a look at "batxml.c" as suggested here: http://old.nabble.com/users-list-Digest,-Vol-3,-Issue-30-td34736774.html but don't see what I'm doing wrong.
What do I have to do to implement a custom aggregate function?
Thanks, Sebastian
users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
Glad you made it work.
I've been asking the Monet team to write a thorough tutorial of a custom aggregate function; I would be happy to help it out. Also, a thorough explanation of the call stack involved in the aggregate would be nice :) there's alot of macros on the custom aggr.
On 06/20/2013 07:38 AM, Sebastian Dorok wrote:
Hi Miguel,
thanks for your reply and your hint at the signatures. A took a deeper look at the MAL execution plan and changed all signatures in MAL, .c and .h files accordingly. Now it works like a charm ;)
On 17.06.2013 18:09, Miguel Ping wrote:
Hi Sebastian,
I understand your pain, had to go through the same process. What version of Monet are you using? The aggr definition changed a bit between the two last versions I think. Also, one problem I hit was that the bat signatures have to be precise, otherwise even if the code compiles, it will throw an error.
On 06/17/2013 01:59 PM, Sebastian Dorok wrote:
Hi all,
I am evaluating MonetDB in the context of genomic data. Therefore I'm trying to implement a custom aggregation function.
To get a feeling for the system I set up a UDF that just returns identical values. Now I'm trying to implement this simple function as aggregate function for groupings but always run into an error when executing this:
select grpident(base) from test.bases group by pos;
"TypeException:user.s1_1[18]:'udf.subident' undefined in: _30:bat[:any,:str] := udf.subident(_15:bat[:oid,:str], _27:bat[:oid,:oid], r1_27:bat[:oid,:oid], _1:bit) program contains errors"
Here is the MAL plan: +-----------------------------------------------------------------+ | mal | +=================================================================+ | function user.s1_1{autoCommit=true}():void; | | X_2 := sql.mvc(); | | X_3:bat[:oid,:oid] := sql.tid(X_2,"test","bases"); | | X_6 := sql.bind(X_2,"test","bases","base",0); | | (X_9,r1_9) := sql.bind(X_2,"test","bases","base",2); | | X_12 := sql.bind(X_2,"test","bases","base",1); | | X_14 := sql.delta(X_6,X_9,r1_9,X_12); | | X_15 := algebra.leftfetchjoin(X_3,X_14); | | calc.str("bases"); | | calc.str("base"); | | X_18 := sql.bind(X_2,"test","bases","pos",0); | | (X_20,r1_20) := sql.bind(X_2,"test","bases","pos",2); | | X_22 := sql.bind(X_2,"test","bases","pos",1); | | X_23 := sql.delta(X_18,X_20,r1_20,X_22); | | X_24 := algebra.leftfetchjoin(X_3,X_23); | | calc.str("bases"); | | calc.str("pos"); | | (X_27,r1_27,r2_27) := group.subgroupdone(X_24); | | X_30:bat[:any,:str] := udf.subident(X_15,X_27,r1_27,true); | | calc.str("bases"); | | calc.str("L1"); | | calc.str("bases"); | | calc.str("L1"); | | X_36 := sql.resultSet(1,1,X_30); | | sql.rsColumn(X_36,"test.bases","L1","clob",1,0,X_30); | | X_40 := io.stdout(); | | sql.exportResult(X_40,X_36); | | end s1_1; | +-----------------------------------------------------------------+
I'm using following setup: MonetDB 5 server v11.15.7 "Feb2013-SP2" (64-bit, 64-bit oids) Found 1.9GiB available memory, 2 available cpu cores Libraries: libpcre: 8.31 2012-07-06 (compiled with 8.31) openssl: OpenSSL 1.0.1c 10 May 2012 (compiled with OpenSSL 1.0.1c 10 May 2012) libxml2: 2.9.0 (compiled with 2.9.0)
Here's what my related code looks like:
udf.c ##### str GRPident(int *ret, int *bid, int *eid){ ... }
udf.h ##### udf_export str GRPident(int *ret, int *bid, int *eid);
udf.mal ####### module udf ... command ident(b:bat[:oid,:str], e:bat[:oid,:str]) :bat[:oid,:str] address GRPident comment "identity of groups";
udf.sql ####### create aggregate grpident(src string) returns string external name udf.ident;
I already renamed the MAL command to "subident" and changed the SQL expression accordingly without success. Furthermore I already had a look at "batxml.c" as suggested here: http://old.nabble.com/users-list-Digest,-Vol-3,-Issue-30-td34736774.html but don't see what I'm doing wrong.
What do I have to do to implement a custom aggregate function?
Thanks, Sebastian
users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
participants (2)
-
Miguel Ping -
Sebastian Dorok