Problems executing custom aggregation function on large datasets

Lefteris lsidir at gmail.com
Wed Jul 3 15:11:55 CEST 2013


Hi, the error that you are getting has nothing to do in my opinion
with your function. There are other reasons. Try to run your queries
through SQL mclient, passing one MAL instruction by hand in general
will not work.

I see in your MAL explain plan

     X_16:bat[:any,:str]  := udf.subgenotype(X_10,X_14,r1_27,true);

which is your function. X_10 here is the "base" column you want to
aggregate and X_14 is the result of subgroupby of the "pos" column.
Try to run monetdb with gdk_nr_threads=1 so you only have 1 thread and
avoid fragmentation of the base tables, so is simpler to see what is
going on.



On Wed, Jul 3, 2013 at 2:48 PM, Sebastian Dorok
<sebastian.dorok at st.ovgu.de> wrote:
> Hi all,
>
> I'm developing a custom aggregation function. Everything works fine until I
> use it on large datasets (ca. 270 Mio. rows).
>
> Here is what my function signature looks like:
>
> str AGGSimpleGenotype(int *ret, int *bid, int *gid, int *eid, bit
> ignore_nils);
>
> I assume that the BATs identified by bid and gid have the same length.
> That's why my function checks this circumstance. On my test dataset (ca. 200
> rows) everything works fine. When I use my production dataset my aggregation
> function returns an error complaining that the BATs have not the same length
> and stops aggregating (by design)! How can this happen?
>
> The SQL query is:
> select grpgenotype(base) from test.bases group by pos;
> I attached the MAL plan generated for the large dataset. When I execute it
> step by step I get an error in line 39 saying:
>
> mal>X_106 := sql.delta(X_56,X_70,X_71,X_8);
> MAPI  = (monetdb) /tmp/.s.monetdb.50000
> QUERY = X_106 := sql.delta(X_56,X_70,X_71,X_8);
> ERROR = !TypeException:user.main[1]:'sql.delta' undefined in: X_106:any :=
> sql.delta(X_56:bat[:oid,:str], X_70:bat[:oid,:oid], X_71:bat[:oid,:str],
> X_8:bat[:oid,:any])
>
> This error occurs before my function is entered but I think it must be
> related with my problem. Is it an internal MonetDB problem?
>
> I'm using the Feb2013-SP2 release.
>
> Any ideas? Thx.
>
> _______________________________________________
> users-list mailing list
> users-list at monetdb.org
> http://mail.monetdb.org/mailman/listinfo/users-list
>



More information about the users-list mailing list