MX file expansion

Tapomay Dey tapomay at yahoo.com
Wed Dec 12 17:04:54 CET 2012


Hi,
Thanks for the guidance.
Justification: I am trying to add group_concat support. (http://www.mysqlperformanceblog.com/2006/09/04/group_concat-useful-group-by-extension/)
Tried to follow the string reverse example using UDF. But got stuck while trying to figure out the semantics of accessing the group-related BATs returned by "group.done" MAL call(ref: explain <group by query>).
This took me to the aggr module implementation in /monetdb5/modules/kernel.
Did some reverse engineering to figure out that the 15K lines of code in the C sources were not hand-written but generated from the mx :).

Also the aggregation algorithms handle a lot of bat access cases (like head or tail bats being dense etc.) and some lookup cases like void/scan/merge/hash lookup etc. I have no knowledge about their internals, but I didn't want to end up with an incomplete implementation.

As I am completely new to the semantics and policies of BAT access I decided to refer aggr_bge_avg and came up with the attached mx for reference.
Also I am still planning to define it as a UDF and internally make a call to the generated function in kernel module. Not sure if that's possible just by including the generated header in the UDF source.


If you still discourage this approach, 
Is it fine if I ignore the above discussed cases and go for a simpler implementation(what should I refer for the same?).
One solution (simpler) could be to extract the most common case from the generated C sources of average function.


Attached the mx for reference. Its a two-pass approach where in first pass we calculate the string lengths required per group and in second pass we allocate memory for the destination and strcat the strings into destination.
Please note that the attached file may still be logically incomplete but the basic changes are in place. Need to test before finalizing.
Thanks again and Regards,
Tapomay.


________________________________
 From: Fabian Groffen <fabian at monetdb.org>
To: Tapomay Dey <tapomay at yahoo.com> 
Cc: "developers-list at monetdb.org" <developers-list at monetdb.org> 
Sent: Wednesday, December 12, 2012 8:54 PM
Subject: Re: MX file expansion
 
On 12-12-2012 07:18:10 -0800, Tapomay Dey wrote:
> How do I generate the sources from a new mx file.

We strongly discourage people adding new .mx files.  Just use normal .c,
.h and .mal files.


-- 
Fabian Groffen                              fabian at monetdb.org
column-store developer             http://www.monetdb.org/Home
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.monetdb.org/pipermail/developers-list/attachments/20121212/8f91c762/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: aggr_bge_concat.mx
Type: application/octet-stream
Size: 9329 bytes
Desc: not available
URL: <http://www.monetdb.org/pipermail/developers-list/attachments/20121212/8f91c762/attachment.obj>
-------------- next part --------------
_______________________________________________
developers-list mailing list
developers-list at monetdb.org
http://mail.monetdb.org/mailman/listinfo/developers-list


More information about the developers-list mailing list