Create R function that can be applied to an entire table

Willem Ligtenberg willem.ligtenberg at openanalytics.eu
Thu Apr 7 09:52:22 CEST 2016


Just to confirm on the list.
This indeed solves the issue.
Right now, I just went for the same lazy fix, just increase the size.
But I agree a proper fix would be nice. :)

Willem

----- Original Message -----
> From: "Willem Ligtenberg" <willem.ligtenberg at openanalytics.eu>
> To: "users-list" <users-list at monetdb.org>
> Sent: Thursday, April 7, 2016 8:18:28 AM
> Subject: Re: Create R function that can be applied to an entire table

> Hi Mark,
> 
> I will fix that, I had to compile from source anyway to get R going.
> I have ~ 1000 columns and the names are long...
> 
> I will apply the fix and see if we can get that small miracle. :)
> 
> Although for some of my use cases, extending the argument types to also allow to
> pass a table would be really nice. That table could then be exposed as either a
> data.frame or data.table. (I prefer data.table)
> 
> Kind regards,
> 
> Willem
> 
> ----- Original Message -----
>> From: "Mark Raasveldt" <m.raasveldt at cwi.nl>
>> To: "users-list" <users-list at monetdb.org>
>> Sent: Wednesday, April 6, 2016 10:13:55 PM
>> Subject: Re: Create R function that can be applied to an entire table
> 
>> Hey Willem,
>> 
>> This is a bug, or more accurately, lazy programming ;) the function just assume
>> nobody will need more than 10,000 characters for the argument names. If you
>> want to do a lazy fix you can just increase the size of the argnames array. If
>> you want to do a not lazy fix you can check the size of each of the argument
>> names and then allocate enough space for them.
>> 
>> I have to ask, though, how many columns are you using at once to trigger this
>> error? If this is the only issue you run into when operating on that many
>> columns in the same query I think that would be a small miracle ;)
>> 
>> Regards,
>> 
>> Mark
>> 
>> 
>> 
>> ----- Original Message -----
>> From: "Willem Ligtenberg" <willem.ligtenberg at openanalytics.eu>
>> To: "users-list" <users-list at monetdb.org>
>> Sent: Wednesday, April 6, 2016 9:45:26 PM
>> Subject: Create R function that can be applied to an entire table
>> 
>> Hi,
>> 
>> I guess I am pushing the user defined R functions to their limits.
>> I know want to implement some batch correcting in R and then apply that to an
>> entire table.
>> I have written R code, that will generate a function definition that has
>> matching parameters.
>> And then I want to call the function as follows:
>> 
>> CREATE TABLE data_bc AS SELECT * FROM batch_correction( (SELECT * FROM data) )
>> WITH DATA;
>> 
>> I then get the following error:
>> Error in .local(conn, statement, ...) :
>>  Unable to execute statement 'CREATE TABLE hcs_batch AS SELECT * FROM
>>  batch_correction( (SELECT * FROM hcs_norm) ) WITH DATA;'.
>> Server says '!Command too large'
>> 
>> Looking at the source, this error is raised in rapi.c
>> on either line 467 or 473.
>> 
>> After editing the source to figure out which of the lines was triggered, I
>> figured out is was on line:
>> 467 it is triggered, so the following code raises the issue that the command is
>> too large:
>> 
>>        pos = 0;
>>        for (i = pci->retc + 2; i < pci->argc && pos < sizeof(argnames); i++) {
>>                pos += snprintf(argnames + pos, sizeof(argnames) - pos, "%s%s",
>>                                                args[i], i < pci->argc - 1 ? ", " : "");
>>        }
>>        if (pos >= sizeof(argnames)) {
>>                msg = createException(MAL, "rapi.eval", "Command too large1");
>>                goto wrapup;
>>        }
>> 
>> (yes, I added the 1 to large, to make it unique)
>> 
>> If this is not a bug, any help in getting my code to work is greatly
>> appreciated.
>> 
>> Kind regards,
>> 
>> Willem
>> _______________________________________________
>> users-list mailing list
>> users-list at monetdb.org
>> https://www.monetdb.org/mailman/listinfo/users-list
>> _______________________________________________
>> users-list mailing list
>> users-list at monetdb.org
>> https://www.monetdb.org/mailman/listinfo/users-list
> _______________________________________________
> users-list mailing list
> users-list at monetdb.org
> https://www.monetdb.org/mailman/listinfo/users-list


More information about the users-list mailing list