Create R function that can be applied to an entire table

Willem Ligtenberg willem.ligtenberg at openanalytics.eu
Thu Apr 7 08:18:28 CEST 2016


Hi Mark,

I will fix that, I had to compile from source anyway to get R going.
I have ~ 1000 columns and the names are long...

I will apply the fix and see if we can get that small miracle. :)

Although for some of my use cases, extending the argument types to also allow to pass a table would be really nice. That table could then be exposed as either a data.frame or data.table. (I prefer data.table)

Kind regards,

Willem

----- Original Message -----
> From: "Mark Raasveldt" <m.raasveldt at cwi.nl>
> To: "users-list" <users-list at monetdb.org>
> Sent: Wednesday, April 6, 2016 10:13:55 PM
> Subject: Re: Create R function that can be applied to an entire table

> Hey Willem,
> 
> This is a bug, or more accurately, lazy programming ;) the function just assume
> nobody will need more than 10,000 characters for the argument names. If you
> want to do a lazy fix you can just increase the size of the argnames array. If
> you want to do a not lazy fix you can check the size of each of the argument
> names and then allocate enough space for them.
> 
> I have to ask, though, how many columns are you using at once to trigger this
> error? If this is the only issue you run into when operating on that many
> columns in the same query I think that would be a small miracle ;)
> 
> Regards,
> 
> Mark
> 
> 
> 
> ----- Original Message -----
> From: "Willem Ligtenberg" <willem.ligtenberg at openanalytics.eu>
> To: "users-list" <users-list at monetdb.org>
> Sent: Wednesday, April 6, 2016 9:45:26 PM
> Subject: Create R function that can be applied to an entire table
> 
> Hi,
> 
> I guess I am pushing the user defined R functions to their limits.
> I know want to implement some batch correcting in R and then apply that to an
> entire table.
> I have written R code, that will generate a function definition that has
> matching parameters.
> And then I want to call the function as follows:
> 
> CREATE TABLE data_bc AS SELECT * FROM batch_correction( (SELECT * FROM data) )
> WITH DATA;
> 
> I then get the following error:
> Error in .local(conn, statement, ...) :
>  Unable to execute statement 'CREATE TABLE hcs_batch AS SELECT * FROM
>  batch_correction( (SELECT * FROM hcs_norm) ) WITH DATA;'.
> Server says '!Command too large'
> 
> Looking at the source, this error is raised in rapi.c
> on either line 467 or 473.
> 
> After editing the source to figure out which of the lines was triggered, I
> figured out is was on line:
> 467 it is triggered, so the following code raises the issue that the command is
> too large:
> 
>        pos = 0;
>        for (i = pci->retc + 2; i < pci->argc && pos < sizeof(argnames); i++) {
>                pos += snprintf(argnames + pos, sizeof(argnames) - pos, "%s%s",
>                                                args[i], i < pci->argc - 1 ? ", " : "");
>        }
>        if (pos >= sizeof(argnames)) {
>                msg = createException(MAL, "rapi.eval", "Command too large1");
>                goto wrapup;
>        }
> 
> (yes, I added the 1 to large, to make it unique)
> 
> If this is not a bug, any help in getting my code to work is greatly
> appreciated.
> 
> Kind regards,
> 
> Willem
> _______________________________________________
> users-list mailing list
> users-list at monetdb.org
> https://www.monetdb.org/mailman/listinfo/users-list
> _______________________________________________
> users-list mailing list
> users-list at monetdb.org
> https://www.monetdb.org/mailman/listinfo/users-list


More information about the users-list mailing list