Performing a quantile in R from a MonetDB.R db with 300million rows (UNCLASSIFIED)
Classification: UNCLASSIFIED Caveats: NONE
Hi all,
I'm currently trying to compare the performance of R's quantile function to that of MonetDB's. I have a table loaded with the TPC-H benchmark's data (scale factor of 50). I'm trying to return ~300 million rows of data (select l_extendedprice from lineitem), and although I have 256G of RAM, I'm getting the following:
Error in paste(0)("",resp, collapse="") : Result would exceed 2^31-1 bytes
The R statement is : x = dbGetQuery(conn, "select l_extendedprice from lineitem")
Forgive me if this is a simplistic questions, as I am fairly new to R and don't know which avenue to take next. I took a look at bigmemory, but don't know quite how to return a result from monetdb and have it use a big.matrix. Note: the l_extendedprice is a floating (double) value.
Any suggestions?
Cheers.
Glover George
Classification: UNCLASSIFIED Caveats: NONE
howdy, after you hit the error type
sessionInfo()
into the R console and send the output? thanks
On Fri, Jun 12, 2015 at 4:49 PM, George, Glover E ERD-MS < Glover.E.George@usace.army.mil> wrote:
Classification: UNCLASSIFIED Caveats: NONE
Hi all,
I'm currently trying to compare the performance of R's quantile function to that of MonetDB's. I have a table loaded with the TPC-H benchmark's data (scale factor of 50). I'm trying to return ~300 million rows of data (select l_extendedprice from lineitem), and although I have 256G of RAM, I'm getting the following:
Error in paste(0)("",resp, collapse="") : Result would exceed 2^31-1 bytes
The R statement is : x = dbGetQuery(conn, "select l_extendedprice from lineitem")
Forgive me if this is a simplistic questions, as I am fairly new to R and don't know which avenue to take next. I took a look at bigmemory, but don't know quite how to return a result from monetdb and have it use a big.matrix. Note: the l_extendedprice is a floating (double) value.
Any suggestions?
Cheers.
Glover George
Classification: UNCLASSIFIED Caveats: NONE
users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
Classification: UNCLASSIFIED Caveats: NONE
R version 3.2.0 (2015-04-16) Platform: x86_64-unknown-linux-gnu (64-bit) Running under: Ubuntu 14.04.2 LTS
locale: .... (US.UTF-8 and C ... ellipsis because I'm having to retype the error message without cut-and-paste, if locale data is important, I can re-send)
Attached base packages: [1] stats graphics grDevices utils datasets methods base
Other attached packages: [1] MonetDB.R_0.9.7 digest_0.6.8 DBI_0.3.1
-----Original Message----- From: users-list [mailto:users-list-bounces+glover.e.george=usace.army.mil@monetdb.org] On Behalf Of Anthony Damico Sent: Friday, June 12, 2015 3:57 PM To: Communication channel for MonetDB users Subject: [EXTERNAL] Re: Performing a quantile in R from a MonetDB.R db with 300million rows (UNCLASSIFIED)
howdy, after you hit the error type
sessionInfo()
into the R console and send the output? thanks
On Fri, Jun 12, 2015 at 4:49 PM, George, Glover E ERD-MS Glover.E.George@usace.army.mil wrote:
Classification: UNCLASSIFIED Caveats: NONE Hi all, I'm currently trying to compare the performance of R's quantile function to that of MonetDB's. I have a table loaded with the TPC-H benchmark's data (scale factor of 50). I'm trying to return ~300 million rows of data (select l_extendedprice from lineitem), and although I have 256G of RAM, I'm getting the following: Error in paste(0)("",resp, collapse="") : Result would exceed 2^31-1 bytes The R statement is : x = dbGetQuery(conn, "select l_extendedprice from lineitem") Forgive me if this is a simplistic questions, as I am fairly new to R and don't know which avenue to take next. I took a look at bigmemory, but don't know quite how to return a result from monetdb and have it use a big.matrix. Note: the l_extendedprice is a floating (double) value. Any suggestions? Cheers. Glover George Classification: UNCLASSIFIED Caveats: NONE _______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
Classification: UNCLASSIFIED Caveats: NONE
participants (2)
-
Anthony Damico -
George, Glover E ERD-MS