Re: [MonetDB-users] I'm looking for a fast way to write data to the DB from a scripting language.
Good morning Jay,
you can use the Ruby MAPI Wrapper => MAPI calls through Ruby. It's elegant and simple and was used to implement the Ruby on Rails adapter for MonetDB. Plus, from what I see in the SWIG input file, the insert and copy_from functions should be implemented in the Ruby MAPI Wrapper. If you want to go the Ruby way, you may find the files in the MonetDB clients package or at http://rubyforge.org/projects/monetdb-ror/ as part of the Ruby on Rails MonetDB adapter.
Regards, Michalis
----------------------------------------------------------------------------------------------- Hi,
I have 50 million rows and 15 columns that I'd like to put into one table. My first attempt to transfer them was through PHP using the monetdb_query method. This is causing a lot of disk thrashing and my throughput is about 100 rows/sec.
I get an "undefined function" error for both monetdb_insert and monetdb_copy_from.
I would like to find a fast method, but from inside a common scripting language.
Thanks, Jay
Thank you for replying quickly. Ruby would be a good choice.
I'm not clear about MAPI. I only know of Mail API. Does this mean Monet API?
I see a reference to the MAPI include files. Can you please outline the first steps I can take to get this going in Ruby? Can I compile in Cygwin or do I need to do it on Linux?
Thanks, Jay
On Sat, Sep 6, 2008 at 11:20 PM, Michalis Polakis mpolakis@gmail.comwrote:
Good morning Jay,
you can use the Ruby MAPI Wrapper => MAPI calls through Ruby. It's elegant and simple and was used to implement the Ruby on Rails adapter for MonetDB. Plus, from what I see in the SWIG input file, the insert and copy_from functions should be implemented in the Ruby MAPI Wrapper. If you want to go the Ruby way, you may find the files in the MonetDB clients package or at http://rubyforge.org/projects/monetdb-ror/ as part of the Ruby on Rails MonetDB adapter.
Regards, Michalis
Hi,
I have 50 million rows and 15 columns that I'd like to put into one table. My first attempt to transfer them was through PHP using the monetdb_query method. This is causing a lot of disk thrashing and my throughput is about 100 rows/sec.
I get an "undefined function" error for both monetdb_insert and monetdb_copy_from.
I would like to find a fast method, but from inside a common scripting language.
Thanks, Jay
Dear JJ
Thanks for using MonetDB. Please could you identify the platform you use MonetDB on and its version? Moreover, the message below indicates a possible problem we are not aware of. Can you provide a micro example that illustrates the failed calls, e.g. a 5 line PHP program to localize the bug.
thanks in advance in improving the code base.
regards, Martin
JJ Jakosky wrote:
Thank you for replying quickly. Ruby would be a good choice.
I'm not clear about MAPI. I only know of Mail API. Does this mean Monet API?
I see a reference to the MAPI include files. Can you please outline the first steps I can take to get this going in Ruby? Can I compile in Cygwin or do I need to do it on Linux?
Thanks, Jay
On Sat, Sep 6, 2008 at 11:20 PM, Michalis Polakis <mpolakis@gmail.com mailto:mpolakis@gmail.com> wrote:
Good morning Jay, you can use the Ruby MAPI Wrapper => MAPI calls through Ruby. It's elegant and simple and was used to implement the Ruby on Rails adapter for MonetDB. Plus, from what I see in the SWIG input file, the insert and copy_from functions should be implemented in the Ruby MAPI Wrapper. If you want to go the Ruby way, you may find the files in the MonetDB clients package or at http://rubyforge.org/projects/monetdb-ror/ as part of the Ruby on Rails MonetDB adapter. Regards, Michalis ----------------------------------------------------------------------------------------------- Hi, I have 50 million rows and 15 columns that I'd like to put into one table. My first attempt to transfer them was through PHP using the monetdb_query method. This is causing a lot of disk thrashing and my throughput is about 100 rows/sec. I get an "undefined function" error for both monetdb_insert and monetdb_copy_from. I would like to find a fast method, but from inside a common scripting language. Thanks, Jay-- JJ Jakosky 720.244.0137
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
On Sun, Sep 7, 2008 at 9:02 AM, JJ Jakosky jay.jakosky@gmail.com wrote:
Thank you for replying quickly. Ruby would be a good choice.
I'm not clear about MAPI. I only know of Mail API. Does this mean Monet API?
Yes. For details on MAPI please see: http://monetdb.cwi.nl/projects/monetdb/SQL/Documentation/The-Mapi-Library.ht...
I see a reference to the MAPI include files. Can you please outline the first steps I can take to get this going in Ruby? Can I compile in Cygwin or do I need to do it on Linux?
Copying from MonetDB's adapter for Rails README :
1. Wrapping the Monet API to Ruby --------------------------------- MonetDB's API, called MAPI provides the functions an application needs to communicate with a MonetDB server. We need to wrap this API in Ruby, so that the MonetDB adapter can utilize it to communicate with the database and we are going to use the Simplified Wrapper and Interface Generator (SWIG) to do it.
The necessary files for this task reside in the mapi_wrapper directory. MapiLib.ruby.i is the input file for SWIG extconf.rb is a ruby makefile generator script.
By now, you should have already installed MonetDB and its client package that includes the MAPI library.
NOTES for MapiLib.ruby.i : %module mapi denotes the name our ruby module will have. There are two include statements. One is in %{ .. %} and the other as %include .. that both include the Mapi header file. MAKE SURE IT'S THE CORRECT PATH. The default path is the default MonetDB installation path, though. Some functions cannot be wrapped in Ruby by SWIG and are therefore ignored. It's ok though.
After you've set the parameters right, you execute :
swig MapiLib.ruby.i ruby extconf.rb make make install (you should typically have superuser rights for this one)
You have now installed the wrapped in Ruby MonetDB API. You can use it in your Ruby applications to connect and interact with MonetDB databases.
Short example : require 'mapi' include Mapi mid = Mapi::mapi_connect("localhost", 50000, "monetdb", "monetdb", "sql", "db_name_here") ----------- The above procedure has been tried and tested on Ubuntu Linux. It should be possible to do it with cygwin, but I'm not aware of any (dirty) details. For an alternative approach in installing MAPI for Ruby check the clients package makefile etc.
Regards, Michalis
Thanks, Jay
On Sat, Sep 6, 2008 at 11:20 PM, Michalis Polakis mpolakis@gmail.com wrote:
Good morning Jay,
you can use the Ruby MAPI Wrapper => MAPI calls through Ruby. It's elegant and simple and was used to implement the Ruby on Rails adapter for MonetDB. Plus, from what I see in the SWIG input file, the insert and copy_from functions should be implemented in the Ruby MAPI Wrapper. If you want to go the Ruby way, you may find the files in the MonetDB clients package or at http://rubyforge.org/projects/monetdb-ror/ as part of the Ruby on Rails MonetDB adapter.
Regards, Michalis
Hi,
I have 50 million rows and 15 columns that I'd like to put into one table. My first attempt to transfer them was through PHP using the monetdb_query method. This is causing a lot of disk thrashing and my throughput is about 100 rows/sec.
I get an "undefined function" error for both monetdb_insert and monetdb_copy_from.
I would like to find a fast method, but from inside a common scripting language.
Thanks, Jay
-- JJ Jakosky 720.244.0137
participants (3)
-
JJ Jakosky -
Martin Kersten -
Michalis Polakis