Hi,
I am wondering whether you are able to solve the issue of slow insert sppeds once it reaches more than 5 million records..
Do you have still this problem..
We will be inserting close to that amount every day (may be) and curious how MonetDB is going to handle the huge no.of records..
Sateesh
chinarinde wrote:
Hi all,
I am new to monetDB and ran into the following problem (I'll try to be as verbose as I can, perhaps it helps helping me):
I've installed MonetDB server 5.0.0, clients and commons 1.18.2 and sql 2.18.2 on Ubuntu Feisty. Then I've taken the JDBC example which is available online and modified it to be able to insert data into a test database. More specifically I'm using prepared statements and executeBatch() together with the monetdb-1.6 JDBC driver.
Which is quite successful until about 5 million records are inserted - afterwards the speed of the inserts drops dramatically. I've experimented with different batch sizes (10000 records seems to work ok), and I've tried to understand the chapter about memory tuning for MonetDB (not sure yet about how to use mem_cursize() though). In any case at first inserting 2 million records only takes about 5 minutes (measured with Javas System.timemillis), later on this time increases and the server gets rather unresponsive.
I found only two posts concerning the speed of JDBC inserts (from January 2007 I think) and I would be interested if a solution or explanation has been found off-list.
Thank you