[MonetDB-users] Slow Data Load

Paulo Jorge Dias paulojorgedias at gmail.com
Mon Dec 3 00:18:44 CET 2007


Hi Jos,

It was not fast ... it was light speed ;-) It works very well. Thanks for
you help.

I understand your concern regarding 64 bits, but it's not the case.

I have loaded 7million rows. After the load the DB has 2GB in total but the
major file has 62MB, which is not a big deal if we compare to the text file
with the data, which has 500MB.

Saying that, you have any explanation why using jdbc results in slower
performance after about 4million of loaded rows? Till there it was like 200
rows/sec and after that it was like 1 row/sec. Note: I have used batch
update.

A final comment. I have to confess that I am very impressed with the
performance of MonetDB. My first contact with “Column-Oriented DB” was with
Sybase IQ (in a presentation), but I don’t believe at the first place and
now I have the opportunity to “touch the thing” ;-) and it was amazing.
Congratulations.

Best regards,
Paulo Jorge Dias


Jos van Dongen wrote:
> 
> Dear Paulo,
> 
> The fastest way to get large amounts of data into MonetDB is through a
> flat
> file and the use of the 'copy' command. This is even much faster than e.g.
> Oracle Bulk Loader. Note however that your 32bit environment will cause
> problems with large files (that's what I experienced). It's better to
> switch
> to 64 bit. 
> 
> Met vriendelijke groet/kind regards,
> 
> Jos van Dongen
> Tholis Consulting
> T: 06-51169606
> F: 0842-231136
> M: jvdongen at tholis.com
> www.tholis.com 
> 
> -----Original Message-----
> From: monetdb-users-bounces at lists.sourceforge.net
> [mailto:monetdb-users-bounces at lists.sourceforge.net] On Behalf Of Paulo
> Jorge Dias
> Sent: zaterdag 1 december 2007 20:26
> To: monetdb-users at lists.sourceforge.net
> Subject: [MonetDB-users] Slow Data Load
> 
> 
> Hi,
> 
> I am working on a Windows XP SP2 32bit with MonetDB v5.2.2 kernel v1.20.0
> 
> I have built a java program to upload data from an Oracle Database Table
> with 6milion rows.
> 
> After about 4milion of rows loaded the performance decreases dramatically.
> 
> I have tried to use other way, which is generating all insert statements
> to
> a file and then using mclient to execute all the insert statements.
> 
> Slower that the java option because the autocommit=on and parsing of each
> statement, but the result is the same.
> 
> After about 4milion of rows loaded the performance decreases dramatically.
> 
> Can someone give an explanation for this behavior?
> 
> Thanks,
> 
> Paulo Jorge Dias
> 
> -- 
> View this message in context:
> http://www.nabble.com/Slow-Data-Load-tf4929292.html#a14108678
> Sent from the monetdb-users mailing list archive at Nabble.com.
> 
> 
> -------------------------------------------------------------------------
> SF.Net email is sponsored by: The Future of Linux Business White Paper
> from Novell.  From the desktop to the data center, Linux is going
> mainstream.  Let it simplify your IT future.
> http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4
> _______________________________________________
> MonetDB-users mailing list
> MonetDB-users at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/monetdb-users
> 
> No virus found in this incoming message.
> Checked by AVG Free Edition. 
> Version: 7.5.503 / Virus Database: 269.16.12/1162 - Release Date:
> 30-11-2007
> 21:26
>  
> 
> No virus found in this outgoing message.
> Checked by AVG Free Edition. 
> Version: 7.5.503 / Virus Database: 269.16.12/1163 - Release Date:
> 1-12-2007
> 12:05
>  
> 
> 
> -------------------------------------------------------------------------
> SF.Net email is sponsored by: The Future of Linux Business White Paper
> from Novell.  From the desktop to the data center, Linux is going
> mainstream.  Let it simplify your IT future.
> http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4
> _______________________________________________
> MonetDB-users mailing list
> MonetDB-users at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/monetdb-users
> 
> 

-- 
View this message in context: http://www.nabble.com/Slow-Data-Load-tf4929292.html#a14121441
Sent from the monetdb-users mailing list archive at Nabble.com.





More information about the users-list mailing list