Big Slowdown (15-20X ) on import functions in Monetdb October 2014 Release

Ashish Kumar Singh ashishkumar.singh at altair.com
Mon Nov 17 12:54:25 CET 2014


Hi,


Below are the steps.


Unzip archive in to /opt/monetdb_test/ directroy to minimise changes in the script datbase can live anywhere in the file system.

Single Big File Import:

1. First create a database using Monetdb October2014 build called pbsworks
2. Create data base schema using schema provided in file schema.sql.

Cat all files present inside testdata directory in to on single file by using following command:

cat * > ../merged_data

Load this merged file in to database using script single_import.sql from mclient

Note down the timing, usually 3-5 minutes on 4 core 17 GB machine on amazon.

Multiple small File Import:
1. First create a database using Monetdb October2014 build called pbsworks
2. Create data base schema using schema provided in file schema.sql.
3. Load files in to database using multiple_import.sql using mclient



Regards,

Ashish

From: Anthony Damico <ajdamico at gmail.com<mailto:ajdamico at gmail.com>>
Reply-To: "users-list at monetdb.org<mailto:users-list at monetdb.org>" <users-list at monetdb.org<mailto:users-list at monetdb.org>>
Date: Monday, 17 November 2014 3:51 pm
To: "users-list at monetdb.org<mailto:users-list at monetdb.org>" <users-list at monetdb.org<mailto:users-list at monetdb.org>>
Cc: "developers-list at monetdb.org<mailto:developers-list at monetdb.org>" <developers-list at monetdb.org<mailto:developers-list at monetdb.org>>
Subject: Re: Big Slowdown (15-20X ) on import functions in Monetdb October 2014 Release

please provide a list of every command that you executed to reproduce the problem.  you need to make the list of commands as short as possible, so people trying to help you are not wasting time on irrelevant code.. thanks!!

On Mon, Nov 17, 2014 at 4:22 AM, Ashish Kumar Singh <ashishkumar.singh at altair.com<mailto:ashishkumar.singh at altair.com>> wrote:
Thanks for looking in to it,

I have uploaded required test files to you at below link:

 https://ftin.india.altair.com:8443/message/eJCZlisrascBO00ffEVsQu<https://ftin.india.altair.com:8443/message/eJCZlisrascBO00ffEVsQu>


Regards,
Ashish

From: Anthony Damico <ajdamico at gmail.com<mailto:ajdamico at gmail.com>>
Reply-To: "users-list at monetdb.org<mailto:users-list at monetdb.org>" <users-list at monetdb.org<mailto:users-list at monetdb.org>>
Date: Saturday, 15 November 2014 2:40 pm
To: "users-list at monetdb.org<mailto:users-list at monetdb.org>" <users-list at monetdb.org<mailto:users-list at monetdb.org>>
Cc: "developers-list at monetdb.org<mailto:developers-list at monetdb.org>" <developers-list at monetdb.org<mailto:developers-list at monetdb.org>>

Subject: Re: Big Slowdown (15-20X ) on import functions in Monetdb October 2014 Release

could you provide a minimal reproducible example?  the less code that re-creates the problem, the better.  thanks

http://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducible-example

On Sat, Nov 15, 2014 at 1:45 AM, Ashish Kumar Singh <ashishkumar.singh at altair.com<mailto:ashishkumar.singh at altair.com>> wrote:
Additional information here,


If I merge all these files in to one file and then import to monetdb then time taken is 3-5 minutes as compared to 10-15 hours with October 2014 branch.

Hopefully some one from moentdb team will look in to this. I am glad to see that inspite of this slowness ( due to which this build is unusable) the segfaults are gone!


Regards,
Ashish


From: Ashish Singh <ashishkumar.singh at altair.com<mailto:ashishkumar.singh at altair.com>>
Date: Friday, 14 November 2014 7:41 pm
To: "developers-list at monetdb.org<mailto:developers-list at monetdb.org>" <developers-list at monetdb.org<mailto:developers-list at monetdb.org>>
Cc: "users-list at monetdb.org<mailto:users-list at monetdb.org>" <users-list at monetdb.org<mailto:users-list at monetdb.org>>
Subject: Re: Big Slowdown (15-20X ) on import functions in Monetdb October 2014 Release

Is any body else seeing this problem with October,2014  release?

I wish developer team can provide some insights in this area.

Regards,
Ashish


From: Ashish Singh <ashishkumar.singh at altair.com<mailto:ashishkumar.singh at altair.com>>
Date: Thursday, 13 November 2014 11:22 pm
To: "developers-list at monetdb.org<mailto:developers-list at monetdb.org>" <developers-list at monetdb.org<mailto:developers-list at monetdb.org>>
Cc: "users-list at monetdb.org<mailto:users-list at monetdb.org>" <users-list at monetdb.org<mailto:users-list at monetdb.org>>
Subject: Big Slowdown (15-20X ) on import functions in Monetdb October 2014 Release

All,

With October,2014  release we are stucked in to very strange performance issue while importing data on virtualised environments.

On Virtual machines in our local VMware servers and amazon machine we are seeing import performance has downgraded to close to 15 to 20 while importing  ~ 2000 files  as compared to 1 hour time being taken with old release. This we are seeing consistently with CentOS, SLES vm and configuration is 4 Core 33 GB machine. We tried on quite a few amazon vm  just to make sure its not an issue with a particular instance setup.

On a physical machine of 4 core 24 GB we are able to match speed of import with old monetdb. We are issuing  copy into command through JDBC connection.

Can you guys help me with pointers on this issue, I will be happy to share any debug info.

Regards,
Ashish Kumar Singh
Mobile: +91-9972045095<tel:%2B91-9972045095>
Skype: toashishk at gmail.com<mailto:toashishk at gmail.com>



_______________________________________________
users-list mailing list
users-list at monetdb.org<mailto:users-list at monetdb.org>
https://www.monetdb.org/mailman/listinfo/users-list



_______________________________________________
users-list mailing list
users-list at monetdb.org<mailto:users-list at monetdb.org>
https://www.monetdb.org/mailman/listinfo/users-list


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.monetdb.org/pipermail/developers-list/attachments/20141117/9c73ed01/attachment.html>


More information about the developers-list mailing list