Big Slowdown (15-20X ) on import functions in Monetdb October 2014 Release

Stefan Manegold Stefan.Manegold at cwi.nl
Tue Nov 18 17:56:36 CET 2014


Hi Ashish,

please bear with us!

We highly appreciate your feedback and will have a closer look at the reported problem as soon as our time and other obligations allow so.

Please be aware that the members of this mailing-list provide free-of-charge voluntary spare-time best-effort services.

For commercial support, please also consider
https://www.monetdbsolutions.com/

Thanks!
Stefan

----- Original Message -----
> Hi all,
> 
> Are you guys able to reproduce this issue? Any help required from my side do
> let me know.
> 
> 
> Regards,
> Ashish
> 
> Sent from my iPad
> 
> On 17-Nov-2014, at 5:24 pm, Ashish Kumar Singh < ashishkumar.singh at altair.com
> > wrote:
> 
> 
> 
> 
> 
> 
> Hi,
> 
> 
> 
> 
> Below are the steps.
> 
> 
> 
> 
> Unzip archive in to /opt/monetdb_test/ directroy to minimise changes in the
> script datbase can live anywhere in the file system.
> 
> Single Big File Import:
> 
> 1. First create a database using Monetdb October2014 build called pbsworks
> 2. Create data base schema using schema provided in file schema.sql.
> 
> Cat all files present inside testdata directory in to on single file by using
> following command:
> 
> cat * > ../merged_data
> 
> Load this merged file in to database using script single_import.sql from
> mclient
> 
> Note down the timing, usually 3-5 minutes on 4 core 17 GB machine on amazon.
> 
> Multiple small File Import:
> 1. First create a database using Monetdb October2014 build called pbsworks
> 2. Create data base schema using schema provided in file schema.sql.
> 3. Load files in to database using multiple_import.sql using mclient
> 
> 
> 
> 
> 
> 
> 
> Regards,
> 
> Ashish
> 
> From: Anthony Damico < ajdamico at gmail.com >
> Reply-To: " users-list at monetdb.org " < users-list at monetdb.org >
> Date: Monday, 17 November 2014 3:51 pm
> To: " users-list at monetdb.org " < users-list at monetdb.org >
> Cc: " developers-list at monetdb.org " < developers-list at monetdb.org >
> Subject: Re: Big Slowdown (15-20X ) on import functions in Monetdb October
> 2014 Release
> 
> 
> 
> 
> please provide a list of every command that you executed to reproduce the
> problem. you need to make the list of commands as short as possible, so
> people trying to help you are not wasting time on irrelevant code.. thanks!!
> 
> On Mon, Nov 17, 2014 at 4:22 AM, Ashish Kumar Singh <
> ashishkumar.singh at altair.com > wrote:
> 
> 
> 
> Thanks for looking in to it,
> 
> I have uploaded required test files to you at below link:
> 
> https://ftin.india.altair.com:8443/message/eJCZlisrascBO00ffEVsQu
> 
> 
> Regards,
> Ashish
> 
> From: Anthony Damico < ajdamico at gmail.com >
> Reply-To: " users-list at monetdb.org " < users-list at monetdb.org >
> Date: Saturday, 15 November 2014 2:40 pm
> To: " users-list at monetdb.org " < users-list at monetdb.org >
> Cc: " developers-list at monetdb.org " < developers-list at monetdb.org >
> 
> Subject: Re: Big Slowdown (15-20X ) on import functions in Monetdb October
> 2014 Release
> 
> 
> 
> 
> could you provide a minimal reproducible example? the less code that
> re-creates the problem, the better. thanks
> 
> http://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducible-example
> 
> On Sat, Nov 15, 2014 at 1:45 AM, Ashish Kumar Singh <
> ashishkumar.singh at altair.com > wrote:
> 
> 
> 
> Additional information here,
> 
> 
> If I merge all these files in to one file and then import to monetdb then
> time taken is 3-5 minutes as compared to 10-15 hours with October 2014
> branch.
> 
> Hopefully some one from moentdb team will look in to this. I am glad to see
> that inspite of this slowness ( due to which this build is unusable) the
> segfaults are gone!
> 
> 
> Regards,
> Ashish
> 
> 
> From: Ashish Singh < ashishkumar.singh at altair.com >
> Date: Friday, 14 November 2014 7:41 pm
> To: " developers-list at monetdb.org " < developers-list at monetdb.org >
> Cc: " users-list at monetdb.org " < users-list at monetdb.org >
> Subject: Re: Big Slowdown (15-20X ) on import functions in Monetdb October
> 2014 Release
> 
> 
> 
> 
> Is any body else seeing this problem with October,2014 release?
> 
> I wish developer team can provide some insights in this area.
> 
> Regards,
> Ashish
> 
> 
> From: Ashish Singh < ashishkumar.singh at altair.com >
> Date: Thursday, 13 November 2014 11:22 pm
> To: " developers-list at monetdb.org " < developers-list at monetdb.org >
> Cc: " users-list at monetdb.org " < users-list at monetdb.org >
> Subject: Big Slowdown (15-20X ) on import functions in Monetdb October 2014
> Release
> 
> 
> 
> 
> All,
> 
> With October,2014 release we are stucked in to very strange performance issue
> while importing data on virtualised environments.
> 
> On Virtual machines in our local VMware servers and amazon machine we are
> seeing import performance has downgraded to close to 15 to 20 while
> importing ~ 2000 files as compared to 1 hour time being taken with old
> release. This we are seeing consistently with CentOS, SLES vm and
> configuration is 4 Core 33 GB machine. We tried on quite a few amazon vm
> just to make sure its not an issue with a particular instance setup.
> 
> On a physical machine of 4 core 24 GB we are able to match speed of import
> with old monetdb. We are issuing copy into command through JDBC connection.
> 
> Can you guys help me with pointers on this issue, I will be happy to share
> any debug info.
> 
> Regards,
> Ashish Kumar Singh
> Mobile: +91-9972045095
> Skype: toashishk at gmail.com
> 
> 
> 
> _______________________________________________
> users-list mailing list
> users-list at monetdb.org
> https://www.monetdb.org/mailman/listinfo/users-list
> 
> 
> 
> _______________________________________________
> users-list mailing list
> users-list at monetdb.org
> https://www.monetdb.org/mailman/listinfo/users-list
> 
> 
> 
> _______________________________________________
> developers-list mailing list
> developers-list at monetdb.org
> https://www.monetdb.org/mailman/listinfo/developers-list
> 

-- 
| Stefan.Manegold at CWI.nl | DB Architectures   (DA) |
| www.CWI.nl/~manegold/  | Science Park 123 (L321) |
| +31 (0)20 592-4212     | 1098 XG Amsterdam  (NL) |




More information about the developers-list mailing list