Big Slowdown (15-20X ) on import functions in Monetdb October 2014 Release

Ashish Kumar Singh ashishkumar.singh at altair.com
Tue Nov 18 18:12:42 CET 2014


Sure, take your time! We have older release of monetdb.

Whenever you guys plan to have a look do let me know i can help in terms providing a test environment, machine etc.. For couple days if that helps.


Sent from my iPad

> On 18-Nov-2014, at 10:27 pm, Stefan Manegold <Stefan.Manegold at cwi.nl> wrote:
> 
> Hi Ashish,
> 
> please bear with us!
> 
> We highly appreciate your feedback and will have a closer look at the reported problem as soon as our time and other obligations allow so.
> 
> Please be aware that the members of this mailing-list provide free-of-charge voluntary spare-time best-effort services.
> 
> For commercial support, please also consider
> https://www.monetdbsolutions.com/
> 
> Thanks!
> Stefan
> 
> ----- Original Message -----
>> Hi all,
>> 
>> Are you guys able to reproduce this issue? Any help required from my side do
>> let me know.
>> 
>> 
>> Regards,
>> Ashish
>> 
>> Sent from my iPad
>> 
>> On 17-Nov-2014, at 5:24 pm, Ashish Kumar Singh < ashishkumar.singh at altair.com
>>> wrote:
>> 
>> 
>> 
>> 
>> 
>> 
>> Hi,
>> 
>> 
>> 
>> 
>> Below are the steps.
>> 
>> 
>> 
>> 
>> Unzip archive in to /opt/monetdb_test/ directroy to minimise changes in the
>> script datbase can live anywhere in the file system.
>> 
>> Single Big File Import:
>> 
>> 1. First create a database using Monetdb October2014 build called pbsworks
>> 2. Create data base schema using schema provided in file schema.sql.
>> 
>> Cat all files present inside testdata directory in to on single file by using
>> following command:
>> 
>> cat * > ../merged_data
>> 
>> Load this merged file in to database using script single_import.sql from
>> mclient
>> 
>> Note down the timing, usually 3-5 minutes on 4 core 17 GB machine on amazon.
>> 
>> Multiple small File Import:
>> 1. First create a database using Monetdb October2014 build called pbsworks
>> 2. Create data base schema using schema provided in file schema.sql.
>> 3. Load files in to database using multiple_import.sql using mclient
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> Regards,
>> 
>> Ashish
>> 
>> From: Anthony Damico < ajdamico at gmail.com >
>> Reply-To: " users-list at monetdb.org " < users-list at monetdb.org >
>> Date: Monday, 17 November 2014 3:51 pm
>> To: " users-list at monetdb.org " < users-list at monetdb.org >
>> Cc: " developers-list at monetdb.org " < developers-list at monetdb.org >
>> Subject: Re: Big Slowdown (15-20X ) on import functions in Monetdb October
>> 2014 Release
>> 
>> 
>> 
>> 
>> please provide a list of every command that you executed to reproduce the
>> problem. you need to make the list of commands as short as possible, so
>> people trying to help you are not wasting time on irrelevant code.. thanks!!
>> 
>> On Mon, Nov 17, 2014 at 4:22 AM, Ashish Kumar Singh <
>> ashishkumar.singh at altair.com > wrote:
>> 
>> 
>> 
>> Thanks for looking in to it,
>> 
>> I have uploaded required test files to you at below link:
>> 
>> https://ftin.india.altair.com:8443/message/eJCZlisrascBO00ffEVsQu
>> 
>> 
>> Regards,
>> Ashish
>> 
>> From: Anthony Damico < ajdamico at gmail.com >
>> Reply-To: " users-list at monetdb.org " < users-list at monetdb.org >
>> Date: Saturday, 15 November 2014 2:40 pm
>> To: " users-list at monetdb.org " < users-list at monetdb.org >
>> Cc: " developers-list at monetdb.org " < developers-list at monetdb.org >
>> 
>> Subject: Re: Big Slowdown (15-20X ) on import functions in Monetdb October
>> 2014 Release
>> 
>> 
>> 
>> 
>> could you provide a minimal reproducible example? the less code that
>> re-creates the problem, the better. thanks
>> 
>> http://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducible-example
>> 
>> On Sat, Nov 15, 2014 at 1:45 AM, Ashish Kumar Singh <
>> ashishkumar.singh at altair.com > wrote:
>> 
>> 
>> 
>> Additional information here,
>> 
>> 
>> If I merge all these files in to one file and then import to monetdb then
>> time taken is 3-5 minutes as compared to 10-15 hours with October 2014
>> branch.
>> 
>> Hopefully some one from moentdb team will look in to this. I am glad to see
>> that inspite of this slowness ( due to which this build is unusable) the
>> segfaults are gone!
>> 
>> 
>> Regards,
>> Ashish
>> 
>> 
>> From: Ashish Singh < ashishkumar.singh at altair.com >
>> Date: Friday, 14 November 2014 7:41 pm
>> To: " developers-list at monetdb.org " < developers-list at monetdb.org >
>> Cc: " users-list at monetdb.org " < users-list at monetdb.org >
>> Subject: Re: Big Slowdown (15-20X ) on import functions in Monetdb October
>> 2014 Release
>> 
>> 
>> 
>> 
>> Is any body else seeing this problem with October,2014 release?
>> 
>> I wish developer team can provide some insights in this area.
>> 
>> Regards,
>> Ashish
>> 
>> 
>> From: Ashish Singh < ashishkumar.singh at altair.com >
>> Date: Thursday, 13 November 2014 11:22 pm
>> To: " developers-list at monetdb.org " < developers-list at monetdb.org >
>> Cc: " users-list at monetdb.org " < users-list at monetdb.org >
>> Subject: Big Slowdown (15-20X ) on import functions in Monetdb October 2014
>> Release
>> 
>> 
>> 
>> 
>> All,
>> 
>> With October,2014 release we are stucked in to very strange performance issue
>> while importing data on virtualised environments.
>> 
>> On Virtual machines in our local VMware servers and amazon machine we are
>> seeing import performance has downgraded to close to 15 to 20 while
>> importing ~ 2000 files as compared to 1 hour time being taken with old
>> release. This we are seeing consistently with CentOS, SLES vm and
>> configuration is 4 Core 33 GB machine. We tried on quite a few amazon vm
>> just to make sure its not an issue with a particular instance setup.
>> 
>> On a physical machine of 4 core 24 GB we are able to match speed of import
>> with old monetdb. We are issuing copy into command through JDBC connection.
>> 
>> Can you guys help me with pointers on this issue, I will be happy to share
>> any debug info.
>> 
>> Regards,
>> Ashish Kumar Singh
>> Mobile: +91-9972045095
>> Skype: toashishk at gmail.com
>> 
>> 
>> 
>> _______________________________________________
>> users-list mailing list
>> users-list at monetdb.org
>> https://www.monetdb.org/mailman/listinfo/users-list
>> 
>> 
>> 
>> _______________________________________________
>> users-list mailing list
>> users-list at monetdb.org
>> https://www.monetdb.org/mailman/listinfo/users-list
>> 
>> 
>> 
>> _______________________________________________
>> developers-list mailing list
>> developers-list at monetdb.org
>> https://www.monetdb.org/mailman/listinfo/developers-list
>> 
> 
> -- 
> | Stefan.Manegold at CWI.nl | DB Architectures   (DA) |
> | www.CWI.nl/~manegold/  | Science Park 123 (L321) |
> | +31 (0)20 592-4212     | 1098 XG Amsterdam  (NL) |
> 
> _______________________________________________
> developers-list mailing list
> developers-list at monetdb.org
> https://www.monetdb.org/mailman/listinfo/developers-list



More information about the developers-list mailing list