Big Slowdown (15-20X ) on import functions in Monetdb October 2014 Release

Ashish Kumar Singh ashishkumar.singh at altair.com
Fri Nov 21 18:54:25 CET 2014


Hey guys,

Just double checking with you guys if there are any plans to  see if reported behaviour is an issue, and how can i help?



Regards,
Ashish

Sent from my iPad

> On 18-Nov-2014, at 10:42 pm, Ashish Kumar Singh <ashishkumar.singh at altair.com> wrote:
> 
> Sure, take your time! We have older release of monetdb.
> 
> Whenever you guys plan to have a look do let me know i can help in terms providing a test environment, machine etc.. For couple days if that helps.
> 
> 
> Sent from my iPad
> 
>> On 18-Nov-2014, at 10:27 pm, Stefan Manegold <Stefan.Manegold at cwi.nl> wrote:
>> 
>> Hi Ashish,
>> 
>> please bear with us!
>> 
>> We highly appreciate your feedback and will have a closer look at the reported problem as soon as our time and other obligations allow so.
>> 
>> Please be aware that the members of this mailing-list provide free-of-charge voluntary spare-time best-effort services.
>> 
>> For commercial support, please also consider
>> https://www.monetdbsolutions.com/
>> 
>> Thanks!
>> Stefan
>> 
>> ----- Original Message -----
>>> Hi all,
>>> 
>>> Are you guys able to reproduce this issue? Any help required from my side do
>>> let me know.
>>> 
>>> 
>>> Regards,
>>> Ashish
>>> 
>>> Sent from my iPad
>>> 
>>> On 17-Nov-2014, at 5:24 pm, Ashish Kumar Singh < ashishkumar.singh at altair.com
>>>> wrote:
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> Hi,
>>> 
>>> 
>>> 
>>> 
>>> Below are the steps.
>>> 
>>> 
>>> 
>>> 
>>> Unzip archive in to /opt/monetdb_test/ directroy to minimise changes in the
>>> script datbase can live anywhere in the file system.
>>> 
>>> Single Big File Import:
>>> 
>>> 1. First create a database using Monetdb October2014 build called pbsworks
>>> 2. Create data base schema using schema provided in file schema.sql.
>>> 
>>> Cat all files present inside testdata directory in to on single file by using
>>> following command:
>>> 
>>> cat * > ../merged_data
>>> 
>>> Load this merged file in to database using script single_import.sql from
>>> mclient
>>> 
>>> Note down the timing, usually 3-5 minutes on 4 core 17 GB machine on amazon.
>>> 
>>> Multiple small File Import:
>>> 1. First create a database using Monetdb October2014 build called pbsworks
>>> 2. Create data base schema using schema provided in file schema.sql.
>>> 3. Load files in to database using multiple_import.sql using mclient
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> Regards,
>>> 
>>> Ashish
>>> 
>>> From: Anthony Damico < ajdamico at gmail.com >
>>> Reply-To: " users-list at monetdb.org " < users-list at monetdb.org >
>>> Date: Monday, 17 November 2014 3:51 pm
>>> To: " users-list at monetdb.org " < users-list at monetdb.org >
>>> Cc: " developers-list at monetdb.org " < developers-list at monetdb.org >
>>> Subject: Re: Big Slowdown (15-20X ) on import functions in Monetdb October
>>> 2014 Release
>>> 
>>> 
>>> 
>>> 
>>> please provide a list of every command that you executed to reproduce the
>>> problem. you need to make the list of commands as short as possible, so
>>> people trying to help you are not wasting time on irrelevant code.. thanks!!
>>> 
>>> On Mon, Nov 17, 2014 at 4:22 AM, Ashish Kumar Singh <
>>> ashishkumar.singh at altair.com > wrote:
>>> 
>>> 
>>> 
>>> Thanks for looking in to it,
>>> 
>>> I have uploaded required test files to you at below link:
>>> 
>>> https://ftin.india.altair.com:8443/message/eJCZlisrascBO00ffEVsQu
>>> 
>>> 
>>> Regards,
>>> Ashish
>>> 
>>> From: Anthony Damico < ajdamico at gmail.com >
>>> Reply-To: " users-list at monetdb.org " < users-list at monetdb.org >
>>> Date: Saturday, 15 November 2014 2:40 pm
>>> To: " users-list at monetdb.org " < users-list at monetdb.org >
>>> Cc: " developers-list at monetdb.org " < developers-list at monetdb.org >
>>> 
>>> Subject: Re: Big Slowdown (15-20X ) on import functions in Monetdb October
>>> 2014 Release
>>> 
>>> 
>>> 
>>> 
>>> could you provide a minimal reproducible example? the less code that
>>> re-creates the problem, the better. thanks
>>> 
>>> http://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducible-example
>>> 
>>> On Sat, Nov 15, 2014 at 1:45 AM, Ashish Kumar Singh <
>>> ashishkumar.singh at altair.com > wrote:
>>> 
>>> 
>>> 
>>> Additional information here,
>>> 
>>> 
>>> If I merge all these files in to one file and then import to monetdb then
>>> time taken is 3-5 minutes as compared to 10-15 hours with October 2014
>>> branch.
>>> 
>>> Hopefully some one from moentdb team will look in to this. I am glad to see
>>> that inspite of this slowness ( due to which this build is unusable) the
>>> segfaults are gone!
>>> 
>>> 
>>> Regards,
>>> Ashish
>>> 
>>> 
>>> From: Ashish Singh < ashishkumar.singh at altair.com >
>>> Date: Friday, 14 November 2014 7:41 pm
>>> To: " developers-list at monetdb.org " < developers-list at monetdb.org >
>>> Cc: " users-list at monetdb.org " < users-list at monetdb.org >
>>> Subject: Re: Big Slowdown (15-20X ) on import functions in Monetdb October
>>> 2014 Release
>>> 
>>> 
>>> 
>>> 
>>> Is any body else seeing this problem with October,2014 release?
>>> 
>>> I wish developer team can provide some insights in this area.
>>> 
>>> Regards,
>>> Ashish
>>> 
>>> 
>>> From: Ashish Singh < ashishkumar.singh at altair.com >
>>> Date: Thursday, 13 November 2014 11:22 pm
>>> To: " developers-list at monetdb.org " < developers-list at monetdb.org >
>>> Cc: " users-list at monetdb.org " < users-list at monetdb.org >
>>> Subject: Big Slowdown (15-20X ) on import functions in Monetdb October 2014
>>> Release
>>> 
>>> 
>>> 
>>> 
>>> All,
>>> 
>>> With October,2014 release we are stucked in to very strange performance issue
>>> while importing data on virtualised environments.
>>> 
>>> On Virtual machines in our local VMware servers and amazon machine we are
>>> seeing import performance has downgraded to close to 15 to 20 while
>>> importing ~ 2000 files as compared to 1 hour time being taken with old
>>> release. This we are seeing consistently with CentOS, SLES vm and
>>> configuration is 4 Core 33 GB machine. We tried on quite a few amazon vm
>>> just to make sure its not an issue with a particular instance setup.
>>> 
>>> On a physical machine of 4 core 24 GB we are able to match speed of import
>>> with old monetdb. We are issuing copy into command through JDBC connection.
>>> 
>>> Can you guys help me with pointers on this issue, I will be happy to share
>>> any debug info.
>>> 
>>> Regards,
>>> Ashish Kumar Singh
>>> Mobile: +91-9972045095
>>> Skype: toashishk at gmail.com
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> users-list mailing list
>>> users-list at monetdb.org
>>> https://www.monetdb.org/mailman/listinfo/users-list
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> users-list mailing list
>>> users-list at monetdb.org
>>> https://www.monetdb.org/mailman/listinfo/users-list
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> developers-list mailing list
>>> developers-list at monetdb.org
>>> https://www.monetdb.org/mailman/listinfo/developers-list
>> 
>> -- 
>> | Stefan.Manegold at CWI.nl | DB Architectures   (DA) |
>> | www.CWI.nl/~manegold/  | Science Park 123 (L321) |
>> | +31 (0)20 592-4212     | 1098 XG Amsterdam  (NL) |
>> 
>> _______________________________________________
>> developers-list mailing list
>> developers-list at monetdb.org
>> https://www.monetdb.org/mailman/listinfo/developers-list



More information about the developers-list mailing list