Monetdb copy binary time varys very much!
357416268 at qq.com
Mon Jul 29 11:18:12 CEST 2013
Yes, I use the server only myself only to run the MonetDB test and notify other users not to use the server in advance.
and I reboot the server before tests .
------------------ Original ------------------
From: "Ying Zhang"<Y.Zhang at cwi.nl>;
Date: Mon, Jul 29, 2013 04:28 PM
To: "Communication channel for MonetDB users"<users-list at monetdb.org>;
Subject: Re: Monetdb copy binary time varys very much!
On Jul 29, 2013, at 05:57, integrity wrote:
> Hi Stefan,
> Due to all the data in RAM having to be written to disk when it's full, if the RAM grows larger and i don't change hard disk, the data volume become larger, will it make the COPY BINARY INTO much slower compared to the small RAM?
> Contrarily, if the RAM get smaller, will it make the COPY BINARY INTO quicker?
> I just want to lower the peak value of COPY, the 10+ seconds every about a hundred times.
Have you made sure that there are no processes running other than the ones strictly necessary? Have you closed all user applications, such as mail client, browser, and chat clients?
> For the most time COPY BINARY INTO took only less than 1 second, is it because they are written to RAM not disk when the RAM is not full? if so, can i write to disk more frequently before the disk is full so i could lower down the peak value of COPY.
> ------------------ Original ------------------
> From: "Stefan Manegold"<Stefan.Manegold at cwi.nl>;
> Date: Fri, Jul 26, 2013 11:10 PM
> To: "Communication channel for MonetDB users"<users-list at monetdb.org>;
> Subject: Re: Monetdb copy binary time varys very much!
> I'm not sure whether I understand correct what you are doing.
> If you repeat the test 1000 times, does that mean that (1) 10000 times you
> re-create (or empty) the table and thus always copy into an empty table, or
> (2) 10000 times you copy into the same (growing) table, i.e., resulting in a
> table of 10,000 times 200,000 rows, i.e., 2,000,000,000 rows, i.e., ~16 GB
> per column, i.e., ~336 GB in total?
> (Only) in case (1) the binary files to be imported are simply moved at zero
> In case (2), only the first copy into (into the empty table) can simply move
> the files at zero costs; all subsequent copy into (into a no longe empty
> table) must copy the files (and delete them afterwards to mimic the same
> behavior as the initial copy into), which is of cause not "for free".
> Also, as Martin explained, unless your machine has (significantly) more RAM
> than the ~336 GB of data you copy, the data needs to be written to disk in
> between, making some copy into's "slower" than others. There's not much to
> do about that other than (a) getting more RAM, or (b) improving I/O
> bandwidth by using either a high performance RAID or SSDs.
> users-list mailing list
> users-list at monetdb.org
users-list mailing list
users-list at monetdb.org
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the users-list