Monetdb copy binary time varys very much!

integrity 357416268 at qq.com
Fri Jul 26 10:48:21 CEST 2013


------------------ Original ------------------
From:  "Martin Kersten"<Martin.Kersten at cwi.nl>;
Date:  Fri, Jul 26, 2013 04:46 PM
To:  "users-list"<users-list at monetdb.org>; 

Subject:  Re: Monetdb copy binary time varys very much!



Hi,
You should plot the times of each of your runs to see the trend.
This single run may be an outlier, which could come from anything
in your system environment. Even a seemingly harmless concurrent
program using significant memory could compete with MonetDB.

And indeed, at some point you will see disk IO.


regards, Martin

On 7/26/13 10:40 AM, integrity wrote:
> Hi dear all,
>
> I did a test of insert 2,000,000,000 rows into MonetDB with "COPY BINARY
> INTO FROM binary_file",
> in this test
> 1.  i generate 21 files, each file represents a table column and has
> 200000 rows.
> I created them with rand() number and use fwrite() binary write method.
>
> the table creation sql command is :
> *create table tmatch(id bigint,a float,b float,c float, d float, e float,*
> *f float,g float,h float, i float,j float,*
> *k float,l float,m float, n float,o float,*
> *p float,q float,r float, s float,t float);*
> the table has 21 columns,each column has 8 bytes, so each column file is
> c1=200000*21*8 Byte= 268800000 Byte=3.2MB
> 2. I use "COPY BINARY INTO FROM above_binary" to load each binary file
> into tmatch.
> The test was run 10000 times repeatedly.
>
> the average time of 10000 times is only 1.0635589558955727,
> but when at 9043th time,  it cost 227m36.136 ,some times later the time
> value increase to a large number,  is it because of flush data from
> cache into database after the cache is full?
> The problem is since we have to control the total process time within 15
> seconds , I am wondering if you can help me reduce the maximum time to a
> lower point?
>
>
> Thanks very much!
> Meng
>
>
>
> _______________________________________________
> users-list mailing list
> users-list at monetdb.org
> http://mail.monetdb.org/mailman/listinfo/users-list
>
_______________________________________________
users-list mailing list
users-list at monetdb.org
http://mail.monetdb.org/mailman/listinfo/users-list
.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.monetdb.org/pipermail/users-list/attachments/20130726/2c4f7f6b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: EEC11D22 at BE32F20E.D537F251.png
Type: application/octet-stream
Size: 52635 bytes
Desc: not available
URL: <http://www.monetdb.org/pipermail/users-list/attachments/20130726/2c4f7f6b/attachment.obj>


More information about the users-list mailing list