Monetdb copy binary time varys very much!

Martin Kersten Martin.Kersten at cwi.nl
Fri Jul 26 11:30:08 CEST 2013


This runs shows expected flushing.

You might get somewhat better performance when you follow
the advice in 
http://www.monetdb.org/Documentation/Cookbooks/SQLrecipes/LoadingBulkData
e.g. use COPY 44000000 RECORDS ....

The COPY INTO forms mentioned so far share one important drawback.  When 
a lot of data is to be inserted, the server doesn't know how much memory 
to allocate for the tables, and so will likely allocate too little. 
This means that during the insertion process, the server has to grow the 
allocated memory area.  This can be an expensive operation.  Therefore, 
it is better to give the server a count of how many records are to be 
inserted:

COPY n RECORDS INTO table FROM 'file';

Here n should be a number that is at least as large as the actual number 
of records to be inserted.  If the file contains more than n records, 
only n will be inserted, if the file contains fewer, all values will be 
inserted.  Giving a higher number is especially useful if multiple COPY 
INTO queries are to be done on the same table.  The first COPY INTO, 
when the table is still empty, should be the total count of 
to-be-inserted values so that the server will allocate enough memory 
when the tables are first created (they are only really created once 
data is inserted).
On 7/26/13 10:58 AM, integrity wrote:
>
>
> ------------------ Original ------------------
> *From: * "Martin Kersten"<Martin.Kersten at cwi.nl>;
> *Date: * Fri, Jul 26, 2013 04:46 PM
> *To: * "users-list"<users-list at monetdb.org>;
> *Subject: * Re: Monetdb copy binary time varys very much!
>
> Hi,
> You should plot the times of each of your runs to see the trend.
> This single run may be an outlier, which could come from anything
> in your system environment. Even a seemingly harmless concurrent
> program using significant memory could compete with MonetDB.
>
> And indeed, at some point you will see disk IO.
>
>
> regards, Martin
>
>
>
>
> _______________________________________________
> users-list mailing list
> users-list at monetdb.org
> http://mail.monetdb.org/mailman/listinfo/users-list
>



More information about the users-list mailing list