[MonetDB-users] Loading directly into memory.
Hi,
The first query after a load action takes longer because the new data is still 'cold'. Is there a way in MonetDB to load data directly with a 'hot' status?
Thank in advance.
Hi,
no, MonetDB/SQL does not provide any means to explicitly load data from disk into memory.
If you need that, e.g., after a server restart to get data "hot" before the first user query runs, you could of course run some queries that touch (only) the data you want to make "hot".
Suitable example include simple (global) aggregates over each coulm you want to make hot. min() & max() are fine (i.e, perform a sequential scan) for columns that are not sorted --- if MonetDB know (after bulkload) that a column is sorted, min() and max() obviously need to touch only the first and last value, respectively. For sorted numerical columns, sum() is a suitable alternative.
Hope this helps you further, Stefan
On Thu, Jul 08, 2010 at 03:39:27AM -0700, Nozhup wrote:
Hi,
The first query after a load action takes longer because the new data is still 'cold'. Is there a way in MonetDB to load data directly with a 'hot' status?
Thank in advance.
View this message in context: http://old.nabble.com/Loading-directly-into-memory.-tp29105569p29105569.html Sent from the monetdb-users mailing list archive at Nabble.com.
This SF.net email is sponsored by Sprint What will you do first with EVO, the first 4G phone? Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Nozhup wrote:
Hi,
The first query after a load action takes longer because the new data is still 'cold'. Is there a way in MonetDB to load data directly with a 'hot' status?
Thank in advance.
So, assuming you are on a Unix platform, there are some pretty simple and effective ways to efficiently do this. The simples is to have a program that touches (read one byte) each page of the index file (typically every 4K in Linux) to force it to paged in to the FS cache (actually, this trick works pretty well on non-Unix platforms as well). For best efficiency, write a program that memory maps the file, then uses mlock() to force it in to RAM, and then exits.
--Chris
participants (3)
-
Christopher Smith -
Nozhup -
Stefan Manegold