Hi Ahmad, Given your response it seems the starting point is to understand memory mapped files http://en.wikipedia.org/wiki/Memory-mapped_file and the distinction between cold/hot database states, i.e. data is (not) cached into memory as a side effect of the memory mapped file functionality.
MonetDB uses this facility.
regards, Martin
On 3/27/13 4:41 PM, Hassan, Ahmad wrote:
HI Martin,
Thanks for pointing to the science library. These are very interesting papers. I have read few papers about monetdb i.e Breaking the Memory Wall in MonetDB But I didn't get the information that I am looking for. Please can you point me to the relevant paper which explains how monetdb makes use of RAM memory and why monetdb only use ~3GB of RAM space for the dataset of size 45GB. Is it configurable in monetdb that all the data should be brought into the RAM and so act as a completely in-memory database?
One thing I noticed that when I run the query first time then it takes quite a bit of time on 40gb dataset but for the subsequent execution of same query is faster. Is it because monetdb store results in some views?
This information will be really helpful for me in understanding how monetdb makes use of the main memory.
Thanks.
Kind Regards, Ahmad
-----Original Message----- From: users-list-bounces+ahmad.hassan=sap.com@monetdb.org [mailto:users-list-bounces+ahmad.hassan=sap.com@monetdb.org] On Behalf Of Martin Kersten Sent: 21 March 2013 12:48 To: Communication channel for MonetDB users Subject: Re: RAM consumption of MonetDB
Hi,
In a nutshell, (1) a column store will only touch the columns on disk as needed and (2) MonetDB will memory map them upon need leaving the details to the operating system.
For the full story, see the (scientific) background literature. http://www.monetdb.org/Home/ScienceLibrary
regards, Martin
On 3/21/13 1:43 PM, Hassan, Ahmad wrote:
Hi Team,
Please may I ask what main memory management strategy has been adopted for MonetDB in terms of memory utilization. For example I loaded the TPCH benchmark dataset in MonetDB (v11.13.5) with scale factor 40. The data on disk is 43Gb in size. But when I run the TPCH queries then the main memory used by mserver stays in the range of 0.5GB to 3GB.
I was thinking that the monetDB would load the tables into memory because TPCH queries are doing joins over various tables but it seems that monetdb loads only small subset of the complete dataset.
Please can someone explain what is going in the background and why I always see 0.5GB to 3GB memory resident size of mserver process even for dataset of size 43G and if this is the normal expected behaviour of monetdb?
The farm size on disk is:
farms/f1/db1/bat# du . -hs
48G .
Thanks.
Kind Regards, Ahmad
users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list _______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list