Hi Stefan, Martin,
Thanks. That's very helpful indeed.
Kind Regards, Ahmad
-----Original Message----- From: users-list-bounces+ahmad.hassan=sap.com@monetdb.org [mailto:users-list-bounces+ahmad.hassan=sap.com@monetdb.org] On Behalf Of Stefan Manegold Sent: 27 March 2013 16:10 To: Communication channel for MonetDB users Subject: Re: RAM consumption of MonetDB
Hi Ahmad,
in addition to what Martin wrote and suggested, please also have a look at http://www.monetdb.org/Documentation/UserGuide/resources
In a nutshell, as opposed to many (all?) other DBMSs --- other than pure in-memory systems --- MonetDB fully relies on the OS's virtual memory management. This also means that only the data that is actually use by your queries is loaded into main memory. There is not need to load all, say, 45 GB, if the queries touch at most 3 GB (per query or in total). It depends on your OS's VM policies (as well as other processes running on the same machine), whether / when data that was loaded into memory will be dropped, again. Thus, if you see a peak of 3 GB RSS of mserver5, this may or may not mean that not more data was touched. You will see that your mserver5's virtual size is (much) larger than the resident size, indicating that not all memory mapped data is indeed accessed. As said, there is no need to load data that is not required. Please also be aware of the fact that --- depending on your OS and RAM size --- more data (relevant) than indicated by your mserver5's resident size might be physically in main memory, as it might be in your OS's file system cache.
Best, Stefan
----- Original Message -----
Hi Ahmad, Given your response it seems the starting point is to understand memory mapped files http://en.wikipedia.org/wiki/Memory-mapped_file and the distinction between cold/hot database states, i.e. data is (not) cached into memory as a side effect of the memory mapped file functionality.
MonetDB uses this facility.
regards, Martin
On 3/27/13 4:41 PM, Hassan, Ahmad wrote:
HI Martin,
Thanks for pointing to the science library. These are very interesting papers. I have read few papers about monetdb i.e Breaking the Memory Wall in MonetDB But I didn't get the information that I am looking for. Please can you point me to the relevant paper which explains how monetdb makes use of RAM memory and why monetdb only use ~3GB of RAM space for the dataset of size 45GB. Is it configurable in monetdb that all the data should be brought into the RAM and so act as a completely in-memory database?
One thing I noticed that when I run the query first time then it takes quite a bit of time on 40gb dataset but for the subsequent execution of same query is faster. Is it because monetdb store results in some views?
This information will be really helpful for me in understanding how monetdb makes use of the main memory.
Thanks.
Kind Regards, Ahmad
-----Original Message----- From: users-list-bounces+ahmad.hassan=sap.com@monetdb.org [mailto:users-list-bounces+ahmad.hassan=sap.com@monetdb.org] On Behalf Of Martin Kersten Sent: 21 March 2013 12:48 To: Communication channel for MonetDB users Subject: Re: RAM consumption of MonetDB
Hi,
In a nutshell, (1) a column store will only touch the columns on disk as needed and (2) MonetDB will memory map them upon need leaving the details to the operating system.
For the full story, see the (scientific) background literature. http://www.monetdb.org/Home/ScienceLibrary
regards, Martin
On 3/21/13 1:43 PM, Hassan, Ahmad wrote:
Hi Team,
Please may I ask what main memory management strategy has been adopted for MonetDB in terms of memory utilization. For example I loaded the TPCH benchmark dataset in MonetDB (v11.13.5) with scale factor 40. The data on disk is 43Gb in size. But when I run the TPCH queries then the main memory used by mserver stays in the range of 0.5GB to 3GB.
I was thinking that the monetDB would load the tables into memory because TPCH queries are doing joins over various tables but it seems that monetdb loads only small subset of the complete dataset.
Please can someone explain what is going in the background and why I always see 0.5GB to 3GB memory resident size of mserver process even for dataset of size 43G and if this is the normal expected behaviour of monetdb?
The farm size on disk is:
farms/f1/db1/bat# du . -hs
48G .
Thanks.
Kind Regards, Ahmad
users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list _______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list