GDK_mmap_minsize again

Hannes Mühleisen hannes.muehleisen at cwi.nl
Thu Sep 22 10:44:58 CEST 2016


Hi Foteini,

> On 20 Sep 2016, at 23:45, Foteini Alvanaki <F.Alvanaki at cwi.nl> wrote:
> 
> It is interesting that malloc is faster than mmap.
Yeah this is one of the magic MonetDB parameters…

> Could you give more details about the setting of your experiment?

just running select (i*2) from table; 
i is a column containing 100M integers. Ran on my laptop (OSX) with 4 threads.

> How many threads, how many concurrent files (BATs), size of files,
> type of data, kind of accesses etc.


> ----- Original Message -----
> From: Roberto Cornacchia <roberto.cornacchia at gmail.com>
> To: Communication channel for developers of the MonetDB suite. <developers-list at monetdb.org>
> Sent: Tue, 20 Sep 2016 18:40:41 +0200 (CEST)
> Subject: Re: GDK_mmap_minsize again
> 
> If I may add, that is indeed the default behaviour of the kernel, which can
> be disabled with
> 
> vm.overcommit_memory = 2
> in /etc/sysctl.conf
> 
> Perhaps MonetDB could check this system setting and decide on which
> strategy to use?
> 
> On 20 September 2016 at 18:24, Sjoerd Mullender <sjoerd at acm.org> wrote:
> 
>> As far as I understand it, malloc on Linux will happily succeed even if
>> there is not enough memory+swap to hold all data.  So you can't rely on
>> malloc failures to tell you to switch to mmap.
>> 
>> On 09/20/2016 06:19 PM, Hannes Mühleisen wrote:
>>> Hello list,
>>> 
>>> we were wondering about the purpose of GDK_mmap_minsize when creating
>> transient columns. The attached patch will always *try* to malloc/realloc a
>> transient column but still fall back to memory-mapped files if malloc
>> should fail. This dramatically improves performance. Any good reason why
>> this should not be the default behaviour?
>>> 
>>> Thanks,
>>> 
>>> Mark and Hannes
>>> 
>>> 
>>> 
>> 
>> --
>> Sjoerd Mullender
>> 
>> 
>> 
>> --
>> Sjoerd Mullender
>> 
>> 
>> _______________________________________________
>> developers-list mailing list
>> developers-list at monetdb.org
>> https://www.monetdb.org/mailman/listinfo/developers-list
>> 
>> 
> 
> _______________________________________________
> developers-list mailing list
> developers-list at monetdb.org
> https://www.monetdb.org/mailman/listinfo/developers-list



More information about the developers-list mailing list