[Monetdb-developers] [Monetdb-checkins] MonetDB/src/gdk gdk_heap.mx, Feb2009, 1.107, 1.107.2.1

Peter Boncz boncz at vectorwise.com
Thu Feb 19 23:52:01 CET 2009


Hi,

The mmap method is very costly on Windows as it does not use a virtual  
file with holes, but really creates it first. This may also be the  
case on macos. However, the pert degradation on Linux is unexpected.  
Maybe newer kernels have introduced different behavior.

I think the principle of falling back on mmap is sound, however the  
threshold needs to be increased. Maybe at a fixed perc (20%?) of  
available ram.

Peter


On Feb 19, 2009, at 10:49 PM, "Stefan Manegold"  
<Stefan.Manegold at cwi.nl> wrote:

> On Tue, Feb 10, 2009 at 09:13:21AM +0000, Stefan Manegold wrote:
>> Update of /cvsroot/monetdb/MonetDB/src/gdk
>> In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv18180/MonetDB/ 
>> src/gdk
>>
>> Modified Files:
>>      Tag: Feb2009
>>    gdk_heap.mx
>> Log Message:
>>
>> [on behalf of Peter]  re-doing
>>
>> 2009/02/05 - boncz: MonetDB/src/gdk/gdk_heap.mx,1.108
>> - make large allocs fall back again on mmap
>
> Peter,
>
> (how) can we make the limit (more) dependend on / adaptive to the  
> actual
> memory size?
>
> Shouldn't this have an effect only with blocks > gdk_vm_minsize ==  
> 128 GB ?
>
> I have sample scenarios (on Linux) where, e.g., queries produce  
> intermediate
> results of ~ 5GB --- on my 64-bit 8GB machine those queries run in   
> 40 secs
> before this change (i.e., with h->storage = STORE_MMAP;), but take  
> 200 secs
> (i.e., 5 times as long) after this change (i.e., with h->newstorage =
> STORE_MMAP;) ...
>
> Jan R. has seen similar effects with XMark on his Mac ...
>
> Thanks,
>
> Stefan
>
>> now only the actually intended changes
>> and as bug fix, i.e., in the Feb2009 release branch
>>
>>
>> Index: gdk_heap.mx
>> ===================================================================
>> RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_heap.mx,v
>> retrieving revision 1.107
>> retrieving revision 1.107.2.1
>> diff -u -d -r1.107 -r1.107.2.1
>> --- gdk_heap.mx    22 Jan 2009 14:47:53 -0000    1.107
>> +++ gdk_heap.mx    10 Feb 2009 09:13:19 -0000    1.107.2.1
>> @@ -115,7 +115,7 @@
>>        fp = GDKfilelocate(nme, "wb", ext);
>>        if (fp != NULL) {
>>            fclose(fp);
>> -            h->storage = STORE_MMAP;
>> +            h->newstorage = STORE_MMAP;
>>            HEAPload(h, nme, ext, FALSE);
>>        }
>>        GDKfree(of);
>>
>>
>> --- 
>> --- 
>> --- 
>> ---------------------------------------------------------------------
>> Create and Deploy Rich Internet Apps outside the browser with  
>> Adobe(R)AIR(TM)
>> software. With Adobe AIR, Ajax developers can use existing skills  
>> and code to
>> build responsive, highly engaging applications that combine the  
>> power of local
>> resources and data with the reach of the web. Download the Adobe  
>> AIR SDK and
>> Ajax docs to start building applications today-http://p.sf.net/sfu/ 
>> adobe-com
>> _______________________________________________
>> Monetdb-checkins mailing list
>> Monetdb-checkins at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
>
> -- 
> | Dr. Stefan Manegold | mailto:Stefan.Manegold at cwi.nl |
> | CWI,  P.O.Box 94079 | http://www.cwi.nl/~manegold/  |
> | 1090 GB Amsterdam   | Tel.: +31 (20) 592-4212       |
> | The Netherlands     | Fax : +31 (20) 592-4312       |
>
> No virus found in this incoming message.
> Checked by AVG - www.avg.com
> Version: 8.0.237 / Virus Database: 270.10.24/1954 - Release Date: 02/17/09 07 
> :07:00




More information about the developers-list mailing list