On Thu, Feb 19, 2009 at 10:49:24PM +0100, Stefan Manegold wrote:

On Tue, Feb 10, 2009 at 09:13:21AM +0000, Stefan Manegold wrote:

...

Update of /cvsroot/monetdb/MonetDB/src/gdk In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv18180/MonetDB/src/gdk

Modified Files: Tag: Feb2009 gdk_heap.mx Log Message:

[on behalf of Peter] re-doing

2009/02/05 - boncz: MonetDB/src/gdk/gdk_heap.mx,1.108

• make large allocs fall back again on mmap

Peter,

(how) can we make the limit (more) dependend on / adaptive to the actual memory size?

Shouldn't this have an effect only with blocks > gdk_vm_minsize == 128 GB ?

I have sample scenarios (on Linux) where, e.g., queries produce intermediate results of ~ 5GB --- on my 64-bit 8GB machine those queries run in 40 secs before this change (i.e., with h->storage = STORE_MMAP;), but take 200 secs (i.e., 5 times as long) after this change (i.e., with h->newstorage = STORE_MMAP;) ...

... on a 64GB Linux machine, I even see a 10x degradation: 30s -> 300s ...

Jan R. has seen similar effects with XMark on his Mac ...

Thanks,

Stefan

...

now only the actually intended changes and as bug fix, i.e., in the Feb2009 release branch

Index: gdk_heap.mx

RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_heap.mx,v retrieving revision 1.107 retrieving revision 1.107.2.1 diff -u -d -r1.107 -r1.107.2.1 --- gdk_heap.mx 22 Jan 2009 14:47:53 -0000 1.107 +++ gdk_heap.mx 10 Feb 2009 09:13:19 -0000 1.107.2.1 @@ -115,7 +115,7 @@ fp = GDKfilelocate(nme, "wb", ext); if (fp != NULL) { fclose(fp);

• 	h->storage = STORE_MMAP;
  

• 	h->newstorage = STORE_MMAP;
  HEAPload(h, nme, ext, FALSE);
  

  } GDKfree(of);

---

Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins

-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

You said 5gb on 8gb works better on swap memory.. wow. Maybe go for 100% of ram then for the threshold

Sent from my iPhone

On Feb 19, 2009, at 10:49 PM, "Stefan Manegold" Stefan.Manegold@cwi.nl wrote:

On Tue, Feb 10, 2009 at 09:13:21AM +0000, Stefan Manegold wrote:

...

Update of /cvsroot/monetdb/MonetDB/src/gdk In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv18180/MonetDB/ src/gdk

Modified Files: Tag: Feb2009 gdk_heap.mx Log Message:

[on behalf of Peter] re-doing

2009/02/05 - boncz: MonetDB/src/gdk/gdk_heap.mx,1.108

• make large allocs fall back again on mmap

Peter,

(how) can we make the limit (more) dependend on / adaptive to the actual memory size?

Shouldn't this have an effect only with blocks > gdk_vm_minsize == 128 GB ?

I have sample scenarios (on Linux) where, e.g., queries produce intermediate results of ~ 5GB --- on my 64-bit 8GB machine those queries run in 40 secs before this change (i.e., with h->storage = STORE_MMAP;), but take 200 secs (i.e., 5 times as long) after this change (i.e., with h->newstorage = STORE_MMAP;) ...

Jan R. has seen similar effects with XMark on his Mac ...

Thanks,

Stefan

...

now only the actually intended changes and as bug fix, i.e., in the Feb2009 release branch

Index: gdk_heap.mx

RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_heap.mx,v retrieving revision 1.107 retrieving revision 1.107.2.1 diff -u -d -r1.107 -r1.107.2.1 --- gdk_heap.mx 22 Jan 2009 14:47:53 -0000 1.107 +++ gdk_heap.mx 10 Feb 2009 09:13:19 -0000 1.107.2.1 @@ -115,7 +115,7 @@ fp = GDKfilelocate(nme, "wb", ext); if (fp != NULL) { fclose(fp);

•        h->storage = STORE_MMAP;
  

•        h->newstorage = STORE_MMAP;
       HEAPload(h, nme, ext, FALSE);
   }
   GDKfree(of);
  
  
  

---

---

---

---

Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/ adobe-com _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins

-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.0.237 / Virus Database: 270.10.24/1954 - Release Date: 02/17/09 07 :07:00