[Monetdb-developers] mem_bigsize

Stefan Manegold Stefan.Manegold at cwi.nl
Mon Feb 11 18:37:06 CET 2008


On Mon, Feb 11, 2008 at 05:07:11PM +0100, Peter Boncz wrote:
> Hi Stefan,
> 
> Thanks for so much Grundlichkeit. It is appreciated.

Welcome ;-)

> It seems that the comment
> in the XQuery documentation describing mem_bigsize is not correct. Given the
> current disabled status of mem_bigsize I will remove it.

Ok. Thanks.

> The behavior to limit mem_bigsize to the values of both mem_maxsize and
> vm_minsize I did not recall; but it probably makes sense.

Ok. Well, my main point was to be aware of this part of code, and to keep it
aligned/consistent between M4 & M5. 
Given your explanation, I suggest to re-anble this part in M5, since
mem_bigsize is still (as your say below correctly) used in other parts of
the code; only in GDKmallocmax() it is ignored, now.
I'll take care of that.

> The rest is.. indeed correct.

Thanks. That's what I wanted to have (double-) checked by an expert.

> The repitition of configuration values in the
> python and monet_options has always bothered me, but I recall there was some
> reason why it cannot be avoided (though not what it was).

The repitition is not a problem. Main point was, whether mem_bigsize should
appear there at all, but since it's still used, (outside GDKmallocmax()),
the asnwer seems to be a "yes".

> However, I have spent the entire night&day, re-thinking the mmap-heap
> situation; and there are still some problems. I think it can be fixed once and
> for all, but it requires abandoning the X.priv approach to STORE_PRIV
> copy-on-write heaps in favor of always mapping the X file and writing into
> X.new files (which take preference over X files if encountered). But that also
> affects the BBP, the commit protocol.

Well, I guess, that's some major change to be consired for the next
release...

Stefan 

> Peter
> 
> 
> 
> 
> Dear all,
> 
> On Mon, Feb 11, 2008 at 02:54:59PM +0100, Peter Boncz wrote:
> > Hi Stefan,
> >
> > The pftijah bug was unrelated to vmalloc(), a relationship that I never
> > assumed.
> 
> Indeed, that was (only) my mis-interpretation. Sorry.
> 
> > The vmalloc() troubles occured, as you may recall, during the data
> > preparation of the sorted 100GB TPC-H.
> >
> > As it is entirely foreseeable that we at some point again will re-encounter an
> > OS with a malloc implementation that is prone to fragmentation, I'd keep the
> > vmalloc() code in for the moment (though in disabled state). Linux and Windows
> > do not need it.
> >
> > So, my advise is to for the monet let vmalloc() be disabled, until the issue
> > is properly investigated and a bugfix is found. Let's call it option (0).
> 
> Ok.
> 
> Just to sum up the current state.
> Please check, whether this is OK and consistent:
> 
> 
> a)
> Documentation in MonetDB4/conf/MonetDB.conf.in &
> MonetDB5/conf/monetdb5.conf.in says:
> "
> # gdk_mem_bigsize & gdk_vm_minsize will be set/limited to
> # 1/2 of the physically available amount of main-memory
> # during start-up in src/tools/Mserver.mx
> # memory chunks of size >= gdk_mem_bigsize (in bytes) will be mmaped anonymously
> #gdk_mem_bigsize=262144
> # memory chunks of size >= gdk_vm_minsize (in bytes) will be mmaped;
> #gdk_vm_minsize=137438953472
> "
> 
> b)
> Documentation in ./monetweb/Docs/XQuery/MonetDB.conf.texi says
> "
> @item @code{gdk_mem_bigsize}: minimum size for memory-mapped columns, e.g.
> @code{262144}.
> @item @code{gdk_vm_minsize}: column size above which memory mapped files are
> used always, e.g. @code{1749291171}
> "
> 
> c)
> MonetDB/src/common/monet_options.mx sets
> "
>         set[i].kind = opt_builtin;
>         set[i].name = strdup("gdk_mem_bigsize");
>         set[i].value = strdup("262144");
> "
> 
> d)
> MonetDB/src/common/monet_options.py.in sets
> "
>     # gdk_mem_bigsize & gdk_vm_minsize will be set/limited to
>     # 1/2 of the physically available amount of main-memory
>     # during start-up in src/tools/Mserver.mx
>     gdk_mem_bigsize = '256K'
>     gdk_vm_minsize = '128G'
> "
> 
> e)
> GDKinit() in MonetDB/src/gdk/gdk_utils.mx sets
> "
>         if ((p = GDKgetenv("gdk_mem_bigsize"))) {
>                 /* when allocating >6% of all RAM; do so using vmalloc() iso
> malloc() */
>                 lng max_mem_bigsize = GDK_mem_maxsize/16;
> 
>                 /* sanity check to avoid memory fragmentation */
>                 GDK_mem_bigsize = (size_t) MIN(max_mem_bigsize, strtol(p,
> NULL, 10));
>         }
> "
> 
> f)
> GDKmallocmax() in MonetDB/src/gdk/gdk_utils.mx ignores GDK_mem_bigsize
> 
> g)
> HEAPalloc() in MonetDB/src/gdk/gdk_heap.mx uses GDK_mem_bigsize as follows:
> "
>         /* when using anonymous vm we malloc we need 64K chunks, also we
>          * 20% extra malloc */
>         if (h->size > GDK_mem_bigsize) {
>                 h->maxsize = (size_t) ((double) h->maxsize * BATMARGIN) - 1;
>                 h->maxsize = (1 + (h->maxsize >> 16)) << 16;
>         }
> "
> 
> h)
> HEAPextend() in MonetDB/src/gdk/gdk_heap.mx uses GDK_mem_bigsize as follows:
> "
>                 /* extend a malloced heap, possibly switching over to
> file-mapped storage */
>                 Heap bak = *h;
>                 int can_mmap = (h->filename && size >= GDK_mem_bigsize);
>                 int must_mmap = can_mmap && (size >= GDK_vm_minsize ||
> (h->newstorage != STORE_MEM));
> [...]
>                 if (can_mmap) {
>                         /* in anonymous vm, if have to realloc anyway, we
> reserve some extra space */
>                         if (size > h->maxsize) {
>                                 h->maxsize = (size_t) ((double) size *
> BATMARGIN);
>                         }
>                         /* when using anonymous vm we malloc we need 64K
> chunks */
>                         h->maxsize = (1 + ((h->maxsize - 1) >> 16)) << 16;
> [...]
>                 /* too big: convert it to a disk-based temporary heap */
>                 if (can_mmap) {
>                         char privext[PATHLENGTH], *of = h->filename;
>                         FILE *fp;
> 
>                         h->filename = NULL;
>                         sprintf(privext, "%s.priv", ext);
>                         fp = GDKfilelocate(nme, "wb", privext);
>                         if (fp != NULL) {
>                                 fclose(fp);
>                                 /* a non-persistent heap: we create a .priv
> but *not* MMAP_PRIV !!! */
>                                 h->storage = STORE_MMAP;
>                                 h->base = NULL;
>                                 if (HEAPload(h, nme, privext, FALSE) >= 0) {
>                                         memcpy(h->base, bak.base, bak.free);
>                                         HEAPfree(&bak);
>                                         return 0;
>                                 }
>                         }
>                         GDKfree(of);
>                 }
> "
> 
> i)
> MIL offers mem_bigsize() & mem_bigsize(lng) to get / set mem_bigsize
> in MonetDB4/src/modules/plain/sys.mx
> 
> j)
> both MIL & MAL offer commands to set mem_maxsize and vm_minsize, however,
> only the MIL variants (still) related mem_maxsize & vm_minsize with mem_bigsize,
> while the MAL variants no longer do so:
> 
> MonetDB4/src/modules/plain/sys.mx:
> "
> int
> set_mem_maxsize(lng *num)
> {
>         @:num2sze(mem_maxsize)@
>         if (sze < GDK_mem_bigsize)
>                 set_mem_bigsize(num);
>         GDK_mem_maxsize = MAX(GDK_mem_bigsize, sze);
>         return GDK_SUCCEED;
> }
> 
> int
> set_vm_minsize(lng *num)
> {
>         @:num2sze(vm_minsize)@
>         if (sze < GDK_mem_bigsize)
>                 set_mem_bigsize(num);
>         GDK_vm_minsize = MAX(GDK_mem_bigsize, sze);
>         return GDK_SUCCEED;
> }
> "
> 
> MonetDB5/src/modules/kernel/status.mx
> "
> int
> set_mem_maxsize(lng *num)
> {
>         @:num2sze(mem_maxsize)@
> #if 0
>         if (sze < GDK_mem_bigsize)
>                 set_mem_bigsize(num);
>         GDK_mem_maxsize = MAX(GDK_mem_bigsize, sze);
> #endif
>         GDK_mem_maxsize = sze;
>         return GDK_SUCCEED;
> }
> 
> int
> set_vm_minsize(lng *num)
> {
>         @:num2sze(vm_minsize)@
> #if 0
>         if (sze < GDK_mem_bigsize)
>                 set_mem_bigsize(num);
>         GDK_vm_minsize = MAX(GDK_mem_bigsize, sze);
> #endif
>         GDK_vm_minsize = sze;
>         return GDK_SUCCEED;
> }
> "
> 
> 
> If all this is considered OK and consistent, we're done.
> Otherwise, we need to do what ever is required to get the
> code and documentation consistent, preferably before the release.
> 
> Stefan
> 
> 
> > Peter
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Dear all,
> >
> > since Roberto's bug appears to be fixed by Peter's ".priv" (vm_minsize
> > related) fixes (Thanks!), and disabling of vmalloc() (mem_bigsize related)
> > does not appear have any impact on this bug (just tested successfully both
> > with vmalloc() disabled and vmalloc() enabled), there are two ways to finish
> > this release (more or less) quickly:
> >
> > (1) finish the disabling of vmalloc() in all parts of the code as indicated
> >     below (or better: remove all remains completely) --- this might need
> >     some extra testing to ensure that it does not have any (significant)
> >     impact on performance ...
> >
> > (2) for now (i.e., in the release branch) re-enable vmalloc(), and then
> >     finish the removal of it (incl. extensive testing) in the development
> >     trunk for th next release.
> >
> > Since I'm not completely aware of the impact of vmalloc(), I cannot really
> > make this decision --- the "save" back-up would IMHO be (2).
> >
> > Any other comments, opinions, suggestions, expertise?
> >
> > Thanks in advance!
> >
> > Stefan
> >
> >
> > On Sun, Feb 10, 2008 at 01:33:04PM +0100, Stefan Manegold wrote:
> > > Peter: Thanks for disabling mem_bigsize / vmalloc().
> > >
> > > All:
> > > (a)
> > > Unfortunately, this does not (yet) solve
> > > [ 1872685 ] PF(tijah): HEAPsetmmap: Assertion `p' failed.
> > > http://sourceforge.net/tracker/index.php?func=detail&aid=1872685&group_id=56967&atid=482468
> > >
> > > But this appears to be more vm_minsize related than mem_bigsize related.
> > >
> > >
> > > (b)
> > > We should check all occurances of mem_bigsize and remove / "hide" those that
> > > have become irrelevant with the deactivation of mem_bigsize / vmalloc():
> > > ========
> > > $ cvsfiles * | egrep -v ' |/Tests/' | xargs grep -n --color 'mem_bigsize'
> > > --------
> > > MonetDB4/conf/MonetDB.conf.in:29:# gdk_mem_bigsize & gdk_vm_minsize will be
> > set/limited to
> > > MonetDB4/conf/MonetDB.conf.in:32:# memory chunks of size >= gdk_mem_bigsize
> > (in bytes) will be mmaped anonymously
> > > MonetDB4/conf/MonetDB.conf.in:33:#gdk_mem_bigsize=262144
> > > MonetDB5/conf/monetdb5.conf.in:33:# gdk_mem_bigsize & gdk_vm_minsize will be
> > set/limited to
> > > MonetDB5/conf/monetdb5.conf.in:36:# memory chunks of size >= gdk_mem_bigsize
> > (in bytes) will be mmaped anonymously
> > > MonetDB5/conf/monetdb5.conf.in:37:#gdk_mem_bigsize=262144
> > > MonetDB/src/common/monet_options.mx:410:        set[i].name =
> > strdup("gdk_mem_bigsize");
> > > MonetDB/src/common/monet_options.py.in:39:    # gdk_mem_bigsize &
> > gdk_vm_minsize will be set/limited to
> > > MonetDB/src/common/monet_options.py.in:42:    gdk_mem_bigsize = '256K'
> > > MonetDB/src/gdk/gdk_heap.mx:121:        if (h->size > GDK_mem_bigsize) {
> > > MonetDB/src/gdk/gdk_heap.mx:189:                int can_mmap = (h->filename
> > && size >= GDK_mem_bigsize);
> > > MonetDB/src/gdk/gdk.mx:2210:gdk_export size_t GDK_mem_bigsize;  /* size
> > after which we use VM rather than heap */
> > > MonetDB/src/gdk/gdk_utils.mx:396:size_t GDK_mem_bigsize = 1 << 20;
> > > MonetDB/src/gdk/gdk_utils.mx:964:larger than GDK_mem_bigsize) to anonymous
> > virtual memory. The seamless
> > > MonetDB/src/gdk/gdk_utils.mx:1054:      if (size > GDK_mem_bigsize) {
> > > MonetDB/src/gdk/gdk_utils.mx:1191:      if (size <= GDK_mem_bigsize) {
> > > MonetDB/src/gdk/gdk_utils.mx:1529:      if ((p =
> > GDKgetenv("gdk_mem_bigsize"))) {
> > > MonetDB/src/gdk/gdk_utils.mx:1531:              lng max_mem_bigsize =
> > GDK_mem_maxsize/16;
> > > MonetDB/src/gdk/gdk_utils.mx:1534:              GDK_mem_bigsize = (size_t)
> > MIN(max_mem_bigsize, strtol(p, NULL, 10));
> > > monetweb/Docs/XQuery/MonetDB.conf.texi:115:@item @code{gdk_mem_bigsize}:
> > minimum size for memory-mapped columns, e.g. @code{262144}.
> > > MonetDB4/src/modules/plain/sys.mx:96:.COMMAND mem_bigsize() : lng =
> > get_mem_bigsize;
> > > MonetDB4/src/modules/plain/sys.mx:98:.COMMAND mem_bigsize(lng) : void =
> > set_mem_bigsize;
> > > MonetDB4/src/modules/plain/sys.mx:1430:get_mem_bigsize(lng *num)
> > > MonetDB4/src/modules/plain/sys.mx:1432: *num = GDK_mem_bigsize;
> > > MonetDB4/src/modules/plain/sys.mx:1437:set_mem_bigsize(lng *num)
> > > MonetDB4/src/modules/plain/sys.mx:1439: @:num2sze(mem_bigsize)@
> > > MonetDB4/src/modules/plain/sys.mx:1440: GDK_mem_bigsize = MAX(32768, sze);
> > > MonetDB4/src/modules/plain/sys.mx:1462: if (sze < GDK_mem_bigsize)
> > > MonetDB4/src/modules/plain/sys.mx:1463:         set_mem_bigsize(num);
> > > MonetDB4/src/modules/plain/sys.mx:1464: GDK_mem_maxsize =
> > MAX(GDK_mem_bigsize, sze);
> > > MonetDB4/src/modules/plain/sys.mx:1486: if (sze < GDK_mem_bigsize)
> > > MonetDB4/src/modules/plain/sys.mx:1487:         set_mem_bigsize(num);
> > > MonetDB4/src/modules/plain/sys.mx:1488: GDK_vm_minsize =
> > MAX(GDK_mem_bigsize, sze);
> > > MonetDB5/src/modules/kernel/status.mx:405:get_mem_bigsize(lng *num)
> > > MonetDB5/src/modules/kernel/status.mx:407:      *num = GDK_mem_bigsize;
> > > MonetDB5/src/modules/kernel/status.mx:412:set_mem_bigsize(lng *num)
> > > MonetDB5/src/modules/kernel/status.mx:414:      @:num2sze(mem_bigsize)@
> > > MonetDB5/src/modules/kernel/status.mx:415:      GDK_mem_bigsize = MAX(32768,
> > sze);
> > > MonetDB5/src/modules/kernel/status.mx:439:      if (sze < GDK_mem_bigsize)
> > > MonetDB5/src/modules/kernel/status.mx:440:              set_mem_bigsize(num);
> > > MonetDB5/src/modules/kernel/status.mx:441:      GDK_mem_maxsize =
> > MAX(GDK_mem_bigsize, sze);
> > > MonetDB5/src/modules/kernel/status.mx:466:      if (sze < GDK_mem_bigsize)
> > > MonetDB5/src/modules/kernel/status.mx:467:              set_mem_bigsize(num);
> > > MonetDB5/src/modules/kernel/status.mx:468:      GDK_vm_minsize =
> > MAX(GDK_mem_bigsize, sze);
> > > monetweb/MonetDB/QuickTour/HelloWorld/content.shtml:42:[ "gdk_mem_bigsize",
> >     "262144"
> > > ========
> > >
> > > more tomorrow during MADAM ...
> > >
> > > Stefan
> > >
> > >
> > > On Sun, Feb 10, 2008 at 10:37:54AM +0100, Peter Boncz wrote:
> > > > no consequences.
> > > >
> > > >
> > > >
> > > >
> > > > Peter,
> > > >
> > > > What about the consequences in:
> > > > build/MonetDB/src/gdk/gdk_heap.c
> > > > build/MonetDB5/src/modules/kernel/status.c
> > > > build/MonetDB/src/gdk/gdk.h
> > > >
> > > > Peter Boncz wrote:
> > > > > Update of /cvsroot/monetdb/MonetDB/src/gdk
> > > > > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv29651
> > > > >
> > > > > Modified Files:
> > > > >       Tag: MonetDB_1-22
> > > > > 	gdk_utils.mx
> > > > > Log Message:
> > > > > disable use of vmalloc()
> > > > >
> > > > >
> > > > >
> > > > > Index: gdk_utils.mx
> > > > > ===================================================================
> > > > > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_utils.mx,v
> > > > > retrieving revision 1.206
> > > > > retrieving revision 1.206.2.1
> > > > > diff -u -d -r1.206 -r1.206.2.1
> > > > > --- gdk_utils.mx	14 Jan 2008 10:48:58 -0000	1.206
> > > > > +++ gdk_utils.mx	9 Feb 2008 15:38:11 -0000	1.206.2.1
> > > > > @@ -1050,6 +1050,7 @@
> > > > >  #endif
> > > > >  	}
> > > > >  	size = (size + 7) & ~7;	/* round up to a multiple of eight */
> > > > > +#if 0
> > > > >  	if (size > GDK_mem_bigsize) {
> > > > >  		size_t newsize = size + sizeof(size_t) + sizeof(size_t);
> > > > >  		size_t newmax = MAX(*maxsize, newsize);
> > > > > @@ -1064,6 +1065,7 @@
> > > > >  		*maxsize = newmax - (sizeof(size_t) + sizeof(size_t));
> > > > >  		return (void *) s;
> > > > >  	}
> > > > > +#endif
> > > > >  	CHKMEM(size, 0);
> > > > >  	GDKmalloc_prefixsize(s, size);
> > > > >  	if (s == NULL) {
> > > > >
> > > > >
> > > > > -------------------------------------------------------------------------
> > > > > This SF.net email is sponsored by: Microsoft
> > > > > Defy all challenges. Microsoft(R) Visual Studio 2008.
> > > > > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> > > > > _______________________________________________
> > > > > Monetdb-checkins mailing list
> > > > > Monetdb-checkins at lists.sourceforge.net
> > > > > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
> > > > >
> > > >
> > > >
> > > > -------------------------------------------------------------------------
> > > > This SF.net email is sponsored by: Microsoft
> > > > Defy all challenges. Microsoft(R) Visual Studio 2008.
> > > > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> > > > _______________________________________________
> > > > Monetdb-developers mailing list
> > > > Monetdb-developers at lists.sourceforge.net
> > > > https://lists.sourceforge.net/lists/listinfo/monetdb-developers
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > > --
> > > | Dr. Stefan Manegold | mailto:Stefan.Manegold at cwi.nl |
> > > | CWI,  P.O.Box 94079 | http://www.cwi.nl/~manegold/  |
> > > | 1090 GB Amsterdam   | Tel.: +31 (20) 592-4212       |
> > > | The Netherlands     | Fax : +31 (20) 592-4312       |
> >
> > --
> > | Dr. Stefan Manegold | mailto:Stefan.Manegold at cwi.nl |
> > | CWI,  P.O.Box 94079 | http://www.cwi.nl/~manegold/  |
> > | 1090 GB Amsterdam   | Tel.: +31 (20) 592-4212       |
> > | The Netherlands     | Fax : +31 (20) 592-4312       |
> >
> >
> 
> -- 
> | Dr. Stefan Manegold | mailto:Stefan.Manegold at cwi.nl |
> | CWI,  P.O.Box 94079 | http://www.cwi.nl/~manegold/  |
> | 1090 GB Amsterdam   | Tel.: +31 (20) 592-4212       |
> | The Netherlands     | Fax : +31 (20) 592-4312       |
> 
> 

-- 
| Dr. Stefan Manegold | mailto:Stefan.Manegold at cwi.nl |
| CWI,  P.O.Box 94079 | http://www.cwi.nl/~manegold/  |
| 1090 GB Amsterdam   | Tel.: +31 (20) 592-4212       |
| The Netherlands     | Fax : +31 (20) 592-4312       |




More information about the developers-list mailing list