[Monetdb-developers] [Monetdb-checkins]MonetDB/src/gdkgdk_utils.mx, MonetDB_1-22, 1.206, 1.206.2.1

Stefan Manegold Stefan.Manegold at cwi.nl
Mon Feb 11 20:03:37 CET 2008


On Mon, Feb 11, 2008 at 07:19:09PM +0100, Sjoerd Mullender wrote:
> On 2008-02-11 19:05, Peter Boncz wrote:
>> Hi Stefan
>>
>> Thanks for your response. However, as I have said, but let me say it as clear
>> as possble: I am *sure* there is a bug still unfound in the way re-allocations
>> are done when mem_bigsize redirects it into virtual memory.
>>
>> For this reason, at this stage, with the goal of a stable release, it is just
>> my recommendation *not* to re-enable it.
>>
>> In M5 nor in M5.
>
> I think Stefan meant reenable access to the variable in M5 that Martin 
> turned off, not reenable the GDKvmalloc switch that you turned off.

right. exactly that. no more, no less ;-)

Stefan

>> Peter
>>
>>
>>
>>
>> Dear all,
>>
>> On Mon, Feb 11, 2008 at 02:54:59PM +0100, Peter Boncz wrote:
>>> Hi Stefan,
>>>
>>> The pftijah bug was unrelated to vmalloc(), a relationship that I never
>>> assumed.
>>
>> Indeed, that was (only) my mis-interpretation. Sorry.
>>
>>> The vmalloc() troubles occured, as you may recall, during the data
>>> preparation of the sorted 100GB TPC-H.
>>>
>>> As it is entirely foreseeable that we at some point again will re-encounter an
>>> OS with a malloc implementation that is prone to fragmentation, I'd keep the
>>> vmalloc() code in for the moment (though in disabled state). Linux and Windows
>>> do not need it.
>>>
>>> So, my advise is to for the monet let vmalloc() be disabled, until the issue
>>> is properly investigated and a bugfix is found. Let's call it option (0).
>>
>> Ok.
>>
>> Just to sum up the current state.
>> Please check, whether this is OK and consistent:
>>
>>
>> a)
>> Documentation in MonetDB4/conf/MonetDB.conf.in &
>> MonetDB5/conf/monetdb5.conf.in says:
>> "
>> # gdk_mem_bigsize & gdk_vm_minsize will be set/limited to
>> # 1/2 of the physically available amount of main-memory
>> # during start-up in src/tools/Mserver.mx
>> # memory chunks of size >= gdk_mem_bigsize (in bytes) will be mmaped anonymously
>> #gdk_mem_bigsize=262144
>> # memory chunks of size >= gdk_vm_minsize (in bytes) will be mmaped;
>> #gdk_vm_minsize=137438953472
>> "
>>
>> b)
>> Documentation in ./monetweb/Docs/XQuery/MonetDB.conf.texi says
>> "
>> @item @code{gdk_mem_bigsize}: minimum size for memory-mapped columns, e.g.
>> @code{262144}.
>> @item @code{gdk_vm_minsize}: column size above which memory mapped files are
>> used always, e.g. @code{1749291171}
>> "
>>
>> c)
>> MonetDB/src/common/monet_options.mx sets
>> "
>>         set[i].kind = opt_builtin;
>>         set[i].name = strdup("gdk_mem_bigsize");
>>         set[i].value = strdup("262144");
>> "
>>
>> d)
>> MonetDB/src/common/monet_options.py.in sets
>> "
>>     # gdk_mem_bigsize & gdk_vm_minsize will be set/limited to
>>     # 1/2 of the physically available amount of main-memory
>>     # during start-up in src/tools/Mserver.mx
>>     gdk_mem_bigsize = '256K'
>>     gdk_vm_minsize = '128G'
>> "
>>
>> e)
>> GDKinit() in MonetDB/src/gdk/gdk_utils.mx sets
>> "
>>         if ((p = GDKgetenv("gdk_mem_bigsize"))) {
>>                 /* when allocating >6% of all RAM; do so using vmalloc() iso
>> malloc() */
>>                 lng max_mem_bigsize = GDK_mem_maxsize/16;
>>
>>                 /* sanity check to avoid memory fragmentation */
>>                 GDK_mem_bigsize = (size_t) MIN(max_mem_bigsize, strtol(p,
>> NULL, 10));
>>         }
>> "
>>
>> f)
>> GDKmallocmax() in MonetDB/src/gdk/gdk_utils.mx ignores GDK_mem_bigsize
>>
>> g)
>> HEAPalloc() in MonetDB/src/gdk/gdk_heap.mx uses GDK_mem_bigsize as follows:
>> "
>>         /* when using anonymous vm we malloc we need 64K chunks, also we
>>          * 20% extra malloc */
>>         if (h->size > GDK_mem_bigsize) {
>>                 h->maxsize = (size_t) ((double) h->maxsize * BATMARGIN) - 1;
>>                 h->maxsize = (1 + (h->maxsize >> 16)) << 16;
>>         }
>> "
>>
>> h)
>> HEAPextend() in MonetDB/src/gdk/gdk_heap.mx uses GDK_mem_bigsize as follows:
>> "
>>                 /* extend a malloced heap, possibly switching over to
>> file-mapped storage */
>>                 Heap bak = *h;
>>                 int can_mmap = (h->filename && size >= GDK_mem_bigsize);
>>                 int must_mmap = can_mmap && (size >= GDK_vm_minsize ||
>> (h->newstorage != STORE_MEM));
>> [...]
>>                 if (can_mmap) {
>>                         /* in anonymous vm, if have to realloc anyway, we
>> reserve some extra space */
>>                         if (size > h->maxsize) {
>>                                 h->maxsize = (size_t) ((double) size *
>> BATMARGIN);
>>                         }
>>                         /* when using anonymous vm we malloc we need 64K
>> chunks */
>>                         h->maxsize = (1 + ((h->maxsize - 1) >> 16)) << 16;
>> [...]
>>                 /* too big: convert it to a disk-based temporary heap */
>>                 if (can_mmap) {
>>                         char privext[PATHLENGTH], *of = h->filename;
>>                         FILE *fp;
>>
>>                         h->filename = NULL;
>>                         sprintf(privext, "%s.priv", ext);
>>                         fp = GDKfilelocate(nme, "wb", privext);
>>                         if (fp != NULL) {
>>                                 fclose(fp);
>>                                 /* a non-persistent heap: we create a .priv
>> but *not* MMAP_PRIV !!! */
>>                                 h->storage = STORE_MMAP;
>>                                 h->base = NULL;
>>                                 if (HEAPload(h, nme, privext, FALSE) >= 0) {
>>                                         memcpy(h->base, bak.base, bak.free);
>>                                         HEAPfree(&bak);
>>                                         return 0;
>>                                 }
>>                         }
>>                         GDKfree(of);
>>                 }
>> "
>>
>> i)
>> MIL offers mem_bigsize() & mem_bigsize(lng) to get / set mem_bigsize
>> in MonetDB4/src/modules/plain/sys.mx
>>
>> j)
>> both MIL & MAL offer commands to set mem_maxsize and vm_minsize, however,
>> only the MIL variants (still) related mem_maxsize & vm_minsize with mem_bigsize,
>> while the MAL variants no longer do so:
>>
>> MonetDB4/src/modules/plain/sys.mx:
>> "
>> int
>> set_mem_maxsize(lng *num)
>> {
>>         @:num2sze(mem_maxsize)@
>>         if (sze < GDK_mem_bigsize)
>>                 set_mem_bigsize(num);
>>         GDK_mem_maxsize = MAX(GDK_mem_bigsize, sze);
>>         return GDK_SUCCEED;
>> }
>>
>> int
>> set_vm_minsize(lng *num)
>> {
>>         @:num2sze(vm_minsize)@
>>         if (sze < GDK_mem_bigsize)
>>                 set_mem_bigsize(num);
>>         GDK_vm_minsize = MAX(GDK_mem_bigsize, sze);
>>         return GDK_SUCCEED;
>> }
>> "
>>
>> MonetDB5/src/modules/kernel/status.mx
>> "
>> int
>> set_mem_maxsize(lng *num)
>> {
>>         @:num2sze(mem_maxsize)@
>> #if 0
>>         if (sze < GDK_mem_bigsize)
>>                 set_mem_bigsize(num);
>>         GDK_mem_maxsize = MAX(GDK_mem_bigsize, sze);
>> #endif
>>         GDK_mem_maxsize = sze;
>>         return GDK_SUCCEED;
>> }
>>
>> int
>> set_vm_minsize(lng *num)
>> {
>>         @:num2sze(vm_minsize)@
>> #if 0
>>         if (sze < GDK_mem_bigsize)
>>                 set_mem_bigsize(num);
>>         GDK_vm_minsize = MAX(GDK_mem_bigsize, sze);
>> #endif
>>         GDK_vm_minsize = sze;
>>         return GDK_SUCCEED;
>> }
>> "
>>
>>
>> If all this is considered OK and consistent, we're done.
>> Otherwise, we need to do what ever is required to get the
>> code and documentation consistent, preferably before the release.
>>
>> Stefan
>>
>>
>>> Peter
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Dear all,
>>>
>>> since Roberto's bug appears to be fixed by Peter's ".priv" (vm_minsize
>>> related) fixes (Thanks!), and disabling of vmalloc() (mem_bigsize related)
>>> does not appear have any impact on this bug (just tested successfully both
>>> with vmalloc() disabled and vmalloc() enabled), there are two ways to finish
>>> this release (more or less) quickly:
>>>
>>> (1) finish the disabling of vmalloc() in all parts of the code as indicated
>>>     below (or better: remove all remains completely) --- this might need
>>>     some extra testing to ensure that it does not have any (significant)
>>>     impact on performance ...
>>>
>>> (2) for now (i.e., in the release branch) re-enable vmalloc(), and then
>>>     finish the removal of it (incl. extensive testing) in the development
>>>     trunk for th next release.
>>>
>>> Since I'm not completely aware of the impact of vmalloc(), I cannot really
>>> make this decision --- the "save" back-up would IMHO be (2).
>>>
>>> Any other comments, opinions, suggestions, expertise?
>>>
>>> Thanks in advance!
>>>
>>> Stefan
>>>
>>>
>>> On Sun, Feb 10, 2008 at 01:33:04PM +0100, Stefan Manegold wrote:
>>>> Peter: Thanks for disabling mem_bigsize / vmalloc().
>>>>
>>>> All:
>>>> (a)
>>>> Unfortunately, this does not (yet) solve
>>>> [ 1872685 ] PF(tijah): HEAPsetmmap: Assertion `p' failed.
>>>> http://sourceforge.net/tracker/index.php?func=detail&aid=1872685&group_id=56967&atid=482468
>>>>
>>>> But this appears to be more vm_minsize related than mem_bigsize related.
>>>>
>>>>
>>>> (b)
>>>> We should check all occurances of mem_bigsize and remove / "hide" those that
>>>> have become irrelevant with the deactivation of mem_bigsize / vmalloc():
>>>> ========
>>>> $ cvsfiles * | egrep -v ' |/Tests/' | xargs grep -n --color 'mem_bigsize'
>>>> --------
>>>> MonetDB4/conf/MonetDB.conf.in:29:# gdk_mem_bigsize & gdk_vm_minsize will be
>>> set/limited to
>>>> MonetDB4/conf/MonetDB.conf.in:32:# memory chunks of size >= gdk_mem_bigsize
>>> (in bytes) will be mmaped anonymously
>>>> MonetDB4/conf/MonetDB.conf.in:33:#gdk_mem_bigsize=262144
>>>> MonetDB5/conf/monetdb5.conf.in:33:# gdk_mem_bigsize & gdk_vm_minsize will be
>>> set/limited to
>>>> MonetDB5/conf/monetdb5.conf.in:36:# memory chunks of size >= gdk_mem_bigsize
>>> (in bytes) will be mmaped anonymously
>>>> MonetDB5/conf/monetdb5.conf.in:37:#gdk_mem_bigsize=262144
>>>> MonetDB/src/common/monet_options.mx:410:        set[i].name =
>>> strdup("gdk_mem_bigsize");
>>>> MonetDB/src/common/monet_options.py.in:39:    # gdk_mem_bigsize &
>>> gdk_vm_minsize will be set/limited to
>>>> MonetDB/src/common/monet_options.py.in:42:    gdk_mem_bigsize = '256K'
>>>> MonetDB/src/gdk/gdk_heap.mx:121:        if (h->size > GDK_mem_bigsize) {
>>>> MonetDB/src/gdk/gdk_heap.mx:189:                int can_mmap = (h->filename
>>> && size >= GDK_mem_bigsize);
>>>> MonetDB/src/gdk/gdk.mx:2210:gdk_export size_t GDK_mem_bigsize;  /* size
>>> after which we use VM rather than heap */
>>>> MonetDB/src/gdk/gdk_utils.mx:396:size_t GDK_mem_bigsize = 1 << 20;
>>>> MonetDB/src/gdk/gdk_utils.mx:964:larger than GDK_mem_bigsize) to anonymous
>>> virtual memory. The seamless
>>>> MonetDB/src/gdk/gdk_utils.mx:1054:      if (size > GDK_mem_bigsize) {
>>>> MonetDB/src/gdk/gdk_utils.mx:1191:      if (size <= GDK_mem_bigsize) {
>>>> MonetDB/src/gdk/gdk_utils.mx:1529:      if ((p =
>>> GDKgetenv("gdk_mem_bigsize"))) {
>>>> MonetDB/src/gdk/gdk_utils.mx:1531:              lng max_mem_bigsize =
>>> GDK_mem_maxsize/16;
>>>> MonetDB/src/gdk/gdk_utils.mx:1534:              GDK_mem_bigsize = (size_t)
>>> MIN(max_mem_bigsize, strtol(p, NULL, 10));
>>>> monetweb/Docs/XQuery/MonetDB.conf.texi:115:@item @code{gdk_mem_bigsize}:
>>> minimum size for memory-mapped columns, e.g. @code{262144}.
>>>> MonetDB4/src/modules/plain/sys.mx:96:.COMMAND mem_bigsize() : lng =
>>> get_mem_bigsize;
>>>> MonetDB4/src/modules/plain/sys.mx:98:.COMMAND mem_bigsize(lng) : void =
>>> set_mem_bigsize;
>>>> MonetDB4/src/modules/plain/sys.mx:1430:get_mem_bigsize(lng *num)
>>>> MonetDB4/src/modules/plain/sys.mx:1432: *num = GDK_mem_bigsize;
>>>> MonetDB4/src/modules/plain/sys.mx:1437:set_mem_bigsize(lng *num)
>>>> MonetDB4/src/modules/plain/sys.mx:1439: @:num2sze(mem_bigsize)@
>>>> MonetDB4/src/modules/plain/sys.mx:1440: GDK_mem_bigsize = MAX(32768, sze);
>>>> MonetDB4/src/modules/plain/sys.mx:1462: if (sze < GDK_mem_bigsize)
>>>> MonetDB4/src/modules/plain/sys.mx:1463:         set_mem_bigsize(num);
>>>> MonetDB4/src/modules/plain/sys.mx:1464: GDK_mem_maxsize =
>>> MAX(GDK_mem_bigsize, sze);
>>>> MonetDB4/src/modules/plain/sys.mx:1486: if (sze < GDK_mem_bigsize)
>>>> MonetDB4/src/modules/plain/sys.mx:1487:         set_mem_bigsize(num);
>>>> MonetDB4/src/modules/plain/sys.mx:1488: GDK_vm_minsize =
>>> MAX(GDK_mem_bigsize, sze);
>>>> MonetDB5/src/modules/kernel/status.mx:405:get_mem_bigsize(lng *num)
>>>> MonetDB5/src/modules/kernel/status.mx:407:      *num = GDK_mem_bigsize;
>>>> MonetDB5/src/modules/kernel/status.mx:412:set_mem_bigsize(lng *num)
>>>> MonetDB5/src/modules/kernel/status.mx:414:      @:num2sze(mem_bigsize)@
>>>> MonetDB5/src/modules/kernel/status.mx:415:      GDK_mem_bigsize = MAX(32768,
>>> sze);
>>>> MonetDB5/src/modules/kernel/status.mx:439:      if (sze < GDK_mem_bigsize)
>>>> MonetDB5/src/modules/kernel/status.mx:440:              set_mem_bigsize(num);
>>>> MonetDB5/src/modules/kernel/status.mx:441:      GDK_mem_maxsize =
>>> MAX(GDK_mem_bigsize, sze);
>>>> MonetDB5/src/modules/kernel/status.mx:466:      if (sze < GDK_mem_bigsize)
>>>> MonetDB5/src/modules/kernel/status.mx:467:              set_mem_bigsize(num);
>>>> MonetDB5/src/modules/kernel/status.mx:468:      GDK_vm_minsize =
>>> MAX(GDK_mem_bigsize, sze);
>>>> monetweb/MonetDB/QuickTour/HelloWorld/content.shtml:42:[ "gdk_mem_bigsize",
>>>     "262144"
>>>> ========
>>>>
>>>> more tomorrow during MADAM ...
>>>>
>>>> Stefan
>>>>
>>>>
>>>> On Sun, Feb 10, 2008 at 10:37:54AM +0100, Peter Boncz wrote:
>>>>> no consequences.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Peter,
>>>>>
>>>>> What about the consequences in:
>>>>> build/MonetDB/src/gdk/gdk_heap.c
>>>>> build/MonetDB5/src/modules/kernel/status.c
>>>>> build/MonetDB/src/gdk/gdk.h
>>>>>
>>>>> Peter Boncz wrote:
>>>>>> Update of /cvsroot/monetdb/MonetDB/src/gdk
>>>>>> In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv29651
>>>>>>
>>>>>> Modified Files:
>>>>>>       Tag: MonetDB_1-22
>>>>>> 	gdk_utils.mx
>>>>>> Log Message:
>>>>>> disable use of vmalloc()
>>>>>>
>>>>>>
>>>>>>
>>>>>> Index: gdk_utils.mx
>>>>>> ===================================================================
>>>>>> RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_utils.mx,v
>>>>>> retrieving revision 1.206
>>>>>> retrieving revision 1.206.2.1
>>>>>> diff -u -d -r1.206 -r1.206.2.1
>>>>>> --- gdk_utils.mx	14 Jan 2008 10:48:58 -0000	1.206
>>>>>> +++ gdk_utils.mx	9 Feb 2008 15:38:11 -0000	1.206.2.1
>>>>>> @@ -1050,6 +1050,7 @@
>>>>>>  #endif
>>>>>>  	}
>>>>>>  	size = (size + 7) & ~7;	/* round up to a multiple of eight */
>>>>>> +#if 0
>>>>>>  	if (size > GDK_mem_bigsize) {
>>>>>>  		size_t newsize = size + sizeof(size_t) + sizeof(size_t);
>>>>>>  		size_t newmax = MAX(*maxsize, newsize);
>>>>>> @@ -1064,6 +1065,7 @@
>>>>>>  		*maxsize = newmax - (sizeof(size_t) + sizeof(size_t));
>>>>>>  		return (void *) s;
>>>>>>  	}
>>>>>> +#endif
>>>>>>  	CHKMEM(size, 0);
>>>>>>  	GDKmalloc_prefixsize(s, size);
>>>>>>  	if (s == NULL) {
>>>>>>
>>>>>>
>>>>>> -------------------------------------------------------------------------
>>>>>> This SF.net email is sponsored by: Microsoft
>>>>>> Defy all challenges. Microsoft(R) Visual Studio 2008.
>>>>>> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
>>>>>> _______________________________________________
>>>>>> Monetdb-checkins mailing list
>>>>>> Monetdb-checkins at lists.sourceforge.net
>>>>>> https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
>>>>>>
>>>>>
>>>>> -------------------------------------------------------------------------
>>>>> This SF.net email is sponsored by: Microsoft
>>>>> Defy all challenges. Microsoft(R) Visual Studio 2008.
>>>>> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
>>>>> _______________________________________________
>>>>> Monetdb-developers mailing list
>>>>> Monetdb-developers at lists.sourceforge.net
>>>>> https://lists.sourceforge.net/lists/listinfo/monetdb-developers
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>> --
>>>> | Dr. Stefan Manegold | mailto:Stefan.Manegold at cwi.nl |
>>>> | CWI,  P.O.Box 94079 | http://www.cwi.nl/~manegold/  |
>>>> | 1090 GB Amsterdam   | Tel.: +31 (20) 592-4212       |
>>>> | The Netherlands     | Fax : +31 (20) 592-4312       |
>>> --
>>> | Dr. Stefan Manegold | mailto:Stefan.Manegold at cwi.nl |
>>> | CWI,  P.O.Box 94079 | http://www.cwi.nl/~manegold/  |
>>> | 1090 GB Amsterdam   | Tel.: +31 (20) 592-4212       |
>>> | The Netherlands     | Fax : +31 (20) 592-4312       |
>>>
>>>
>>
>
>
> -- 
> Sjoerd Mullender
>
>

-- 
| Dr. Stefan Manegold | mailto:Stefan.Manegold at cwi.nl |
| CWI,  P.O.Box 94079 | http://www.cwi.nl/~manegold/  |
| 1090 GB Amsterdam   | Tel.: +31 (20) 592-4212       |
| The Netherlands     | Fax : +31 (20) 592-4312       |




More information about the developers-list mailing list