[Monetdb-developers] [Monetdb-checkins] MonetDB/src/gdk gdk_utils.mx, MonetDB_1-22, 1.206, 1.206.2.1

Stefan Manegold Stefan.Manegold at cwi.nl
Mon Feb 11 14:46:02 CET 2008


Dear all,

since Roberto's bug appears to be fixed by Peter's ".priv" (vm_minsize
related) fixes (Thanks!), and disabling of vmalloc() (mem_bigsize related)
does not appear have any impact on this bug (just tested successfully both
with vmalloc() disabled and vmalloc() enabled), there are two ways to finish
this release (more or less) quickly:

(1) finish the disabling of vmalloc() in all parts of the code as indicated
    below (or better: remove all remains completely) --- this might need
    some extra testing to ensure that it does not have any (significant)
    impact on performance ...

(2) for now (i.e., in the release branch) re-enable vmalloc(), and then
    finish the removal of it (incl. extensive testing) in the development
    trunk for th next release.

Since I'm not completely aware of the impact of vmalloc(), I cannot really
make this decision --- the "save" back-up would IMHO be (2).

Any other comments, opinions, suggestions, expertise?

Thanks in advance!

Stefan


On Sun, Feb 10, 2008 at 01:33:04PM +0100, Stefan Manegold wrote:
> Peter: Thanks for disabling mem_bigsize / vmalloc().
> 
> All:
> (a) 
> Unfortunately, this does not (yet) solve 
> [ 1872685 ] PF(tijah): HEAPsetmmap: Assertion `p' failed.
> http://sourceforge.net/tracker/index.php?func=detail&aid=1872685&group_id=56967&atid=482468
> 
> But this appears to be more vm_minsize related than mem_bigsize related.
> 
> 
> (b)
> We should check all occurances of mem_bigsize and remove / "hide" those that
> have become irrelevant with the deactivation of mem_bigsize / vmalloc():
> ========
> $ cvsfiles * | egrep -v ' |/Tests/' | xargs grep -n --color 'mem_bigsize'
> --------
> MonetDB4/conf/MonetDB.conf.in:29:# gdk_mem_bigsize & gdk_vm_minsize will be set/limited to 
> MonetDB4/conf/MonetDB.conf.in:32:# memory chunks of size >= gdk_mem_bigsize (in bytes) will be mmaped anonymously
> MonetDB4/conf/MonetDB.conf.in:33:#gdk_mem_bigsize=262144
> MonetDB5/conf/monetdb5.conf.in:33:# gdk_mem_bigsize & gdk_vm_minsize will be set/limited to
> MonetDB5/conf/monetdb5.conf.in:36:# memory chunks of size >= gdk_mem_bigsize (in bytes) will be mmaped anonymously
> MonetDB5/conf/monetdb5.conf.in:37:#gdk_mem_bigsize=262144
> MonetDB/src/common/monet_options.mx:410:        set[i].name = strdup("gdk_mem_bigsize");
> MonetDB/src/common/monet_options.py.in:39:    # gdk_mem_bigsize & gdk_vm_minsize will be set/limited to
> MonetDB/src/common/monet_options.py.in:42:    gdk_mem_bigsize = '256K'
> MonetDB/src/gdk/gdk_heap.mx:121:        if (h->size > GDK_mem_bigsize) {
> MonetDB/src/gdk/gdk_heap.mx:189:                int can_mmap = (h->filename && size >= GDK_mem_bigsize);
> MonetDB/src/gdk/gdk.mx:2210:gdk_export size_t GDK_mem_bigsize;  /* size after which we use VM rather than heap */
> MonetDB/src/gdk/gdk_utils.mx:396:size_t GDK_mem_bigsize = 1 << 20;
> MonetDB/src/gdk/gdk_utils.mx:964:larger than GDK_mem_bigsize) to anonymous virtual memory. The seamless
> MonetDB/src/gdk/gdk_utils.mx:1054:      if (size > GDK_mem_bigsize) {
> MonetDB/src/gdk/gdk_utils.mx:1191:      if (size <= GDK_mem_bigsize) {
> MonetDB/src/gdk/gdk_utils.mx:1529:      if ((p = GDKgetenv("gdk_mem_bigsize"))) {
> MonetDB/src/gdk/gdk_utils.mx:1531:              lng max_mem_bigsize = GDK_mem_maxsize/16;
> MonetDB/src/gdk/gdk_utils.mx:1534:              GDK_mem_bigsize = (size_t) MIN(max_mem_bigsize, strtol(p, NULL, 10));
> monetweb/Docs/XQuery/MonetDB.conf.texi:115:@item @code{gdk_mem_bigsize}: minimum size for memory-mapped columns, e.g. @code{262144}.
> MonetDB4/src/modules/plain/sys.mx:96:.COMMAND mem_bigsize() : lng = get_mem_bigsize;
> MonetDB4/src/modules/plain/sys.mx:98:.COMMAND mem_bigsize(lng) : void = set_mem_bigsize;
> MonetDB4/src/modules/plain/sys.mx:1430:get_mem_bigsize(lng *num)
> MonetDB4/src/modules/plain/sys.mx:1432: *num = GDK_mem_bigsize;
> MonetDB4/src/modules/plain/sys.mx:1437:set_mem_bigsize(lng *num)
> MonetDB4/src/modules/plain/sys.mx:1439: @:num2sze(mem_bigsize)@
> MonetDB4/src/modules/plain/sys.mx:1440: GDK_mem_bigsize = MAX(32768, sze);
> MonetDB4/src/modules/plain/sys.mx:1462: if (sze < GDK_mem_bigsize)
> MonetDB4/src/modules/plain/sys.mx:1463:         set_mem_bigsize(num);
> MonetDB4/src/modules/plain/sys.mx:1464: GDK_mem_maxsize = MAX(GDK_mem_bigsize, sze);
> MonetDB4/src/modules/plain/sys.mx:1486: if (sze < GDK_mem_bigsize)
> MonetDB4/src/modules/plain/sys.mx:1487:         set_mem_bigsize(num);
> MonetDB4/src/modules/plain/sys.mx:1488: GDK_vm_minsize = MAX(GDK_mem_bigsize, sze);
> MonetDB5/src/modules/kernel/status.mx:405:get_mem_bigsize(lng *num)
> MonetDB5/src/modules/kernel/status.mx:407:      *num = GDK_mem_bigsize;
> MonetDB5/src/modules/kernel/status.mx:412:set_mem_bigsize(lng *num)
> MonetDB5/src/modules/kernel/status.mx:414:      @:num2sze(mem_bigsize)@
> MonetDB5/src/modules/kernel/status.mx:415:      GDK_mem_bigsize = MAX(32768, sze);
> MonetDB5/src/modules/kernel/status.mx:439:      if (sze < GDK_mem_bigsize)
> MonetDB5/src/modules/kernel/status.mx:440:              set_mem_bigsize(num);
> MonetDB5/src/modules/kernel/status.mx:441:      GDK_mem_maxsize = MAX(GDK_mem_bigsize, sze);
> MonetDB5/src/modules/kernel/status.mx:466:      if (sze < GDK_mem_bigsize)
> MonetDB5/src/modules/kernel/status.mx:467:              set_mem_bigsize(num);
> MonetDB5/src/modules/kernel/status.mx:468:      GDK_vm_minsize = MAX(GDK_mem_bigsize, sze);
> monetweb/MonetDB/QuickTour/HelloWorld/content.shtml:42:[ "gdk_mem_bigsize",      "262144"
> ========
> 
> more tomorrow during MADAM ...
> 
> Stefan
> 
> 
> On Sun, Feb 10, 2008 at 10:37:54AM +0100, Peter Boncz wrote:
> > no consequences.
> > 
> > 
> > 
> > 
> > Peter,
> > 
> > What about the consequences in:
> > build/MonetDB/src/gdk/gdk_heap.c
> > build/MonetDB5/src/modules/kernel/status.c
> > build/MonetDB/src/gdk/gdk.h
> > 
> > Peter Boncz wrote:
> > > Update of /cvsroot/monetdb/MonetDB/src/gdk
> > > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv29651
> > >
> > > Modified Files:
> > >       Tag: MonetDB_1-22
> > > 	gdk_utils.mx
> > > Log Message:
> > > disable use of vmalloc()
> > >
> > >
> > >
> > > Index: gdk_utils.mx
> > > ===================================================================
> > > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_utils.mx,v
> > > retrieving revision 1.206
> > > retrieving revision 1.206.2.1
> > > diff -u -d -r1.206 -r1.206.2.1
> > > --- gdk_utils.mx	14 Jan 2008 10:48:58 -0000	1.206
> > > +++ gdk_utils.mx	9 Feb 2008 15:38:11 -0000	1.206.2.1
> > > @@ -1050,6 +1050,7 @@
> > >  #endif
> > >  	}
> > >  	size = (size + 7) & ~7;	/* round up to a multiple of eight */
> > > +#if 0
> > >  	if (size > GDK_mem_bigsize) {
> > >  		size_t newsize = size + sizeof(size_t) + sizeof(size_t);
> > >  		size_t newmax = MAX(*maxsize, newsize);
> > > @@ -1064,6 +1065,7 @@
> > >  		*maxsize = newmax - (sizeof(size_t) + sizeof(size_t));
> > >  		return (void *) s;
> > >  	}
> > > +#endif
> > >  	CHKMEM(size, 0);
> > >  	GDKmalloc_prefixsize(s, size);
> > >  	if (s == NULL) {
> > >
> > >
> > > -------------------------------------------------------------------------
> > > This SF.net email is sponsored by: Microsoft
> > > Defy all challenges. Microsoft(R) Visual Studio 2008.
> > > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> > > _______________________________________________
> > > Monetdb-checkins mailing list
> > > Monetdb-checkins at lists.sourceforge.net
> > > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
> > >
> > 
> > 
> > -------------------------------------------------------------------------
> > This SF.net email is sponsored by: Microsoft
> > Defy all challenges. Microsoft(R) Visual Studio 2008.
> > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> > _______________________________________________
> > Monetdb-developers mailing list
> > Monetdb-developers at lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/monetdb-developers
> > 
> > 
> > 
> > 
> > 
> 
> -- 
> | Dr. Stefan Manegold | mailto:Stefan.Manegold at cwi.nl |
> | CWI,  P.O.Box 94079 | http://www.cwi.nl/~manegold/  |
> | 1090 GB Amsterdam   | Tel.: +31 (20) 592-4212       |
> | The Netherlands     | Fax : +31 (20) 592-4312       |

-- 
| Dr. Stefan Manegold | mailto:Stefan.Manegold at cwi.nl |
| CWI,  P.O.Box 94079 | http://www.cwi.nl/~manegold/  |
| 1090 GB Amsterdam   | Tel.: +31 (20) 592-4212       |
| The Netherlands     | Fax : +31 (20) 592-4312       |




More information about the developers-list mailing list