Re: [Monetdb-developers] [Monetdb-checkins]MonetDB/src/gdkgdk_utils.mx, MonetDB_1-22, 1.206, 1.206.2.1
Hi Stefan
Thanks for your response. However, as I have said, but let me say it as clear as possble: I am *sure* there is a bug still unfound in the way re-allocations are done when mem_bigsize redirects it into virtual memory.
For this reason, at this stage, with the goal of a stable release, it is just my recommendation *not* to re-enable it.
In M5 nor in M5.
Peter
Dear all,
On Mon, Feb 11, 2008 at 02:54:59PM +0100, Peter Boncz wrote:
Hi Stefan,
The pftijah bug was unrelated to vmalloc(), a relationship that I never assumed.
Indeed, that was (only) my mis-interpretation. Sorry.
The vmalloc() troubles occured, as you may recall, during the data preparation of the sorted 100GB TPC-H.
As it is entirely foreseeable that we at some point again will re-encounter an OS with a malloc implementation that is prone to fragmentation, I'd keep the vmalloc() code in for the moment (though in disabled state). Linux and Windows do not need it.
So, my advise is to for the monet let vmalloc() be disabled, until the issue is properly investigated and a bugfix is found. Let's call it option (0).
Ok.
Just to sum up the current state. Please check, whether this is OK and consistent:
a) Documentation in MonetDB4/conf/MonetDB.conf.in & MonetDB5/conf/monetdb5.conf.in says: " # gdk_mem_bigsize & gdk_vm_minsize will be set/limited to # 1/2 of the physically available amount of main-memory # during start-up in src/tools/Mserver.mx # memory chunks of size >= gdk_mem_bigsize (in bytes) will be mmaped anonymously #gdk_mem_bigsize=262144 # memory chunks of size >= gdk_vm_minsize (in bytes) will be mmaped; #gdk_vm_minsize=137438953472 "
b) Documentation in ./monetweb/Docs/XQuery/MonetDB.conf.texi says " @item @code{gdk_mem_bigsize}: minimum size for memory-mapped columns, e.g. @code{262144}. @item @code{gdk_vm_minsize}: column size above which memory mapped files are used always, e.g. @code{1749291171} "
c) MonetDB/src/common/monet_options.mx sets " set[i].kind = opt_builtin; set[i].name = strdup("gdk_mem_bigsize"); set[i].value = strdup("262144"); "
d) MonetDB/src/common/monet_options.py.in sets " # gdk_mem_bigsize & gdk_vm_minsize will be set/limited to # 1/2 of the physically available amount of main-memory # during start-up in src/tools/Mserver.mx gdk_mem_bigsize = '256K' gdk_vm_minsize = '128G' "
e) GDKinit() in MonetDB/src/gdk/gdk_utils.mx sets " if ((p = GDKgetenv("gdk_mem_bigsize"))) { /* when allocating >6% of all RAM; do so using vmalloc() iso malloc() */ lng max_mem_bigsize = GDK_mem_maxsize/16;
/* sanity check to avoid memory fragmentation */ GDK_mem_bigsize = (size_t) MIN(max_mem_bigsize, strtol(p, NULL, 10)); } "
f) GDKmallocmax() in MonetDB/src/gdk/gdk_utils.mx ignores GDK_mem_bigsize
g) HEAPalloc() in MonetDB/src/gdk/gdk_heap.mx uses GDK_mem_bigsize as follows: " /* when using anonymous vm we malloc we need 64K chunks, also we * 20% extra malloc */ if (h->size > GDK_mem_bigsize) { h->maxsize = (size_t) ((double) h->maxsize * BATMARGIN) - 1; h->maxsize = (1 + (h->maxsize >> 16)) << 16; } "
h) HEAPextend() in MonetDB/src/gdk/gdk_heap.mx uses GDK_mem_bigsize as follows: " /* extend a malloced heap, possibly switching over to file-mapped storage */ Heap bak = *h; int can_mmap = (h->filename && size >= GDK_mem_bigsize); int must_mmap = can_mmap && (size >= GDK_vm_minsize || (h->newstorage != STORE_MEM)); [...] if (can_mmap) { /* in anonymous vm, if have to realloc anyway, we reserve some extra space */ if (size > h->maxsize) { h->maxsize = (size_t) ((double) size * BATMARGIN); } /* when using anonymous vm we malloc we need 64K chunks */ h->maxsize = (1 + ((h->maxsize - 1) >> 16)) << 16; [...] /* too big: convert it to a disk-based temporary heap */ if (can_mmap) { char privext[PATHLENGTH], *of = h->filename; FILE *fp;
h->filename = NULL; sprintf(privext, "%s.priv", ext); fp = GDKfilelocate(nme, "wb", privext); if (fp != NULL) { fclose(fp); /* a non-persistent heap: we create a .priv but *not* MMAP_PRIV !!! */ h->storage = STORE_MMAP; h->base = NULL; if (HEAPload(h, nme, privext, FALSE) >= 0) { memcpy(h->base, bak.base, bak.free); HEAPfree(&bak); return 0; } } GDKfree(of); } "
i) MIL offers mem_bigsize() & mem_bigsize(lng) to get / set mem_bigsize in MonetDB4/src/modules/plain/sys.mx
j) both MIL & MAL offer commands to set mem_maxsize and vm_minsize, however, only the MIL variants (still) related mem_maxsize & vm_minsize with mem_bigsize, while the MAL variants no longer do so:
MonetDB4/src/modules/plain/sys.mx: " int set_mem_maxsize(lng *num) { @:num2sze(mem_maxsize)@ if (sze < GDK_mem_bigsize) set_mem_bigsize(num); GDK_mem_maxsize = MAX(GDK_mem_bigsize, sze); return GDK_SUCCEED; }
int set_vm_minsize(lng *num) { @:num2sze(vm_minsize)@ if (sze < GDK_mem_bigsize) set_mem_bigsize(num); GDK_vm_minsize = MAX(GDK_mem_bigsize, sze); return GDK_SUCCEED; } "
MonetDB5/src/modules/kernel/status.mx " int set_mem_maxsize(lng *num) { @:num2sze(mem_maxsize)@ #if 0 if (sze < GDK_mem_bigsize) set_mem_bigsize(num); GDK_mem_maxsize = MAX(GDK_mem_bigsize, sze); #endif GDK_mem_maxsize = sze; return GDK_SUCCEED; }
int set_vm_minsize(lng *num) { @:num2sze(vm_minsize)@ #if 0 if (sze < GDK_mem_bigsize) set_mem_bigsize(num); GDK_vm_minsize = MAX(GDK_mem_bigsize, sze); #endif GDK_vm_minsize = sze; return GDK_SUCCEED; } "
If all this is considered OK and consistent, we're done. Otherwise, we need to do what ever is required to get the code and documentation consistent, preferably before the release.
Stefan
Peter
Dear all,
since Roberto's bug appears to be fixed by Peter's ".priv" (vm_minsize related) fixes (Thanks!), and disabling of vmalloc() (mem_bigsize related) does not appear have any impact on this bug (just tested successfully both with vmalloc() disabled and vmalloc() enabled), there are two ways to finish this release (more or less) quickly:
(1) finish the disabling of vmalloc() in all parts of the code as indicated below (or better: remove all remains completely) --- this might need some extra testing to ensure that it does not have any (significant) impact on performance ...
(2) for now (i.e., in the release branch) re-enable vmalloc(), and then finish the removal of it (incl. extensive testing) in the development trunk for th next release.
Since I'm not completely aware of the impact of vmalloc(), I cannot really make this decision --- the "save" back-up would IMHO be (2).
Any other comments, opinions, suggestions, expertise?
Thanks in advance!
Stefan
On Sun, Feb 10, 2008 at 01:33:04PM +0100, Stefan Manegold wrote:
Peter: Thanks for disabling mem_bigsize / vmalloc().
All: (a) Unfortunately, this does not (yet) solve [ 1872685 ] PF(tijah): HEAPsetmmap: Assertion `p' failed. http://sourceforge.net/tracker/index.php?func=detail&aid=1872685&gro...
But this appears to be more vm_minsize related than mem_bigsize related.
(b) We should check all occurances of mem_bigsize and remove / "hide" those that have become irrelevant with the deactivation of mem_bigsize / vmalloc(): ======== $ cvsfiles * | egrep -v ' |/Tests/' | xargs grep -n --color 'mem_bigsize'
MonetDB4/conf/MonetDB.conf.in:29:# gdk_mem_bigsize & gdk_vm_minsize will be
set/limited to
MonetDB4/conf/MonetDB.conf.in:32:# memory chunks of size >= gdk_mem_bigsize
(in bytes) will be mmaped anonymously
MonetDB4/conf/MonetDB.conf.in:33:#gdk_mem_bigsize=262144 MonetDB5/conf/monetdb5.conf.in:33:# gdk_mem_bigsize & gdk_vm_minsize will be
set/limited to
MonetDB5/conf/monetdb5.conf.in:36:# memory chunks of size >= gdk_mem_bigsize
(in bytes) will be mmaped anonymously
MonetDB5/conf/monetdb5.conf.in:37:#gdk_mem_bigsize=262144 MonetDB/src/common/monet_options.mx:410: set[i].name =
strdup("gdk_mem_bigsize");
MonetDB/src/common/monet_options.py.in:39: # gdk_mem_bigsize &
gdk_vm_minsize will be set/limited to
MonetDB/src/common/monet_options.py.in:42: gdk_mem_bigsize = '256K' MonetDB/src/gdk/gdk_heap.mx:121: if (h->size > GDK_mem_bigsize) { MonetDB/src/gdk/gdk_heap.mx:189: int can_mmap = (h->filename
&& size >= GDK_mem_bigsize);
MonetDB/src/gdk/gdk.mx:2210:gdk_export size_t GDK_mem_bigsize; /* size
after which we use VM rather than heap */
MonetDB/src/gdk/gdk_utils.mx:396:size_t GDK_mem_bigsize = 1 << 20; MonetDB/src/gdk/gdk_utils.mx:964:larger than GDK_mem_bigsize) to anonymous
virtual memory. The seamless
MonetDB/src/gdk/gdk_utils.mx:1054: if (size > GDK_mem_bigsize) { MonetDB/src/gdk/gdk_utils.mx:1191: if (size <= GDK_mem_bigsize) { MonetDB/src/gdk/gdk_utils.mx:1529: if ((p =
GDKgetenv("gdk_mem_bigsize"))) {
MonetDB/src/gdk/gdk_utils.mx:1531: lng max_mem_bigsize =
GDK_mem_maxsize/16;
MonetDB/src/gdk/gdk_utils.mx:1534: GDK_mem_bigsize = (size_t)
MIN(max_mem_bigsize, strtol(p, NULL, 10));
monetweb/Docs/XQuery/MonetDB.conf.texi:115:@item @code{gdk_mem_bigsize}:
minimum size for memory-mapped columns, e.g. @code{262144}.
MonetDB4/src/modules/plain/sys.mx:96:.COMMAND mem_bigsize() : lng =
get_mem_bigsize;
MonetDB4/src/modules/plain/sys.mx:98:.COMMAND mem_bigsize(lng) : void =
set_mem_bigsize;
MonetDB4/src/modules/plain/sys.mx:1430:get_mem_bigsize(lng *num) MonetDB4/src/modules/plain/sys.mx:1432: *num = GDK_mem_bigsize; MonetDB4/src/modules/plain/sys.mx:1437:set_mem_bigsize(lng *num) MonetDB4/src/modules/plain/sys.mx:1439: @:num2sze(mem_bigsize)@ MonetDB4/src/modules/plain/sys.mx:1440: GDK_mem_bigsize = MAX(32768, sze); MonetDB4/src/modules/plain/sys.mx:1462: if (sze < GDK_mem_bigsize) MonetDB4/src/modules/plain/sys.mx:1463: set_mem_bigsize(num); MonetDB4/src/modules/plain/sys.mx:1464: GDK_mem_maxsize =
MAX(GDK_mem_bigsize, sze);
MonetDB4/src/modules/plain/sys.mx:1486: if (sze < GDK_mem_bigsize) MonetDB4/src/modules/plain/sys.mx:1487: set_mem_bigsize(num); MonetDB4/src/modules/plain/sys.mx:1488: GDK_vm_minsize =
MAX(GDK_mem_bigsize, sze);
MonetDB5/src/modules/kernel/status.mx:405:get_mem_bigsize(lng *num) MonetDB5/src/modules/kernel/status.mx:407: *num = GDK_mem_bigsize; MonetDB5/src/modules/kernel/status.mx:412:set_mem_bigsize(lng *num) MonetDB5/src/modules/kernel/status.mx:414: @:num2sze(mem_bigsize)@ MonetDB5/src/modules/kernel/status.mx:415: GDK_mem_bigsize = MAX(32768,
sze);
MonetDB5/src/modules/kernel/status.mx:439: if (sze < GDK_mem_bigsize) MonetDB5/src/modules/kernel/status.mx:440: set_mem_bigsize(num); MonetDB5/src/modules/kernel/status.mx:441: GDK_mem_maxsize =
MAX(GDK_mem_bigsize, sze);
MonetDB5/src/modules/kernel/status.mx:466: if (sze < GDK_mem_bigsize) MonetDB5/src/modules/kernel/status.mx:467: set_mem_bigsize(num); MonetDB5/src/modules/kernel/status.mx:468: GDK_vm_minsize =
MAX(GDK_mem_bigsize, sze);
monetweb/MonetDB/QuickTour/HelloWorld/content.shtml:42:[ "gdk_mem_bigsize",
"262144"========
more tomorrow during MADAM ...
Stefan
On Sun, Feb 10, 2008 at 10:37:54AM +0100, Peter Boncz wrote:
no consequences.
Peter,
What about the consequences in: build/MonetDB/src/gdk/gdk_heap.c build/MonetDB5/src/modules/kernel/status.c build/MonetDB/src/gdk/gdk.h
Peter Boncz wrote:
Update of /cvsroot/monetdb/MonetDB/src/gdk In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv29651
Modified Files: Tag: MonetDB_1-22 gdk_utils.mx Log Message: disable use of vmalloc()
Index: gdk_utils.mx
RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_utils.mx,v retrieving revision 1.206 retrieving revision 1.206.2.1 diff -u -d -r1.206 -r1.206.2.1 --- gdk_utils.mx 14 Jan 2008 10:48:58 -0000 1.206 +++ gdk_utils.mx 9 Feb 2008 15:38:11 -0000 1.206.2.1 @@ -1050,6 +1050,7 @@ #endif } size = (size + 7) & ~7; /* round up to a multiple of eight */ +#if 0 if (size > GDK_mem_bigsize) { size_t newsize = size + sizeof(size_t) + sizeof(size_t); size_t newmax = MAX(*maxsize, newsize); @@ -1064,6 +1065,7 @@ *maxsize = newmax - (sizeof(size_t) + sizeof(size_t)); return (void *) s; } +#endif CHKMEM(size, 0); GDKmalloc_prefixsize(s, size); if (s == NULL) {
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
On 2008-02-11 19:05, Peter Boncz wrote:
Hi Stefan
Thanks for your response. However, as I have said, but let me say it as clear as possble: I am *sure* there is a bug still unfound in the way re-allocations are done when mem_bigsize redirects it into virtual memory.
For this reason, at this stage, with the goal of a stable release, it is just my recommendation *not* to re-enable it.
In M5 nor in M5.
I think Stefan meant reenable access to the variable in M5 that Martin turned off, not reenable the GDKvmalloc switch that you turned off.
Peter
Dear all,
On Mon, Feb 11, 2008 at 02:54:59PM +0100, Peter Boncz wrote:
Hi Stefan,
The pftijah bug was unrelated to vmalloc(), a relationship that I never assumed.
Indeed, that was (only) my mis-interpretation. Sorry.
The vmalloc() troubles occured, as you may recall, during the data preparation of the sorted 100GB TPC-H.
As it is entirely foreseeable that we at some point again will re-encounter an OS with a malloc implementation that is prone to fragmentation, I'd keep the vmalloc() code in for the moment (though in disabled state). Linux and Windows do not need it.
So, my advise is to for the monet let vmalloc() be disabled, until the issue is properly investigated and a bugfix is found. Let's call it option (0).
Ok.
Just to sum up the current state. Please check, whether this is OK and consistent:
a) Documentation in MonetDB4/conf/MonetDB.conf.in & MonetDB5/conf/monetdb5.conf.in says: " # gdk_mem_bigsize & gdk_vm_minsize will be set/limited to # 1/2 of the physically available amount of main-memory # during start-up in src/tools/Mserver.mx # memory chunks of size >= gdk_mem_bigsize (in bytes) will be mmaped anonymously #gdk_mem_bigsize=262144 # memory chunks of size >= gdk_vm_minsize (in bytes) will be mmaped; #gdk_vm_minsize=137438953472 "
b) Documentation in ./monetweb/Docs/XQuery/MonetDB.conf.texi says " @item @code{gdk_mem_bigsize}: minimum size for memory-mapped columns, e.g. @code{262144}. @item @code{gdk_vm_minsize}: column size above which memory mapped files are used always, e.g. @code{1749291171} "
c) MonetDB/src/common/monet_options.mx sets " set[i].kind = opt_builtin; set[i].name = strdup("gdk_mem_bigsize"); set[i].value = strdup("262144"); "
d) MonetDB/src/common/monet_options.py.in sets " # gdk_mem_bigsize & gdk_vm_minsize will be set/limited to # 1/2 of the physically available amount of main-memory # during start-up in src/tools/Mserver.mx gdk_mem_bigsize = '256K' gdk_vm_minsize = '128G' "
e) GDKinit() in MonetDB/src/gdk/gdk_utils.mx sets " if ((p = GDKgetenv("gdk_mem_bigsize"))) { /* when allocating >6% of all RAM; do so using vmalloc() iso malloc() */ lng max_mem_bigsize = GDK_mem_maxsize/16;
/* sanity check to avoid memory fragmentation */ GDK_mem_bigsize = (size_t) MIN(max_mem_bigsize, strtol(p,NULL, 10)); } "
f) GDKmallocmax() in MonetDB/src/gdk/gdk_utils.mx ignores GDK_mem_bigsize
g) HEAPalloc() in MonetDB/src/gdk/gdk_heap.mx uses GDK_mem_bigsize as follows: " /* when using anonymous vm we malloc we need 64K chunks, also we * 20% extra malloc */ if (h->size > GDK_mem_bigsize) { h->maxsize = (size_t) ((double) h->maxsize * BATMARGIN) - 1; h->maxsize = (1 + (h->maxsize >> 16)) << 16; } "
h) HEAPextend() in MonetDB/src/gdk/gdk_heap.mx uses GDK_mem_bigsize as follows: " /* extend a malloced heap, possibly switching over to file-mapped storage */ Heap bak = *h; int can_mmap = (h->filename && size >= GDK_mem_bigsize); int must_mmap = can_mmap && (size >= GDK_vm_minsize || (h->newstorage != STORE_MEM)); [...] if (can_mmap) { /* in anonymous vm, if have to realloc anyway, we reserve some extra space */ if (size > h->maxsize) { h->maxsize = (size_t) ((double) size * BATMARGIN); } /* when using anonymous vm we malloc we need 64K chunks */ h->maxsize = (1 + ((h->maxsize - 1) >> 16)) << 16; [...] /* too big: convert it to a disk-based temporary heap */ if (can_mmap) { char privext[PATHLENGTH], *of = h->filename; FILE *fp;
h->filename = NULL; sprintf(privext, "%s.priv", ext); fp = GDKfilelocate(nme, "wb", privext); if (fp != NULL) { fclose(fp); /* a non-persistent heap: we create a .privbut *not* MMAP_PRIV !!! */ h->storage = STORE_MMAP; h->base = NULL; if (HEAPload(h, nme, privext, FALSE) >= 0) { memcpy(h->base, bak.base, bak.free); HEAPfree(&bak); return 0; } } GDKfree(of); } "
i) MIL offers mem_bigsize() & mem_bigsize(lng) to get / set mem_bigsize in MonetDB4/src/modules/plain/sys.mx
j) both MIL & MAL offer commands to set mem_maxsize and vm_minsize, however, only the MIL variants (still) related mem_maxsize & vm_minsize with mem_bigsize, while the MAL variants no longer do so:
MonetDB4/src/modules/plain/sys.mx: " int set_mem_maxsize(lng *num) { @:num2sze(mem_maxsize)@ if (sze < GDK_mem_bigsize) set_mem_bigsize(num); GDK_mem_maxsize = MAX(GDK_mem_bigsize, sze); return GDK_SUCCEED; }
int set_vm_minsize(lng *num) { @:num2sze(vm_minsize)@ if (sze < GDK_mem_bigsize) set_mem_bigsize(num); GDK_vm_minsize = MAX(GDK_mem_bigsize, sze); return GDK_SUCCEED; } "
MonetDB5/src/modules/kernel/status.mx " int set_mem_maxsize(lng *num) { @:num2sze(mem_maxsize)@ #if 0 if (sze < GDK_mem_bigsize) set_mem_bigsize(num); GDK_mem_maxsize = MAX(GDK_mem_bigsize, sze); #endif GDK_mem_maxsize = sze; return GDK_SUCCEED; }
int set_vm_minsize(lng *num) { @:num2sze(vm_minsize)@ #if 0 if (sze < GDK_mem_bigsize) set_mem_bigsize(num); GDK_vm_minsize = MAX(GDK_mem_bigsize, sze); #endif GDK_vm_minsize = sze; return GDK_SUCCEED; } "
If all this is considered OK and consistent, we're done. Otherwise, we need to do what ever is required to get the code and documentation consistent, preferably before the release.
Stefan
Peter
Dear all,
since Roberto's bug appears to be fixed by Peter's ".priv" (vm_minsize related) fixes (Thanks!), and disabling of vmalloc() (mem_bigsize related) does not appear have any impact on this bug (just tested successfully both with vmalloc() disabled and vmalloc() enabled), there are two ways to finish this release (more or less) quickly:
(1) finish the disabling of vmalloc() in all parts of the code as indicated below (or better: remove all remains completely) --- this might need some extra testing to ensure that it does not have any (significant) impact on performance ...
(2) for now (i.e., in the release branch) re-enable vmalloc(), and then finish the removal of it (incl. extensive testing) in the development trunk for th next release.
Since I'm not completely aware of the impact of vmalloc(), I cannot really make this decision --- the "save" back-up would IMHO be (2).
Any other comments, opinions, suggestions, expertise?
Thanks in advance!
Stefan
On Sun, Feb 10, 2008 at 01:33:04PM +0100, Stefan Manegold wrote:
Peter: Thanks for disabling mem_bigsize / vmalloc().
All: (a) Unfortunately, this does not (yet) solve [ 1872685 ] PF(tijah): HEAPsetmmap: Assertion `p' failed. http://sourceforge.net/tracker/index.php?func=detail&aid=1872685&gro...
But this appears to be more vm_minsize related than mem_bigsize related.
(b) We should check all occurances of mem_bigsize and remove / "hide" those that have become irrelevant with the deactivation of mem_bigsize / vmalloc(): ======== $ cvsfiles * | egrep -v ' |/Tests/' | xargs grep -n --color 'mem_bigsize'
MonetDB4/conf/MonetDB.conf.in:29:# gdk_mem_bigsize & gdk_vm_minsize will be
set/limited to
MonetDB4/conf/MonetDB.conf.in:32:# memory chunks of size >= gdk_mem_bigsize
(in bytes) will be mmaped anonymously
MonetDB4/conf/MonetDB.conf.in:33:#gdk_mem_bigsize=262144 MonetDB5/conf/monetdb5.conf.in:33:# gdk_mem_bigsize & gdk_vm_minsize will be
set/limited to
MonetDB5/conf/monetdb5.conf.in:36:# memory chunks of size >= gdk_mem_bigsize
(in bytes) will be mmaped anonymously
MonetDB5/conf/monetdb5.conf.in:37:#gdk_mem_bigsize=262144 MonetDB/src/common/monet_options.mx:410: set[i].name =
strdup("gdk_mem_bigsize");
MonetDB/src/common/monet_options.py.in:39: # gdk_mem_bigsize &
gdk_vm_minsize will be set/limited to
MonetDB/src/common/monet_options.py.in:42: gdk_mem_bigsize = '256K' MonetDB/src/gdk/gdk_heap.mx:121: if (h->size > GDK_mem_bigsize) { MonetDB/src/gdk/gdk_heap.mx:189: int can_mmap = (h->filename
&& size >= GDK_mem_bigsize);
MonetDB/src/gdk/gdk.mx:2210:gdk_export size_t GDK_mem_bigsize; /* size
after which we use VM rather than heap */
MonetDB/src/gdk/gdk_utils.mx:396:size_t GDK_mem_bigsize = 1 << 20; MonetDB/src/gdk/gdk_utils.mx:964:larger than GDK_mem_bigsize) to anonymous
virtual memory. The seamless
MonetDB/src/gdk/gdk_utils.mx:1054: if (size > GDK_mem_bigsize) { MonetDB/src/gdk/gdk_utils.mx:1191: if (size <= GDK_mem_bigsize) { MonetDB/src/gdk/gdk_utils.mx:1529: if ((p =
GDKgetenv("gdk_mem_bigsize"))) {
MonetDB/src/gdk/gdk_utils.mx:1531: lng max_mem_bigsize =
GDK_mem_maxsize/16;
MonetDB/src/gdk/gdk_utils.mx:1534: GDK_mem_bigsize = (size_t)
MIN(max_mem_bigsize, strtol(p, NULL, 10));
monetweb/Docs/XQuery/MonetDB.conf.texi:115:@item @code{gdk_mem_bigsize}:
minimum size for memory-mapped columns, e.g. @code{262144}.
MonetDB4/src/modules/plain/sys.mx:96:.COMMAND mem_bigsize() : lng =
get_mem_bigsize;
MonetDB4/src/modules/plain/sys.mx:98:.COMMAND mem_bigsize(lng) : void =
set_mem_bigsize;
MonetDB4/src/modules/plain/sys.mx:1430:get_mem_bigsize(lng *num) MonetDB4/src/modules/plain/sys.mx:1432: *num = GDK_mem_bigsize; MonetDB4/src/modules/plain/sys.mx:1437:set_mem_bigsize(lng *num) MonetDB4/src/modules/plain/sys.mx:1439: @:num2sze(mem_bigsize)@ MonetDB4/src/modules/plain/sys.mx:1440: GDK_mem_bigsize = MAX(32768, sze); MonetDB4/src/modules/plain/sys.mx:1462: if (sze < GDK_mem_bigsize) MonetDB4/src/modules/plain/sys.mx:1463: set_mem_bigsize(num); MonetDB4/src/modules/plain/sys.mx:1464: GDK_mem_maxsize =
MAX(GDK_mem_bigsize, sze);
MonetDB4/src/modules/plain/sys.mx:1486: if (sze < GDK_mem_bigsize) MonetDB4/src/modules/plain/sys.mx:1487: set_mem_bigsize(num); MonetDB4/src/modules/plain/sys.mx:1488: GDK_vm_minsize =
MAX(GDK_mem_bigsize, sze);
MonetDB5/src/modules/kernel/status.mx:405:get_mem_bigsize(lng *num) MonetDB5/src/modules/kernel/status.mx:407: *num = GDK_mem_bigsize; MonetDB5/src/modules/kernel/status.mx:412:set_mem_bigsize(lng *num) MonetDB5/src/modules/kernel/status.mx:414: @:num2sze(mem_bigsize)@ MonetDB5/src/modules/kernel/status.mx:415: GDK_mem_bigsize = MAX(32768,
sze);
MonetDB5/src/modules/kernel/status.mx:439: if (sze < GDK_mem_bigsize) MonetDB5/src/modules/kernel/status.mx:440: set_mem_bigsize(num); MonetDB5/src/modules/kernel/status.mx:441: GDK_mem_maxsize =
MAX(GDK_mem_bigsize, sze);
MonetDB5/src/modules/kernel/status.mx:466: if (sze < GDK_mem_bigsize) MonetDB5/src/modules/kernel/status.mx:467: set_mem_bigsize(num); MonetDB5/src/modules/kernel/status.mx:468: GDK_vm_minsize =
MAX(GDK_mem_bigsize, sze);
monetweb/MonetDB/QuickTour/HelloWorld/content.shtml:42:[ "gdk_mem_bigsize",
"262144"========
more tomorrow during MADAM ...
Stefan
On Sun, Feb 10, 2008 at 10:37:54AM +0100, Peter Boncz wrote:
no consequences.
Peter,
What about the consequences in: build/MonetDB/src/gdk/gdk_heap.c build/MonetDB5/src/modules/kernel/status.c build/MonetDB/src/gdk/gdk.h
Peter Boncz wrote:
Update of /cvsroot/monetdb/MonetDB/src/gdk In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv29651
Modified Files: Tag: MonetDB_1-22 gdk_utils.mx Log Message: disable use of vmalloc()
Index: gdk_utils.mx
RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_utils.mx,v retrieving revision 1.206 retrieving revision 1.206.2.1 diff -u -d -r1.206 -r1.206.2.1 --- gdk_utils.mx 14 Jan 2008 10:48:58 -0000 1.206 +++ gdk_utils.mx 9 Feb 2008 15:38:11 -0000 1.206.2.1 @@ -1050,6 +1050,7 @@ #endif } size = (size + 7) & ~7; /* round up to a multiple of eight */ +#if 0 if (size > GDK_mem_bigsize) { size_t newsize = size + sizeof(size_t) + sizeof(size_t); size_t newmax = MAX(*maxsize, newsize); @@ -1064,6 +1065,7 @@ *maxsize = newmax - (sizeof(size_t) + sizeof(size_t)); return (void *) s; } +#endif CHKMEM(size, 0); GDKmalloc_prefixsize(s, size); if (s == NULL) {
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
On Mon, Feb 11, 2008 at 07:19:09PM +0100, Sjoerd Mullender wrote:
On 2008-02-11 19:05, Peter Boncz wrote:
Hi Stefan
Thanks for your response. However, as I have said, but let me say it as clear as possble: I am *sure* there is a bug still unfound in the way re-allocations are done when mem_bigsize redirects it into virtual memory.
For this reason, at this stage, with the goal of a stable release, it is just my recommendation *not* to re-enable it.
In M5 nor in M5.
I think Stefan meant reenable access to the variable in M5 that Martin turned off, not reenable the GDKvmalloc switch that you turned off.
right. exactly that. no more, no less ;-)
Stefan
Peter
Dear all,
On Mon, Feb 11, 2008 at 02:54:59PM +0100, Peter Boncz wrote:
Hi Stefan,
The pftijah bug was unrelated to vmalloc(), a relationship that I never assumed.
Indeed, that was (only) my mis-interpretation. Sorry.
The vmalloc() troubles occured, as you may recall, during the data preparation of the sorted 100GB TPC-H.
As it is entirely foreseeable that we at some point again will re-encounter an OS with a malloc implementation that is prone to fragmentation, I'd keep the vmalloc() code in for the moment (though in disabled state). Linux and Windows do not need it.
So, my advise is to for the monet let vmalloc() be disabled, until the issue is properly investigated and a bugfix is found. Let's call it option (0).
Ok.
Just to sum up the current state. Please check, whether this is OK and consistent:
a) Documentation in MonetDB4/conf/MonetDB.conf.in & MonetDB5/conf/monetdb5.conf.in says: " # gdk_mem_bigsize & gdk_vm_minsize will be set/limited to # 1/2 of the physically available amount of main-memory # during start-up in src/tools/Mserver.mx # memory chunks of size >= gdk_mem_bigsize (in bytes) will be mmaped anonymously #gdk_mem_bigsize=262144 # memory chunks of size >= gdk_vm_minsize (in bytes) will be mmaped; #gdk_vm_minsize=137438953472 "
b) Documentation in ./monetweb/Docs/XQuery/MonetDB.conf.texi says " @item @code{gdk_mem_bigsize}: minimum size for memory-mapped columns, e.g. @code{262144}. @item @code{gdk_vm_minsize}: column size above which memory mapped files are used always, e.g. @code{1749291171} "
c) MonetDB/src/common/monet_options.mx sets " set[i].kind = opt_builtin; set[i].name = strdup("gdk_mem_bigsize"); set[i].value = strdup("262144"); "
d) MonetDB/src/common/monet_options.py.in sets " # gdk_mem_bigsize & gdk_vm_minsize will be set/limited to # 1/2 of the physically available amount of main-memory # during start-up in src/tools/Mserver.mx gdk_mem_bigsize = '256K' gdk_vm_minsize = '128G' "
e) GDKinit() in MonetDB/src/gdk/gdk_utils.mx sets " if ((p = GDKgetenv("gdk_mem_bigsize"))) { /* when allocating >6% of all RAM; do so using vmalloc() iso malloc() */ lng max_mem_bigsize = GDK_mem_maxsize/16;
/* sanity check to avoid memory fragmentation */ GDK_mem_bigsize = (size_t) MIN(max_mem_bigsize, strtol(p,NULL, 10)); } "
f) GDKmallocmax() in MonetDB/src/gdk/gdk_utils.mx ignores GDK_mem_bigsize
g) HEAPalloc() in MonetDB/src/gdk/gdk_heap.mx uses GDK_mem_bigsize as follows: " /* when using anonymous vm we malloc we need 64K chunks, also we * 20% extra malloc */ if (h->size > GDK_mem_bigsize) { h->maxsize = (size_t) ((double) h->maxsize * BATMARGIN) - 1; h->maxsize = (1 + (h->maxsize >> 16)) << 16; } "
h) HEAPextend() in MonetDB/src/gdk/gdk_heap.mx uses GDK_mem_bigsize as follows: " /* extend a malloced heap, possibly switching over to file-mapped storage */ Heap bak = *h; int can_mmap = (h->filename && size >= GDK_mem_bigsize); int must_mmap = can_mmap && (size >= GDK_vm_minsize || (h->newstorage != STORE_MEM)); [...] if (can_mmap) { /* in anonymous vm, if have to realloc anyway, we reserve some extra space */ if (size > h->maxsize) { h->maxsize = (size_t) ((double) size * BATMARGIN); } /* when using anonymous vm we malloc we need 64K chunks */ h->maxsize = (1 + ((h->maxsize - 1) >> 16)) << 16; [...] /* too big: convert it to a disk-based temporary heap */ if (can_mmap) { char privext[PATHLENGTH], *of = h->filename; FILE *fp;
h->filename = NULL; sprintf(privext, "%s.priv", ext); fp = GDKfilelocate(nme, "wb", privext); if (fp != NULL) { fclose(fp); /* a non-persistent heap: we create a .privbut *not* MMAP_PRIV !!! */ h->storage = STORE_MMAP; h->base = NULL; if (HEAPload(h, nme, privext, FALSE) >= 0) { memcpy(h->base, bak.base, bak.free); HEAPfree(&bak); return 0; } } GDKfree(of); } "
i) MIL offers mem_bigsize() & mem_bigsize(lng) to get / set mem_bigsize in MonetDB4/src/modules/plain/sys.mx
j) both MIL & MAL offer commands to set mem_maxsize and vm_minsize, however, only the MIL variants (still) related mem_maxsize & vm_minsize with mem_bigsize, while the MAL variants no longer do so:
MonetDB4/src/modules/plain/sys.mx: " int set_mem_maxsize(lng *num) { @:num2sze(mem_maxsize)@ if (sze < GDK_mem_bigsize) set_mem_bigsize(num); GDK_mem_maxsize = MAX(GDK_mem_bigsize, sze); return GDK_SUCCEED; }
int set_vm_minsize(lng *num) { @:num2sze(vm_minsize)@ if (sze < GDK_mem_bigsize) set_mem_bigsize(num); GDK_vm_minsize = MAX(GDK_mem_bigsize, sze); return GDK_SUCCEED; } "
MonetDB5/src/modules/kernel/status.mx " int set_mem_maxsize(lng *num) { @:num2sze(mem_maxsize)@ #if 0 if (sze < GDK_mem_bigsize) set_mem_bigsize(num); GDK_mem_maxsize = MAX(GDK_mem_bigsize, sze); #endif GDK_mem_maxsize = sze; return GDK_SUCCEED; }
int set_vm_minsize(lng *num) { @:num2sze(vm_minsize)@ #if 0 if (sze < GDK_mem_bigsize) set_mem_bigsize(num); GDK_vm_minsize = MAX(GDK_mem_bigsize, sze); #endif GDK_vm_minsize = sze; return GDK_SUCCEED; } "
If all this is considered OK and consistent, we're done. Otherwise, we need to do what ever is required to get the code and documentation consistent, preferably before the release.
Stefan
Peter
Dear all,
since Roberto's bug appears to be fixed by Peter's ".priv" (vm_minsize related) fixes (Thanks!), and disabling of vmalloc() (mem_bigsize related) does not appear have any impact on this bug (just tested successfully both with vmalloc() disabled and vmalloc() enabled), there are two ways to finish this release (more or less) quickly:
(1) finish the disabling of vmalloc() in all parts of the code as indicated below (or better: remove all remains completely) --- this might need some extra testing to ensure that it does not have any (significant) impact on performance ...
(2) for now (i.e., in the release branch) re-enable vmalloc(), and then finish the removal of it (incl. extensive testing) in the development trunk for th next release.
Since I'm not completely aware of the impact of vmalloc(), I cannot really make this decision --- the "save" back-up would IMHO be (2).
Any other comments, opinions, suggestions, expertise?
Thanks in advance!
Stefan
On Sun, Feb 10, 2008 at 01:33:04PM +0100, Stefan Manegold wrote:
Peter: Thanks for disabling mem_bigsize / vmalloc().
All: (a) Unfortunately, this does not (yet) solve [ 1872685 ] PF(tijah): HEAPsetmmap: Assertion `p' failed. http://sourceforge.net/tracker/index.php?func=detail&aid=1872685&gro...
But this appears to be more vm_minsize related than mem_bigsize related.
(b) We should check all occurances of mem_bigsize and remove / "hide" those that have become irrelevant with the deactivation of mem_bigsize / vmalloc(): ======== $ cvsfiles * | egrep -v ' |/Tests/' | xargs grep -n --color 'mem_bigsize'
MonetDB4/conf/MonetDB.conf.in:29:# gdk_mem_bigsize & gdk_vm_minsize will be
set/limited to
MonetDB4/conf/MonetDB.conf.in:32:# memory chunks of size >= gdk_mem_bigsize
(in bytes) will be mmaped anonymously
MonetDB4/conf/MonetDB.conf.in:33:#gdk_mem_bigsize=262144 MonetDB5/conf/monetdb5.conf.in:33:# gdk_mem_bigsize & gdk_vm_minsize will be
set/limited to
MonetDB5/conf/monetdb5.conf.in:36:# memory chunks of size >= gdk_mem_bigsize
(in bytes) will be mmaped anonymously
MonetDB5/conf/monetdb5.conf.in:37:#gdk_mem_bigsize=262144 MonetDB/src/common/monet_options.mx:410: set[i].name =
strdup("gdk_mem_bigsize");
MonetDB/src/common/monet_options.py.in:39: # gdk_mem_bigsize &
gdk_vm_minsize will be set/limited to
MonetDB/src/common/monet_options.py.in:42: gdk_mem_bigsize = '256K' MonetDB/src/gdk/gdk_heap.mx:121: if (h->size > GDK_mem_bigsize) { MonetDB/src/gdk/gdk_heap.mx:189: int can_mmap = (h->filename
&& size >= GDK_mem_bigsize);
MonetDB/src/gdk/gdk.mx:2210:gdk_export size_t GDK_mem_bigsize; /* size
after which we use VM rather than heap */
MonetDB/src/gdk/gdk_utils.mx:396:size_t GDK_mem_bigsize = 1 << 20; MonetDB/src/gdk/gdk_utils.mx:964:larger than GDK_mem_bigsize) to anonymous
virtual memory. The seamless
MonetDB/src/gdk/gdk_utils.mx:1054: if (size > GDK_mem_bigsize) { MonetDB/src/gdk/gdk_utils.mx:1191: if (size <= GDK_mem_bigsize) { MonetDB/src/gdk/gdk_utils.mx:1529: if ((p =
GDKgetenv("gdk_mem_bigsize"))) {
MonetDB/src/gdk/gdk_utils.mx:1531: lng max_mem_bigsize =
GDK_mem_maxsize/16;
MonetDB/src/gdk/gdk_utils.mx:1534: GDK_mem_bigsize = (size_t)
MIN(max_mem_bigsize, strtol(p, NULL, 10));
monetweb/Docs/XQuery/MonetDB.conf.texi:115:@item @code{gdk_mem_bigsize}:
minimum size for memory-mapped columns, e.g. @code{262144}.
MonetDB4/src/modules/plain/sys.mx:96:.COMMAND mem_bigsize() : lng =
get_mem_bigsize;
MonetDB4/src/modules/plain/sys.mx:98:.COMMAND mem_bigsize(lng) : void =
set_mem_bigsize;
MonetDB4/src/modules/plain/sys.mx:1430:get_mem_bigsize(lng *num) MonetDB4/src/modules/plain/sys.mx:1432: *num = GDK_mem_bigsize; MonetDB4/src/modules/plain/sys.mx:1437:set_mem_bigsize(lng *num) MonetDB4/src/modules/plain/sys.mx:1439: @:num2sze(mem_bigsize)@ MonetDB4/src/modules/plain/sys.mx:1440: GDK_mem_bigsize = MAX(32768, sze); MonetDB4/src/modules/plain/sys.mx:1462: if (sze < GDK_mem_bigsize) MonetDB4/src/modules/plain/sys.mx:1463: set_mem_bigsize(num); MonetDB4/src/modules/plain/sys.mx:1464: GDK_mem_maxsize =
MAX(GDK_mem_bigsize, sze);
MonetDB4/src/modules/plain/sys.mx:1486: if (sze < GDK_mem_bigsize) MonetDB4/src/modules/plain/sys.mx:1487: set_mem_bigsize(num); MonetDB4/src/modules/plain/sys.mx:1488: GDK_vm_minsize =
MAX(GDK_mem_bigsize, sze);
MonetDB5/src/modules/kernel/status.mx:405:get_mem_bigsize(lng *num) MonetDB5/src/modules/kernel/status.mx:407: *num = GDK_mem_bigsize; MonetDB5/src/modules/kernel/status.mx:412:set_mem_bigsize(lng *num) MonetDB5/src/modules/kernel/status.mx:414: @:num2sze(mem_bigsize)@ MonetDB5/src/modules/kernel/status.mx:415: GDK_mem_bigsize = MAX(32768,
sze);
MonetDB5/src/modules/kernel/status.mx:439: if (sze < GDK_mem_bigsize) MonetDB5/src/modules/kernel/status.mx:440: set_mem_bigsize(num); MonetDB5/src/modules/kernel/status.mx:441: GDK_mem_maxsize =
MAX(GDK_mem_bigsize, sze);
MonetDB5/src/modules/kernel/status.mx:466: if (sze < GDK_mem_bigsize) MonetDB5/src/modules/kernel/status.mx:467: set_mem_bigsize(num); MonetDB5/src/modules/kernel/status.mx:468: GDK_vm_minsize =
MAX(GDK_mem_bigsize, sze);
monetweb/MonetDB/QuickTour/HelloWorld/content.shtml:42:[ "gdk_mem_bigsize",
"262144"========
more tomorrow during MADAM ...
Stefan
On Sun, Feb 10, 2008 at 10:37:54AM +0100, Peter Boncz wrote:
no consequences.
Peter,
What about the consequences in: build/MonetDB/src/gdk/gdk_heap.c build/MonetDB5/src/modules/kernel/status.c build/MonetDB/src/gdk/gdk.h
Peter Boncz wrote:
Update of /cvsroot/monetdb/MonetDB/src/gdk In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv29651
Modified Files: Tag: MonetDB_1-22 gdk_utils.mx Log Message: disable use of vmalloc()
Index: gdk_utils.mx
RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_utils.mx,v retrieving revision 1.206 retrieving revision 1.206.2.1 diff -u -d -r1.206 -r1.206.2.1 --- gdk_utils.mx 14 Jan 2008 10:48:58 -0000 1.206 +++ gdk_utils.mx 9 Feb 2008 15:38:11 -0000 1.206.2.1 @@ -1050,6 +1050,7 @@ #endif } size = (size + 7) & ~7; /* round up to a multiple of eight */ +#if 0 if (size > GDK_mem_bigsize) { size_t newsize = size + sizeof(size_t) + sizeof(size_t); size_t newmax = MAX(*maxsize, newsize); @@ -1064,6 +1065,7 @@ *maxsize = newmax - (sizeof(size_t) + sizeof(size_t)); return (void *) s; } +#endif CHKMEM(size, 0); GDKmalloc_prefixsize(s, size); if (s == NULL) {
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
-- Sjoerd Mullender
participants (3)
-
Peter Boncz -
Sjoerd Mullender -
Stefan Manegold