[Monetdb-developers] [Monetdb-checkins] MonetDB5/src/modules/kernel batcast.mx, Feb2010, 1.31, 1.31.2.1 batstr.mx, Feb2010, 1.68, 1.68.2.1

Stefan Manegold Stefan.Manegold at cwi.nl
Sat Feb 6 11:42:07 CET 2010


On Sat, Feb 06, 2010 at 10:17:54AM +0000, Martin Kersten wrote:
> Update of /cvsroot/monetdb/MonetDB5/src/modules/kernel
> In directory sfp-cvsdas-1.v30.ch3.sourceforge.com:/tmp/cvs-serv685
> 
> Modified Files:
>       Tag: Feb2010
> 	batcast.mx batstr.mx 
> Log Message:
> All string BATs should be preloaded to avoid random access to disk.

Preloading their string heap(s) is indeed a good idea (unless they exceeds
memory size, anyway) due to their inherently random access.

However, preloading sequentially traversed BUN heaps might just create an
unnecessarily huge memory footprint, in particular with large BATs.

Hence, we might want to consider extending BATpreload to not uncoditionally
preload all heaps of a BAT (up to four right now, since we ignore hash
tables),
but rather allow selective preload of those heaps that are (a) accessed at
all, and (b) access randomly (or not accessed sequentially).

I'll look into this later ...

Stefan

> 
> Index: batstr.mx
> ===================================================================
> RCS file: /cvsroot/monetdb/MonetDB5/src/modules/kernel/batstr.mx,v
> retrieving revision 1.68
> retrieving revision 1.68.2.1
> diff -u -d -r1.68 -r1.68.2.1
> --- batstr.mx	7 Jan 2010 15:24:26 -0000	1.68
> +++ batstr.mx	6 Feb 2010 10:17:49 -0000	1.68.2.1
> @@ -239,6 +239,7 @@
>  
>  	bi = bat_iterator(b);
>  
> +	BATpreload(b);
>  	BATloop(b, p, q) {
>  		ptr h = BUNhead(bi,p);
>  		x = (@3) BUNtail(bi,p);
> @@ -275,6 +276,7 @@
>  
>  	bi = bat_iterator(b);
>  
> +	BATpreload(b);
>  	BATloop(b, p, q) {
>  		ptr h = BUNhead(bi,p);
>  		str y = (str)str_nil, *yp = &y;
> @@ -322,6 +324,7 @@
>  	
>  	if (strcmp(*cst, str_nil) == 0) 
>  		isnil = 1;
> +	BATpreload(b);
>  	BATloop(b, p, q) {
>  		ptr h = BUNhead(bi, p);
>  
> @@ -369,6 +372,8 @@
>  	lefti = bat_iterator(left);
>  	righti = bat_iterator(right);
>  
> +	BATpreload(left);
> +	BATpreload(right);
>  	BATloop(left, p, q) {
>  		bit y = TRUE;
>  		ptr h = BUNhead(lefti, p);
> @@ -411,6 +416,8 @@
>  	lefti = bat_iterator(left);
>  	righti = bat_iterator(right);
>  
> +	BATpreload(left);
> +	BATpreload(right);
>  	BATloop(left, p, q) {
>  		ptr h = BUNhead(lefti,p);
>  		ptr tl = BUNtail(lefti,p);
> @@ -444,6 +451,7 @@
>  
>  	lefti = bat_iterator(left);
>  
> +	BATpreload(left);
>  	BATloop(left, p, q) {
>  		ptr h = BUNhead(lefti,p);
>  		ptr tl = BUNtail(lefti,p);
> @@ -476,6 +484,8 @@
>  	lefti = bat_iterator(left);
>  	righti = bat_iterator(right);
>  
> +	BATpreload(left);
> +	BATpreload(right);
>  	BATloop(left, p, q) {
>  		ptr h = BUNhead(lefti,p);
>  		str tl = (str) BUNtail(lefti,p);
> @@ -509,6 +519,7 @@
>  
>  	lefti = bat_iterator(left);
>  
> +	BATpreload(left);
>  	BATloop(left, p, q) {
>  		ptr h = BUNhead(lefti,p);
>  		str tl = (str) BUNtail(lefti,p);
> @@ -539,6 +550,7 @@
>  
>  	righti = bat_iterator(right);
>  
> +	BATpreload(right);
>  	BATloop(right, p, q) {
>  		ptr h = BUNhead(righti,p);
>  		str tr = (str) BUNtail(righti,p);
> @@ -579,6 +591,7 @@
>  
>  	bi = bat_iterator(b);
>  
> +	BATpreload(b);
>  	BATloop(b, p, q) {
>  		ptr h = BUNhead(bi, p);
>  		str y = (str)str_nil, *yp = &y;
> @@ -622,6 +635,7 @@
>  
>  	bi = bat_iterator(b);
>  
> +	BATpreload(b);
>  	BATloop(b, p, q) {
>  		ptr h = BUNhead(bi, p);
>  		ptr t = BUNtail(bi, p);
> @@ -666,6 +680,7 @@
>  	bn->tsorted = b->tsorted;
>  
>  	bi = bat_iterator(b);
> +	BATpreload(b);
>  	BATloop(b, p, q) {
>  		ptr h = BUNhead(bi, p);
>  		str t =  (str) BUNtail(bi, p);
> @@ -720,6 +735,7 @@
>  	lefti = bat_iterator(left);
>  	starti = bat_iterator(start);
>  	lengthi = bat_iterator(length);
> +	BATpreload(left);
>  	BATloop(left, p, q) {
>  		ptr h = BUNhead(lefti,p);
>  		str tl = (str) BUNtail(lefti,p);
> @@ -764,6 +780,7 @@
>  	bn->tsorted=0; 
>  
>  	li = bat_iterator(left);
> +	BATpreload(left);
>  	BATloop(left, p, q) {
>  		ptr h = BUNhead(li,p);
>  		str tl = (str) BUNtail(li,p);
> 
> Index: batcast.mx
> ===================================================================
> RCS file: /cvsroot/monetdb/MonetDB5/src/modules/kernel/batcast.mx,v
> retrieving revision 1.31
> retrieving revision 1.31.2.1
> diff -u -d -r1.31 -r1.31.2.1
> --- batcast.mx	7 Jan 2010 15:24:26 -0000	1.31
> +++ batcast.mx	6 Feb 2010 10:17:47 -0000	1.31.2.1
> @@ -253,6 +253,7 @@
>  	@:voidresultBAT(TYPE_ at 1,"batcalc. at 1")@
>  	bi = bat_iterator(b);
>  
> +	BATpreload(b);
>  	BATloop(b, p, q) {
>  		ptr v = NULL; 
>  		int len = 0;
> 
> 
> ------------------------------------------------------------------------------
> The Planet: dedicated and managed hosting, cloud storage, colocation
> Stay online with enterprise data centers and the best network in the business
> Choose flexible plans and management services without long-term contracts
> Personal 24x7 support from experience hosting pros just a phone call away.
> http://p.sf.net/sfu/theplanet-com
> _______________________________________________
> Monetdb-checkins mailing list
> Monetdb-checkins at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
> 

-- 
| Dr. Stefan Manegold | mailto:Stefan.Manegold at cwi.nl |
| CWI,  P.O.Box 94079 | http://www.cwi.nl/~manegold/  |
| 1090 GB Amsterdam   | Tel.: +31 (20) 592-4212       |
| The Netherlands     | Fax : +31 (20) 592-4199       |




More information about the developers-list mailing list