[Monetdb-developers] [Monetdb-checkins] MonetDB5/src/modules/kernel batcast.mx, Feb2010, 1.31, 1.31.2.1 batstr.mx, Feb2010, 1.68, 1.68.2.1

Martin Kersten Martin.Kersten at cwi.nl
Sat Feb 6 11:52:12 CET 2010


Stefan Manegold wrote:
> On Sat, Feb 06, 2010 at 10:17:54AM +0000, Martin Kersten wrote:
>> Update of /cvsroot/monetdb/MonetDB5/src/modules/kernel
>> In directory sfp-cvsdas-1.v30.ch3.sourceforge.com:/tmp/cvs-serv685
>>
>> Modified Files:
>>       Tag: Feb2010
>> 	batcast.mx batstr.mx 
>> Log Message:
>> All string BATs should be preloaded to avoid random access to disk.
> 
> Preloading their string heap(s) is indeed a good idea (unless they exceeds
> memory size, anyway) due to their inherently random access.
Again, the pre-loading is always limited to the amount of main memory available!
> 
> However, preloading sequentially traversed BUN heaps might just create an
> unnecessarily huge memory footprint, in particular with large BATs.
Again, it preloads items that are needed anyway and the admission policy
'ensures' that all operands together fit in memory unless unavoidable.
The negative effect is a memory probe extra per page as part of the loop.

The preloading gives preference to the vheaps, ensuring random probing
there does not trigger io.
> 
> Hence, we might want to consider extending BATpreload to not uncoditionally
> preload all heaps of a BAT (up to four right now, since we ignore hash
> tables),
> but rather allow selective preload of those heaps that are (a) accessed at
> all, and (b) access randomly (or not accessed sequentially).
Yes, a refinement is possible. For now it marks the places where it should
be considered. I am sure there are more places.

Martin
> 
> I'll look into this later ...
> 
> Stefan
> 
>> Index: batstr.mx
>> ===================================================================
>> RCS file: /cvsroot/monetdb/MonetDB5/src/modules/kernel/batstr.mx,v
>> retrieving revision 1.68
>> retrieving revision 1.68.2.1
>> diff -u -d -r1.68 -r1.68.2.1
>> --- batstr.mx	7 Jan 2010 15:24:26 -0000	1.68
>> +++ batstr.mx	6 Feb 2010 10:17:49 -0000	1.68.2.1
>> @@ -239,6 +239,7 @@
>>  
>>  	bi = bat_iterator(b);
>>  
>> +	BATpreload(b);
>>  	BATloop(b, p, q) {
>>  		ptr h = BUNhead(bi,p);
>>  		x = (@3) BUNtail(bi,p);
>> @@ -275,6 +276,7 @@
>>  
>>  	bi = bat_iterator(b);
>>  
>> +	BATpreload(b);
>>  	BATloop(b, p, q) {
>>  		ptr h = BUNhead(bi,p);
>>  		str y = (str)str_nil, *yp = &y;
>> @@ -322,6 +324,7 @@
>>  	
>>  	if (strcmp(*cst, str_nil) == 0) 
>>  		isnil = 1;
>> +	BATpreload(b);
>>  	BATloop(b, p, q) {
>>  		ptr h = BUNhead(bi, p);
>>  
>> @@ -369,6 +372,8 @@
>>  	lefti = bat_iterator(left);
>>  	righti = bat_iterator(right);
>>  
>> +	BATpreload(left);
>> +	BATpreload(right);
>>  	BATloop(left, p, q) {
>>  		bit y = TRUE;
>>  		ptr h = BUNhead(lefti, p);
>> @@ -411,6 +416,8 @@
>>  	lefti = bat_iterator(left);
>>  	righti = bat_iterator(right);
>>  
>> +	BATpreload(left);
>> +	BATpreload(right);
>>  	BATloop(left, p, q) {
>>  		ptr h = BUNhead(lefti,p);
>>  		ptr tl = BUNtail(lefti,p);
>> @@ -444,6 +451,7 @@
>>  
>>  	lefti = bat_iterator(left);
>>  
>> +	BATpreload(left);
>>  	BATloop(left, p, q) {
>>  		ptr h = BUNhead(lefti,p);
>>  		ptr tl = BUNtail(lefti,p);
>> @@ -476,6 +484,8 @@
>>  	lefti = bat_iterator(left);
>>  	righti = bat_iterator(right);
>>  
>> +	BATpreload(left);
>> +	BATpreload(right);
>>  	BATloop(left, p, q) {
>>  		ptr h = BUNhead(lefti,p);
>>  		str tl = (str) BUNtail(lefti,p);
>> @@ -509,6 +519,7 @@
>>  
>>  	lefti = bat_iterator(left);
>>  
>> +	BATpreload(left);
>>  	BATloop(left, p, q) {
>>  		ptr h = BUNhead(lefti,p);
>>  		str tl = (str) BUNtail(lefti,p);
>> @@ -539,6 +550,7 @@
>>  
>>  	righti = bat_iterator(right);
>>  
>> +	BATpreload(right);
>>  	BATloop(right, p, q) {
>>  		ptr h = BUNhead(righti,p);
>>  		str tr = (str) BUNtail(righti,p);
>> @@ -579,6 +591,7 @@
>>  
>>  	bi = bat_iterator(b);
>>  
>> +	BATpreload(b);
>>  	BATloop(b, p, q) {
>>  		ptr h = BUNhead(bi, p);
>>  		str y = (str)str_nil, *yp = &y;
>> @@ -622,6 +635,7 @@
>>  
>>  	bi = bat_iterator(b);
>>  
>> +	BATpreload(b);
>>  	BATloop(b, p, q) {
>>  		ptr h = BUNhead(bi, p);
>>  		ptr t = BUNtail(bi, p);
>> @@ -666,6 +680,7 @@
>>  	bn->tsorted = b->tsorted;
>>  
>>  	bi = bat_iterator(b);
>> +	BATpreload(b);
>>  	BATloop(b, p, q) {
>>  		ptr h = BUNhead(bi, p);
>>  		str t =  (str) BUNtail(bi, p);
>> @@ -720,6 +735,7 @@
>>  	lefti = bat_iterator(left);
>>  	starti = bat_iterator(start);
>>  	lengthi = bat_iterator(length);
>> +	BATpreload(left);
>>  	BATloop(left, p, q) {
>>  		ptr h = BUNhead(lefti,p);
>>  		str tl = (str) BUNtail(lefti,p);
>> @@ -764,6 +780,7 @@
>>  	bn->tsorted=0; 
>>  
>>  	li = bat_iterator(left);
>> +	BATpreload(left);
>>  	BATloop(left, p, q) {
>>  		ptr h = BUNhead(li,p);
>>  		str tl = (str) BUNtail(li,p);
>>
>> Index: batcast.mx
>> ===================================================================
>> RCS file: /cvsroot/monetdb/MonetDB5/src/modules/kernel/batcast.mx,v
>> retrieving revision 1.31
>> retrieving revision 1.31.2.1
>> diff -u -d -r1.31 -r1.31.2.1
>> --- batcast.mx	7 Jan 2010 15:24:26 -0000	1.31
>> +++ batcast.mx	6 Feb 2010 10:17:47 -0000	1.31.2.1
>> @@ -253,6 +253,7 @@
>>  	@:voidresultBAT(TYPE_ at 1,"batcalc. at 1")@
>>  	bi = bat_iterator(b);
>>  
>> +	BATpreload(b);
>>  	BATloop(b, p, q) {
>>  		ptr v = NULL; 
>>  		int len = 0;
>>
>>
>> ------------------------------------------------------------------------------
>> The Planet: dedicated and managed hosting, cloud storage, colocation
>> Stay online with enterprise data centers and the best network in the business
>> Choose flexible plans and management services without long-term contracts
>> Personal 24x7 support from experience hosting pros just a phone call away.
>> http://p.sf.net/sfu/theplanet-com
>> _______________________________________________
>> Monetdb-checkins mailing list
>> Monetdb-checkins at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
>>
> 





More information about the developers-list mailing list