[Monetdb-developers] [Monetdb-checkins] MonetDB/src/gdk gdk_storage.mx, Feb2010, 1.149.2.5, 1.149.2.6

Martin Kersten Martin.Kersten at cwi.nl
Sun Jan 31 19:01:22 CET 2010


Stefan Manegold wrote:
> On Sun, Jan 31, 2010 at 04:43:29PM +0000, Martin Kersten wrote:
>> Update of /cvsroot/monetdb/MonetDB/src/gdk
>> In directory sfp-cvsdas-1.v30.ch3.sourceforge.com:/tmp/cvs-serv28035
>>
>> Modified Files:
>>       Tag: Feb2010
>> 	gdk_storage.mx 
>> Log Message:
>> A pity that the first code was not properly checked, because it did not
>> even tried to force the preload by memory touching properly.
>>
> 
> As per request (though admittedly belated ...):
nothing is too late and indeed this was the previous code.
It could be that the BATs you touch will be filled, then the size
is the option.
> 
> Wouldn't it be sufficient (and faster and less memory consuming) to preload
The total amount of memory consumed is controlled by the dataflow admission
policy. See the respective check-ins last two weeks. Monitoring the run
of Q10 shows that with the current setting we stay properly within the memory
bounds most of the time.

> only that part of the heaps that is indeed used (i.e., holds useful/relevant
> data) instead of preloading the whole (possibly consciously over-allocated)
> heaps?
> 
> I.e., replacing ...heap->size by ...heap->free
> 
> (as it was in the original code, only that the original code incorrecly
> assumed ...heap->free to be an absolute address instead of a relative byte
> offset ...)
original code incorrectly used b->hheap and b->theap, which also need to
be preloaded upon need.
Martin
> 
> Stefan
> 
>> Index: gdk_storage.mx
>> ===================================================================
>> RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_storage.mx,v
>> retrieving revision 1.149.2.5
>> retrieving revision 1.149.2.6
>> diff -u -d -r1.149.2.5 -r1.149.2.6
>> --- gdk_storage.mx	31 Jan 2010 15:50:19 -0000	1.149.2.5
>> +++ gdk_storage.mx	31 Jan 2010 16:43:27 -0000	1.149.2.6
>> @@ -703,28 +703,32 @@
>>  @c
>>  size_t 
>>  BATpreload_(BAT *b) {
>> -	size_t dummy = 0, *i;
>> +	size_t dummy = 0, *i, *limit;
>>  	size_t step = MT_pagesize()/sizeof(size_t);
>>  	size_t pages = (size_t) (0.8 * MT_npages());
>>  
>> -	if ( b->T->vheap && b->T->vheap->base ){
>> -		IODEBUG THRprintf(GDKout,"#BATpreload(%s): T->vheap\n", BATgetId(b));
>> -		for ( i = (size_t *) b->T->vheap->base; i < (size_t *) b->T->vheap->free  && pages > 0; i+= step, pages--)
>> +	if ( b->T->vheap && b->T->vheap->base  && pages){
>> +		IODEBUG THRprintf(GDKout,"#BATpreloaded(%s): T->vheap\n", BATgetId(b));
>> +		limit = (size_t *) (b->T->vheap->base + b->T->vheap->size);
>> +		for ( i = (size_t *) b->T->vheap->base; i < limit  && pages > 0; i+= step, pages--)
>>  			dummy += *i;
>>  	}
>> -	if ( b->H->vheap && b->H->vheap->base ){
>> -		IODEBUG THRprintf(GDKout,"#BATpreload(%s): H->vheap\n", BATgetId(b));
>> -		for ( i = (size_t *) b->H->vheap->base; i < (size_t *) b->H->vheap->free && pages > 0 ; i+= step, pages--)
>> +	if ( b->T->heap.base && pages ){
>> +		IODEBUG THRprintf(GDKout,"#BATpreload(%s): T->heap\n", BATgetId(b));
>> +		limit = (size_t *) (b->T->heap.base + b->T->heap.size);
>> +		for ( i = (size_t *) b->T->heap.base; i < limit && pages > 0; i+= step, pages--)
>>  			dummy += *i;
>>  	}
>> -	if ( b->hheap && b->hheap->base ){
>> -		IODEBUG THRprintf(GDKout,"#BATpreload(%s): hheap\n", BATgetId(b));
>> -		for ( i = (size_t *) b->hheap->base; i <  (size_t *)b->hheap->free && pages > 0 ; i+= step, pages--)
>> +	if ( b->H->heap.base && pages ){
>> +		IODEBUG THRprintf(GDKout,"#BATpreload(%s): H->heap\n", BATgetId(b));
>> +		limit = (size_t *) (b->H->heap.base + b->H->heap.size);
>> +		for ( i = (size_t *) b->H->heap.base; i <  limit && pages > 0 ; i+= step, pages--)
>>  			dummy += *i;
>>  	}
>> -	if ( b->theap && b->theap->base ){
>> -		IODEBUG THRprintf(GDKout,"#BATpreload(%s): theap\n", BATgetId(b));
>> -		for ( i = (size_t *) b->theap->base; i < (size_t *) b->theap->free  && pages > 0; i+= step, pages--)
>> +	if ( b->H->vheap && b->H->vheap->base && pages ){
>> +		IODEBUG THRprintf(GDKout,"#BATpreload(%s): H->vheap\n", BATgetId(b));
>> +		limit = (size_t *) (b->H->vheap->base + b->H->vheap->size);
>> +		for ( i = (size_t *) b->H->vheap->base; i < limit && pages > 0 ; i+= step, pages--)
>>  			dummy += *i;
>>  	}
>>  	return dummy;
>>
>>
>> ------------------------------------------------------------------------------
>> The Planet: dedicated and managed hosting, cloud storage, colocation
>> Stay online with enterprise data centers and the best network in the business
>> Choose flexible plans and management services without long-term contracts
>> Personal 24x7 support from experience hosting pros just a phone call away.
>> http://p.sf.net/sfu/theplanet-com
>> _______________________________________________
>> Monetdb-checkins mailing list
>> Monetdb-checkins at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
>>
>>
>>
>>
> 





More information about the developers-list mailing list