MonetDB: Feb2013 - Limit the number of concurrent queries

Stefan Manegold Stefan.Manegold at cwi.nl
Fri Jan 4 13:06:41 CET 2013


> Changeset: e79b2edeba71 for MonetDB
> URL:
> http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=e79b2edeba71
> Modified Files:
> 	monetdb5/mal/mal.c
> 	monetdb5/mal/mal.h
> 	monetdb5/mal/mal_interpreter.c
> Branch: Feb2013
> Log Message:
> 
> Limit the number of concurrent queries
> The total throughput for small (and large) queries is
> improved if we limit the number of concurrent queries
> being executed. This patch is geared at SQL, wrapping
> callMAL with a semaphore to limit the level of parallelism
> to number of physical cores.
> 
> 
> diffs (59 lines):
> 
> diff --git a/monetdb5/mal/mal.c b/monetdb5/mal/mal.c
> --- a/monetdb5/mal/mal.c
> +++ b/monetdb5/mal/mal.c
> @@ -195,6 +195,7 @@ MT_Lock     mal_remoteLock;
>  MT_Lock  	mal_profileLock ;
>  MT_Lock     mal_copyLock;
>  MT_Lock     mal_delayLock;
> +MT_Sema		mal_parallelism;
>  /*
>   * Initialization of the MAL context
>   * The compiler directive STRUCT_ALIGNED tells that the
> @@ -233,6 +234,7 @@ int mal_init(void){
>  	MT_lock_init( &mal_profileLock, "mal_profileLock");
>  	MT_lock_init( &mal_copyLock, "mal_copyLock");
>  	MT_lock_init( &mal_delayLock, "mal_delayLock");
> +	MT_sema_init( &mal_parallelism, (GDKnr_threads ? GDKnr_threads/2 : 4), "mal_parallelism");
                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

With GDKnr_threads == 1 (single threaded execution or single-core system),
this results in mal_parallelism == 0,
which in turn results in all SQL tests failing (hanging until timeout) with `Mtest.py -n1` (single-threaded).

Stefan

>  
>  	tstAligned();
>  	MCinit();
> diff --git a/monetdb5/mal/mal.h b/monetdb5/mal/mal.h
> --- a/monetdb5/mal/mal.h
> +++ b/monetdb5/mal/mal.h
> @@ -118,6 +118,7 @@ mal_export MT_Lock  mal_remoteLock;
>  mal_export MT_Lock  mal_profileLock ;
>  mal_export MT_Lock  mal_copyLock ;
>  mal_export MT_Lock  mal_delayLock ;
> +mal_export MT_Sema	mal_parallelism;
>  
>  
>  mal_export int mal_init(void);
> diff --git a/monetdb5/mal/mal_interpreter.c
> b/monetdb5/mal/mal_interpreter.c
> --- a/monetdb5/mal/mal_interpreter.c
> +++ b/monetdb5/mal/mal_interpreter.c
> @@ -479,7 +479,13 @@ callMAL(Client cntxt, MalBlkPtr mb, MalS
>  	ValPtr lhs;
>  	InstrPtr pci = getInstrPtr(mb, 0);
>  	RuntimeProfileRecord runtimeProfile;
> -
> +
> +/*
> + * Control the level of parallelism. The maximum number of
> concurrent MAL plans
> + * is determined by an environment variable. It is initially set
> equal to the
> + * number of cores, which may be too coarse.
> + */
> +	MT_sema_down(&mal_parallelism,"mal_parallelism");
>  	runtimeProfileInit(mb, &runtimeProfile, cntxt->flags & memoryFlag);
>  #ifdef DEBUG_CALLMAL
>  	mnstr_printf(cntxt->fdout, "callMAL\n");
> @@ -518,8 +524,10 @@ callMAL(Client cntxt, MalBlkPtr mb, MalS
>  	case PATcall:
>  	case CMDcall:
>  	default:
> +		MT_sema_up(&mal_parallelism,"mal_parallelism");
>  		throw(MAL, "mal.interpreter", RUNTIME_UNKNOWN_INSTRUCTION);
>  	}
> +	MT_sema_up(&mal_parallelism,"mal_parallelism");
>  	if (cntxt->qtimeout && time(NULL) - stk->clock.tv_usec >
>  	cntxt->qtimeout)
>  		throw(MAL, "mal.interpreter", RUNTIME_QRY_TIMEOUT);
>  	return ret;
> _______________________________________________
> checkin-list mailing list
> checkin-list at monetdb.org
> http://mail.monetdb.org/mailman/listinfo/checkin-list
> 
_______________________________________________
developers-list mailing list
developers-list at monetdb.org
http://mail.monetdb.org/mailman/listinfo/developers-list



More information about the developers-list mailing list