MonetDB: Feb2013 - Limit the number of concurrent queries

Stefan Manegold Stefan.Manegold at cwi.nl
Fri Jan 4 13:19:54 CET 2013


> > Changeset: e79b2edeba71 for MonetDB
> > URL:
> > http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=e79b2edeba71
> > Modified Files:
> > 	monetdb5/mal/mal.c
> > 	monetdb5/mal/mal.h
> > 	monetdb5/mal/mal_interpreter.c
> > Branch: Feb2013
> > Log Message:
> > 
> > Limit the number of concurrent queries
> > The total throughput for small (and large) queries is
> > improved if we limit the number of concurrent queries
> > being executed. This patch is geared at SQL, wrapping
> > callMAL with a semaphore to limit the level of parallelism
> > to number of physical cores.
> > 
> > 
> > diffs (59 lines):
> > 
> > diff --git a/monetdb5/mal/mal.c b/monetdb5/mal/mal.c
> > --- a/monetdb5/mal/mal.c
> > +++ b/monetdb5/mal/mal.c
> > @@ -195,6 +195,7 @@ MT_Lock     mal_remoteLock;
> >  MT_Lock  	mal_profileLock ;
> >  MT_Lock     mal_copyLock;
> >  MT_Lock     mal_delayLock;
> > +MT_Sema		mal_parallelism;
> >  /*
> >   * Initialization of the MAL context
> >   * The compiler directive STRUCT_ALIGNED tells that the
> > @@ -233,6 +234,7 @@ int mal_init(void){
> >  	MT_lock_init( &mal_profileLock, "mal_profileLock");
> >  	MT_lock_init( &mal_copyLock, "mal_copyLock");
> >  	MT_lock_init( &mal_delayLock, "mal_delayLock");
> > +	MT_sema_init( &mal_parallelism, (GDKnr_threads ? GDKnr_threads/2 : 4), "mal_parallelism");
>                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> 
> With GDKnr_threads == 1 (single threaded execution or single-core
> system),
> this results in mal_parallelism == 0,
> which in turn results in all SQL tests failing (hanging until
> timeout) with `Mtest.py -n1` (single-threaded).

Maybe you intended one of (I cannot guess which one):

	MT_sema_init( &mal_parallelism, (GDKnr_threads > 8 ? GDKnr_threads/2 : 4), "mal_parallelism");

	MT_sema_init( &mal_parallelism, (GDKnr_threads > 4 ? GDKnr_threads/2 : 2), "mal_parallelism");

	MT_sema_init( &mal_parallelism, (GDKnr_threads > 2 ? GDKnr_threads/2 : 1), "mal_parallelism");

Stefan

> Stefan
> 
> >  
> >  	tstAligned();
> >  	MCinit();
> > diff --git a/monetdb5/mal/mal.h b/monetdb5/mal/mal.h
> > --- a/monetdb5/mal/mal.h
> > +++ b/monetdb5/mal/mal.h
> > @@ -118,6 +118,7 @@ mal_export MT_Lock  mal_remoteLock;
> >  mal_export MT_Lock  mal_profileLock ;
> >  mal_export MT_Lock  mal_copyLock ;
> >  mal_export MT_Lock  mal_delayLock ;
> > +mal_export MT_Sema	mal_parallelism;
> >  
> >  
> >  mal_export int mal_init(void);
> > diff --git a/monetdb5/mal/mal_interpreter.c
> > b/monetdb5/mal/mal_interpreter.c
> > --- a/monetdb5/mal/mal_interpreter.c
> > +++ b/monetdb5/mal/mal_interpreter.c
> > @@ -479,7 +479,13 @@ callMAL(Client cntxt, MalBlkPtr mb, MalS
> >  	ValPtr lhs;
> >  	InstrPtr pci = getInstrPtr(mb, 0);
> >  	RuntimeProfileRecord runtimeProfile;
> > -
> > +
> > +/*
> > + * Control the level of parallelism. The maximum number of
> > concurrent MAL plans
> > + * is determined by an environment variable. It is initially set
> > equal to the
> > + * number of cores, which may be too coarse.
> > + */
> > +	MT_sema_down(&mal_parallelism,"mal_parallelism");
> >  	runtimeProfileInit(mb, &runtimeProfile, cntxt->flags &
> >  	memoryFlag);
> >  #ifdef DEBUG_CALLMAL
> >  	mnstr_printf(cntxt->fdout, "callMAL\n");
> > @@ -518,8 +524,10 @@ callMAL(Client cntxt, MalBlkPtr mb, MalS
> >  	case PATcall:
> >  	case CMDcall:
> >  	default:
> > +		MT_sema_up(&mal_parallelism,"mal_parallelism");
> >  		throw(MAL, "mal.interpreter", RUNTIME_UNKNOWN_INSTRUCTION);
> >  	}
> > +	MT_sema_up(&mal_parallelism,"mal_parallelism");
> >  	if (cntxt->qtimeout && time(NULL) - stk->clock.tv_usec >
> >  	cntxt->qtimeout)
> >  		throw(MAL, "mal.interpreter", RUNTIME_QRY_TIMEOUT);
> >  	return ret;
> > _______________________________________________
> > checkin-list mailing list
> > checkin-list at monetdb.org
> > http://mail.monetdb.org/mailman/listinfo/checkin-list
> > 
> _______________________________________________
> developers-list mailing list
> developers-list at monetdb.org
> http://mail.monetdb.org/mailman/listinfo/developers-list
> 
_______________________________________________
developers-list mailing list
developers-list at monetdb.org
http://mail.monetdb.org/mailman/listinfo/developers-list



More information about the developers-list mailing list