[MonetDB-users] MonetDB server thread architecture
Hi,
does anyone know whether there is any doc on how the server is implemented in Pthreads, such as how many threads are forked and what job each different thread is responsible for?
Thanks a lot.
xxiao wrote:
Hi,
does anyone know whether there is any doc on how the server is implemented in Pthreads, such as how many threads are forked and what job each different thread is responsible for?
The source is the most complete documentation in this ;) The number of threads depend on the back end used and the advice coming from the frontends.
In the M5/SQL setting each client connection receives its own thread of control. Within a single query plan multiple threads of control may become active when the dataflow optimizer is activated.
There is one thread awaiting connection requests and a separate thread to flush memory.
Please explain what you plan to do in this area.
regards, Martin
Thanks a lot.
Hi, Martin
We are doing some research on how MonetDB will perform on multi-core machines. So, in the first place, we would like to understand how MonetDB utilize multiple threads to process a query.
According to you, M5/SQL with dataflow optimizer forks multiple threads to process a single query plan. To activate dataflow optimizer, do I use flag "--enable-optimize" when I build M5? Do you know what files I may look at to get a overall view of the thread implementation? Thanks a lot.
Xiang
Martin Kersten wrote:
xxiao wrote:
Hi,
does anyone know whether there is any doc on how the server is implemented in Pthreads, such as how many threads are forked and what job each different thread is responsible for?
The source is the most complete documentation in this ;) The number of threads depend on the back end used and the advice coming from the frontends.
In the M5/SQL setting each client connection receives its own thread of control. Within a single query plan multiple threads of control may become active when the dataflow optimizer is activated.
There is one thread awaiting connection requests and a separate thread to flush memory.
Please explain what you plan to do in this area.
regards, Martin
Thanks a lot.
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
xxiao wrote:
Hi, Martin
We are doing some research on how MonetDB will perform on multi-core machines. So, in the first place, we would like to understand how MonetDB utilize multiple threads to process a query.
This is part of an on-going research project we should be aware of?
According to you, M5/SQL with dataflow optimizer forks multiple threads to process a single query plan. To activate dataflow optimizer, do I use flag "--enable-optimize" when I build M5? Do you know what files I may look at to get a overall view of the thread implementation? Thanks a lot.
--enable-optimize drives the parameters of the C-compiler only
The dataflow optimizer is part of CURRENT, it is enabled by default for about four months nowe. This is covered by the file sql_optimizer.mx. The optimizer is src/optimizer/opt_dataflow.mx en supported in the interpreter src/mal/mal_interpreter.mx
You can control the maximum number of threads using the argument --set gdk_nr_threads=N
Xiang
Martin Kersten wrote:
xxiao wrote:
Hi,
does anyone know whether there is any doc on how the server is implemented in Pthreads, such as how many threads are forked and what job each different thread is responsible for?
The source is the most complete documentation in this ;) The number of threads depend on the back end used and the advice coming from the frontends.
In the M5/SQL setting each client connection receives its own thread of control. Within a single query plan multiple threads of control may become active when the dataflow optimizer is activated.
There is one thread awaiting connection requests and a separate thread to flush memory.
Please explain what you plan to do in this area.
regards, Martin
Thanks a lot.
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Use "--set gdk_nr_threads=N", when I invoke mserver5? Can you give me a sample command?
I used the source code checked out in January. Do I need to check out the new source and rebuild Monetdb5
Thanks a lot.
Xiang
Martin Kersten wrote:
xxiao wrote:
Hi, Martin
We are doing some research on how MonetDB will perform on multi-core machines. So, in the first place, we would like to understand how MonetDB utilize multiple threads to process a query.
This is part of an on-going research project we should be aware of?
According to you, M5/SQL with dataflow optimizer forks multiple threads to process a single query plan. To activate dataflow optimizer, do I use flag "--enable-optimize" when I build M5? Do you know what files I may look at to get a overall view of the thread implementation? Thanks a lot.
--enable-optimize drives the parameters of the C-compiler only
The dataflow optimizer is part of CURRENT, it is enabled by default for about four months nowe. This is covered by the file sql_optimizer.mx. The optimizer is src/optimizer/opt_dataflow.mx en supported in the interpreter src/mal/mal_interpreter.mx
You can control the maximum number of threads using the argument --set gdk_nr_threads=N
Xiang
Martin Kersten wrote:
xxiao wrote:
Hi,
does anyone know whether there is any doc on how the server is implemented in Pthreads, such as how many threads are forked and what job each different thread is responsible for?
The source is the most complete documentation in this ;) The number of threads depend on the back end used and the advice coming from the frontends.
In the M5/SQL setting each client connection receives its own thread of control. Within a single query plan multiple threads of control may become active when the dataflow optimizer is activated.
There is one thread awaiting connection requests and a separate thread to flush memory.
Please explain what you plan to do in this area.
regards, Martin
Thanks a lot.
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
xxiao wrote:
Use "--set gdk_nr_threads=N", when I invoke mserver5? Can you give me a sample command?
I used the source code checked out in January. Do I need to check out the new source and rebuild Monetdb5
That Januari code base is way to old for the functions you try to analyse.
Thanks a lot.
Xiang
Martin Kersten wrote:
xxiao wrote:
Hi, Martin
We are doing some research on how MonetDB will perform on multi-core machines. So, in the first place, we would like to understand how MonetDB utilize multiple threads to process a query.
This is part of an on-going research project we should be aware of?
According to you, M5/SQL with dataflow optimizer forks multiple threads to process a single query plan. To activate dataflow optimizer, do I use flag "--enable-optimize" when I build M5? Do you know what files I may look at to get a overall view of the thread implementation? Thanks a lot.
--enable-optimize drives the parameters of the C-compiler only
The dataflow optimizer is part of CURRENT, it is enabled by default for about four months nowe. This is covered by the file sql_optimizer.mx. The optimizer is src/optimizer/opt_dataflow.mx en supported in the interpreter src/mal/mal_interpreter.mx
You can control the maximum number of threads using the argument --set gdk_nr_threads=N
Xiang
Martin Kersten wrote:
xxiao wrote:
Hi,
does anyone know whether there is any doc on how the server is implemented in Pthreads, such as how many threads are forked and what job each different thread is responsible for?
The source is the most complete documentation in this ;) The number of threads depend on the back end used and the advice coming from the frontends.
In the M5/SQL setting each client connection receives its own thread of control. Within a single query plan multiple threads of control may become active when the dataflow optimizer is activated.
There is one thread awaiting connection requests and a separate thread to flush memory.
Please explain what you plan to do in this area.
regards, Martin
Thanks a lot.
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Hi, Martin
I am running MonetDB on Simics simulator. Rebuilding the modules takes very long time. So can I just rebuild Monetdb and Monetdb5 in order to enable the multi-thread dataflow optimizer. Thanks a lot for your inputs.
Xiang
Martin Kersten wrote:
xxiao wrote:
Use "--set gdk_nr_threads=N", when I invoke mserver5? Can you give me a sample command?
I used the source code checked out in January. Do I need to check out the new source and rebuild Monetdb5
That Januari code base is way to old for the functions you try to analyse.
Thanks a lot.
Xiang
Martin Kersten wrote:
xxiao wrote:
Hi, Martin
We are doing some research on how MonetDB will perform on multi-core machines. So, in the first place, we would like to understand how MonetDB utilize multiple threads to process a query.
This is part of an on-going research project we should be aware of?
According to you, M5/SQL with dataflow optimizer forks multiple threads to process a single query plan. To activate dataflow optimizer, do I use flag "--enable-optimize" when I build M5? Do you know what files I may look at to get a overall view of the thread implementation? Thanks a lot.
--enable-optimize drives the parameters of the C-compiler only
The dataflow optimizer is part of CURRENT, it is enabled by default for about four months nowe. This is covered by the file sql_optimizer.mx. The optimizer is src/optimizer/opt_dataflow.mx en supported in the interpreter src/mal/mal_interpreter.mx
You can control the maximum number of threads using the argument --set gdk_nr_threads=N
Xiang
Martin Kersten wrote:
xxiao wrote:
Hi,
does anyone know whether there is any doc on how the server is implemented in Pthreads, such as how many threads are forked and what job each different thread is responsible for?
The source is the most complete documentation in this ;) The number of threads depend on the back end used and the advice coming from the frontends.
In the M5/SQL setting each client connection receives its own thread of control. Within a single query plan multiple threads of control may become active when the dataflow optimizer is activated.
There is one thread awaiting connection requests and a separate thread to flush memory.
Please explain what you plan to do in this area.
regards, Martin
Thanks a lot.
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
xxiao wrote:
Hi, Martin
I am running MonetDB on Simics simulator. Rebuilding the modules takes very long time. So can I just rebuild Monetdb and Monetdb5 in order to enable the multi-thread dataflow optimizer. Thanks a lot for your inputs.
you can obtain the nightly tar ball which contains the C-code files the client code is not absolutely necessary, provided you run experiments from the mserver5 console, otherwise you need at leasts mclient you need the SQL tree.
Xiang
Martin Kersten wrote:
xxiao wrote:
Use "--set gdk_nr_threads=N", when I invoke mserver5? Can you give me a sample command?
I used the source code checked out in January. Do I need to check out the new source and rebuild Monetdb5
That Januari code base is way to old for the functions you try to analyse.
Thanks a lot.
Xiang
Martin Kersten wrote:
xxiao wrote:
Hi, Martin
We are doing some research on how MonetDB will perform on multi-core machines. So, in the first place, we would like to understand how MonetDB utilize multiple threads to process a query.
This is part of an on-going research project we should be aware of?
According to you, M5/SQL with dataflow optimizer forks multiple threads to process a single query plan. To activate dataflow optimizer, do I use flag "--enable-optimize" when I build M5? Do you know what files I may look at to get a overall view of the thread implementation? Thanks a lot.
--enable-optimize drives the parameters of the C-compiler only
The dataflow optimizer is part of CURRENT, it is enabled by default for about four months nowe. This is covered by the file sql_optimizer.mx. The optimizer is src/optimizer/opt_dataflow.mx en supported in the interpreter src/mal/mal_interpreter.mx
You can control the maximum number of threads using the argument --set gdk_nr_threads=N
Xiang
Martin Kersten wrote:
xxiao wrote:
> Hi, > > does anyone know whether there is any doc on how the server is > implemented > in Pthreads, such as how many threads are forked and what job each > different > thread is responsible for? > > > > > The source is the most complete documentation in this ;) The number of threads depend on the back end used and the advice coming from the frontends.
In the M5/SQL setting each client connection receives its own thread of control. Within a single query plan multiple threads of control may become active when the dataflow optimizer is activated.
There is one thread awaiting connection requests and a separate thread to flush memory.
Please explain what you plan to do in this area.
regards, Martin
> Thanks a lot. > > >
>
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
is it possible to know how many threads are used for a given query at run-time? thanks,
Xiang
Martin Kersten wrote:
xxiao wrote:
Hi, Martin
I am running MonetDB on Simics simulator. Rebuilding the modules takes very long time. So can I just rebuild Monetdb and Monetdb5 in order to enable the multi-thread dataflow optimizer. Thanks a lot for your inputs.
you can obtain the nightly tar ball which contains the C-code files the client code is not absolutely necessary, provided you run experiments from the mserver5 console, otherwise you need at leasts mclient you need the SQL tree.
Xiang
Martin Kersten wrote:
xxiao wrote:
Use "--set gdk_nr_threads=N", when I invoke mserver5? Can you give me a sample command?
I used the source code checked out in January. Do I need to check out the new source and rebuild Monetdb5
That Januari code base is way to old for the functions you try to analyse.
Thanks a lot.
Xiang
Martin Kersten wrote:
xxiao wrote:
Hi, Martin
We are doing some research on how MonetDB will perform on multi-core machines. So, in the first place, we would like to understand how MonetDB utilize multiple threads to process a query.
This is part of an on-going research project we should be aware of?
According to you, M5/SQL with dataflow optimizer forks multiple threads to process a single query plan. To activate dataflow optimizer, do I use flag "--enable-optimize" when I build M5? Do you know what files I may look at to get a overall view of the thread implementation? Thanks a lot.
--enable-optimize drives the parameters of the C-compiler only
The dataflow optimizer is part of CURRENT, it is enabled by default for about four months nowe. This is covered by the file sql_optimizer.mx. The optimizer is src/optimizer/opt_dataflow.mx en supported in the interpreter src/mal/mal_interpreter.mx
You can control the maximum number of threads using the argument --set gdk_nr_threads=N
Xiang
Martin Kersten wrote:
> xxiao wrote: > > > >> Hi, >> >> does anyone know whether there is any doc on how the server is >> implemented >> in Pthreads, such as how many threads are forked and what job each >> different >> thread is responsible for? >> >> >> >> >> > The source is the most complete documentation in this ;) > The number of threads depend on the back end used and > the advice coming from the frontends. > > In the M5/SQL setting each client connection receives > its own thread of control. Within a single query plan > multiple threads of control may become active when > the dataflow optimizer is activated. > > There is one thread awaiting connection requests > and a separate thread to flush memory. > > Please explain what you plan to do in this area. > > regards, Martin > > > >> Thanks a lot. >> >> >> >> > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's > challenge > Build the coolest Linux based applications with Moblin SDK & win > great > prizes > Grand prize is a trip for two to an Open Source event anywhere in > the > world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > MonetDB-users mailing list > MonetDB-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-users > > > > >
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
xxiao wrote:
is it possible to know how many threads are used for a given query at run-time? thanks,
This depends on the dataflow dependencies and will fluctuate during the query execution. The MAL trace flags give an indication what individual threads are doing.
Xiang
Martin Kersten wrote:
xxiao wrote:
Hi, Martin
I am running MonetDB on Simics simulator. Rebuilding the modules takes very long time. So can I just rebuild Monetdb and Monetdb5 in order to enable the multi-thread dataflow optimizer. Thanks a lot for your inputs.
you can obtain the nightly tar ball which contains the C-code files the client code is not absolutely necessary, provided you run experiments from the mserver5 console, otherwise you need at leasts mclient you need the SQL tree.
Xiang
Martin Kersten wrote:
xxiao wrote:
Use "--set gdk_nr_threads=N", when I invoke mserver5? Can you give me a sample command?
I used the source code checked out in January. Do I need to check out the new source and rebuild Monetdb5
That Januari code base is way to old for the functions you try to analyse.
Thanks a lot.
Xiang
Martin Kersten wrote:
xxiao wrote:
> Hi, Martin > > We are doing some research on how MonetDB will perform on multi-core > machines. So, in the first place, we would like to understand how > MonetDB > utilize multiple threads to process a query. > > > This is part of an on-going research project we should be aware of?
> > > According to you, M5/SQL with dataflow optimizer forks multiple > threads > to > process a single query plan. To activate dataflow optimizer, do I use > flag > "--enable-optimize" when I build M5? Do you know what files I may > look > at > to > get a overall view of the thread implementation? Thanks a lot. > > > > --enable-optimize drives the parameters of the C-compiler only
The dataflow optimizer is part of CURRENT, it is enabled by default for about four months nowe. This is covered by the file sql_optimizer.mx. The optimizer is src/optimizer/opt_dataflow.mx en supported in the interpreter src/mal/mal_interpreter.mx
You can control the maximum number of threads using the argument --set gdk_nr_threads=N
> Xiang > > > Martin Kersten wrote: > > > > >> xxiao wrote: >> >> >> >> >>> Hi, >>> >>> does anyone know whether there is any doc on how the server is >>> implemented >>> in Pthreads, such as how many threads are forked and what job each >>> different >>> thread is responsible for? >>> >>> >>> >>> >>> >>> >> The source is the most complete documentation in this ;) >> The number of threads depend on the back end used and >> the advice coming from the frontends. >> >> In the M5/SQL setting each client connection receives >> its own thread of control. Within a single query plan >> multiple threads of control may become active when >> the dataflow optimizer is activated. >> >> There is one thread awaiting connection requests >> and a separate thread to flush memory. >> >> Please explain what you plan to do in this area. >> >> regards, Martin >> >> >> >> >>> Thanks a lot. >>> >>> >>> >>> >>> >> ------------------------------------------------------------------------- >> This SF.Net email is sponsored by the Moblin Your Move Developer's >> challenge >> Build the coolest Linux based applications with Moblin SDK & win >> great >> prizes >> Grand prize is a trip for two to an Open Source event anywhere in >> the >> world >> http://moblin-contest.org/redirect.php?banner_id=100&url=/ >> _______________________________________________ >> MonetDB-users mailing list >> MonetDB-users@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/monetdb-users >> >> >> >> >> >> > > >
>
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
participants (2)
-
Martin Kersten -
xxiao