Hi Team,
Has anyone tried compiling/running MonetDB under LLVM/CLANG compiler framework please?
Normally the use case is:
1) Produce set of llvm bitcode files *.bc from C code
2) Apply LLVM Pass to instrument *.bc files through llvm 'OPT' compiler tool
3) Link all *.bc files and produce executables of monetdb
I am going to try monetdb under llvm. If anyone hasn't done it before then I may ask for the initial pointers to compile monedb under llvm instrumentation.
Thanks.
Kind Regards, Ahmad
_______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
Hi Ahmad
a simple `.../configure CC=clang ...` used to work fine, though we haven't tested it recently ...
Stefan
----- Original Message -----
Hi Team,
Has anyone tried compiling/running MonetDB under LLVM/CLANG compiler framework please?
Normally the use case is:
Produce set of llvm bitcode files *.bc from C codeApply LLVM Pass to instrument *.bc files through llvm 'OPT'compiler tool
Link all *.bc files and produce executables of monetdbI am going to try monetdb under llvm. If anyone hasn't done it before then I may ask for the initial pointers to compile monedb under llvm instrumentation.
Thanks.
Kind Regards, Ahmad
users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
_______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
Hi Stefan,
Yes Fabian told me this earlier. I tried that option but it only uses 'clang' to compile the source code. It doesn't produce LLVM bitcode files. It produces standard object files.
If I pass CFLAGS=-emit-llvm -c, then I get config error:
./configure CC=clang CFLAGS="-emit-llvm -c"
checking for gcc... clang checking for C compiler default output file name... configure: error: in `/data/ahmad/test/MonetDB-11.13.5': configure: error: C compiler cannot create executables See `config.log' for more details.
Config.log says:
configure:3976: clang -v >&5 clang version 3.1 (tags/RELEASE_31/final 170126) Target: x86_64-unknown-linux-gnu Thread model: posix configure:3980: $? = 0 configure:3987: clang -V >&5 clang: error: argument to '-V' is missing (expected 1 value)
Kind Regards, Ahmad
-----Original Message----- From: users-list-bounces+ahmad.hassan=sap.com@monetdb.org [mailto:users-list-bounces+ahmad.hassan=sap.com@monetdb.org] On Behalf Of Stefan Manegold Sent: 10 January 2013 16:32 To: Communication channel for MonetDB users Subject: Re: MonetDB under LLVM
Hi Ahmad
a simple `.../configure CC=clang ...` used to work fine, though we haven't tested it recently ...
Stefan
----- Original Message -----
Hi Team,
Has anyone tried compiling/running MonetDB under LLVM/CLANG compiler framework please?
Normally the use case is:
Produce set of llvm bitcode files *.bc from C codeApply LLVM Pass to instrument *.bc files through llvm 'OPT'compiler tool
Link all *.bc files and produce executables of monetdbI am going to try monetdb under llvm. If anyone hasn't done it before then I may ask for the initial pointers to compile monedb under llvm instrumentation.
Thanks.
Kind Regards, Ahmad
users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
_______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list _______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
On 10-01-2013 17:23:27 +0000, Hassan, Ahmad wrote:
Yes Fabian told me this earlier. I tried that option but it only uses 'clang' to compile the source code. It doesn't produce LLVM bitcode files. It produces standard object files.
A quick search with your favourite search engine would have told you that you'll need at least some intrusive changes to the build-system, and a different way of building.
Since you seem not to have reached that conclusion, I doubt whether you are the one to actually find out what you need to get it working. It is not of a concern to MonetDB, at the moment.
Hi Team,
I have modified the monetdb build system to generate executable using LLVM Bitcode and added instrumentation rather than generating exe directly from Object files. At the moment the changes are limited to following executables:
monetdbd: tools/merovingian/daemon mserver5: tools/mserver
I have a question please. I want to analyse the storage manager of monetdb that implements the data structures for holding Table's data and Indexes. More precisely, I want to analyse all the data structures involved during Query processing explicitly i.e indexes, data structures holding tables data in main memory. May I ask which sub-tree of Monet source tree deals with that. Being naïve in monetdb internals, I assumed that monetdbd and mserver5 would be relevant because they are server side part of the code.
If I get any documentation on that, then it will be very useful.
Thanks.
Kind Regards, Ahmad
-----Original Message----- From: users-list-bounces+ahmad.hassan=sap.com@monetdb.org [mailto:users-list-bounces+ahmad.hassan=sap.com@monetdb.org] On Behalf Of Fabian Groffen Sent: 10 January 2013 17:35 To: users-list@monetdb.org Subject: Re: MonetDB under LLVM
On 10-01-2013 17:23:27 +0000, Hassan, Ahmad wrote:
Yes Fabian told me this earlier. I tried that option but it only uses 'clang' to compile the source code. It doesn't produce LLVM bitcode files. It produces standard object files.
A quick search with your favourite search engine would have told you that you'll need at least some intrusive changes to the build-system, and a different way of building.
Since you seem not to have reached that conclusion, I doubt whether you are the one to actually find out what you need to get it working. It is not of a concern to MonetDB, at the moment.
Hi Ahmad,
Such an analysis calls for a deep study of the code base, which you already have at your disposal. For datastructures related to all table operations you can start by studying the .../gdk library to raise from MonetDB naive to MonetDB informed user. Thereafter, the MAL interpreter becomes your target for studies. Once done, you might browse around in the SQL code base for further details on catalog datastructures.
Success, regards, Martin ps. As ever, the source code is your most detailed documentation.
On 1/18/13 1:09 PM, Hassan, Ahmad wrote:
Hi Team,
I have modified the monetdb build system to generate executable using LLVM Bitcode and added instrumentation rather than generating exe directly from Object files. At the moment the changes are limited to following executables:
monetdbd: tools/merovingian/daemon mserver5: tools/mserver
I have a question please. I want to analyse the storage manager of monetdb that implements the data structures for holding Table's data and Indexes. More precisely, I want to analyse all the data structures involved during Query processing explicitly i.e indexes, data structures holding tables data in main memory. May I ask which sub-tree of Monet source tree deals with that. Being naïve in monetdb internals, I assumed that monetdbd and mserver5 would be relevant because they are server side part of the code.
If I get any documentation on that, then it will be very useful.
Thanks.
Kind Regards, Ahmad
-----Original Message----- From: users-list-bounces+ahmad.hassan=sap.com@monetdb.org [mailto:users-list-bounces+ahmad.hassan=sap.com@monetdb.org] On Behalf Of Fabian Groffen Sent: 10 January 2013 17:35 To: users-list@monetdb.org Subject: Re: MonetDB under LLVM
On 10-01-2013 17:23:27 +0000, Hassan, Ahmad wrote:
Yes Fabian told me this earlier. I tried that option but it only uses 'clang' to compile the source code. It doesn't produce LLVM bitcode files. It produces standard object files.
A quick search with your favourite search engine would have told you that you'll need at least some intrusive changes to the build-system, and a different way of building.
Since you seem not to have reached that conclusion, I doubt whether you are the one to actually find out what you need to get it working. It is not of a concern to MonetDB, at the moment.
Hi Martin,
Thanks for pointing out the relevant sections. I read some of your lecture slides on MonetDB (http://homepages.cwi.nl/~manegold/teaching/adt/lectures/lecture2.pdf) and it seems like BAT (Binary Association Tables) data structures are the relevant code regions for me to analyse. In the beginning, I am interested in analysing the main memory access pattern of BAT data structures for a database benchmark schema (i.e. TPC-H) and any temporary BAT structures that gets created during the workload execution.
So in order to focus only on this part, would you please point me to relevant codebase where the BAT structures for certain schema are initialised and declared?
After looking at the GDK kernel source, I can see that it has gdk_bat.c implementation and other database operator implementation. As my focus is on BAT structures memory access pattern, so it will really helpful if I can get the information about where BAT data structures for certain database schema and workload are declared and initialised. For example if I run TPC-H workload on monetDB, which part of the code initialises BAT structures for TPC-H schema and where temporary BAT structures (if any) are creating during query execution.
Notes on Building MonetDB under LLVM Framework:
At the moment I directly modified 'Makefile' of monetdbd and mserver5 source sub tree. Compiling and Instrumenting under LLVM framework involves the following steps:
1) Compile C/C++ files into LLVM bitcode files 2) Link individual LLVM bitcode files into one aggregated bitcode file 3) Apply LLVM based instrumentation on bitcode file 4) Perform linking with shared libraries and generate executable
So following the above sequence, I added the following rule in the monetdbd Makefile:
CC=clang CFLAGS = -g -O2 -emit-llvm -c $(X_CFLAGS) //-emit-llvm -c for generating bitcode instead of object files monetdbd_LLVMMERGE = llvm-link -o merge-$@.bc monetdbd_LLVMOPT = opt -o opt-$@.bc -load mypass.so -mypass < merge-$@.bc //Only required if adding instrumentation
monetdbd$(EXEEXT): $(monetdbd_OBJECTS) $(monetdbd_DEPENDENCIES) $(EXTRA_monetdbd_DEPENDENCIES) @rm -f monetdbd$(EXEEXT) @rm -f opt-$@.bc merge-$@.bc $(monetdbd_LLVMMERGE) $(monetdbd_OBJECTS) $(monetdbd_LLVMOPT) //Only required if adding instrumentation $(monetdbd_LINK) opt-$@.bc $(monetdbd_LDADD) $(LIBS)
At the moment, it is very hacky way of building monetdb under llvm but this approach can eventually go to 'configure' and 'Makefile.in' to add support for llvm based compilation.
Thanks.
Kind Regards, Ahmad
-----Original Message----- From: users-list-bounces+ahmad.hassan=sap.com@monetdb.org [mailto:users-list-bounces+ahmad.hassan=sap.com@monetdb.org] On Behalf Of Martin Kersten Sent: 18 January 2013 13:42 To: users-list@monetdb.org Subject: Re: MonetDB under LLVM
Hi Ahmad,
Such an analysis calls for a deep study of the code base, which you already have at your disposal. For datastructures related to all table operations you can start by studying the .../gdk library to raise from MonetDB naive to MonetDB informed user. Thereafter, the MAL interpreter becomes your target for studies. Once done, you might browse around in the SQL code base for further details on catalog datastructures.
Success, regards, Martin ps. As ever, the source code is your most detailed documentation.
On 1/18/13 1:09 PM, Hassan, Ahmad wrote:
Hi Team,
I have modified the monetdb build system to generate executable using LLVM Bitcode and added instrumentation rather than generating exe directly from Object files. At the moment the changes are limited to following executables:
monetdbd: tools/merovingian/daemon mserver5: tools/mserver
I have a question please. I want to analyse the storage manager of monetdb that implements the data structures for holding Table's data and Indexes. More precisely, I want to analyse all the data structures involved during Query processing explicitly i.e indexes, data structures holding tables data in main memory. May I ask which sub-tree of Monet source tree deals with that. Being naïve in monetdb internals, I assumed that monetdbd and mserver5 would be relevant because they are server side part of the code.
If I get any documentation on that, then it will be very useful.
Thanks.
Kind Regards, Ahmad
-----Original Message----- From: users-list-bounces+ahmad.hassan=sap.com@monetdb.org [mailto:users-list-bounces+ahmad.hassan=sap.com@monetdb.org] On Behalf Of Fabian Groffen Sent: 10 January 2013 17:35 To: users-list@monetdb.org Subject: Re: MonetDB under LLVM
On 10-01-2013 17:23:27 +0000, Hassan, Ahmad wrote:
Yes Fabian told me this earlier. I tried that option but it only uses 'clang' to compile the source code. It doesn't produce LLVM bitcode files. It produces standard object files.
A quick search with your favourite search engine would have told you that you'll need at least some intrusive changes to the build-system, and a different way of building.
Since you seem not to have reached that conclusion, I doubt whether you are the one to actually find out what you need to get it working. It is not of a concern to MonetDB, at the moment.
_______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
See http://www.monetdb.org/Documentation/Cookbooks/SQLrecipies/storage-model for a gross overview.
On 1/18/13 5:37 PM, Hassan, Ahmad wrote:
Hi Martin,
Thanks for pointing out the relevant sections. I read some of your lecture slides on MonetDB (http://homepages.cwi.nl/~manegold/teaching/adt/lectures/lecture2.pdf) and it seems like BAT (Binary Association Tables) data structures are the relevant code regions for me to analyse. In the beginning, I am interested in analysing the main memory access pattern of BAT data structures for a database benchmark schema (i.e. TPC-H) and any temporary BAT structures that gets created during the workload execution.
So in order to focus only on this part, would you please point me to relevant codebase where the BAT structures for certain schema are initialised and declared?
After looking at the GDK kernel source, I can see that it has gdk_bat.c implementation and other database operator implementation. As my focus is on BAT structures memory access pattern, so it will really helpful if I can get the information about where BAT data structures for certain database schema and workload are declared and initialised. For example if I run TPC-H workload on monetDB, which part of the code initialises BAT structures for TPC-H schema and where temporary BAT structures (if any) are creating during query execution.
Notes on Building MonetDB under LLVM Framework:
At the moment I directly modified 'Makefile' of monetdbd and mserver5 source sub tree. Compiling and Instrumenting under LLVM framework involves the following steps:
- Compile C/C++ files into LLVM bitcode files
- Link individual LLVM bitcode files into one aggregated bitcode file
- Apply LLVM based instrumentation on bitcode file
- Perform linking with shared libraries and generate executable
So following the above sequence, I added the following rule in the monetdbd Makefile:
CC=clang CFLAGS = -g -O2 -emit-llvm -c $(X_CFLAGS) //-emit-llvm -c for generating bitcode instead of object files monetdbd_LLVMMERGE = llvm-link -o merge-$@.bc monetdbd_LLVMOPT = opt -o opt-$@.bc -load mypass.so -mypass < merge-$@.bc //Only required if adding instrumentation
monetdbd$(EXEEXT): $(monetdbd_OBJECTS) $(monetdbd_DEPENDENCIES) $(EXTRA_monetdbd_DEPENDENCIES) @rm -f monetdbd$(EXEEXT) @rm -f opt-$@.bc merge-$@.bc $(monetdbd_LLVMMERGE) $(monetdbd_OBJECTS) $(monetdbd_LLVMOPT) //Only required if adding instrumentation $(monetdbd_LINK) opt-$@.bc $(monetdbd_LDADD) $(LIBS)
At the moment, it is very hacky way of building monetdb under llvm but this approach can eventually go to 'configure' and 'Makefile.in' to add support for llvm based compilation.
Thanks.
Kind Regards, Ahmad
-----Original Message----- From: users-list-bounces+ahmad.hassan=sap.com@monetdb.org [mailto:users-list-bounces+ahmad.hassan=sap.com@monetdb.org] On Behalf Of Martin Kersten Sent: 18 January 2013 13:42 To: users-list@monetdb.org Subject: Re: MonetDB under LLVM
Hi Ahmad,
Such an analysis calls for a deep study of the code base, which you already have at your disposal. For datastructures related to all table operations you can start by studying the .../gdk library to raise from MonetDB naive to MonetDB informed user. Thereafter, the MAL interpreter becomes your target for studies. Once done, you might browse around in the SQL code base for further details on catalog datastructures.
Success, regards, Martin ps. As ever, the source code is your most detailed documentation.
On 1/18/13 1:09 PM, Hassan, Ahmad wrote:
Hi Team,
I have modified the monetdb build system to generate executable using LLVM Bitcode and added instrumentation rather than generating exe directly from Object files. At the moment the changes are limited to following executables:
monetdbd: tools/merovingian/daemon mserver5: tools/mserver
I have a question please. I want to analyse the storage manager of monetdb that implements the data structures for holding Table's data and Indexes. More precisely, I want to analyse all the data structures involved during Query processing explicitly i.e indexes, data structures holding tables data in main memory. May I ask which sub-tree of Monet source tree deals with that. Being naïve in monetdb internals, I assumed that monetdbd and mserver5 would be relevant because they are server side part of the code.
If I get any documentation on that, then it will be very useful.
Thanks.
Kind Regards, Ahmad
-----Original Message----- From: users-list-bounces+ahmad.hassan=sap.com@monetdb.org [mailto:users-list-bounces+ahmad.hassan=sap.com@monetdb.org] On Behalf Of Fabian Groffen Sent: 10 January 2013 17:35 To: users-list@monetdb.org Subject: Re: MonetDB under LLVM
On 10-01-2013 17:23:27 +0000, Hassan, Ahmad wrote:
Yes Fabian told me this earlier. I tried that option but it only uses 'clang' to compile the source code. It doesn't produce LLVM bitcode files. It produces standard object files.
A quick search with your favourite search engine would have told you that you'll need at least some intrusive changes to the build-system, and a different way of building.
Since you seem not to have reached that conclusion, I doubt whether you are the one to actually find out what you need to get it working. It is not of a concern to MonetDB, at the moment.
users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list _______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
On 1/18/13 1:09 PM, Hassan, Ahmad wrote:
Hi Team,
I have modified the monetdb build system to generate executable using LLVM Bitcode and added instrumentation rather than generating exe directly from Object files. At the moment the changes are limited to following executables:
monetdbd: tools/merovingian/daemon mserver5: tools/mserver
Hi Ahmad,
You might want to share within the community what changes you have made to reach this point.
regards, Martin
Thanks.
Kind Regards, Ahmad
participants (5)
-
Fabian Groffen -
Hassan, Ahmad -
Martin Kersten -
Martin Kersten -
Stefan Manegold