[Monetdb-developers] configure & *_config.h include files
Dear fellow developers,
while trying to fix portability issues with the pftijah modules of pathfinder, I noticed the following problem:
To ensure portability, each source file should know about and adhere to what configure found out; hence, the respective X_config.h (with X one of monetdb, sql, pf, etc,) must be the first include in each .c file.
However, with the front-ends, some files (especially those implementing extension modules) also need to MonetDB's monet.h or gdk.h to get the respective definition.
Since monet.h includes gdk.h which includes monet_utils.h which in turn includes monetdb_config.h, these files the include both the "local" X_config.h (with X one of sql, pf, etc,) and the monetdb_config.h.
This might trigger two problems:
1) If we compile sql, pf, etc, a against a binary install of MonetDB which was configured and generated on an binary compatible but potentially (slightly) differently configured/installed system, this might lead to conflicting definition/declarations in the two X_config.h.
2) Even if we compile MonetDB and the front-end on the same system, there might be conflicts since the respective configure are not completely identical.
In pathfinder/runtime we avoid (2) be not including pf_config.h but only monetdb_config.h (via monet.h or gdk.h) which might actually lead to problems similar to (1).
In pathfinder/module/pftijah, we run into (2) with _FILE_OFFSET_BITS (at least) on our Solaris 10 x86 machine, since only MonetDB' configure.ag checks AC_SYS_LARGEFILE() while pathfinder's configure.ag does not.
To solve the problem (2) "properly", we could either make all configures identical (e.g., by moving all "local" checks from the various configure.ag to MonetDB/conf.monet.m4). But that still leaves us with (1).
To solve (1) (and (2)) "properly", we could avoid the implicit inclusion of monetdb_config.h in monet_utils.h (and hence gdk.h & monet.h) by removing the respective include from monet_utils.h. But then, we'd have to explicitly include monetdb_config.h in each .c file that now includes one of monet_utils.h, gdk.h, monet.h (directly or via some other include) --- a lot of work, with a high risk to overlook some locations...
A "better"(?) (at least simpler) solution could be to guard the #include "monetdb_config.h" in MonetDB' monet_utils.h by, say, #ifndef NO_MONETDB_CONFIG ... #endif, and then define NO_MONETDB_CONFIG in X_config.h (with X one of sql, pf, etc,).
I'd strongly vote for the last solution. ;-)
Comments, ideas, complaints, objections?
Stefan
On Fri, Sep 22, 2006 at 02:25:21PM +0200, Stefan Manegold wrote:
Dear fellow developers,
while trying to fix portability issues with the pftijah modules of pathfinder, I noticed the following problem:
To ensure portability, each source file should know about and adhere to what configure found out; hence, the respective X_config.h (with X one of monetdb, sql, pf, etc,) must be the first include in each .c file.
However, with the front-ends, some files (especially those implementing extension modules) also need to MonetDB's monet.h or gdk.h to get the respective definition.
Since monet.h includes gdk.h which includes monet_utils.h which in turn includes monetdb_config.h, these files the include both the "local" X_config.h (with X one of sql, pf, etc,) and the monetdb_config.h.
This might trigger two problems:
- If we compile sql, pf, etc, a against a binary install of MonetDB which
was configured and generated on an binary compatible but potentially (slightly) differently configured/installed system, this might lead to conflicting definition/declarations in the two X_config.h.
- Even if we compile MonetDB and the front-end on the same system, there
might be conflicts since the respective configure are not completely identical.
In pathfinder/runtime we avoid (2) be not including pf_config.h but only monetdb_config.h (via monet.h or gdk.h) which might actually lead to problems similar to (1).
In pathfinder/module/pftijah, we run into (2) with _FILE_OFFSET_BITS (at least) on our Solaris 10 x86 machine, since only MonetDB' configure.ag checks AC_SYS_LARGEFILE() while pathfinder's configure.ag does not.
To solve the problem (2) "properly", we could either make all configures identical (e.g., by moving all "local" checks from the various configure.ag to MonetDB/conf.monet.m4). But that still leaves us with (1).
To solve (1) (and (2)) "properly", we could avoid the implicit inclusion of monetdb_config.h in monet_utils.h (and hence gdk.h & monet.h) by removing the respective include from monet_utils.h. But then, we'd have to explicitly include monetdb_config.h in each .c file that now includes one of monet_utils.h, gdk.h, monet.h (directly or via some other include) --- a lot of work, with a high risk to overlook some locations...
A "better"(?) (at least simpler) solution could be to guard the #include "monetdb_config.h" in MonetDB' monet_utils.h by, say, #ifndef NO_MONETDB_CONFIG ... #endif, and then define NO_MONETDB_CONFIG in X_config.h (with X one of sql, pf, etc,).
Of course, "even" this solution does not keep us from making sure that all results of MonetDB's configure that are use in ("expoerted") MonetDB include files are also checked by the other configures, i.e., the checks must be in the shared MonetDB/conf/monet.m4 .
Stefan
I'd strongly vote for the last solution. ;-)
Comments, ideas, complaints, objections?
Stefan
On Fri, Sep 22, 2006 at 02:25:21PM +0200, Stefan Manegold wrote:
Dear fellow developers,
while trying to fix portability issues with the pftijah modules of pathfinder, I noticed the following problem:
To ensure portability, each source file should know about and adhere to what configure found out; hence, the respective X_config.h (with X one of monetdb, sql, pf, etc,) must be the first include in each .c file.
This is a longstanding problem. I actually "complained" about this more than two years ago (see, e.g., bug #957994). On the Pathfinder side, there is (more or less) silence since I ripped out all references to monetdb_config.h in July 2004 (despite people telling me that that such a reference is required and that it cannot simply be avoided :-)).
[...]
This might trigger two problems:
- If we compile sql, pf, etc, a against a binary install of MonetDB which
was configured and generated on an binary compatible but potentially (slightly) differently configured/installed system, this might lead to conflicting definition/declarations in the two X_config.h.
- Even if we compile MonetDB and the front-end on the same system, there
might be conflicts since the respective configure are not completely identical.
There's one more instance of this problem:
3) The user is free to use different ./configure options for MonetDB and, e.g., Pathfinder. For instance, he/she may want to compile one package with icc, the other one with gcc.
This definitely rules out any approach that tries to align all the ./configure scripts in any way. The same ./configure script may give different results also depending on its options!
So this rules out a solution proposed by Stefan:
To solve the problem (2) "properly", we could either make all configures identical (e.g., by moving all "local" checks from the various configure.ag to MonetDB/conf.monet.m4). But that still leaves us with (1).
But it also rules out the other solution proposed by Stefan:
A "better"(?) (at least simpler) solution could be to guard the #include "monetdb_config.h" in MonetDB' monet_utils.h by, say, #ifndef NO_MONETDB_CONFIG ... #endif, and then define NO_MONETDB_CONFIG in X_config.h (with X one of sql, pf, etc,).
This would mean that MonetDB was compiled with some option. Then, e.g., Pathfinder is linked against MonetDB under the assumption that MonetDB uses some some different options.
So the problem is definitely not a trivial one. Let me throw in some thoughts:
The only thing we need to access from some MonetDB add-on (e.g., Pathfinder) are its *interfaces* to specific functionality. So the first question is: Must these interfaces be dependent on the outcome of ./configure (that is, are there any type declarations, function declarations, etc. in the *interfaces* from an add-on to MonetDB that depend on ./configure?). If they aren't, we should factor out stable interfaces (that are independent of the ./configure run and, hence, don't include monetdb_config.h) and only reference those from outside.
If they *are* dependent, we may still take the same road and *generate* interfaces (i.e., include files) when compiling MonetDB that only depend on the ./configure outcome once (during generation). I'm thinking of a file that only contains
typeA_t foo (...)
and not
#if HAVE_FOO typeA_t foo (...) #else typeB_t foo (...) #endif
These ideas pretty much go into the direction of this proposal of Stefan:
To solve (1) (and (2)) "properly", we could avoid the implicit inclusion of monetdb_config.h in monet_utils.h (and hence gdk.h & monet.h) by removing the respective include from monet_utils.h. But then, we'd have to explicitly include monetdb_config.h in each .c file that now includes one of monet_utils.h, gdk.h, monet.h (directly or via some other include) --- a lot of work, with a high risk to overlook some locations...
So these are my comments, ideas, complaints, objections. :-)
Jens
Just to throw it in once more...
... or make it easier for users and yourself and make it all one package, where you just do --{enable,disable}-{sql,xquery,...} at the configure call. Greatly eases packaging too.
On 24-09-2006 10:48:51 +0200, Jens Teubner wrote:
On Fri, Sep 22, 2006 at 02:25:21PM +0200, Stefan Manegold wrote:
[snip]
On Sun, Sep 24, 2006 at 03:04:55PM +0200, Fabian Groffen wrote:
Just to throw it in once more...
... or make it easier for users and yourself and make it all one package, where you just do --{enable,disable}-{sql,xquery,...} at the configure call. Greatly eases packaging too.
No.
Why?
-- I want to spend my development time with programming and *not* with compiling MonetDB over and over, just because there have been other people's commits that force me to recompile once more. The ideal situation would be to have a stable (with respect to its functionality and its interfaces) MonetDB core package against which the different front-ends are coded. Way too often we are suffering from quite the opposite situation already: careless commits break core functionality, which hinders too many people in their development efforts.
-- Pathfinder is a standalone and retargetable compiler. Most of my coding is not related to back-end changes at all. And when I do changes that do not directly involve the MonetDB back-end, there is no need to have my development efforts slowed down by a package as huge as MonetDB including all its front-ends.
Throwing all our code into a single big pot would only heal the single symptom that we are currently observing. Even worse, it would lead to a bigger mess in our code, instead of enforcing discipline and stable, well-designed interfaces.
Other open source projects, by the way, have had to spend a lot of effort to go just the opposite way, from a bloated monolithic project to a nice, modularized software (I'm thinking of the XFree project here).
Regards,
Jens
This post is OT, but I felt the need to formulate an answer.
On 24-09-2006 21:01:30 +0200, Jens Teubner wrote:
On Sun, Sep 24, 2006 at 03:04:55PM +0200, Fabian Groffen wrote:
... or make it easier for users and yourself and make it all one package, where you just do --{enable,disable}-{sql,xquery,...} at the configure call. Greatly eases packaging too.
No.
We simply disagree. Not only we. But then again, I'm not even surprised anymore when people can't get MonetDB to do something useful, let alone just use it, because people seem to "on purpose" make it as hard as possible to make the system usable from a normal user perspective.
No pun intended. This is not a personal remark.
It's just weird that it can be partly done for windows users, but not for a much bigger group of our users.
-- I want to spend my development time with programming and *not* with compiling MonetDB over and over, just because there have been other
As long as other people commit to your code, this will always be the case, no matter how large or small.
-- Pathfinder is a standalone and retargetable compiler. Most of my
The part of pathfinder that makes MonetDB/XQuery is a piece that should be easily obtainable and usable, to make the pathfinder sources as useful as possible to people who are not experienced developers.
Throwing all our code into a single big pot would only heal the single symptom that we are currently observing. Even worse, it would lead to a bigger mess in our code, instead of enforcing discipline and stable, well-designed interfaces.
Good API/interfacing doesn't seem to be much related to where the code is located, but more to who is writing it, IMHO.
Other open source projects, by the way, have had to spend a lot of effort to go just the opposite way, from a bloated monolithic project to a nice, modularized software (I'm thinking of the XFree project here).
(Modular) XOrg is a couple of times bigger than MonetDB. Sure, I've nothing against modularising of code. But it should not harm the usability of the code. With MonetDB, it is clear to me that the current setup greatly harms the general usability of the software, and results in all kinds of insane decisions (e.g. MapiClient supports all languages and has semantic knowledge on those, while Mserver doesn't know anything and needs to be instructed using a sequence of commands in order to let it use such language.).
I believe that for MonetDB's existance in the database world it would be good if people (devs) would see the product as a whole. Ok, I admit immediately, that has close to nothing to do with research.
I think the ideal solution to the problems would be something along the lines of this:
Projects that use MonetDB should not depend on monetdb_config.h, but instead in their own configure step figure out what the relevant parameters are with which MonetDB was compiled. This means that there should be a way for those configure scripts to figure this information out, possibly be including monetdb_config.h in a configure test.
We can still have these other projects share the code base for configure (i.e. MonetDB's conf/monet.m4), but they should not use the output of a different run of that code base.
This solution involves figuring out *all* the relevant parameters. I can think of some off-hand (sizeof(long), sizeof(oid)), but there may be more.
On 2006-09-24 21:31, Fabian Groffen wrote:
This post is OT, but I felt the need to formulate an answer.
On 24-09-2006 21:01:30 +0200, Jens Teubner wrote:
On Sun, Sep 24, 2006 at 03:04:55PM +0200, Fabian Groffen wrote:
... or make it easier for users and yourself and make it all one package, where you just do --{enable,disable}-{sql,xquery,...} at the configure call. Greatly eases packaging too.
No.
We simply disagree. Not only we. But then again, I'm not even surprised anymore when people can't get MonetDB to do something useful, let alone just use it, because people seem to "on purpose" make it as hard as possible to make the system usable from a normal user perspective.
No pun intended. This is not a personal remark.
It's just weird that it can be partly done for windows users, but not for a much bigger group of our users.
-- I want to spend my development time with programming and *not* with compiling MonetDB over and over, just because there have been other
As long as other people commit to your code, this will always be the case, no matter how large or small.
-- Pathfinder is a standalone and retargetable compiler. Most of my
The part of pathfinder that makes MonetDB/XQuery is a piece that should be easily obtainable and usable, to make the pathfinder sources as useful as possible to people who are not experienced developers.
Throwing all our code into a single big pot would only heal the single symptom that we are currently observing. Even worse, it would lead to a bigger mess in our code, instead of enforcing discipline and stable, well-designed interfaces.
Good API/interfacing doesn't seem to be much related to where the code is located, but more to who is writing it, IMHO.
Other open source projects, by the way, have had to spend a lot of effort to go just the opposite way, from a bloated monolithic project to a nice, modularized software (I'm thinking of the XFree project here).
(Modular) XOrg is a couple of times bigger than MonetDB. Sure, I've nothing against modularising of code. But it should not harm the usability of the code. With MonetDB, it is clear to me that the current setup greatly harms the general usability of the software, and results in all kinds of insane decisions (e.g. MapiClient supports all languages and has semantic knowledge on those, while Mserver doesn't know anything and needs to be instructed using a sequence of commands in order to let it use such language.).
I believe that for MonetDB's existance in the database world it would be good if people (devs) would see the product as a whole. Ok, I admit immediately, that has close to nothing to do with research.
Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=D... _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
This is now also a problem for M5, and it is more or less blocking my development right now.
On 24-09-2006 10:48:51 +0200, Jens Teubner wrote: [snip]
This would mean that MonetDB was compiled with some option. Then, e.g., Pathfinder is linked against MonetDB under the assumption that MonetDB uses some some different options.
So the problem is definitely not a trivial one. Let me throw in some thoughts:
The only thing we need to access from some MonetDB add-on (e.g., Pathfinder) are its *interfaces* to specific functionality. So the first question is: Must these interfaces be dependent on the outcome of ./configure (that is, are there any type declarations, function declarations, etc. in the *interfaces* from an add-on to MonetDB that depend on ./configure?). If they aren't, we should factor out stable interfaces (that are independent of the ./configure run and, hence, don't include monetdb_config.h) and only reference those from outside.
If they *are* dependent, we may still take the same road and *generate* interfaces (i.e., include files) when compiling MonetDB that only depend on the ./configure outcome once (during generation). I'm thinking of a file that only contains
typeA_t foo (...)
and not
#if HAVE_FOO typeA_t foo (...) #else typeB_t foo (...) #endif
These ideas pretty much go into the direction of this proposal of Stefan:
To solve (1) (and (2)) "properly", we could avoid the implicit inclusion of monetdb_config.h in monet_utils.h (and hence gdk.h & monet.h) by removing the respective include from monet_utils.h. But then, we'd have to explicitly include monetdb_config.h in each .c file that now includes one of monet_utils.h, gdk.h, monet.h (directly or via some other include) --- a lot of work, with a high risk to overlook some locations...
So these are my comments, ideas, complaints, objections. :-)
I think this is "The" only way to do it right. But M4 is not worth it to spend the time on. Since M5 depends on M4 at the moment and has the same problem, it might be a better solution to solve this in M5 now.
I cannot define VERSION in M5 for instance, because I get the contents of the M4 configure and defines.
participants (4)
-
Fabian Groffen -
Jens Teubner -
Sjoerd Mullender -
Stefan Manegold