Hi Joel,
About X100. In my opinion this is a very cool project that actually constitutes an entirely new query engine. It is more scalable than the current Monet4 engine, and also than the Monet5 engine. There is no main memory limitation anymore. Let's think about it as Monet6 (though that is not sure). The system is quite functional, but especially regarding APIs, database updates and also binary/source distribution of the code, a lot has to be done. Currently, we are concentrating on obtaining scientific results. Some functioanlities (like database updates) still have to be developed. I think we are more than a year off from a releasable system. Also, it has not been decided yet whether that will be an open-source release. Maybe X100 has some commercial future..
Then more general. I just read your posts on this mailing list and I do not have a full grasp of your plans in order to properly advise you.
Some general tips: - try to keep all bulk operations in MIL, with interfaces to the lisp engine that move only little data in and out. - use unary bats preferable. i/e/ those that have a void head type - try to seperate data and metadata. Thus, the schema you use, and also the results, are best kept outside MonetDB. Maybe store them in flat files XML or something. It makes the architecture easier to maintain. Also, your application does not depend that strongly on MonetDB repositories.
As for special purpose indexing, often it is possible to represent them as a set of bats. you will have to do some coding in a new monet extension module (a .mx file as found in the src/modules/plain/ directory) to write some primitives that access those bats (that are your index) efficiently.
just my 2 cts.
Peter