On Thu, 31 Mar 2005 20:26:08 +0200, Martin Kersten Martin.Kersten@cwi.nl wrote:
We are playing with TPCH database of up to 100Gb right now, but are actively moving into the next realm. Two approaches are on the way, one where we replace the kernel with MonetDB/X100, specially designed for this kind of problems (actually, we have some radio astronomy database in mind, which runs in the terabytes) and using BAT groups in Monet/Five.
The latter you can easily built on top of the currently released system, i.e. partition the BAT using the void ranges and adjust your algorithms to work on the partitions one (few) at a time. This is what we do to manipulate largers Multimedia Database problems.
Well, a little background. This is a system for managing events coming out of systems, everything from web traffic to security information. It presents a couple of unique challenges:
* Huge volumes of data * High rate of data insertion (situations can demand 20,000 records/second)
Both of these make traditional RDBMS a losing proposition, as they are tuned for transactional performance, which is something I don't care about. I care about durability, but data is never updated, it is only inserted and then queries (which is a bit more like an OLAP application, which is where my interest in MonetDB came from).
Are you planning to use the kernel from SQL or directly from MIL? Tell us a little more of the kind of application you are working on. Perhaps we can give you a few helpful pointers.
Actually I was planning to use MIL, since SQL tends to become logrithmically complex. Also, I'll be using multiple backends to deal with storage requirements, so breaking MIL up seems actually a bit easier, from my view.
say "this is the wrong approach" or system for this problem, as this is obviously a non-normal case.
We love non-normal cases and unconventional use. (But also can;t promises to solve all problems arising within 24 hours)
Unless I'm paying I never expect promises :-) This is mostly an exploratory exercise right now, since I know from previous implementations, where the system collapses, and want to shore up that section.
Chris