[Monetdb-developers] Lisp trading systems kernel on top of GDK
joelr1 at gmail.com
Thu Jul 28 13:49:34 CEST 2005
On Jul 28, 2005, at 1:19 PM, Martin Kersten wrote:
>>> MIL is likely
>>> to be moved into a corner next year when MonetDB Version 5 is
>> What is the expected replacement?
> An assembler-like version, intended as a target language for front-end
> compilers and optimizers, not for end-user programming.
Then I could actually translate things into this new MIL. Sound good.
>> Can I screw it [BAT properties] up by using GDK functions?
> Mostly not, but you are really linking into a kernel and this requires
> a lot more administration to take care off. Not to mention proper
> garbage collection,..., i.e. all the stuff hidden by the language
Fortunately for me garbage collection is taken care of by Lisp.
It's also a mixed interpreted _and_ compiled environment
which gives me an additional leg up.
> The structure of the table is known and the integrity constraints
> allow direct updates on the BATs.
Yes, my updates are fixed indeed. Where can I find the batch-loading
code? Is that the ascii_io module?
> Then a simple IP-channel can be used
> to quickly pump data into the system.
It does not need to be an IP channel, right? I could just as well
pump data into the system directly, right?
> Depending on the batching size
> implied in a separate data gatherer, you should be able to handle
> anywhere between 5K-200K inserts/sec on a normal PC.
> It also bypasses the SQL logger, which may not be what you want.
I will have to run some experiments here as loosing incoming data
would be very bad for me on the one hand and on the other hand
I need to store data as quickly as possible. I have a Mac OSX laptop.
> A time-series module over BATs would be the preferred way to go.
What should this module be written in?
> Such a module would have primitives for moving-statistics,
> window-based selections, window-based signal analysis, and
> schemes for efficient temporal joins (including interpolation).
I have more interest in rolling out a trading platform but once
the first version is out I could definitely look into optimizations
like building a time series module.
I recon from your "Efficient k-NN Search on Vertically Decomposed Data"
paper that I could calculate the Euclidian distance between
time series on the fly since vector ops are very fast. At least
I could try. I'm also talking to the http://www.cs.purdue.edu/spgist/
folks and other scientists to see if I could use one of the indexing
schemes as a MonetDB search accelerator (hope I have the terminology
I did not see an answer on my other post so I still don't know if a
search can be efficiently done on a lng bat. If they can then I could
do with that as my queries are mostly run this strategy on this range
using a sliding window of X BUNs.
The range of data would be a subset of the price BAT, limited by a range
of dates in a different BAT. But then I also need to consider a symbol
BAT to take only MSFT or IBM, etc. into account. It seems a lot like
the problem that you tackled in the k-NN paper.
I don't fully understand how indexes work in MonetDB. I don't understand
if I need to manually create them on my columns for example. I suppose
this does not make sense with MonetDB. It seems that only one index
the one that links all BATs in a table so that particular BUNs can be
Let me know if I got it right please.
> Unfortunately, there is no free lunch and I guess a more top-down
> architectural design is now what you should be looking for.
I'm reading through Peter Alexander Boncz's dissertation at the moment.
> Before you jump into the MonetDB code.
> Compare it with riding a Ferrari the first time around, you can get
> good speed-up, but also easily get killed.
Yes, I'm aware of the dangers but think that I'm on to something
Your help and advice is greatly appreciated!
More information about the developers-list