[Monetdb-developers] Lisp trading systems kernel on top of GDK

Joel Reymont joelr1 at gmail.com
Thu Jul 28 13:49:34 CEST 2005

On Jul 28, 2005, at 1:19 PM, Martin Kersten wrote:

>>> MIL is likely
>>> to be moved into a corner next year when MonetDB Version 5 is   
>>> released.
>> What is the expected replacement?
> An assembler-like version, intended as a target language for front-end
> compilers and optimizers, not for end-user programming.

Then I could actually translate things into this new MIL. Sound good.

>> Can I screw it [BAT properties] up by using GDK functions?
> Mostly not, but you are really linking into a kernel and this requires
> a lot more administration to take care off. Not to mention proper
> garbage collection,..., i.e. all the stuff hidden by the language
> interpreters.

Fortunately for me garbage collection is taken care of by Lisp.
It's also a mixed interpreted _and_ compiled environment
which gives me an additional leg up.

> The structure of the table is known and the integrity constraints
> allow direct updates on the BATs.

Yes, my updates are fixed indeed. Where can I find the batch-loading
code? Is that the ascii_io module?

> Then a simple IP-channel can be used
> to quickly pump data into the system.

It does not need to be an IP channel, right? I could just as well
pump data into the system directly, right?

> Depending on the batching size
> implied in a separate data gatherer, you should be able to handle
> anywhere between 5K-200K inserts/sec on a normal PC.
> It also bypasses the SQL logger, which may not be what you want.

I will have to run some experiments here as loosing incoming data
would be very bad for me on the one hand and on the other hand
I need to store data as quickly as possible. I have a Mac OSX laptop.

> A time-series module over BATs would be the preferred way to go.

What should this module be written in?

> Such a module would have primitives for moving-statistics,
> window-based selections, window-based signal analysis, and
> schemes for efficient temporal joins (including interpolation).

I have more interest in rolling out a trading platform but once
the first version is out I could definitely look into optimizations
like building a time series module.

I recon from your "Efficient k-NN Search on Vertically Decomposed Data"
paper that I could calculate the Euclidian distance between
time series on the fly since vector ops are very fast. At least
I could try. I'm also talking to the http://www.cs.purdue.edu/spgist/
folks and other scientists to see if I could use one of the indexing
schemes as a MonetDB search accelerator (hope I have the terminology  

I did not see an answer on my other post so I still don't know if a  
search can be efficiently done on a lng bat. If they can then I could  
do with that as my queries are mostly run this strategy on this range  
of data,
using a sliding window of X BUNs.

The range of data would be a subset of the price BAT, limited by a range
of dates in a different BAT. But then I also need to consider a symbol
BAT to take only MSFT or IBM, etc. into account. It seems a lot like
the problem that you tackled in the k-NN paper.

I don't fully understand how indexes work in MonetDB. I don't understand
if I need to manually create them on my columns for example. I suppose
this does not make sense with MonetDB. It seems that only one index  
is used,
the one that links all BATs in a table so that particular BUNs can be  
Let me know if I got it right please.

> Unfortunately, there is no free lunch and I guess a more top-down
> architectural design is now what you should be looking for.

I'm reading through Peter Alexander Boncz's dissertation at the moment.

> Before you jump into the MonetDB code.
> Compare it with riding a Ferrari the first time around, you can get
> good speed-up, but also easily get killed.

Yes, I'm aware of the dangers but think that I'm on to something  

Your help and advice is greatly appreciated!

     Thanks, Joel


More information about the developers-list mailing list