[Monetdb-developers] Lisp trading systems kernel on top of GDK

Joel Reymont joelr1 at gmail.com
Thu Jul 28 13:49:34 CEST 2005


On Jul 28, 2005, at 1:19 PM, Martin Kersten wrote:

>>> MIL is likely
>>> to be moved into a corner next year when MonetDB Version 5 is   
>>> released.
>>>
>> What is the expected replacement?
>>
> An assembler-like version, intended as a target language for front-end
> compilers and optimizers, not for end-user programming.

Then I could actually translate things into this new MIL. Sound good.

>> Can I screw it [BAT properties] up by using GDK functions?
>>
> Mostly not, but you are really linking into a kernel and this requires
> a lot more administration to take care off. Not to mention proper
> garbage collection,..., i.e. all the stuff hidden by the language
> interpreters.

Fortunately for me garbage collection is taken care of by Lisp.
It's also a mixed interpreted _and_ compiled environment
which gives me an additional leg up.

> The structure of the table is known and the integrity constraints
> allow direct updates on the BATs.

Yes, my updates are fixed indeed. Where can I find the batch-loading
code? Is that the ascii_io module?

> Then a simple IP-channel can be used
> to quickly pump data into the system.

It does not need to be an IP channel, right? I could just as well
pump data into the system directly, right?

> Depending on the batching size
> implied in a separate data gatherer, you should be able to handle
> anywhere between 5K-200K inserts/sec on a normal PC.
> It also bypasses the SQL logger, which may not be what you want.

I will have to run some experiments here as loosing incoming data
would be very bad for me on the one hand and on the other hand
I need to store data as quickly as possible. I have a Mac OSX laptop.

> A time-series module over BATs would be the preferred way to go.

What should this module be written in?

> Such a module would have primitives for moving-statistics,
> window-based selections, window-based signal analysis, and
> schemes for efficient temporal joins (including interpolation).

I have more interest in rolling out a trading platform but once
the first version is out I could definitely look into optimizations
like building a time series module.

I recon from your "Efficient k-NN Search on Vertically Decomposed Data"
paper that I could calculate the Euclidian distance between
time series on the fly since vector ops are very fast. At least
I could try. I'm also talking to the http://www.cs.purdue.edu/spgist/
folks and other scientists to see if I could use one of the indexing
schemes as a MonetDB search accelerator (hope I have the terminology  
right).

I did not see an answer on my other post so I still don't know if a  
range
search can be efficiently done on a lng bat. If they can then I could  
just
do with that as my queries are mostly run this strategy on this range  
of data,
using a sliding window of X BUNs.

The range of data would be a subset of the price BAT, limited by a range
of dates in a different BAT. But then I also need to consider a symbol
BAT to take only MSFT or IBM, etc. into account. It seems a lot like
the problem that you tackled in the k-NN paper.

I don't fully understand how indexes work in MonetDB. I don't understand
if I need to manually create them on my columns for example. I suppose
this does not make sense with MonetDB. It seems that only one index  
is used,
the one that links all BATs in a table so that particular BUNs can be  
accessed.
Let me know if I got it right please.

> Unfortunately, there is no free lunch and I guess a more top-down
> architectural design is now what you should be looking for.

I'm reading through Peter Alexander Boncz's dissertation at the moment.

> Before you jump into the MonetDB code.
> Compare it with riding a Ferrari the first time around, you can get
> good speed-up, but also easily get killed.

Yes, I'm aware of the dangers but think that I'm on to something  
exciting.

Your help and advice is greatly appreciated!

     Thanks, Joel

--
http://wagerlabs.com/uptick







More information about the developers-list mailing list