July 2005 - developers-list

[Monetdb-developers] X100
by Joel Reymont 02 Aug '05

02 Aug '05

Is it possible to look at the X100 source code or more detailed information right now? What I'm trying to accomplish is basically the same, in Common Lisp. I read through Peter Boncz's Ph.D. thesis as well as the X100 paper. One thing that I don't understand is whether the X100 scan retrieves vectors from _all_ participating BATs. I'm also wondering how the size of the cache (and thus the size of the vector) is determined. Thanks, Joel -- http://wagerlabs.com/uptick

2 3

[Monetdb-developers] Re: X100
by Peter Boncz 30 Jul '05

30 Jul '05

Hi Joel, About X100. In my opinion this is a very cool project that actually constitutes an entirely new query engine. It is more scalable than the current Monet4 engine, and also than the Monet5 engine. There is no main memory limitation anymore. Let's think about it as Monet6 (though that is not sure). The system is quite functional, but especially regarding APIs, database updates and also binary/source distribution of the code, a lot has to be done. Currently, we are concentrating on obtaining scientific results. Some functioanlities (like database updates) still have to be developed. I think we are more than a year off from a releasable system. Also, it has not been decided yet whether that will be an open-source release. Maybe X100 has some commercial future.. Then more general. I just read your posts on this mailing list and I do not have a full grasp of your plans in order to properly advise you. Some general tips: - try to keep all bulk operations in MIL, with interfaces to the lisp engine that move only little data in and out. - use unary bats preferable. i/e/ those that have a void head type - try to seperate data and metadata. Thus, the schema you use, and also the results, are best kept outside MonetDB. Maybe store them in flat files XML or something. It makes the architecture easier to maintain. Also, your application does not depend that strongly on MonetDB repositories. As for special purpose indexing, often it is possible to represent them as a set of bats. you will have to do some coding in a new monet extension module (a .mx file as found in the src/modules/plain/ directory) to write some primitives that access those bats (that are your index) efficiently. just my 2 cts. Peter

2 1

Re: [Monetdb-developers] enum BATs
by Joel Reymont 29 Jul '05

29 Jul '05

Martin, Thanks for your help and my apologies. I thought there was support for enums in the GDK but I'll take a look at the enum module. I'm trying to implement the pieces of MIL that I need for my trading platform, in Common Lisp. Thanks, Joel On Jul 29, 2005, at 11:52 PM, Martin Kersten wrote: > Joel, > > We apologize that the user-documentation is behind the code. > Nevertheless, the following areas are a good starting point for > gaining additional information. -- http://wagerlabs.com/uptick

1 0

[Monetdb-developers] enum BATs
by Joel Reymont 29 Jul '05

29 Jul '05

Folks, Is there a property I can set on the BAT to mark it as enum? How are num bats implemented in the kernel? I could not find this in the docs. Thanks, Joel -- http://wagerlabs.com/uptick

1 0

Re: [Monetdb-developers] Lisp trading systems kernel on top of GDK
by Joel Reymont 28 Jul '05

28 Jul '05

On Jul 28, 2005, at 1:19 PM, Martin Kersten wrote: >>> MIL is likely >>> to be moved into a corner next year when MonetDB Version 5 is >>> released. >>> >> What is the expected replacement? >> > An assembler-like version, intended as a target language for front-end > compilers and optimizers, not for end-user programming. Then I could actually translate things into this new MIL. Sound good. >> Can I screw it [BAT properties] up by using GDK functions? >> > Mostly not, but you are really linking into a kernel and this requires > a lot more administration to take care off. Not to mention proper > garbage collection,..., i.e. all the stuff hidden by the language > interpreters. Fortunately for me garbage collection is taken care of by Lisp. It's also a mixed interpreted _and_ compiled environment which gives me an additional leg up. > The structure of the table is known and the integrity constraints > allow direct updates on the BATs. Yes, my updates are fixed indeed. Where can I find the batch-loading code? Is that the ascii_io module? > Then a simple IP-channel can be used > to quickly pump data into the system. It does not need to be an IP channel, right? I could just as well pump data into the system directly, right? > Depending on the batching size > implied in a separate data gatherer, you should be able to handle > anywhere between 5K-200K inserts/sec on a normal PC. > It also bypasses the SQL logger, which may not be what you want. I will have to run some experiments here as loosing incoming data would be very bad for me on the one hand and on the other hand I need to store data as quickly as possible. I have a Mac OSX laptop. > A time-series module over BATs would be the preferred way to go. What should this module be written in? > Such a module would have primitives for moving-statistics, > window-based selections, window-based signal analysis, and > schemes for efficient temporal joins (including interpolation). I have more interest in rolling out a trading platform but once the first version is out I could definitely look into optimizations like building a time series module. I recon from your "Efficient k-NN Search on Vertically Decomposed Data" paper that I could calculate the Euclidian distance between time series on the fly since vector ops are very fast. At least I could try. I'm also talking to the http://www.cs.purdue.edu/spgist/ folks and other scientists to see if I could use one of the indexing schemes as a MonetDB search accelerator (hope I have the terminology right). I did not see an answer on my other post so I still don't know if a range search can be efficiently done on a lng bat. If they can then I could just do with that as my queries are mostly run this strategy on this range of data, using a sliding window of X BUNs. The range of data would be a subset of the price BAT, limited by a range of dates in a different BAT. But then I also need to consider a symbol BAT to take only MSFT or IBM, etc. into account. It seems a lot like the problem that you tackled in the k-NN paper. I don't fully understand how indexes work in MonetDB. I don't understand if I need to manually create them on my columns for example. I suppose this does not make sense with MonetDB. It seems that only one index is used, the one that links all BATs in a table so that particular BUNs can be accessed. Let me know if I got it right please. > Unfortunately, there is no free lunch and I guess a more top-down > architectural design is now what you should be looking for. I'm reading through Peter Alexander Boncz's dissertation at the moment. > Before you jump into the MonetDB code. > Compare it with riding a Ferrari the first time around, you can get > good speed-up, but also easily get killed. Yes, I'm aware of the dangers but think that I'm on to something exciting. Your help and advice is greatly appreciated! Thanks, Joel -- http://wagerlabs.com/uptick

1 0

Re: [Monetdb-developers] Lisp trading systems kernel on top of GDK
by Joel Reymont 28 Jul '05

28 Jul '05

Martin, On Jul 28, 2005, at 12:49 PM, Martin Kersten wrote: > 3) is a good option. Separate the database functionality from the > application and only extend the kernel if you can not escape it. > [...] > indeed, accessing points one-by-one nullifies all positive effects of > a database system, you should think set-(list-) at a time I could not see any way of retrieving data in sets or lists using MAPI. Am I missing something? > MIL is likely > to be moved into a corner next year when MonetDB Version 5 is > released. What is the expected replacement? >> or something similar. I'm wondering if MIL would be up to the task. >> #2 is the option that I would prefer as I could hook up into the GDK >> to quickly iterate through BATs and BUNs and use Lisp to write >> > that is feasible and has been done more often. The pitfall here is > that > you may easily screw-up the property infrastructure What is the property infrastructure? Can I screw it up by using GDK functions? > Actually, if your input/updates are fixed, you may consider a much > faster scheme directly interfacing with updates on the BAT structures. > The bulk loading routines may give you the hints how to do this. Just so that I understand you correctly, what do you mean by fixed input/updates? Thanks, Joel -- http://wagerlabs.com/uptick

1 0

[Monetdb-developers] Lisp trading systems kernel on top of GDK
by Joel Reymont 28 Jul '05

28 Jul '05

Folks, The real-time trading platform that I'm building requires running strategies on historical data at very high speed. There could be hundreds of thousands of such runs with different permutations of arguments. I do not want to force my customers to write trading systems in MIL and MIL, according to its description, might not be suitable. So I was wondering if I should 1) try to translate trading systems from Lisp into MIL or if I should 2) build my own Lisp kernel for trading systems execution. There's also 3) the option of not running strategies in MonetDB/MIL. #3 would require using MAPI and extracting data points one by one. I dislike this as it would not let me take advantage of vector processing. #1 would require using MIL to potentially write Fast Fourier Transforms or something similar. I'm wondering if MIL would be up to the task. #2 is the option that I would prefer as I could hook up into the GDK to quickly iterate through BATs and BUNs and use Lisp to write and debug trading systems. With some coordination I think I could even use MonetDB-SQL together with my kernel, so long as I leave all the insertion and update operations outside of Lisp. What do you think? Thanks, Joel -- http://wagerlabs.com/uptick

1 0

[Monetdb-developers] Time series extension module
by Joel Reymont 28 Jul '05

28 Jul '05

Folks, I'm building a real-time, very short-frequency trading platform. The platform needs a backend for storing exchange ticks (quotes) and I think MonetDB could be that platform. My ultimate attraction is the vertical decomposition that sets MonetDB apart. It's the approach taken by KDB http://www.kx.com/products/kdbplusfaq.php http://cs.nyu.edu/cs/faculty/shasha/papers/sigmodpap.pdf and that has already proven immensely effective. My personal need is in searching for matching subsequences in other time series (k-NN?) which would require me to compute the Euclidian distance between points, etc. Martin suggested that I build a time series extension module which is fine with me (see http://wagerlabs.com/resume.pdf) I have several questions before I embark on this project... I don't have a clear understanding of how MonetDB managed indexes. If I'm dealing with timestamp, price and volume and need to search either would I need to build indexes on the three columns? I believe MonetDB supports range queries. Could I just store timestamp as an integer such as YYYYMMDDHHSS and do a range query on that? How would I go about building the time series extension module? Would I need to add custom search accelerators to build a specialized index on the fly? Thanks, Joel -- http://wagerlabs.com/uptick

1 0