[Monetdb-developers] Building accelerators

Martin Kersten Martin.Kersten at cwi.nl
Wed Apr 11 14:48:32 CEST 2007


Agustin Schapira wrote:
>> Both modules are in MonetDB5/src/modules/mal/{bpm.mx,crackers.mx}
>> The former is under active development. It uses an internal structure
>> with import/export to BATs as part of the session startup/exit.
>> The latter is currently subject to reduction and cleanup to improve
>> re-user and maintenance. It uses an AVL tree to keep a catalog of
>> routing information in memory.
> 
> Thanks, Martin. We'll take a look at them.
> 
> Our new datatype is used to store links between objects in the database. 
> It's a simple struct with (link_id, from_id, to_id), which we store in a 
hmm, this sounds like a triple store. If the number of link-ids is small
enough (<100K), I would seriously consider horizontal fragmentation
based on it as a starting point. Then the automatic hash on the
(from_id,to_id) bat gives you the answer quickly.
> central BAT. One common operation is, of course, finding the neighbors 
> of a given object, and we would like to be able to do that efficiently, 
How volatile is your data? Otherwise, simply sort them on the oid and
the hash will bring you quickly to the desired plac.
> without a full table scan every time. We can build internal data 
> structures every time we do a select on the BAT, but it would be more 
> efficient to have a persistent data structure that we can somehow re-use.
This calls for re-use of the BATs as a storage scheme. It also aids
in debugging your accelerators code.
> 
> We've read the old GIS paper, and our needs are pretty similar to those. 
> Is that code still active, somewhere? Could we take a look at it? Is 
That code went to the attic many years ago.
> there any other place where yo do similar things, such as having BATs of 
> a new datatype and searching/joining them efficiently?
> 
> Too many questions, I know ;-) Thanks!
> 
> -- Agustin





More information about the developers-list mailing list