Hi, i am working on monetdb so as it can be used for handling huge streaming data, in the most efficient way. i am having a requirement of sustained inserts of rate around 50k-100k inserts per sec(maybe more).i have tested up the results on big enough data (upto 100Gb) and it's the fastest one, to be frank.Most data is 8 byte longs,1 column is time in secs and 1 column is string based.FYI,Data writer in single threaded. However,the issue is too much data and it will be going on for months( maybe years) and all queries have a range part.Most queries are on current data, while some will be for ages old data.Horizontal partitioning is the way to go as it seems. so what i thought was for efficiency, dump the arrays as per time in secs and then when a query comes, somehow form a new BAT (maybe using FUSE, since copying is unusable), join the data in temporary way and use it and discard.Few secs delay is okay even for the range query of 1 sec.
as i have been digging around, http://www.monetdb.org/Documentation/Cookbooks/SQLrecipies/BinaryBulkLoad it's possible for full size columns.
but since this is going to be a copy,it's inefficient. is there a way to get it done ( already exists) ?? any pointers might help such as BAT Partition Manager and so on.
if NOT, i may try to get down and get it done but some pointers might be useful from the experts.
Thanks for help. mike