On Thu, May 27, 2010 at 12:24 AM, Martin Kersten <Martin.Kersten@cwi.nl> wrote:

Hering Cheng wrote:

Hi,

I was wondering if you can give me some ideas on how to retrieve data from MonetDB at very high throughput. The query will be very simple, issued against a single table. This table contains billions of records and I'd like to retrieve the data at a rate of at least one million records per second. From my naive tests with mclient running on the same 16-core machine as mserver5, I am only able to extract the data at about 20,000 records per second with the Feb2010 release.

Without an indication of the table structure (few columns to thousands)
and a skeleton of the SQL query you are firing, and a synopsis how your
application interface is setup, an answer would be hard to give.

Scanning 1M records inside the kernel is not a big deal.
See http://www.cwi.nl/~mk/ontimeReport
for scanning >100M rows

As a baseline case, with the same data in compressed (gzip'd) ASCII form stored in a regular file on the same (SAN) file system as MonetDB, I am able to read at the desired speed of one million records per second.

A DBMS is not a file store!

I understand that there is communication overhead between mserver5 and mclient (or whatever client I use). Therefore, one possibility is to embed my application within mserver5. The embedded application basically just needs to be able to issue a SQL (or even MAL) query against the enclosing mserver5 and process the result set. If this is a viable approach, I'd like some guidance on where the hooks are.

Thanks.
Hering Cheng

------------------------------------------------------------------------

------------------------------------------------------------------------------

------------------------------------------------------------------------

_______________________________________________
MonetDB-users mailing list
MonetDB-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/monetdb-users