Thank you, Martin. The report is very impressive. I only wish that it were more widely disseminated. I thought it would be great if it is accessible from your "Benchmark" (http://monetdb.cwi.nl/SQL/Benchmark/index.html) page.
On Thu, May 27, 2010 at 12:24 AM, Martin Kersten Martin.Kersten@cwi.nlwrote:
Hering Cheng wrote:
Hi,
I was wondering if you can give me some ideas on how to retrieve data from MonetDB at very high throughput. The query will be very simple, issued against a single table. This table contains billions of records and I'd like to retrieve the data at a rate of at least one million records per second. From my naive tests with mclient running on the same 16-core machine as mserver5, I am only able to extract the data at about 20,000 records per second with the Feb2010 release.
Without an indication of the table structure (few columns to thousands) and a skeleton of the SQL query you are firing, and a synopsis how your application interface is setup, an answer would be hard to give.
Scanning 1M records inside the kernel is not a big deal. See http://www.cwi.nl/~mk/ontimeReporthttp://www.cwi.nl/%7Emk/ontimeReport for scanning >100M rows
As a baseline case, with the same data in compressed (gzip'd) ASCII form stored in a regular file on the same (SAN) file system as MonetDB, I am able to read at the desired speed of one million records per second.
A DBMS is not a file store!
I understand that there is communication overhead between mserver5 and mclient (or whatever client I use). Therefore, one possibility is to embed my application within mserver5. The embedded application basically just needs to be able to issue a SQL (or even MAL) query against the enclosing mserver5 and process the result set. If this is a viable approach, I'd like some guidance on where the hooks are.
Thanks. Hering Cheng
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users