Hi,
I've recently been conducting some performance testing of MonetDB with a variety of servers (from 4 core -8GB RAM to 20 core-64GB RAM) with various data sizes, in an attempt to gain a better understanding of how MonetDB scales.
During the performance tests it became obvious that much of the processing was IO bound due to:
1) Columns being unmapped from memory overly aggressively (even when there was plenty of memory still available). 2) The constant mapping/unmapping of memory mapped bat files for intermediate results.
I've attached a patch which attempts to address both issues. The first patch to (gdk_utils.c) is to update the memory limit at which the GDKvmtrim kicks in to be 80% memory usage. The second patch (gdk_heap.c) limits the number of mmap/munmap calls via the existing 'heap caching' mechanism which was not working at all! In addition to fixing up the caching code, I've also wired in the heap cache into the case where extending a malloced heap results in a swap over to memory mapped storage.
After applying the patches I was seeing approximately 40% performance improvements (your mileage may vary!) If the changes are deemed to be useful, how do I go about getting them accepted into the MonetDB source repository?
Thanks, Alistair Sutherland
________________________________