[MonetDB-users] ODBC connection issues

Stefan de Konink stefan at konink.de
Wed Oct 19 01:53:41 CEST 2011


On Tue, 18 Oct 2011, Jeremy OSullivan wrote:

> till system crash when data volume exceeds total capacity of memory and
> swap.

Since MonetDB uses memory mapping any "data volume" is in essence 'swap', 
it is up to the operation system to load these parts in memory that it is 
processing on.


> So the high performance of MMDB relies on sufficient memory supply. "
> 
> Is this a fair assessment ?

I don't think so; it is about selectivity. First of all MonetDB is a 
column store thus only the columns that are used in a query plan have to 
be in memory, either in paralel or in sequence. The total amount data is 
therefore not the bottle neck, the selectivity of the query is. Since meta 
data on columns exist data can be read from a specific offset.

The memory mapping allows a specific region te be load into memory by the 
Operating System. Obviously this will not really help with a sequential 
scan on a column that is bigger than the available memory of the host 
system, but given that scenario the actual performance issue is in the 
bandwidth of storage to the memory, and not the amount of memory: the 
data is a supply by itself - swap space.

The only thing regarding to 'system crash' I can currently think of: if 
the intermediate results exceed the amount available memory (in physical 
RAM and in swap)...


Stefan




More information about the users-list mailing list