Hi,
I am trying to understand the limitations of MonetDB with regard to memory. It seems that the amount of virtual memory (RAM plus swap space) available to MonetDB must be greater than the amount of data. Before I give up on MonetDB, can someone please confirm or correct my understanding? I did not understand the virtual memory requirement when I read the following in its documentation ( http://monetdb.cwi.nl/MonetDB/Documentation/Size-Limitations.html#Size-Limit... ):
*The maximal database size supported by MonetBD depends on the underlying processing platform, e.g., a 32- or 64-bit processor, and storage device, e.g., the file system and disk raids. *
For example, I started up merovingian (using the Feb 2010 release), limiting it and its child processes to 32 GB of virtual memory (via ulimit -v). I then try to load 115 GB of data into a single table. mclient reports the following:
MAPI = chenher@myserver50000 COPY 472112651 RECORDS INTO mytable FROM 'mydata.txt' DELIMITERS '|', '\n'; ERROR = !SQLException:importTable:failed to import table !ERROR: HEAPextend: failed to extend to 1888450604 for 17/1734tail !ERROR: TABLETcreate_bats: Failed to create bat of size 472112651
I understand if MonetDB must be able to fit all its data within virtual memory. But if this is the case, then unfortunately I cannot use it, since I have terabytes of data and I do not have that much virtual memory on my hardware.
Thank you. Hering