[Monetdb-developers] Large XML instances with MonetDB/XQuery 4.38.5

Christian Grün christian.gruen at gmail.com
Sat Oct 2 01:20:50 CEST 2010


Thanks for the quick feedback, Stefan,

> In case of doubt, we recomment to use a 64-bit version of MonetDB (on a
> 64-bit system); cf.,
> http://monetdb.cwi.nl/XQuery/Documentation/Scalability.html

On this page, I found the information that "The default 64-bits
MonetDB/XQuery binaries are built with 32-bits object identifiers
(OIDs)". I noticed that the "--disable-oid32" isn't recognized by
monetdb-install.sh; but this is probably negligible, as my Mserver
instance shows me the following information:

# MonetDB Server v4.38.5
# based on GDK   v1.38.5
# release Jun2010-SP2
# Copyright (c) 1993-July 2008, CWI. All rights reserved.
# Copyright (c) August 2008-2010, MonetDB B.V.. All rights reserved.
# Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs;
dynamically linked.

…so I guess 64bit is now default?

Next, I've now successfully created an 1GB XMark instance on a 64bit
Linux machine with 32GB RAM; but when I try to shred an 11GB instance,
I get the following output:

> mclient -tlx pfadddoc.xq
#GDKmmap(1336803328) fails, try to free up space [memory in
use=33105304,virtual memory in use=36277911552]
#GDKmmap(1336803328) result [mem=32625976,vm=35689922560]
#GDKmmap: recovery ok. Continuing..
#GDKmmap(1336803328) fails, try to free up space [memory in
use=32626608,virtual memory in use=36971282432]
#GDKmmap(1336803328) result [mem=23481048,vm=36963221504]
MAPI  = monetdb at localhost:50000
QUERY = pf:add-doc("11gb.xml", "xmark")
ERROR = !ERROR: HEAPalloc: Insufficient space for HEAP of 1336803328 bytes.
        !ERROR: CMDleftjoin: operation failed.
        !ERROR: interpret_params: reverse(param 1): evaluation error.
Timer  720853.104 msec


Do you have an idea how to tweak MonetDB, or my system, to parse
documents of that size? Currently, no other processes are running on
the machine, and space on hard disk should be enough as well.

Thanks again,
Christian




More information about the developers-list mailing list