[Monetdb-developers] Large XML instances with MonetDB/XQuery 4.38.5

Stefan Manegold Stefan.Manegold at cwi.nl
Sat Oct 2 07:33:03 CEST 2010


On Sat, Oct 02, 2010 at 01:20:50AM +0200, Christian Grün wrote:
> Thanks for the quick feedback, Stefan,
> 
> > In case of doubt, we recomment to use a 64-bit version of MonetDB (on a
> > 64-bit system); cf.,
> > http://monetdb.cwi.nl/XQuery/Documentation/Scalability.html
> 
> On this page, I found the information that "The default 64-bits
> MonetDB/XQuery binaries are built with 32-bits object identifiers
> (OIDs)". I noticed that the "--disable-oid32" isn't recognized by
> monetdb-install.sh; but this is probably negligible, as my Mserver
> instance shows me the following information:

monetdb-install.sh is to build from a source tar ball on unix-like systems;
i.e., is not at all related to the binary builds we provide for Windows.

The default is (and has always been) to build with 32-bit OIDs on 32-bit
systems and with 64-it OIDs on 64-bit systems.  Alternatively, there is an
option to build with 32-bit OIDs on 64-bit systems to get a slightly smaller
footprint / memory requirement --- at the expense of scalability: BATs
(tables) can then contain at most 2^31 BUNs (tuples) rather than 2^63.
You can enable that option by running ("plain") configure with
"--enable-oid32" (see `monetdb-install.sh --devhelp` for expert help on
monetdb-install.sh).

We did once decide to build the 64-bit Windows installer with that option to
cope with 64-bit Windows machines with (rather) smalle amounts of physical
memory and their inherent vulnarability to address space fragmentation.
We are currently considering to also provide the 64-bit Windows build with
default 64-bit OIDs in the future.

> # MonetDB Server v4.38.5
> # based on GDK   v1.38.5
> # release Jun2010-SP2
> # Copyright (c) 1993-July 2008, CWI. All rights reserved.
> # Copyright (c) August 2008-2010, MonetDB B.V.. All rights reserved.
> # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs;
> dynamically linked.
> 
> …so I guess 64bit is now default?

As said above, not only now.

> Next, I've now successfully created an 1GB XMark instance on a 64bit
> Linux machine with 32GB RAM; but when I try to shred an 11GB instance,
> I get the following output:
> 
> > mclient -tlx pfadddoc.xq
> #GDKmmap(1336803328) fails, try to free up space [memory in
> use=33105304,virtual memory in use=36277911552]
> #GDKmmap(1336803328) result [mem=32625976,vm=35689922560]
> #GDKmmap: recovery ok. Continuing..
> #GDKmmap(1336803328) fails, try to free up space [memory in
> use=32626608,virtual memory in use=36971282432]
> #GDKmmap(1336803328) result [mem=23481048,vm=36963221504]
> MAPI  = monetdb at localhost:50000
> QUERY = pf:add-doc("11gb.xml", "xmark")
> ERROR = !ERROR: HEAPalloc: Insufficient space for HEAP of 1336803328 bytes.
>         !ERROR: CMDleftjoin: operation failed.
>         !ERROR: interpret_params: reverse(param 1): evaluation error.
> Timer  720853.104 msec

Your Mserver is already using ~36 GB (virtual) address space --- I haven't
recently loaded an 11 GB XMark doc., but that might be reasonable ---, and
then (unexpectedly?) fails to allocate an other ~1.3 GB.

How much free disk space it there on the partition that holds your dbfarm?

> Do you have an idea how to tweak MonetDB, or my system, to parse
> documents of that size? Currently, no other processes are running on
> the machine, and space on hard disk should be enough as well.

What does "space on hard disk should be enough as well" mean?  Is there
(considerably) more than 36 GB free (on the partition that holds your
dbfarm)?

if so, please consider filing a bug report via
http://bugs.monetdb.org/

Stefan

> Thanks again,
> Christian
> 
> 

-- 
| Dr. Stefan Manegold | mailto:Stefan.Manegold at cwi.nl |
| CWI,  P.O.Box 94079 | http://www.cwi.nl/~manegold/  |
| 1090 GB Amsterdam   | Tel.: +31 (20) 592-4212       |
| The Netherlands     | Fax : +31 (20) 592-4199       |




More information about the developers-list mailing list