[Monetdb-developers] dealing with 1G XML file in MonetDB/XQuery

Sjoerd Mullender sjoerd at acm.org
Thu Apr 8 10:28:07 CEST 2010

On 2010-04-01 22:49, Mo Zhou wrote:
> Hi,
> When I load a 1G XML file to MonetDB/XQuery, it throws exceptions as
> follows. I did not find related parameters in the monetDB config file
> to enlarge the heap size. I wonder how I can solve this problem? Thank
> you.
> #GDKmmap(1073741824) fails, try to free up space [memory in
> use=156511940,virtual memory in use=1309212672]
> #GDKmmap(1073741824) result [mem=156511940,vm=1309212672]
> MAPI  = mozhou at localhost:40000
> QUERY = pf:add-doc("/.../new1GL1Comp/xmldoc3.xml","1GL1doc3")
> ERROR = !ERROR: [shred_url]: 1 times inserted nil due to errors at tuples 0 at 0.
>         !ERROR: [shred_url]: first error was:
>         !ERROR: HEAPextend: failed to extend to 1073741824 for 03/02/30243theap
>         !ERROR: shredBAT_append_str:
> APPEND-STR[_prop_text](http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/dataFromVendor20/Offer31281/),
> BUNappend fails
>         !ERROR: CMDshred_url: operation failed.

What version are you using.  Please include the output of
Mserver --version

If you are trying to do this on a 32 bit architecture, then the problem
is that you're running out of address space.  In order to load a 1 GB
document, you'll likely need an address space (not necessarily memory)
that is quite a bit larger than that.  A 32 bit system has an address
space of 2 GB (or 3 GB on Linux), and that is probably not enough.  Try
it on a 64 bit architecture with a 64 bit build of MonetDB.

If you're already using a 64 bit build, then the problem is probably not
enough available virtual memory (core memory + swap memory).

Sjoerd Mullender

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 371 bytes
Desc: OpenPGP digital signature
URL: <http://www.monetdb.org/pipermail/developers-list/attachments/20100408/3288a49b/attachment.sig>

More information about the developers-list mailing list