[Monetdb-developers] dealing with 1G XML file in MonetDB/XQuery
sjoerd at acm.org
Thu Apr 8 10:28:07 CEST 2010
On 2010-04-01 22:49, Mo Zhou wrote:
> When I load a 1G XML file to MonetDB/XQuery, it throws exceptions as
> follows. I did not find related parameters in the monetDB config file
> to enlarge the heap size. I wonder how I can solve this problem? Thank
> #GDKmmap(1073741824) fails, try to free up space [memory in
> use=156511940,virtual memory in use=1309212672]
> #GDKmmap(1073741824) result [mem=156511940,vm=1309212672]
> MAPI = mozhou at localhost:40000
> QUERY = pf:add-doc("/.../new1GL1Comp/xmldoc3.xml","1GL1doc3")
> ERROR = !ERROR: [shred_url]: 1 times inserted nil due to errors at tuples 0 at 0.
> !ERROR: [shred_url]: first error was:
> !ERROR: HEAPextend: failed to extend to 1073741824 for 03/02/30243theap
> !ERROR: shredBAT_append_str:
> BUNappend fails
> !ERROR: CMDshred_url: operation failed.
What version are you using. Please include the output of
If you are trying to do this on a 32 bit architecture, then the problem
is that you're running out of address space. In order to load a 1 GB
document, you'll likely need an address space (not necessarily memory)
that is quite a bit larger than that. A 32 bit system has an address
space of 2 GB (or 3 GB on Linux), and that is probably not enough. Try
it on a 64 bit architecture with a 64 bit build of MonetDB.
If you're already using a 64 bit build, then the problem is probably not
enough available virtual memory (core memory + swap memory).
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 371 bytes
Desc: OpenPGP digital signature
More information about the developers-list