[Monetdb-developers] dealing with 1G XML file in MonetDB/XQuery
Hi,
When I load a 1G XML file to MonetDB/XQuery, it throws exceptions as follows. I did not find related parameters in the monetDB config file to enlarge the heap size. I wonder how I can solve this problem? Thank you.
#GDKmmap(1073741824) fails, try to free up space [memory in use=156511940,virtual memory in use=1309212672] #GDKmmap(1073741824) result [mem=156511940,vm=1309212672] MAPI = mozhou@localhost:40000 QUERY = pf:add-doc("/.../new1GL1Comp/xmldoc3.xml","1GL1doc3") ERROR = !ERROR: [shred_url]: 1 times inserted nil due to errors at tuples 0@0. !ERROR: [shred_url]: first error was: !ERROR: HEAPextend: failed to extend to 1073741824 for 03/02/30243theap !ERROR: shredBAT_append_str: APPEND-STR[_prop_text](http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/dataFromVendor20/Of...), BUNappend fails !ERROR: CMDshred_url: operation failed.
Hello,
Could you please reply as soon as possible?
Thanks, Mo On Thu, Apr 1, 2010 at 4:49 PM, Mo Zhou mozhou@umail.iu.edu wrote:
Hi,
When I load a 1G XML file to MonetDB/XQuery, it throws exceptions as follows. I did not find related parameters in the monetDB config file to enlarge the heap size. I wonder how I can solve this problem? Thank you.
#GDKmmap(1073741824) fails, try to free up space [memory in use=156511940,virtual memory in use=1309212672] #GDKmmap(1073741824) result [mem=156511940,vm=1309212672] MAPI = mozhou@localhost:40000 QUERY = pf:add-doc("/.../new1GL1Comp/xmldoc3.xml","1GL1doc3") ERROR = !ERROR: [shred_url]: 1 times inserted nil due to errors at tuples 0@0. !ERROR: [shred_url]: first error was: !ERROR: HEAPextend: failed to extend to 1073741824 for 03/02/30243theap !ERROR: shredBAT_append_str: APPEND-STR[_prop_text](http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/dataFromVendor20/Of...), BUNappend fails !ERROR: CMDshred_url: operation failed.
-- Thanks, Mo
On 2010-04-01 22:49, Mo Zhou wrote:
Hi,
When I load a 1G XML file to MonetDB/XQuery, it throws exceptions as follows. I did not find related parameters in the monetDB config file to enlarge the heap size. I wonder how I can solve this problem? Thank you.
#GDKmmap(1073741824) fails, try to free up space [memory in use=156511940,virtual memory in use=1309212672] #GDKmmap(1073741824) result [mem=156511940,vm=1309212672] MAPI = mozhou@localhost:40000 QUERY = pf:add-doc("/.../new1GL1Comp/xmldoc3.xml","1GL1doc3") ERROR = !ERROR: [shred_url]: 1 times inserted nil due to errors at tuples 0@0. !ERROR: [shred_url]: first error was: !ERROR: HEAPextend: failed to extend to 1073741824 for 03/02/30243theap !ERROR: shredBAT_append_str: APPEND-STR[_prop_text](http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/dataFromVendor20/Of...), BUNappend fails !ERROR: CMDshred_url: operation failed.
What version are you using. Please include the output of Mserver --version
If you are trying to do this on a 32 bit architecture, then the problem is that you're running out of address space. In order to load a 1 GB document, you'll likely need an address space (not necessarily memory) that is quite a bit larger than that. A 32 bit system has an address space of 2 GB (or 3 GB on Linux), and that is probably not enough. Try it on a 64 bit architecture with a 64 bit build of MonetDB.
If you're already using a 64 bit build, then the problem is probably not enough available virtual memory (core memory + swap memory).
participants (2)
-
Mo Zhou -
Sjoerd Mullender