[Monetdb-developers] [MonetDB-users] Large XML instances with MonetDB/XQuery 4.38.5

Stefan Manegold Stefan.Manegold at cwi.nl
Fri Oct 1 23:51:08 CEST 2010


Hi Christian,

quick and simple answer, on a 32-bit Windows system, you are running out of
address space, which is in practice limited to 2GB or at most 3GB if you
configured your system accordingly (cf.,
http://monetdb.cwi.nl/XQuery/Documentation/Scalability.html
http://zone.ni.com/reference/en-XX/help/371361D-01/lvhowto/enable_lrg_ad_aware/
), and know to be very vulnarable to address space fragmentation.

I haven't tried it recently, but you might be able to load the 1 GB XMark
document on a 32-bit Unix-like system.

A 64-bit build of MonetDB/XQuery (on a 64-bit system) should not have any
problems loading a 1 GB XMark document.

Stefan

On Fri, Oct 01, 2010 at 07:13:53PM +0200, Christian Grün wrote:
> Hi everyone,
> 
> is it still possible to shred large XML instances with MonetDB/XQuery,
> or do I need more memory for shredding? I was just trying to the 1GB
> instance of XMark with WinXP 32Bit and 2GB RAM, and this is what I
> got:
> 
> MonetDB>shred_doc("1gb.xml", "xmark");
> #GDKmmap(536870912) fails, try to free up space [memory in use=36143780,virtual
> memory in use=1015480320]
> #GDKmmap(536870912) result [mem=26016940,vm=1005387776]
> !ERROR: [shred_url]: 1 times inserted nil due to errors at tuples 0 at 0.
> !ERROR: [shred_url]: first error was:
> !ERROR: HEAPextend: failed to extend to 536870912 for 06\632theap
> !ERROR: shredBAT_append_str: APPEND-STR[_prop_text]( follow murders prov shoes p
> resent eager delivered defy bend viol here marquis powers complexion pillar cran
> ts agate ), BUNappend fails
> !ERROR: CMDshred_url: operation failed.
> 
> pf:add-doc(...) produces the same output; the XML document itself is
> well-formed.
> 
> Thanks,
> Christian
> 
> ------------------------------------------------------------------------------
> Start uncovering the many advantages of virtual appliances
> and start using them to simplify application deployment and
> accelerate your shift to cloud computing.
> http://p.sf.net/sfu/novell-sfdev2dev
> _______________________________________________
> MonetDB-users mailing list
> MonetDB-users at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/monetdb-users
> 
> 

-- 
| Dr. Stefan Manegold | mailto:Stefan.Manegold at cwi.nl |
| CWI,  P.O.Box 94079 | http://www.cwi.nl/~manegold/  |
| 1090 GB Amsterdam   | Tel.: +31 (20) 592-4212       |
| The Netherlands     | Fax : +31 (20) 592-4199       |




More information about the developers-list mailing list