[Monetdb-developers] Large XML instances with MonetDB/XQuery 4.38.5

Stefan Manegold Stefan.Manegold at cwi.nl
Fri Oct 1 23:57:04 CEST 2010


On Fri, Oct 01, 2010 at 09:35:26PM +0200, Christian Grün wrote:
> To continue… Do you have some information on the current theoretical
> limits for storing XML instances in MonetDB/XQuery?

The theoretical and pratical limit are the available address space and free
disk space on your system, as well as the vulnerability of your OS to
address space fragmentation.  In practice, not only the plain serialized
document size is a factor, but also the number of nodes in your document(s).

In case of doubt, we recomment to use a 64-bit version of MonetDB (on a
64-bit system); cf.,
http://monetdb.cwi.nl/XQuery/Documentation/Scalability.html

Stefan

> Thanks,
> Christian
> 
> 
> 
> On Fri, Oct 1, 2010 at 7:13 PM, Christian Grün
> <christian.gruen at gmail.com> wrote:
> > Hi everyone,
> >
> > is it still possible to shred large XML instances with MonetDB/XQuery,
> > or do I need more memory for shredding? I was just trying to the 1GB
> > instance of XMark with WinXP 32Bit and 2GB RAM, and this is what I
> > got:
> >
> > MonetDB>shred_doc("1gb.xml", "xmark");
> > #GDKmmap(536870912) fails, try to free up space [memory in use=36143780,virtual
> > memory in use=1015480320]
> > #GDKmmap(536870912) result [mem=26016940,vm=1005387776]
> > !ERROR: [shred_url]: 1 times inserted nil due to errors at tuples 0 at 0.
> > !ERROR: [shred_url]: first error was:
> > !ERROR: HEAPextend: failed to extend to 536870912 for 06\632theap
> > !ERROR: shredBAT_append_str: APPEND-STR[_prop_text]( follow murders prov shoes p
> > resent eager delivered defy bend viol here marquis powers complexion pillar cran
> > ts agate ), BUNappend fails
> > !ERROR: CMDshred_url: operation failed.
> >
> > pf:add-doc(...) produces the same output; the XML document itself is
> > well-formed.
> >
> > Thanks,
> > Christian
> >
> 
> ------------------------------------------------------------------------------
> Start uncovering the many advantages of virtual appliances
> and start using them to simplify application deployment and
> accelerate your shift to cloud computing.
> http://p.sf.net/sfu/novell-sfdev2dev
> _______________________________________________
> Monetdb-developers mailing list
> Monetdb-developers at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/monetdb-developers
> 

-- 
| Dr. Stefan Manegold | mailto:Stefan.Manegold at cwi.nl |
| CWI,  P.O.Box 94079 | http://www.cwi.nl/~manegold/  |
| 1090 GB Amsterdam   | Tel.: +31 (20) 592-4212       |
| The Netherlands     | Fax : +31 (20) 592-4199       |




More information about the developers-list mailing list