[Monetdb-developers] Unable to shred 1gb xml file (OS: Not enough space)
Hi,
I tried MonetDB/XQuery to shred a 1gb xml file, but it failed.
Following is the error message: MonetDB>shred_doc("D:xbench/output/DC1000catalog.xml", "DC1000catalog.xml"); !ERROR: MT_mmap: MapViewOfFile(6b4, 2, 0, 0, 442499072, 0) failed !OS: Not enough space !GDKmmap(442499072) fail => BBPtrim(enter) usage[mem=101297568,vm=794361856]
I use windows XP Professional version 2002 service pack 2, Pentium 4 CPU, 2.40GHz, 512 MB of RAM.
Previously I tried to shred 100mb xml file, and it took 18.532 sec. For this 1 gb file, I left it overnight so I don't know how long it takes till the error message occurs.
Can please help me to solve this problem?
Regards,
Klarinda
Below is the complete error message: MonetDB>shred_doc("D:xbench/output/DC1000catalog.xml", "DC1000catalog.xml"); !ERROR: MT_mmap: MapViewOfFile(6b4, 2, 0, 0, 442499072, 0) failed !OS: Not enough space !GDKmmap(442499072) fail => BBPtrim(enter) usage[mem=101297568,vm=794361856] # !mallinfo.arena = 15613828 !mallinfo.ordblks = 46134 !mallinfo.smblks = 15492 !mallinfo.hblkhd = 0 !mallinfo.hblks = 0 !mallinfo.usmblks = 13718408 !mallinfo.fsmblks = 899720 !mallinfo.uordblks = 950740 !mallinfo.fordblks = 44960 #BBPTRIM_ENTER: memsize=101297568,vmsize=794361856 #BBPTRIM: memtarget=0 vmtarget=1073741824 #TRIMSCAN: mem=0 vm=1, start=1, limit=1 #TRIMSCAN: 145030 0=tmp_35 (#0) #TRIMSCAN: 145059 1=tmp_36 (#0) #TRIMSCAN: 145088 2=tmp_37 (#0) #TRIMSCAN: 145146 3=tmp_41 (#0) #TRIMSCAN: 149075 4=doc_query (#0) #TRIMSCAN: 149092 5=doc_sema (#0) #TRIMSCAN: 155215 6=tmp_374 (#0) #TRIMSCAN: 155218 7=prop_pre_39 (#0) #TRIMSCAN: 157895 8=tmp_533 (#0) #TRIMSCAN: 157898 9=prop_pre_310 (#0) #TRIMSCAN: end at 1 (size=628) #TRIMSELECT: dirty = 0 #TRIMSELECT: candidate=tmp_35 BAT*=03D6A230 # (cnt=0, mode=1024, refs=0, wait=0, parent=0, lastused=145030,145030 ,145030) #TRIMSELECT: keep tmp_35 [224,0] bytes [224,0] dirty target(mem=0 vm=1073741824)
#TRIMSELECT: candidate=tmp_36 BAT*=03D66E60 # (cnt=0, mode=1024, refs=0, wait=0, parent=0, lastused=145059,145059 ,145059) #TRIMSELECT: keep tmp_36 [224,0] bytes [224,0] dirty target(mem=0 vm=1073741824)
#TRIMSELECT: candidate=tmp_37 BAT*=058080B0 # (cnt=0, mode=1024, refs=0, wait=0, parent=0, lastused=145088,145088 ,145088) #TRIMSELECT: keep tmp_37 [224,0] bytes [224,0] dirty target(mem=0 vm=1073741824)
#TRIMSELECT: candidate=tmp_41 BAT*=058066F0 # (cnt=0, mode=1024, refs=0, wait=0, parent=0, lastused=145146,145146 ,145146) #TRIMSELECT: keep tmp_41 [224,0] bytes [224,0] dirty target(mem=0 vm=1073741824)
#TRIMSELECT: candidate=doc_query BAT*=057F9D10 # (cnt=0, mode=1024, refs=0, wait=0, parent=0, lastused=149075,149075 ,149075) #TRIMSELECT: keep doc_query [224,0] bytes [224,0] dirty target(mem=0 vm=10737418 24) #TRIMSELECT: candidate=doc_sema BAT*=05845370 # (cnt=0, mode=1024, refs=0, wait=0, parent=0, lastused=149092,149092 ,149092) #TRIMSELECT: keep doc_sema [224,0] bytes [224,0] dirty target(mem=0 vm=107374182 4) #TRIMSELECT: candidate=tmp_374 BAT*=03D722F0 # (cnt=0, mode=4096, refs=0, wait=0, parent=0, lastused=155215,155215 ,155215) #TRIMSELECT: keep tmp_374 [224,0] bytes [0,0] dirty target(mem=0 vm=1073741824) #TRIMSELECT: candidate=prop_pre_39 BAT*=05845CD0 # (cnt=0, mode=4096, refs=0, wait=0, parent=0, lastused=155218,155218 ,155218) #TRIMSELECT: keep prop_pre_39 [224,0] bytes [0,0] dirty target(mem=0 vm=10737418 24) #TRIMSELECT: candidate=tmp_533 BAT*=0583C598 # (cnt=0, mode=4096, refs=0, wait=0, parent=0, lastused=157895,157895 ,157895) #TRIMSELECT: keep tmp_533 [224,0] bytes [0,0] dirty target(mem=0 vm=1073741824) #TRIMSELECT: candidate=prop_pre_310 BAT*=057F7E70 # (cnt=0, mode=4096, refs=0, wait=0, parent=0, lastused=157898,157898 ,157898) #TRIMSELECT: keep prop_pre_310 [224,0] bytes [0,0] dirty target(mem=0 vm=1073741 824) #TRIMSELECT: end #TRIMSELECT: dirty = 1 #TRIMSELECT: candidate=tmp_35 BAT*=03D6A230 # (cnt=0, mode=1024, refs=0, wait=0, parent=0, lastused=145030,145030 ,145030) #TRIMSELECT: delete tmp_35 from trimlist (does not match trim needs) #TRIMSELECT: candidate=tmp_36 BAT*=03D66E60 # (cnt=0, mode=1024, refs=0, wait=0, parent=0, lastused=145059,145059 ,145059) #TRIMSELECT: delete tmp_36 from trimlist (does not match trim needs) #TRIMSELECT: candidate=tmp_37 BAT*=058080B0 # (cnt=0, mode=1024, refs=0, wait=0, parent=0, lastused=145088,145088 ,145088) #TRIMSELECT: delete tmp_37 from trimlist (does not match trim needs) #TRIMSELECT: candidate=tmp_41 BAT*=058066F0 # (cnt=0, mode=1024, refs=0, wait=0, parent=0, lastused=145146,145146 ,145146) #TRIMSELECT: delete tmp_41 from trimlist (does not match trim needs) #TRIMSELECT: candidate=doc_query BAT*=057F9D10 # (cnt=0, mode=1024, refs=0, wait=0, parent=0, lastused=149075,149075 ,149075) #TRIMSELECT: delete doc_query from trimlist (does not match trim needs) #TRIMSELECT: candidate=doc_sema BAT*=05845370 # (cnt=0, mode=1024, refs=0, wait=0, parent=0, lastused=149092,149092 ,149092) #TRIMSELECT: delete doc_sema from trimlist (does not match trim needs) #TRIMSELECT: candidate=tmp_374 BAT*=03D722F0 # (cnt=0, mode=4096, refs=0, wait=0, parent=0, lastused=155215,155215 ,155215) #TRIMSELECT: delete tmp_374 from trimlist (does not match trim needs) #TRIMSELECT: candidate=prop_pre_39 BAT*=05845CD0 # (cnt=0, mode=4096, refs=0, wait=0, parent=0, lastused=155218,155218 ,155218) #TRIMSELECT: delete prop_pre_39 from trimlist (does not match trim needs) #TRIMSELECT: candidate=tmp_533 BAT*=0583C598 # (cnt=0, mode=4096, refs=0, wait=0, parent=0, lastused=157895,157895 ,157895) #TRIMSELECT: delete tmp_533 from trimlist (does not match trim needs) #TRIMSELECT: candidate=prop_pre_310 BAT*=057F7E70 # (cnt=0, mode=4096, refs=0, wait=0, parent=0, lastused=157898,157898 ,157898) #TRIMSELECT: delete prop_pre_310 from trimlist (does not match trim needs) #TRIMSELECT: end #BBPTRIM: no more unload candidates! #BBPTRIM_EXIT: memsize=95140356,vmsize=794361856 !GDKmmap(442499072) fail => BBPtrim(ready) usage[mem=101297568,vm=794361856] # !mallinfo.arena = 15613828 !mallinfo.ordblks = 46134 !mallinfo.smblks = 15492 !mallinfo.hblkhd = 0 !mallinfo.hblks = 0 !mallinfo.usmblks = 13718408 !mallinfo.fsmblks = 899720 !mallinfo.uordblks = 950740 !mallinfo.fordblks = 44960 !ERROR: MT_mmap: MapViewOfFile(6b0, 2, 0, 0, 442499072, 0) failed !OS: Not enough space !ERROR: GDKload: cannot mmap(): name=05\552, ext=theap.priv !OS: Not enough space !ERROR: GDKload failed: name=05\552, ext=theap.priv !ERROR: shredder.mx:append_str2bat: APPEND-STR[PROP_TEXT](final foxes since the silent, quick realms should breach never sheaves--ruthless, daring waters beneat h the close asymptotes c), BUNappend fails !ERROR: CMDshred2bats: operation failed. MonetDB>
Hi Klarinda,
[first of all, our apologies for the late reply --- it's summer and vacation time... ;-)]
Concerning your, it looks as if the 1GB requires more (virtual) memory (and/or disk space) to load than is available on your system. The error indicates that MonetDB/XQuery fails to allocate 442499072 bytes (422 MB), while already 794361856 bytes (757 MB) of your machine's virtual memory (512 MB physical memory plus the size of your Windows installation's "page file") are used by MonetDB. Most probably, your virtual memory is less than 422 MB + 757 MB = 1179 MB, right? It could also be, that you harddisk (at least the partition used by MonetDB) has only less then 422 MB free at the time that MonetDB/XQuery tries to allocate the extra 422 MB.
Given that, it remains open, whether MonetDB/XQuery indeed requires all the memory to load the 1GB document, or whether there is a bug that makes it request more memory than it should require.
To test this, we'd need to have your document --- the pure size of the serialized document is not enough information for us to estimate how big the internal data structure will/must be --- we also need to know (a.o.) how many nodes the document has, what the structure look like, etc.
I tried to generate your document myself in order to analyse the problem, but (while working fine for Text-Centric documents) the XBench/ToXgene generator fails for me with Document-Centric documents (at least with the "large" and "huge" ones): ======== $ perl ./xdbgen.pl ----------------------------------- | XBench Database Generator v1.0 | | (c)2002 by University of Waterloo | -----------------------------------
Database Class: [1]TC/SD [2]TC/MD [3]DC/SD [4]DC/MD Please choose database class (any other key to exit): 3
Database Size: [1]Small [2]Normal [3]Large [4]Huge Please choose database size (default is Normal): 3
Generating template templates/DCSD.tsl==>templates/newDCSD.tsl
Generating TPC-W titles/lastnames ... sh: wgen/tpcw: cannot execute binary file sh: wgen/tpcw: cannot execute binary file
ToXgene 1.1a - (c) 2001 by University of Toronto and IBM Corporation
***** Parsing template: Done!
Generating 250000 elements in items: Done! Reading list titles from input/titles.xml: java.lang.ArrayIndexOutOfBoundsException: 1 at genes.lists.ToxListParser.endElement(ToxListParser.java:125) at org.apache.xerces.parsers.SAXParser.endElement(SAXParser.java:1403) at org.apache.xerces.validators.common.XMLValidator.callEndElement(XMLValidator.java:1480) at org.apache.xerces.framework.XMLDocumentScanner$ContentDispatcher.dispatch(XMLDocumentScanner.java:1204) at org.apache.xerces.framework.XMLDocumentScanner.parseSome(XMLDocumentScanner.java:381) at org.apache.xerces.framework.XMLParser.parse(XMLParser.java:1081) at org.apache.xerces.framework.XMLParser.parse(XMLParser.java:1122) at genes.lists.ToxListParser.parse(ToxListParser.java:53) at genes.lists.ToxList.readFromFile(ToxList.java:354) at genes.lists.ToxList.generate(ToxList.java:230) at toxgene.ToXgene.main(ToXgene.java:160)
***** UNRECOGNIZED ERROR: An unrecognized error has occurred. Please report the following debug information to toxgene-bugs@cs.toronto.edu Please include the template that causes this problem.
java.lang.NullPointerException at genes.lists.ToxList.readFromFile(ToxList.java:419) at genes.lists.ToxList.generate(ToxList.java:230) at toxgene.ToXgene.main(ToXgene.java:160) Total elapsed time: 18759ms. Parsing time: 835ms. List processing time: -1154356332715ms. Done. ========
Hence, could you please send me the (compressed!) document by email (stefan.manegold@cwi.nl), or please it somewhere where I coudl download it from?
One final question (for now): Which version of MonetDB/XQuery are you using?
Kind Regards,
Stefan
On Mon, Jul 24, 2006 at 05:43:33PM +0800, kla gw wrote:
Hi,
I tried MonetDB/XQuery to shred a 1gb xml file, but it failed.
Following is the error message: MonetDB>shred_doc("D:xbench/output/DC1000catalog.xml", "DC1000catalog.xml"); !ERROR: MT_mmap: MapViewOfFile(6b4, 2, 0, 0, 442499072, 0) failed !OS: Not enough space !GDKmmap(442499072) fail => BBPtrim(enter) usage[mem=101297568,vm=794361856]
I use windows XP Professional version 2002 service pack 2, Pentium 4 CPU, 2.40GHz, 512 MB of RAM.
Previously I tried to shred 100mb xml file, and it took 18.532 sec. For this 1 gb file, I left it overnight so I don't know how long it takes till the error message occurs.
Can please help me to solve this problem?
Regards,
Klarinda
Below is the complete error message: MonetDB>shred_doc("D:xbench/output/DC1000catalog.xml", "DC1000catalog.xml"); !ERROR: MT_mmap: MapViewOfFile(6b4, 2, 0, 0, 442499072, 0) failed !OS: Not enough space !GDKmmap(442499072) fail => BBPtrim(enter) usage[mem=101297568,vm=794361856] # !mallinfo.arena = 15613828 !mallinfo.ordblks = 46134 !mallinfo.smblks = 15492 !mallinfo.hblkhd = 0 !mallinfo.hblks = 0 !mallinfo.usmblks = 13718408 !mallinfo.fsmblks = 899720 !mallinfo.uordblks = 950740 !mallinfo.fordblks = 44960 #BBPTRIM_ENTER: memsize=101297568,vmsize=794361856 #BBPTRIM: memtarget=0 vmtarget=1073741824 #TRIMSCAN: mem=0 vm=1, start=1, limit=1 #TRIMSCAN: 145030 0=tmp_35 (#0) #TRIMSCAN: 145059 1=tmp_36 (#0) #TRIMSCAN: 145088 2=tmp_37 (#0) #TRIMSCAN: 145146 3=tmp_41 (#0) #TRIMSCAN: 149075 4=doc_query (#0) #TRIMSCAN: 149092 5=doc_sema (#0) #TRIMSCAN: 155215 6=tmp_374 (#0) #TRIMSCAN: 155218 7=prop_pre_39 (#0) #TRIMSCAN: 157895 8=tmp_533 (#0) #TRIMSCAN: 157898 9=prop_pre_310 (#0) #TRIMSCAN: end at 1 (size=628) #TRIMSELECT: dirty = 0 #TRIMSELECT: candidate=tmp_35 BAT*=03D6A230 # (cnt=0, mode=1024, refs=0, wait=0, parent=0, lastused=145030,145030 ,145030) #TRIMSELECT: keep tmp_35 [224,0] bytes [224,0] dirty target(mem=0 vm=1073741824)
#TRIMSELECT: candidate=tmp_36 BAT*=03D66E60 # (cnt=0, mode=1024, refs=0, wait=0, parent=0, lastused=145059,145059 ,145059) #TRIMSELECT: keep tmp_36 [224,0] bytes [224,0] dirty target(mem=0 vm=1073741824)
#TRIMSELECT: candidate=tmp_37 BAT*=058080B0 # (cnt=0, mode=1024, refs=0, wait=0, parent=0, lastused=145088,145088 ,145088) #TRIMSELECT: keep tmp_37 [224,0] bytes [224,0] dirty target(mem=0 vm=1073741824)
#TRIMSELECT: candidate=tmp_41 BAT*=058066F0 # (cnt=0, mode=1024, refs=0, wait=0, parent=0, lastused=145146,145146 ,145146) #TRIMSELECT: keep tmp_41 [224,0] bytes [224,0] dirty target(mem=0 vm=1073741824)
#TRIMSELECT: candidate=doc_query BAT*=057F9D10 # (cnt=0, mode=1024, refs=0, wait=0, parent=0, lastused=149075,149075 ,149075) #TRIMSELECT: keep doc_query [224,0] bytes [224,0] dirty target(mem=0 vm=10737418 24) #TRIMSELECT: candidate=doc_sema BAT*=05845370 # (cnt=0, mode=1024, refs=0, wait=0, parent=0, lastused=149092,149092 ,149092) #TRIMSELECT: keep doc_sema [224,0] bytes [224,0] dirty target(mem=0 vm=107374182 4) #TRIMSELECT: candidate=tmp_374 BAT*=03D722F0 # (cnt=0, mode=4096, refs=0, wait=0, parent=0, lastused=155215,155215 ,155215) #TRIMSELECT: keep tmp_374 [224,0] bytes [0,0] dirty target(mem=0 vm=1073741824) #TRIMSELECT: candidate=prop_pre_39 BAT*=05845CD0 # (cnt=0, mode=4096, refs=0, wait=0, parent=0, lastused=155218,155218 ,155218) #TRIMSELECT: keep prop_pre_39 [224,0] bytes [0,0] dirty target(mem=0 vm=10737418 24) #TRIMSELECT: candidate=tmp_533 BAT*=0583C598 # (cnt=0, mode=4096, refs=0, wait=0, parent=0, lastused=157895,157895 ,157895) #TRIMSELECT: keep tmp_533 [224,0] bytes [0,0] dirty target(mem=0 vm=1073741824) #TRIMSELECT: candidate=prop_pre_310 BAT*=057F7E70 # (cnt=0, mode=4096, refs=0, wait=0, parent=0, lastused=157898,157898 ,157898) #TRIMSELECT: keep prop_pre_310 [224,0] bytes [0,0] dirty target(mem=0 vm=1073741 824) #TRIMSELECT: end #TRIMSELECT: dirty = 1 #TRIMSELECT: candidate=tmp_35 BAT*=03D6A230 # (cnt=0, mode=1024, refs=0, wait=0, parent=0, lastused=145030,145030 ,145030) #TRIMSELECT: delete tmp_35 from trimlist (does not match trim needs) #TRIMSELECT: candidate=tmp_36 BAT*=03D66E60 # (cnt=0, mode=1024, refs=0, wait=0, parent=0, lastused=145059,145059 ,145059) #TRIMSELECT: delete tmp_36 from trimlist (does not match trim needs) #TRIMSELECT: candidate=tmp_37 BAT*=058080B0 # (cnt=0, mode=1024, refs=0, wait=0, parent=0, lastused=145088,145088 ,145088) #TRIMSELECT: delete tmp_37 from trimlist (does not match trim needs) #TRIMSELECT: candidate=tmp_41 BAT*=058066F0 # (cnt=0, mode=1024, refs=0, wait=0, parent=0, lastused=145146,145146 ,145146) #TRIMSELECT: delete tmp_41 from trimlist (does not match trim needs) #TRIMSELECT: candidate=doc_query BAT*=057F9D10 # (cnt=0, mode=1024, refs=0, wait=0, parent=0, lastused=149075,149075 ,149075) #TRIMSELECT: delete doc_query from trimlist (does not match trim needs) #TRIMSELECT: candidate=doc_sema BAT*=05845370 # (cnt=0, mode=1024, refs=0, wait=0, parent=0, lastused=149092,149092 ,149092) #TRIMSELECT: delete doc_sema from trimlist (does not match trim needs) #TRIMSELECT: candidate=tmp_374 BAT*=03D722F0 # (cnt=0, mode=4096, refs=0, wait=0, parent=0, lastused=155215,155215 ,155215) #TRIMSELECT: delete tmp_374 from trimlist (does not match trim needs) #TRIMSELECT: candidate=prop_pre_39 BAT*=05845CD0 # (cnt=0, mode=4096, refs=0, wait=0, parent=0, lastused=155218,155218 ,155218) #TRIMSELECT: delete prop_pre_39 from trimlist (does not match trim needs) #TRIMSELECT: candidate=tmp_533 BAT*=0583C598 # (cnt=0, mode=4096, refs=0, wait=0, parent=0, lastused=157895,157895 ,157895) #TRIMSELECT: delete tmp_533 from trimlist (does not match trim needs) #TRIMSELECT: candidate=prop_pre_310 BAT*=057F7E70 # (cnt=0, mode=4096, refs=0, wait=0, parent=0, lastused=157898,157898 ,157898) #TRIMSELECT: delete prop_pre_310 from trimlist (does not match trim needs) #TRIMSELECT: end #BBPTRIM: no more unload candidates! #BBPTRIM_EXIT: memsize=95140356,vmsize=794361856 !GDKmmap(442499072) fail => BBPtrim(ready) usage[mem=101297568,vm=794361856] # !mallinfo.arena = 15613828 !mallinfo.ordblks = 46134 !mallinfo.smblks = 15492 !mallinfo.hblkhd = 0 !mallinfo.hblks = 0 !mallinfo.usmblks = 13718408 !mallinfo.fsmblks = 899720 !mallinfo.uordblks = 950740 !mallinfo.fordblks = 44960 !ERROR: MT_mmap: MapViewOfFile(6b0, 2, 0, 0, 442499072, 0) failed !OS: Not enough space !ERROR: GDKload: cannot mmap(): name=05\552, ext=theap.priv !OS: Not enough space !ERROR: GDKload failed: name=05\552, ext=theap.priv !ERROR: shredder.mx:append_str2bat: APPEND-STR[PROP_TEXT](final foxes since the silent, quick realms should breach never sheaves--ruthless, daring waters beneat h the close asymptotes c), BUNappend fails !ERROR: CMDshred2bats: operation failed. MonetDB>
Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=D... _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
Hi Stefan,
Thanks for your reply.
Concerning your, it looks as if the 1GB requires more (virtual) memory (and/or disk space) to load than is available on your system. The error indicates that MonetDB/XQuery fails to allocate 442499072 bytes (422 MB), while already 794361856 bytes (757 MB) of your machine's virtual memory (512 MB physical memory plus the size of your Windows installation's "page file") are used by MonetDB. Most probably, your virtual memory is less than 422 MB + 757 MB = 1179 MB, right? It could also be, that you harddisk (at least the partition used by MonetDB) has only less then 422 MB free at the time that MonetDB/XQuery tries to allocate the extra 422 MB.
I'm sorry, I don't really understand about the memory thing. But when I check in: Right Click My Computer -> Properties -> Advanced -> Performance Settings -> Advanced -> Virtual Memory Change, it is said that Drive C: Paging File Size 768-1536 MB Space Available: 3156MB
My Hard Disk free space in Drive C is 2.33 GB (total is 12 GB).
I use windows XP Professional version 2002 service pack 2, Pentium 4 CPU, 2.40GHz, 512 MB of RAM.
How much memory and space does monetDB use?
How to change the MonetDB/XQuery setting to shred the file in "D:\MyFolder\MonetDB" instead of the default location "C:\Documents and Settings\klarinda\Application Data\MonetDB\dbfarm"?
To test this, we'd need to have your document --- the pure size of the serialized document is not enough information for us to estimate how big the internal data structure will/must be --- we also need to know (a.o.) how many nodes the document has, what the structure look like, etc.
You can download the file (DC1000catalog.xml) from here: http://www.yousendit.com/transfer.php?action=download&ufid=1604D48611A31... (the file is only available for 7 days) The size after zipping is around 214mb.
I tried to generate your document myself in order to analyse the problem, but (while working fine for Text-Centric documents) the XBench/ToXgene generator fails for me with Document-Centric documents (at least with the "large" and "huge" ones):
I don't know why it throws error. I get it from my friend and it's working fine. Have you tried the small and normal? Actually I also never tried the huge one.
One final question (for now): Which version of MonetDB/XQuery are you using?
I'm using version 0.12.0 FYI, previously I installed 0.10.2, then I uninstall it and install a new one.
Thanks,
Klarinda
Hi Klarinda,
On Fri, Aug 04, 2006 at 11:54:47AM +0800, kla gw wrote:
Hi Stefan,
Thanks for your reply.
Concerning your, it looks as if the 1GB requires more (virtual) memory (and/or disk space) to load than is available on your system. The error indicates that MonetDB/XQuery fails to allocate 442499072 bytes (422 MB), while already 794361856 bytes (757 MB) of your machine's virtual memory (512 MB physical memory plus the size of your Windows installation's "page file") are used by MonetDB. Most probably, your virtual memory is less than 422 MB + 757 MB = 1179 MB, right? It could also be, that you harddisk (at least the partition used by MonetDB) has only less then 422 MB free at the time that MonetDB/XQuery tries to allocate the extra 422 MB.
I'm sorry, I don't really understand about the memory thing. But when I check in: Right Click My Computer -> Properties -> Advanced -> Performance Settings -> Advanced -> Virtual Memory Change, it is said that Drive C: Paging File Size 768-1536 MB Space Available: 3156MB
My Hard Disk free space in Drive C is 2.33 GB (total is 12 GB).
I use windows XP Professional version 2002 service pack 2, Pentium 4 CPU, 2.40GHz, 512 MB of RAM.
How much memory and space does monetDB use?
MonetDB tries to use as much memory as available and necessary for running the very task, what ever is less. Obviously, the latter depends (a.o.) on the amount of data that has to be processed.
In principle, your memory settings look fine to me.
I just tried to load your 1.1GB DC1000catalog.xml document on my laptop (Pentium M (Dothan) 1.5 GHz, 2 GB RAM, 2.1 GB free disk space; running Fedora Core 5 Linux (Kernel 2.6.17)) into MonetDB/XQuery 0.12.0, and it did work. MonetDB (Mserver) never required more than 1 GB of (virtual) memory, but is took quite long (2552.377s ~= 42.5 min) (for the expert: I/O-bound!): (I haven't analysed, yet, why it is that slow...)
======== $ Mserver --dbinit='module(pathfinder);' # Monet Database Server V4.12.0 # Copyright (c) 1993-2006, CWI. All rights reserved. # Compiled for i686-redhat-linux-gnu/32bit with 32bit OIDs; dynamically linked. # Visit http://monetdb.cwi.nl/ for further information. MonetDB>shred_doc("/data/StM/DC1000catalog.xml","DC1000catalog.xml"); # Shredded XML doc("DC1000catalog.xml"), total time after commit=2552.377s MonetDB> ========
Hence, I'm not quite sure, why it fails in your case. Unfortunately(??), I'm everything but a Windows expert, and I don't have any Windows machine to test on.
Maybe someone else could try to load the document into MonetDB/XQuery 0.12.0 on Windows?
The document is now also available at http://www.cwi.nl/~manegold/Public/DC1000catalog.xml.zip
How to change the MonetDB/XQuery setting to shred the file in "D:\MyFolder\MonetDB" instead of the default location "C:\Documents and Settings\klarinda\Application Data\MonetDB\dbfarm"?
Well, you have to start the MonetDB/XQuery server (Mserver) with "--dbfarm=D:\MyFolder\MonetDB" --- at least on a Unix system one simply starts Mserver from a command line --- AFAIK, on Windows, the MonetDB/XQuery server is started by clicking on some icon, hence, I'm not sure how to pass extra arguments... sorry...
Sjoerd, could you please help and enlighten me/us?
To test this, we'd need to have your document --- the pure size of the serialized document is not enough information for us to estimate how big the internal data structure will/must be --- we also need to know (a.o.) how many nodes the document has, what the structure look like, etc.
You can download the file (DC1000catalog.xml) from here: http://www.yousendit.com/transfer.php?action=download&ufid=1604D48611A31... (the file is only available for 7 days) The size after zipping is around 214mb.
I tried to generate your document myself in order to analyse the problem, but (while working fine for Text-Centric documents) the XBench/ToXgene generator fails for me with Document-Centric documents (at least with the "large" and "huge" ones):
I don't know why it throws error. I get it from my friend and it's working fine. Have you tried the small and normal? Actually I also never tried the huge one.
For small ans normal documents. it works fine for me. Thank you very much for proving the large document.
Stefan
One final question (for now): Which version of MonetDB/XQuery are you using?
I'm using version 0.12.0 FYI, previously I installed 0.10.2, then I uninstall it and install a new one.
Thanks,
Klarinda
participants (2)
-
kla gw -
Stefan Manegold