[Monetdb-developers] OS: Not enough space (again)

Stefan Manegold Stefan.Manegold at cwi.nl
Fri Sep 8 08:45:50 CEST 2006


Hi Erwin,

as my experiments on Linux showed, shredding 1 GB XML file with
MonetDB/XQuery should not require more than 1 GB of (virtual) memory,
however, in the process, MonetDB/XQuery requires (at least once) to be able
to allocate a consecutive memory area of 436731904 bytes (i.e., ~417 MB). In
particular this means, that a *consecutive* area of ~417 MB needs to be free
in the very process' address space. While this seems to work fine on Linux,
it does not seem to work on Windows. The reasons are (as far as I know)
twofold, 
(1) 32-bit Windows only provides an address space of only 2 GB, while Linux
usually provides 3 GB (theoretically, 32-bit systems could allow up to 4 GB);
hence, even if the physical memory or Windows page file is large than 2 GB,
each process can at most allocate up to 2 GB.
(2) Windows seems to be more vulnerable to address space fragmentation than
Linux, i.e., while there might still be more than 417 MB free in total in
the process' address space, none of the *consecutive* free blocks is large
enough to accommodate the requested 417 MB.

Lacking a Windows machine, I unfortunately cannot verify this practically.


Could any "Windows expert" help?

The document in question is available from
	http://www.cwi.nl/~manegold/Public/DC1000catalog.xml.zip


Concerning MonetDB/XQuery, I'll try to analyze, why/whether we indeed need
to allocate that 417 MB block while shredding this 1 GB document, but I
guess we cannot avoid this, because this might be required by MonetDB
processing model.

Hence, there's quite a chance that MonetDB/XQuery represents quite a
challenge to the above mentioned address space limitations of 32-bit Windows
systems. The short-term solutions (unless we unexpectedly find out that this
is caused by a bug in MonetDB/XQuery) are to either use a different
operating system (e.g., Linux) or "at least" to use a 64-bit Windows system.

My apologies for the inconvenience!

Stefan


On Fri, Sep 08, 2006 at 10:11:06AM +0800, Erwin Leonardi wrote:
> Hi there,
> 
> I tried to shred 1GB XML data into MonetDB. However, I faced the same
> problem as
> Klarinda. The error is as follows.
> 
> -------------- Error ---------------
> ERROR = !ERROR: MT_mmap: MapViewOfFile(68c, 2, 0, 0, 436731904, 0) failed
>        !OS: Not enough space
>        !GDKmmap(436731904) fail => BBPtrim(enter)
> usage[mem=212972396,vm=803209216]
> ---------- End Error ---------------
> 
> I use a PC with 1GB RAM, more than 5GB free hard disk space, and WinXP SP2
> as OS.
> I have tried to set Windows' page file to 2GB. I read the previous postings
> regarding
> the same problem. I notice that Stefan has done similar experiment using his
> laptop
> (Pentium M (Dothan) 1.5 GHz, 2 GB RAM, 2.1 GB free disk space; running
> Fedora
> Core 5 Linux (Kernel 2.6.17)).
> 
> My question is whether I need more than 1GB RAM if I want to shred 1GB XML
> data.
> 
> Thanks
> Erwin Leonardi

> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> Monetdb-developers mailing list
> Monetdb-developers at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/monetdb-developers


-- 
| Dr. Stefan Manegold | mailto:Stefan.Manegold at cwi.nl |
| CWI,  P.O.Box 94079 | http://www.cwi.nl/~manegold/  |
| 1090 GB Amsterdam   | Tel.: +31 (20) 592-4212       |
| The Netherlands     | Fax : +31 (20) 592-4312       |




More information about the developers-list mailing list