[Monetdb-developers] Monetdb-developers Digest, Vol 30, Issue 10

Peter Boncz P.Boncz at cwi.nl
Sun Nov 23 17:58:33 CET 2008


Hi Stefan (the non-german .de one)

> Maybe a very strange idea, could be your better solution. If by default
> these operations are mmap'ed, they should be taken care of by a disk based
> memory extension. Hence being independent of RAM/SWAP but having all pro's
> of both.

You are right. In fact, this is what MonetDB is supposed to do. Memory is
organized in contiguous Heap objects.  This does not only hold for BATs but
also for Hash Tables. At some moment, I did add the possibility to use a
real memory mapped files for their allocation, as a last resort. The problem
was there that the Heap() API did not have a filename parameter, and a
HEAPextend() thus would not know what file to use. In order not to break the
API, I decided to initialize the filename fields at BAT creation time (this
is why you may sometimes see .hhash and .thash files in a just stopped
MonetDB repository -- even though hash tables are not persistent).

With hindsight, this API decision was bad judgement; because the question
whether real file mapping is used now depends on the question whether the
filename was initialized at BAT or Hash table creation. Now, this should be
the case, but it would be my first question to address in a debugging
session.

I also recall that in case of the Skyserver TB dataset, key checking was
disabled to to similar problems.

Of course, if we would find and fix this bug, life will not be rosy for you,
doing random access (key uniqueness checking) into a 14GB heap that will
never fit your 2GB RAM. That goes back to your question whether it is in
fact reasonable to materialize 14GB. Questions like these led to a redesign
that is much more careful with memory consumption (X100) whose spin-off
development regrettably means that my time for such debugging sessions
currently is limited.

Peter

PS
The likely cause for the problem is a shortage of swap space. Increasing
your swap file size (with a few 10s of GB) should be a workaround for the
problem.






More information about the developers-list mailing list