Memory management, 8 years later

Roberto Cornacchia roberto.cornacchia at gmail.com
Thu Jun 28 13:40:03 CEST 2018


On Thu, 28 Jun 2018 at 12:45 Arjen de Rijke <arjen.de.rijke at cwi.nl> wrote:

> > At some point (can't remember when) mserver5 stopped respecting ulimit.
> No
> > matter how I change the limits, it always sees all available physical
> ram. Does
> > any other mechanism exist?
>
> I think this is the correct behaviour of ulimit. The mserver5 process has
> nothing to do with it, ulimit controls how much resources a shell can use,
> it does not "fake" the amount of physical memory of the machine. And the
> behaviour of a process that hits the ulimit setting likely behaves
> differently than a process on a machine that runs out of real memory. And
> you also have to take into account that you can have ulimit settings for
> users, processes and shells.
>
>
Yes, that is the correct behaviour. MonetDB used to exploit that
information to self-enforce limits on the available resources. Now it
doesn't, and it's ok, I'm only poinint out that these is no other mechanism
to enforce limits.


> I think you are right that the only available solution is vm's. Otherwise
> monetdb would have to enforce the memory usage per process itself. But i am
> not sure if this information is available internally. And even if it is,
> how would you handle for example a situation where one of the dbfarms runs
> out of memory during a query that is split over multiple dbfarms?
>
>
Indeed, I wonder why there is no configuration option to have some degree
of control on how aggressively memory should be used. I would expect
something like "max 2GB resident memory", and if this means more memory
mapping is happening (thus lower performance), then be it. But no control
at all with many mserver5 running leaves me a bit frustrated.

Also, do you think there could be recommendations about swap memory (in
particular in this shared environment)?
I confess I don't have very clear in which occasions swap memory is used.
Naively, I always thought that memory mapping (provided you have enough
disk space) would make swap memory not necessary to use. But clearly it's
not really like that.



> Arjen
>
> >
> > Does anyone have experience with / tips about multi-dbfarm environments?
> >
> > Thanks!
> > Roberto
> >
> > _______________________________________________
> > users-list mailing list
> > users-list at monetdb.org
> > https://www.monetdb.org/mailman/listinfo/users-list
> _______________________________________________
> users-list mailing list
> users-list at monetdb.org
> https://www.monetdb.org/mailman/listinfo/users-list
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.monetdb.org/pipermail/users-list/attachments/20180628/ddc09425/attachment.html>


More information about the users-list mailing list