Memory management, 8 years later

Arjen de Rijke arjen.de.rijke at cwi.nl
Thu Jun 28 12:44:01 CEST 2018


Hi Roberto,

----- Original Message -----
> From: "Roberto Cornacchia" <roberto.cornacchia at gmail.com>
> To: "Communication channel for MonetDB users" <users-list at monetdb.org>
> Sent: Thursday, June 28, 2018 11:39:09 AM
> Subject: Memory management, 8 years later

> Hi all,
> 
> 8 yeas after this interesting thread about using ulimit to limit the data
> segment size (
> https://www.monetdb.org/pipermail/users-list/2010-March/004007.html ), a few
> things have changed and I find myself a bit lost again about memory management.
> 
> At some point (can't remember when) mserver5 stopped respecting ulimit. No
> matter how I change the limits, it always sees all available physical ram. Does
> any other mechanism exist?

I think this is the correct behaviour of ulimit. The mserver5 process has nothing to do with it, ulimit controls how much resources a shell can use, it does not "fake" the amount of physical memory of the machine. And the behaviour of a process that hits the ulimit setting likely behaves differently than a process on a machine that runs out of real memory. And you also have to take into account that you can have ulimit settings for users, processes and shells.

> 
> The issue I'm facing all the time, and for which I find very little
> documentation, is:
> - multiple dbfarms, possibly tens of mserver5 instances, running on the same
> (virtual) machine, plus other applications
> - Not necessarily, but most times each dmfarm is served by a Docker container
> - CPU and memory resources need to be share fairly (ideally, controlling
> priorities)
> - No mserver5 should ever be killed because of memory issues
> 
> Now, especially when using Docker, there are means to control the maximum memory
> size allowed to each process. However, the resolution in case of overusage is
> to kill the process. That is of course not a solution.
> 
> What I'm looking for is: can I set a cap to resources *at process level*, in a
> way that mserver5 will just *see* those, make the best out of them, and never
> try to use more? (yes, one way would be a VM, but far from ideal for me)

I think you are right that the only available solution is vm's. Otherwise monetdb would have to enforce the memory usage per process itself. But i am not sure if this information is available internally. And even if it is, how would you handle for example a situation where one of the dbfarms runs out of memory during a query that is split over multiple dbfarms?

Arjen

> 
> Does anyone have experience with / tips about multi-dbfarm environments?
> 
> Thanks!
> Roberto
> 
> _______________________________________________
> users-list mailing list
> users-list at monetdb.org
> https://www.monetdb.org/mailman/listinfo/users-list


More information about the users-list mailing list