High Memory Usage - Sever Not Releasing Resources?
Hi all:
I ran into an issue yesterday when I ran a simple group count query - The query took 7 minutes, 13 seconds to run.
Query was as follows -
SELECT zone, count(*) FROM chronicle.events WHERE event_type = 'mb' GROUP BY zone
This table, which a merge table currently has 546 million events. The merge table is made up of currently two merge table each representing a year (events_2018, and events_2017). Those merge tables are each made up of four tables, events_year_zone_(1-4)
I have a scheduled task which inserts events into the relevant tables every 5 minutes from 5 Microsoft SQL Servers. Roughly 12,000 events broken across 5 inserts (one for each server).
The import process is handled with a C# Windows service that utilizes ODBC to talk to MonetDB.
The server is currently MonetDB v11.27.11 (July2017-SP3) running on VMware 6.5 in a CentOS 7 VM with 64GB of RAM, and 16 CPU cores. When I ran into this issue yesterday, the mserver5 process was utilizing 90% of available RAM.
This was making my insertion task take longer and longer each time, and simple queries like the one above took minutes. Logged into the server via SSH, the input lag as a result of the almost total resource usage caused me to reboot the server.
I restarted the VM, and the same query took 20 seconds.
I'd like to sort where this issue might be coming from, but I'm not sure where to look. This server is currently part of a proof-of-concept project that will soon be going to production.
Any guidance is greatly appreciated.
Adam
Hello Adam,
First of all:
- does this also happen with Mar2018? - Is this repeatable?
If answers to both questions are Yes, can you please provide us a script/test programme/etc to reproduce the problem?
Regards, Jennie
On 8 Jun 2018, at 17:03, Doherty, Adam adam.doherty@esso.ca wrote:
Hi all:
I ran into an issue yesterday when I ran a simple group count query - The query took 7 minutes, 13 seconds to run.
Query was as follows -
SELECT zone, count(*) FROM chronicle.events WHERE event_type = 'mb' GROUP BY zone
This table, which a merge table currently has 546 million events. The merge table is made up of currently two merge table each representing a year (events_2018, and events_2017). Those merge tables are each made up of four tables, events_year_zone_(1-4)
I have a scheduled task which inserts events into the relevant tables every 5 minutes from 5 Microsoft SQL Servers. Roughly 12,000 events broken across 5 inserts (one for each server).
The import process is handled with a C# Windows service that utilizes ODBC to talk to MonetDB.
The server is currently MonetDB v11.27.11 (July2017-SP3) running on VMware 6.5 in a CentOS 7 VM with 64GB of RAM, and 16 CPU cores. When I ran into this issue yesterday, the mserver5 process was utilizing 90% of available RAM.
This was making my insertion task take longer and longer each time, and simple queries like the one above took minutes. Logged into the server via SSH, the input lag as a result of the almost total resource usage caused me to reboot the server.
I restarted the VM, and the same query took 20 seconds.
I'd like to sort where this issue might be coming from, but I'm not sure where to look. This server is currently part of a proof-of-concept project that will soon be going to production.
Any guidance is greatly appreciated.
Adam
users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
Hello Jennie:
The documentation doesn't clearly identify the upgrade path for MonetDB. This server is an offline environment, so I assume that I would lock my database, shutdown the monetdb service and run yum update with the newest RPMs.
I'll install the latest version and report my findings.
Adam
-----Original Message----- From: users-list [mailto:users-list-bounces+adam.doherty=esso.ca@monetdb.org] On Behalf Of Ying Zhang Sent: Friday, June 8, 2018 09:17 To: Communication channel for MonetDB users users-list@monetdb.org Subject: Re: High Memory Usage - Sever Not Releasing Resources?
Hello Adam,
First of all:
- does this also happen with Mar2018? - Is this repeatable?
If answers to both questions are Yes, can you please provide us a script/test programme/etc to reproduce the problem?
Regards, Jennie
On 8 Jun 2018, at 17:03, Doherty, Adam adam.doherty@esso.ca wrote:
Hi all:
I ran into an issue yesterday when I ran a simple group count query - The query took 7 minutes, 13 seconds to run.
Query was as follows -
SELECT zone, count(*) FROM chronicle.events WHERE event_type = 'mb' GROUP BY zone
This table, which a merge table currently has 546 million events. The merge table is made up of currently two merge table each representing a year (events_2018, and events_2017). Those merge tables are each made up of four tables, events_year_zone_(1-4)
I have a scheduled task which inserts events into the relevant tables every 5 minutes from 5 Microsoft SQL Servers. Roughly 12,000 events broken across 5 inserts (one for each server).
The import process is handled with a C# Windows service that utilizes ODBC to talk to MonetDB.
The server is currently MonetDB v11.27.11 (July2017-SP3) running on VMware 6.5 in a CentOS 7 VM with 64GB of RAM, and 16 CPU cores. When I ran into this issue yesterday, the mserver5 process was utilizing 90% of available RAM.
This was making my insertion task take longer and longer each time, and simple queries like the one above took minutes. Logged into the server via SSH, the input lag as a result of the almost total resource usage caused me to reboot the server.
I restarted the VM, and the same query took 20 seconds.
I'd like to sort where this issue might be coming from, but I'm not sure where to look. This server is currently part of a proof-of-concept project that will soon be going to production.
Any guidance is greatly appreciated.
Adam
users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
participants (2)
-
Doherty, Adam -
Ying Zhang