MonetDB crashing due to out of memory - monetdbd[2247] 1.7 (Aug2018-SP2)
Thank you Ying and Jenin. I am going to upgrade the MonetDB as you suggested and then go from there. Can you please let me know where can I find the upgrade procedure.
Thank you for your help in advance.
Suresh Devarakonda 84.51° TDO - Database Support Team D: 513.632.3692 M: 859.512.1839 A: 100 W. 5th Street, Cincinnati, Ohio 45202
Cincinnati | Chicago | New York | Portland Facebook LinkedIn Instagram Twitter Please consider the environment before printing this email.
-----Original Message----- From: users-list users-list-bounces+suresh.devarakonda=8451.com@monetdb.org On Behalf Of users-list-request@monetdb.org Sent: Friday, January 17, 2020 6:00 AM To: users-list@monetdb.org Subject: users-list Digest, Vol 89, Issue 6
Send users-list mailing list submissions to users-list@monetdb.org
To subscribe or unsubscribe via the World Wide Web, visit https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.monetd... or, via email, send a message with subject or body 'help' to users-list-request@monetdb.org
You can reach the person managing the list at users-list-owner@monetdb.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of users-list digest..."
Today's Topics:
1. Re: MonetDB crashing due to out of memory - monetdbd[2247] 1.7 (Aug2018-SP2) (Ying Zhang)
----------------------------------------------------------------------
Message: 1 Date: Thu, 16 Jan 2020 12:01:27 +0100 From: Ying Zhang Y.Zhang@cwi.nl To: Communication channel for MonetDB users users-list@monetdb.org Subject: Re: MonetDB crashing due to out of memory - monetdbd[2247] 1.7 (Aug2018-SP2) Message-ID: E6A3FCCA-8023-4EBD-AF7A-CF89F615B2D8@cwi.nl Content-Type: text/plain; charset=utf-8
Hai Suresh,
First of all, I?d recommend you to upgrade your MonetDB installation to the most recent one, i.e. Nov2019-SP1. There are many improvements related to your problem since Aug2018-SP2. Also, the version you?re using is no longer supported by the open-source team.
Please let us know if upgrade helps. But below, a bit more info. inlined.
On 15 Jan 2020, at 23:44, Suresh Devarakonda Suresh.Devarakonda@8451.com wrote:
Hi,
The Moentdb is crashing every couple of weeks due to out of memory in our environment.
Did it really crash, or did MonetDB give error messages and stop executing the query? Crash always indicates a problem in MonetDB, but if it?s the second case, it?s expected.
What I have noticed that MonetDB is eating memory and not releasing it at all once we brought up the MonetDB server. I am not sure if this is how MonetDB works or is it a bug.
MonetDB is supposed to free up the memory when it no longer use it, which often happens towards the end of a query execution.
For further investigation, we need more information after you have upgraded to Nov2019-SP1. In general, we?ll need information with which we can reproduce the problem, e.g. data + queries.
This is running on monetdbd[2247] 1.7 (Aug2018-SP2) version. Also wondering is there way to limit the memory usage of MonetDB.
Please beware that depending on your data sizes and (concurrent) queries, MonetDB might run faster into out-of-memory problem, if you limit the amount of memory it may use,
How about putting MonetDB in a VM?
There are also some undocumented kernel configurations for this, which we don?t recommend to our users lightly?
Can you guys help me to resolve this issue.
If you need professional support, please contact the supporting company MonetDB Solutions.
Regards, Jennie
[root@xxx ~]# monetdb status name state health remarks INVPRDB R 6h 67% 3d mapi:monetdb://xxxxx:50000/INVPRDB FSPPRDB R 6h 50% 6w mapi:monetdb://xxxxx:50000/FSPPRDB [root@xxxxx ~]# free -g --- Current utilization total used free shared buff/cache available Mem: 31 0 22 0 7 30 Swap: 3 0 3
<image003.png>
<image002.jpg> Suresh Devarakonda 84.51? TDO - Database Support Team D: 513.632.3692 M: 859.512.1839 A: 100 W. 5th Street, Cincinnati, Ohio 45202
Cincinnati | Chicago | New York | Portland Facebook LinkedIn Instagram Twitter Please consider the environment before printing this email.
users-list mailing list users-list@monetdb.org https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww. monetdb.org%2Fmailman%2Flistinfo%2Fusers-list&data=02%7C01%7Csures h.devarakonda%408451.com%7Cae946815e409491cb83d08d79b3c9c2c%7C5f9dc6bd f38a454a864cc803691193c5%7C0%7C0%7C637148556952386018&sdata=3JmWyH MNaeMNrBp8vX%2FAYES%2By%2FDqNHqGmE%2FKX%2F%2F4m%2Bs%3D&reserved=0
------------------------------
Subject: Digest Footer
_______________________________________________ users-list mailing list users-list@monetdb.org https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.monetd...
------------------------------
End of users-list Digest, Vol 89, Issue 6 *****************************************
Hi Ying ,
I couldn't find the procedure to upgrade the MonetDB in documentation. Can you please let me know where can I find that information.
Suresh Devarakonda 84.51° TDO - Database Support Team D: 513.632.3692 M: 859.512.1839 A: 100 W. 5th Street, Cincinnati, Ohio 45202
Cincinnati | Chicago | New York | Portland Facebook LinkedIn Instagram Twitter Please consider the environment before printing this email.
-----Original Message----- From: Suresh Devarakonda Sent: Monday, January 20, 2020 10:21 AM To: users-list@monetdb.org Subject: MonetDB crashing due to out of memory - monetdbd[2247] 1.7 (Aug2018-SP2)
Thank you Ying and Jenin. I am going to upgrade the MonetDB as you suggested and then go from there. Can you please let me know where can I find the upgrade procedure.
Thank you for your help in advance.
Suresh Devarakonda 84.51° TDO - Database Support Team D: 513.632.3692 M: 859.512.1839 A: 100 W. 5th Street, Cincinnati, Ohio 45202
Cincinnati | Chicago | New York | Portland Facebook LinkedIn Instagram Twitter Please consider the environment before printing this email.
-----Original Message----- From: users-list users-list-bounces+suresh.devarakonda=8451.com@monetdb.org On Behalf Of users-list-request@monetdb.org Sent: Friday, January 17, 2020 6:00 AM To: users-list@monetdb.org Subject: users-list Digest, Vol 89, Issue 6
Send users-list mailing list submissions to users-list@monetdb.org
To subscribe or unsubscribe via the World Wide Web, visit https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.monetd... or, via email, send a message with subject or body 'help' to users-list-request@monetdb.org
You can reach the person managing the list at users-list-owner@monetdb.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of users-list digest..."
Today's Topics:
1. Re: MonetDB crashing due to out of memory - monetdbd[2247] 1.7 (Aug2018-SP2) (Ying Zhang)
----------------------------------------------------------------------
Message: 1 Date: Thu, 16 Jan 2020 12:01:27 +0100 From: Ying Zhang Y.Zhang@cwi.nl To: Communication channel for MonetDB users users-list@monetdb.org Subject: Re: MonetDB crashing due to out of memory - monetdbd[2247] 1.7 (Aug2018-SP2) Message-ID: E6A3FCCA-8023-4EBD-AF7A-CF89F615B2D8@cwi.nl Content-Type: text/plain; charset=utf-8
Hai Suresh,
First of all, I?d recommend you to upgrade your MonetDB installation to the most recent one, i.e. Nov2019-SP1. There are many improvements related to your problem since Aug2018-SP2. Also, the version you?re using is no longer supported by the open-source team.
Please let us know if upgrade helps. But below, a bit more info. inlined.
On 15 Jan 2020, at 23:44, Suresh Devarakonda Suresh.Devarakonda@8451.com wrote:
Hi,
The Moentdb is crashing every couple of weeks due to out of memory in our environment.
Did it really crash, or did MonetDB give error messages and stop executing the query? Crash always indicates a problem in MonetDB, but if it?s the second case, it?s expected.
What I have noticed that MonetDB is eating memory and not releasing it at all once we brought up the MonetDB server. I am not sure if this is how MonetDB works or is it a bug.
MonetDB is supposed to free up the memory when it no longer use it, which often happens towards the end of a query execution.
For further investigation, we need more information after you have upgraded to Nov2019-SP1. In general, we?ll need information with which we can reproduce the problem, e.g. data + queries.
This is running on monetdbd[2247] 1.7 (Aug2018-SP2) version. Also wondering is there way to limit the memory usage of MonetDB.
Please beware that depending on your data sizes and (concurrent) queries, MonetDB might run faster into out-of-memory problem, if you limit the amount of memory it may use,
How about putting MonetDB in a VM?
There are also some undocumented kernel configurations for this, which we don?t recommend to our users lightly?
Can you guys help me to resolve this issue.
If you need professional support, please contact the supporting company MonetDB Solutions.
Regards, Jennie
[root@xxx ~]# monetdb status name state health remarks INVPRDB R 6h 67% 3d mapi:monetdb://xxxxx:50000/INVPRDB FSPPRDB R 6h 50% 6w mapi:monetdb://xxxxx:50000/FSPPRDB [root@xxxxx ~]# free -g --- Current utilization total used free shared buff/cache available Mem: 31 0 22 0 7 30 Swap: 3 0 3
<image003.png>
<image002.jpg> Suresh Devarakonda 84.51? TDO - Database Support Team D: 513.632.3692 M: 859.512.1839 A: 100 W. 5th Street, Cincinnati, Ohio 45202
Cincinnati | Chicago | New York | Portland Facebook LinkedIn Instagram Twitter Please consider the environment before printing this email.
users-list mailing list users-list@monetdb.org https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww. monetdb.org%2Fmailman%2Flistinfo%2Fusers-list&data=02%7C01%7Csures h.devarakonda%408451.com%7Cae946815e409491cb83d08d79b3c9c2c%7C5f9dc6bd f38a454a864cc803691193c5%7C0%7C0%7C637148556952386018&sdata=3JmWyH MNaeMNrBp8vX%2FAYES%2By%2FDqNHqGmE%2FKX%2F%2F4m%2Bs%3D&reserved=0
------------------------------
Subject: Digest Footer
_______________________________________________ users-list mailing list users-list@monetdb.org https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.monetd...
------------------------------
End of users-list Digest, Vol 89, Issue 6 *****************************************
There is currently no documented db upgrade procedure as the database upgrade is done automatically after you have downloaded (https://www.monetdb.org/Downloads) and installed the newer MonetDB software version and started the MonetDB server for a specific database. If you have multiple database you will need to start a MonetDB server for each of them to start the database upgrade process.
Important: always make a backup of the database files (in the dbfarm directory) before you install the new software, just in case you need to go back to the previous software version. Before making a copy of the database files, make sure the MonetDB servers (monetdbd and mserver5 processes) are stopped.
See also: https://www.monetdb.org/Documentation/Guide/Installation
On 1/22/20 5:17 PM, Suresh Devarakonda wrote:
Hi Ying ,
I couldn't find the procedure to upgrade the MonetDB in documentation. Can you please let me know where can I find that information.
Suresh Devarakonda 84.51° TDO - Database Support Team D: 513.632.3692 M: 859.512.1839 A: 100 W. 5th Street, Cincinnati, Ohio 45202
Cincinnati | Chicago | New York | Portland Facebook LinkedIn Instagram Twitter Please consider the environment before printing this email.
-----Original Message----- From: Suresh Devarakonda Sent: Monday, January 20, 2020 10:21 AM To: users-list@monetdb.org Subject: MonetDB crashing due to out of memory - monetdbd[2247] 1.7 (Aug2018-SP2)
Thank you Ying and Jenin. I am going to upgrade the MonetDB as you suggested and then go from there. Can you please let me know where can I find the upgrade procedure.
Thank you for your help in advance.
Suresh Devarakonda 84.51° TDO - Database Support Team D: 513.632.3692 M: 859.512.1839 A: 100 W. 5th Street, Cincinnati, Ohio 45202
Cincinnati | Chicago | New York | Portland Facebook LinkedIn Instagram Twitter Please consider the environment before printing this email.
-----Original Message----- From: users-list users-list-bounces+suresh.devarakonda=8451.com@monetdb.org On Behalf Of users-list-request@monetdb.org Sent: Friday, January 17, 2020 6:00 AM To: users-list@monetdb.org Subject: users-list Digest, Vol 89, Issue 6
Send users-list mailing list submissions to users-list@monetdb.org
To subscribe or unsubscribe via the World Wide Web, visit https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.monetd... or, via email, send a message with subject or body 'help' to users-list-request@monetdb.org
You can reach the person managing the list at users-list-owner@monetdb.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of users-list digest..."
Today's Topics:
1. Re: MonetDB crashing due to out of memory - monetdbd[2247] 1.7 (Aug2018-SP2) (Ying Zhang)
Message: 1 Date: Thu, 16 Jan 2020 12:01:27 +0100 From: Ying Zhang Y.Zhang@cwi.nl To: Communication channel for MonetDB users users-list@monetdb.org Subject: Re: MonetDB crashing due to out of memory - monetdbd[2247] 1.7 (Aug2018-SP2) Message-ID: E6A3FCCA-8023-4EBD-AF7A-CF89F615B2D8@cwi.nl Content-Type: text/plain; charset=utf-8
Hai Suresh,
First of all, I?d recommend you to upgrade your MonetDB installation to the most recent one, i.e. Nov2019-SP1. There are many improvements related to your problem since Aug2018-SP2. Also, the version you?re using is no longer supported by the open-source team.
Please let us know if upgrade helps. But below, a bit more info. inlined.
On 15 Jan 2020, at 23:44, Suresh Devarakonda Suresh.Devarakonda@8451.com wrote:
Hi,
The Moentdb is crashing every couple of weeks due to out of memory in our environment.
Did it really crash, or did MonetDB give error messages and stop executing the query? Crash always indicates a problem in MonetDB, but if it?s the second case, it?s expected.
What I have noticed that MonetDB is eating memory and not releasing it at all once we brought up the MonetDB server. I am not sure if this is how MonetDB works or is it a bug.
MonetDB is supposed to free up the memory when it no longer use it, which often happens towards the end of a query execution.
For further investigation, we need more information after you have upgraded to Nov2019-SP1. In general, we?ll need information with which we can reproduce the problem, e.g. data + queries.
This is running on monetdbd[2247] 1.7 (Aug2018-SP2) version. Also wondering is there way to limit the memory usage of MonetDB.
Please beware that depending on your data sizes and (concurrent) queries, MonetDB might run faster into out-of-memory problem, if you limit the amount of memory it may use,
How about putting MonetDB in a VM?
There are also some undocumented kernel configurations for this, which we don?t recommend to our users lightly?
Can you guys help me to resolve this issue.
If you need professional support, please contact the supporting company MonetDB Solutions.
Regards, Jennie
[root@xxx ~]# monetdb status name state health remarks INVPRDB R 6h 67% 3d mapi:monetdb://xxxxx:50000/INVPRDB FSPPRDB R 6h 50% 6w mapi:monetdb://xxxxx:50000/FSPPRDB [root@xxxxx ~]# free -g --- Current utilization total used free shared buff/cache available Mem: 31 0 22 0 7 30 Swap: 3 0 3
<image003.png>
<image002.jpg> Suresh Devarakonda 84.51? TDO - Database Support Team D: 513.632.3692 M: 859.512.1839 A: 100 W. 5th Street, Cincinnati, Ohio 45202
Cincinnati | Chicago | New York | Portland Facebook LinkedIn Instagram Twitter Please consider the environment before printing this email.
users-list mailing list users-list@monetdb.org https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww. monetdb.org%2Fmailman%2Flistinfo%2Fusers-list&data=02%7C01%7Csures h.devarakonda%408451.com%7Cae946815e409491cb83d08d79b3c9c2c%7C5f9dc6bd f38a454a864cc803691193c5%7C0%7C0%7C637148556952386018&sdata=3JmWyH MNaeMNrBp8vX%2FAYES%2By%2FDqNHqGmE%2FKX%2F%2F4m%2Bs%3D&reserved=0
Subject: Digest Footer
users-list mailing list users-list@monetdb.org https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.monetd...
End of users-list Digest, Vol 89, Issue 6
users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
participants (2)
-
Martin van Dinther -
Suresh Devarakonda