Huge merovingian.log after update
HI all –
A week or so ago we ran an upgrade to the lastest monetdb June-2020 patch. Our db was 2 patches old.
While the log showed no specific problems, the upgrade effected our databases in that 2 of them would crash when they were accessed, and the third one (the largest one) I could access but could not see any tables.
I ended up “destroying” all the dbs and recreating them fomr the saved binary files.
Now, we are getting disk full messages (the db is only read – no one is adding to it). I look at the file sizes and there is a huge mervingian.log file:
Is it a problem if I delete this file? What will it hurt? Should I “tail” it and save the end of it? Please advise.
root@cbsudc01 dbfarmBL]# du -h --max-depth=1 1.3M ./testDB_lynn 52G ./maizeChrom10DB 955G ./maizeFullGenomeDB 1.3M ./testDB 6.9T .
[lcj34@cbsudc01 dbfarmBL]$ l -ltr
total 6320304952
-rw-rw---- 1 root buckler 339559112 Oct 9 2018 merovingian.log.old
drwx------ 4 root root 121 Jun 26 13:54 testDB_lynn/
drwx------ 4 root root 121 Jul 1 17:23 maizeChrom10DB/
drwx------ 4 root root 121 Jul 1 17:23 testDB/
-rw-r--r-- 1 root root 6 Jul 1 17:23 merovingian.pid
drwx------ 4 root root 4096 Jul 1 17:24 maizeFullGenomeDB/
-rw-r--r-- 1 root buckler 6471651893248 Jul 6 14:43 merovingian.log
[lcj34@cbsudc01 dbfarmBL]$
Thanks - Lynn
One more note. I just tailed the log and see thousands of message that look like this: Does this imply some endless bad loop of someone trying to access the db?
2020-07-06 12:01:03 ERR maizeFullGenomeDB[30808]: #client26051: createExceptionInternal: !ERROR: SQLException:SQLparser:45000!Result set construction failed 2020-07-06 12:01:03 ERR maizeFullGenomeDB[30808]: #client34997: createExceptionInte 2020-07-06 12:01:03 ERR maizeFullGenomeDB[30808]: rnal: !ERROR: SQLException:SQLparser:45000!Result set construction failed
Is there a setting that limits the size of the meringian.log file?
Thanks - Lynn
From: Lynn Carol Johnson lcj34@cornell.edu Date: Monday, July 6, 2020 at 2:51 PM To: Communication channel for MonetDB users users-list@monetdb.org Subject: Huge merovingian.log after update
HI all –
A week or so ago we ran an upgrade to the lastest monetdb June-2020 patch. Our db was 2 patches old.
While the log showed no specific problems, the upgrade effected our databases in that 2 of them would crash when they were accessed, and the third one (the largest one) I could access but could not see any tables.
I ended up “destroying” all the dbs and recreating them fomr the saved binary files.
Now, we are getting disk full messages (the db is only read – no one is adding to it). I look at the file sizes and there is a huge mervingian.log file:
Is it a problem if I delete this file? What will it hurt? Should I “tail” it and save the end of it? Please advise.
root@cbsudc01 dbfarmBL]# du -h --max-depth=1 1.3M ./testDB_lynn 52G ./maizeChrom10DB 955G ./maizeFullGenomeDB 1.3M ./testDB 6.9T .
[lcj34@cbsudc01 dbfarmBL]$ l -ltr
total 6320304952
-rw-rw---- 1 root buckler 339559112 Oct 9 2018 merovingian.log.old
drwx------ 4 root root 121 Jun 26 13:54 testDB_lynn/
drwx------ 4 root root 121 Jul 1 17:23 maizeChrom10DB/
drwx------ 4 root root 121 Jul 1 17:23 testDB/
-rw-r--r-- 1 root root 6 Jul 1 17:23 merovingian.pid
drwx------ 4 root root 4096 Jul 1 17:24 maizeFullGenomeDB/
-rw-r--r-- 1 root buckler 6471651893248 Jul 6 14:43 merovingian.log
[lcj34@cbsudc01 dbfarmBL]$
Thanks - Lynn
Hai Lynn,
The huge merovingian.log file can be safely removed as the following:
mv /<path-to-your-dbfarm>/merovingian.log /<path-to-your-dbfarm>/merovingian.log.old kill -HUP $(cat /<path-to-your-dbfarm>/merovingian.pid)
i.e. you first move the huge merovingian.log out of the way and tell monetdbd to start with a new log. If you now run "ls -alh /<path-to-your-dbfarm>", you should see a new merovingian.log. Now you can safely remove the old one: "rm /<path-to-your-dbfarm>/merovingian.log.old”
Regards, Jennie
On 6 Jul 2020, at 20:57, Lynn Carol Johnson lcj34@cornell.edu wrote:
One more note. I just tailed the log and see thousands of message that look like this: Does this imply some endless bad loop of someone trying to access the db?
2020-07-06 12:01:03 ERR maizeFullGenomeDB[30808]: #client26051: createExceptionInternal: !ERROR: SQLException:SQLparser:45000!Result set construction failed 2020-07-06 12:01:03 ERR maizeFullGenomeDB[30808]: #client34997: createExceptionInte 2020-07-06 12:01:03 ERR maizeFullGenomeDB[30808]: rnal: !ERROR: SQLException:SQLparser:45000!Result set construction failed
Is there a setting that limits the size of the meringian.log file?
Thanks - Lynn
From: Lynn Carol Johnson lcj34@cornell.edu Date: Monday, July 6, 2020 at 2:51 PM To: Communication channel for MonetDB users users-list@monetdb.org Subject: Huge merovingian.log after update
HI all –
A week or so ago we ran an upgrade to the lastest monetdb June-2020 patch. Our db was 2 patches old.
While the log showed no specific problems, the upgrade effected our databases in that 2 of them would crash when they were accessed, and the third one (the largest one) I could access but could not see any tables.
I ended up “destroying” all the dbs and recreating them fomr the saved binary files.
Now, we are getting disk full messages (the db is only read – no one is adding to it). I look at the file sizes and there is a huge mervingian.log file:
Is it a problem if I delete this file? What will it hurt? Should I “tail” it and save the end of it? Please advise.
root@cbsudc01 dbfarmBL]# du -h --max-depth=1 1.3M ./testDB_lynn 52G ./maizeChrom10DB 955G ./maizeFullGenomeDB 1.3M ./testDB 6.9T .
[lcj34@cbsudc01 dbfarmBL]$ l -ltr total 6320304952 -rw-rw---- 1 root buckler 339559112 Oct 9 2018 merovingian.log.old drwx------ 4 root root 121 Jun 26 13:54 testDB_lynn/ drwx------ 4 root root 121 Jul 1 17:23 maizeChrom10DB/ drwx------ 4 root root 121 Jul 1 17:23 testDB/ -rw-r--r-- 1 root root 6 Jul 1 17:23 merovingian.pid drwx------ 4 root root 4096 Jul 1 17:24 maizeFullGenomeDB/ -rw-r--r-- 1 root buckler 6471651893248 Jul 6 14:43 merovingian.log [lcj34@cbsudc01 dbfarmBL]$
Thanks - Lynn _______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
Thanks, Jennie
Is there a monetdbd setting that limits the size of these log files? 6T basically brought down our machine.
From: users-list users-list-bounces+lcj34=cornell.edu@monetdb.org on behalf of Ying Zhang Y.Zhang@cwi.nl Reply-To: Communication channel for MonetDB users users-list@monetdb.org Date: Tuesday, July 7, 2020 at 12:31 PM To: Communication channel for MonetDB users users-list@monetdb.org Subject: Re: Huge merovingian.log after update
Hai Lynn,
The huge merovingian.log file can be safely removed as the following:
mv /<path-to-your-dbfarm>/merovingian.log /<path-to-your-dbfarm>/merovingian.log.old kill -HUP $(cat /<path-to-your-dbfarm>/merovingian.pid)
i.e. you first move the huge merovingian.log out of the way and tell monetdbd to start with a new log. If you now run "ls -alh /<path-to-your-dbfarm>", you should see a new merovingian.log. Now you can safely remove the old one: "rm /<path-to-your-dbfarm>/merovingian.log.old”
Regards, Jennie
On 6 Jul 2020, at 20:57, Lynn Carol Johnson <lcj34@cornell.edumailto:lcj34@cornell.edu> wrote:
One more note. I just tailed the log and see thousands of message that look like this: Does this imply some endless bad loop of someone trying to access the db?
2020-07-06 12:01:03 ERR maizeFullGenomeDB[30808]: #client26051: createExceptionInternal: !ERROR: SQLException:SQLparser:45000!Result set construction failed 2020-07-06 12:01:03 ERR maizeFullGenomeDB[30808]: #client34997: createExceptionInte 2020-07-06 12:01:03 ERR maizeFullGenomeDB[30808]: rnal: !ERROR: SQLException:SQLparser:45000!Result set construction failed
Is there a setting that limits the size of the meringian.log file?
Thanks - Lynn
From: Lynn Carol Johnson <lcj34@cornell.edumailto:lcj34@cornell.edu> Date: Monday, July 6, 2020 at 2:51 PM To: Communication channel for MonetDB users <users-list@monetdb.orgmailto:users-list@monetdb.org> Subject: Huge merovingian.log after update
HI all –
A week or so ago we ran an upgrade to the lastest monetdb June-2020 patch. Our db was 2 patches old.
While the log showed no specific problems, the upgrade effected our databases in that 2 of them would crash when they were accessed, and the third one (the largest one) I could access but could not see any tables.
I ended up “destroying” all the dbs and recreating them fomr the saved binary files.
Now, we are getting disk full messages (the db is only read – no one is adding to it). I look at the file sizes and there is a huge mervingian.log file:
Is it a problem if I delete this file? What will it hurt? Should I “tail” it and save the end of it? Please advise.
root@cbsudc01 dbfarmBL]# du -h --max-depth=1 1.3M ./testDB_lynn 52G ./maizeChrom10DB 955G ./maizeFullGenomeDB 1.3M ./testDB 6.9T .
[lcj34@cbsudc01 dbfarmBL]$ l -ltr total 6320304952 -rw-rw---- 1 root buckler 339559112 Oct 9 2018 merovingian.log.old drwx------ 4 root root 121 Jun 26 13:54 testDB_lynn/ drwx------ 4 root root 121 Jul 1 17:23 maizeChrom10DB/ drwx------ 4 root root 121 Jul 1 17:23 testDB/ -rw-r--r-- 1 root root 6 Jul 1 17:23 merovingian.pid drwx------ 4 root root 4096 Jul 1 17:24 maizeFullGenomeDB/ -rw-r--r-- 1 root buckler 6471651893248 Jul 6 14:43 merovingian.log [lcj34@cbsudc01 dbfarmBL]$
Thanks - Lynn _______________________________________________ users-list mailing list users-list@monetdb.orgmailto:users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
No, there is no such setting. But what you can do is to every once in a while execute the Jennie's procedure. In fact, this is what is done when you install MonetDB using yum/dnf or apt: the packages include a logrotate configuration that does this. So this is a perfectly good way of keeping the size of the log file in check. Instead of logrotate, you could use a cron job that once a day, or once a week moves the log to the side and sends the SIGHUP to monetdbd. If you then remove the logs that are a few versions old, you limit the size while still keeping some log information in case you need to look into the logs.
On 07/07/2020 18.33, Lynn Carol Johnson wrote:
Thanks, Jennie
Is there a monetdbd setting that limits the size of these log files? 6T basically brought down our machine.
*From: *users-list users-list-bounces+lcj34=cornell.edu@monetdb.org on behalf of Ying Zhang Y.Zhang@cwi.nl *Reply-To: *Communication channel for MonetDB users users-list@monetdb.org *Date: *Tuesday, July 7, 2020 at 12:31 PM *To: *Communication channel for MonetDB users users-list@monetdb.org *Subject: *Re: Huge merovingian.log after update
Hai Lynn,
The huge merovingian.log file can be safely removed as the following:
mv /<path-to-your-dbfarm>/merovingian.log /<path-to-your-dbfarm>/merovingian.log.old
kill -HUP $(cat /<path-to-your-dbfarm>/merovingian.pid)
i.e. you first move the huge merovingian.log out of the way and tell monetdbd to start with a new log. If you now run "ls -alh /<path-to-your-dbfarm>", you should see a new merovingian.log. Now you can safely remove the old one: "rm /<path-to-your-dbfarm>/merovingian.log.old”
Regards,
Jennie
On 6 Jul 2020, at 20:57, Lynn Carol Johnson <lcj34@cornell.edu <mailto:lcj34@cornell.edu>> wrote: One more note. I just tailed the log and see thousands of message that look like this: Does this imply some endless bad loop of someone trying to access the db? 2020-07-06 12:01:03 ERR maizeFullGenomeDB[30808]: #client26051: createExceptionInternal: !ERROR: SQLException:SQLparser:45000!Result set construction failed 2020-07-06 12:01:03 ERR maizeFullGenomeDB[30808]: #client34997: createExceptionInte 2020-07-06 12:01:03 ERR maizeFullGenomeDB[30808]: rnal: !ERROR: SQLException:SQLparser:45000!Result set construction failed Is there a setting that limits the size of the meringian.log file? Thanks - Lynn *From: *Lynn Carol Johnson <lcj34@cornell.edu <mailto:lcj34@cornell.edu>> *Date: *Monday, July 6, 2020 at 2:51 PM *To: *Communication channel for MonetDB users <users-list@monetdb.org <mailto:users-list@monetdb.org>> *Subject: *Huge merovingian.log after update HI all – A week or so ago we ran an upgrade to the lastest monetdb June-2020 patch. Our db was 2 patches old. While the log showed no specific problems, the upgrade effected our databases in that 2 of them would crash when they were accessed, and the third one (the largest one) I could access but could not see any tables. I ended up “destroying” all the dbs and recreating them fomr the saved binary files. Now, we are getting disk full messages (the db is only read – no one is adding to it). I look at the file sizes and there is a huge mervingian.log file: Is it a problem if I delete this file? What will it hurt? Should I “tail” it and save the end of it? Please advise. root@cbsudc01 dbfarmBL]# du -h --max-depth=1 1.3M ./testDB_lynn 52G ./maizeChrom10DB 955G ./maizeFullGenomeDB 1.3M ./testDB 6.9T . [lcj34@cbsudc01 dbfarmBL]$ l -ltr total 6320304952 -rw-rw---- 1 root buckler 339559112 Oct 9 2018 merovingian.log.old drwx------ 4 root root 121 Jun 26 13:54 testDB_lynn/ drwx------ 4 root root 121 Jul 1 17:23 maizeChrom10DB/ drwx------ 4 root root 121 Jul 1 17:23 testDB/ -rw-r--r-- 1 root root 6 Jul 1 17:23 merovingian.pid drwx------ 4 root root 4096 Jul 1 17:24 maizeFullGenomeDB/ -rw-r--r-- 1 root buckler 6471651893248 Jul 6 14:43 merovingian.log [lcj34@cbsudc01 dbfarmBL]$ Thanks - Lynn _______________________________________________ users-list mailing list users-list@monetdb.org <mailto:users-list@monetdb.org> https://www.monetdb.org/mailman/listinfo/users-list
users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
Thanks for the information, Sjoerd. We'll look into a cron job.
On 7/7/20, 2:11 PM, "users-list on behalf of Sjoerd Mullender" <users-list-bounces+lcj34=cornell.edu@monetdb.org on behalf of sjoerd@monetdb.org> wrote:
No, there is no such setting. But what you can do is to every once in a while execute the Jennie's procedure. In fact, this is what is done when you install MonetDB using yum/dnf or apt: the packages include a logrotate configuration that does this. So this is a perfectly good way of keeping the size of the log file in check. Instead of logrotate, you could use a cron job that once a day, or once a week moves the log to the side and sends the SIGHUP to monetdbd. If you then remove the logs that are a few versions old, you limit the size while still keeping some log information in case you need to look into the logs.
On 07/07/2020 18.33, Lynn Carol Johnson wrote: > Thanks, Jennie > > > > Is there a monetdbd setting that limits the size of these log files? 6T > basically brought down our machine. > > > > *From: *users-list users-list-bounces+lcj34=cornell.edu@monetdb.org on > behalf of Ying Zhang Y.Zhang@cwi.nl > *Reply-To: *Communication channel for MonetDB users users-list@monetdb.org > *Date: *Tuesday, July 7, 2020 at 12:31 PM > *To: *Communication channel for MonetDB users users-list@monetdb.org > *Subject: *Re: Huge merovingian.log after update > > > > Hai Lynn, > > > > The huge merovingian.log file can be safely removed as the following: > > > > mv /<path-to-your-dbfarm>/merovingian.log > /<path-to-your-dbfarm>/merovingian.log.old > > kill -HUP $(cat /<path-to-your-dbfarm>/merovingian.pid) > > > > i.e. you first move the huge merovingian.log out of the way and tell > monetdbd to start with a new log. If you now run "ls -alh > /<path-to-your-dbfarm>", you should see a new merovingian.log. Now you > can safely remove the old one: "rm > /<path-to-your-dbfarm>/merovingian.log.old” > > > > Regards, > > Jennie > > > > On 6 Jul 2020, at 20:57, Lynn Carol Johnson <lcj34@cornell.edu > mailto:lcj34@cornell.edu> wrote: > > > > One more note. I just tailed the log and see thousands of message > that look like this: Does this imply some endless bad loop of > someone trying to access the db? > > 2020-07-06 12:01:03 ERR maizeFullGenomeDB[30808]: #client26051: > createExceptionInternal: !ERROR: SQLException:SQLparser:45000!Result > set construction failed > > 2020-07-06 12:01:03 ERR maizeFullGenomeDB[30808]: #client34997: > createExceptionInte > > 2020-07-06 12:01:03 ERR maizeFullGenomeDB[30808]: rnal: !ERROR: > SQLException:SQLparser:45000!Result set construction failed > > > Is there a setting that limits the size of the meringian.log file? > > > > Thanks - Lynn > > > > *From: *Lynn Carol Johnson <lcj34@cornell.edu > mailto:lcj34@cornell.edu> > *Date: *Monday, July 6, 2020 at 2:51 PM > *To: *Communication channel for MonetDB users > <users-list@monetdb.org mailto:users-list@monetdb.org> > *Subject: *Huge merovingian.log after update > > > > HI all – > > > > A week or so ago we ran an upgrade to the lastest monetdb June-2020 > patch. Our db was 2 patches old. > > > > While the log showed no specific problems, the upgrade effected our > databases in that 2 of them would crash when they were accessed, and > the third one (the largest one) I could access but could not see any > tables. > > > > I ended up “destroying” all the dbs and recreating them fomr the > saved binary files. > > > > Now, we are getting disk full messages (the db is only read – no one > is adding to it). I look at the file sizes and there is a huge > mervingian.log file: > > Is it a problem if I delete this file? What will it hurt? Should I > “tail” it and save the end of it? Please advise. > > root@cbsudc01 dbfarmBL]# du -h --max-depth=1 > 1.3M ./testDB_lynn > 52G ./maizeChrom10DB > 955G ./maizeFullGenomeDB > 1.3M ./testDB > 6.9T . > > > [lcj34@cbsudc01 dbfarmBL]$ l -ltr > > total 6320304952 > > -rw-rw---- 1 root buckler 339559112 Oct 9 2018 merovingian.log.old > > drwx------ 4 root root 121 Jun 26 13:54 testDB_lynn/ > > drwx------ 4 root root 121 Jul 1 17:23 maizeChrom10DB/ > > drwx------ 4 root root 121 Jul 1 17:23 testDB/ > > -rw-r--r-- 1 root root 6 Jul 1 17:23 merovingian.pid > > drwx------ 4 root root 4096 Jul 1 17:24 maizeFullGenomeDB/ > > -rw-r--r-- 1 root buckler 6471651893248 Jul 6 14:43 merovingian.log > > [lcj34@cbsudc01 dbfarmBL]$ > > > > > > Thanks - Lynn > > _______________________________________________ > users-list mailing list > users-list@monetdb.org mailto:users-list@monetdb.org > https://www.monetdb.org/mailman/listinfo/users-list > > > > > _______________________________________________ > users-list mailing list > users-list@monetdb.org > https://www.monetdb.org/mailman/listinfo/users-list >
-- Sjoerd Mullender _______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
participants (3)
-
Lynn Carol Johnson -
Sjoerd Mullender -
Ying Zhang