Hey Folks, we are continuing to get deadlocks about once a day and I've noticed this in the logfile:
2009-11-28 14:31:05 ERR reporting_primary[31485]: !SABAOTHgetMyStatus: IOException:sabaoth.getStatus:Unable to open directory /opt/MonetDB5/dbfarm: Too many open files
Not sure how many open files it had when it went south, but immediately after a restart I saw this in files-nr:
# cat /proc/sys/fs/file-nr 1408 0 3164775
That third number being the 'files-max' on the system currently... I think that must be being dynamically set.
This is with the latest release August2009-SP2, the net effect is a deadlock on all connections to merovingian and a segfault of the mserver process.
Neils, this is what led up to that 'log' file being truncated in sql_logs a few days ago... this message occurred just before this happened. I've restarted my database and should see the problem reproduce itself again within 24 hours. I am recording the number of file descriptors in use every minute by mserver, merovingian, and overall for the system so I can know what the numbers were leading up to it.
Is this a file descriptor leak in Monet or is my files-max set too low (even though it appears to be dynamic)?
73, Matthew W. Jones (KI4ZIB) http://matburt.net