Had a strange, and as yet unexplained, 'event' (I wasn't in front of the machine when things went weird) that took place while a system was left running a large rsync over ssh. On returning, a majority of the directories under /var vanished, and a number of services refused to start after a reboot, including auditd, nfsd, system message bus, hpiod, hpssd, mysql, syslogd, httpd, sm-client, and setroubleshootd.
In the cases of most of these services, there seemed to be problems either with orphaned /var/run/*.pid files, or with orphaned /var/lock/subsys/* lock files. Also, many services were reporting 'subsys locked'. Deleting orphaned files, followed by relabeling the filesystem selinux permissions did the trick, with relabeling being the key to getting things going again. Debugging was made more challenging by the fact that I had no logs to refer to.
Now, almost all seems well, but I can't get setroubleshootd to start unless I select 'setroubleshootd_disable_trans'. Without this checked, setroubleshootd seems to start, but then fails:
[root@file1 subsys]# rm setroubleshootd rm: remove regular empty file `setroubleshootd'? y [root@file1 subsys]# service setroubleshoot status setroubleshootd is stopped [root@file1 subsys]# service setroubleshoot start Starting setroubleshootd: [ OK ] [root@file1 subsys]# service setroubleshoot status setroubleshootd dead but subsys locked
Attempting to run setroubleshoot generates the error:
'attempt to open server connection failed: (2, 'No such file or directory')
Since someone might ask about permissions:
[root@file1 subsys]# ls -laRZ /var/log | grep setroubleshoot drwxr-xr-x root root system_u:object_r:setroubleshoot_var_log_t setroubleshoot /var/log/setroubleshoot: drwxr-xr-x root root system_u:object_r:setroubleshoot_var_log_t . -rw-r--r-- root root system_u:object_r:setroubleshoot_var_log_t setroubleshootd.log -rw-r--r-- root root system_u:object_r:setroubleshoot_var_log_t setroubleshootd.log.1 -rw-r--r-- root root system_u:object_r:setroubleshoot_var_log_t setroubleshootd.log.2
Can anyone explain why setroubleshootd_disable_trans should need to be selected? Also, since this entire event seems to have close ties to selinux, would anyone have an idea what might have happened to this system?
Thanks for any ideas; it's been a long day...
Steven Stromer