||Recovering unlinked, but still open files.
Unlinked files that still have open file descriptors can be recovered with relative ease.
If you lose access to a file that is still open in a process (check with lsof(8)), you can recover it without using the "crash the server and let fsck sort 'em out" technique (this is all I could find while googling about).
1. figure out which pid is holding the file descriptor (fd).
2. cd /proc/[pid number]/fd
3. cat [fd number] >new_filename
Here's the real-world example that I needed this for - my syslog script rotated and unlinked /var/log/syslog/syslog.log on my centralized loghost. The script that bounces syslog-ng failed, so syslog-ng continued to log to the now unlinked file. My log rotation script actually moves the file before bouncing syslog-ng, so the filename was syslog.log--rotating at the time of unlinking (with rm). When somebody realized that there was no syslog.log on the system, the first thing I did was the following:
root> lsof |grep syslog
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
syslog-ng 23210 root 6u unix 0xdbb230e0 151727656 /dev/log
syslog-ng 23210 root 7w REG 58,2 1006948 372 /var/log/syslog/syslog.log--rotating (deleted)
You can see that FD 7w is the logfile, which is 1006948 bytes in size and is held open by pid 23210. Of course, this is a reenactment of the event, so the file I dealt with was a bit bigger than this. Here goes the rest:
root> cd /proc/23210/fd
root> ls -l
l-wx------ 1 root root 64 Jun 12 14:58 7 -> /var/log/syslog/syslog.log--rotating (deleted)
root> file 7
7: broken symbolic link to
Looks like a dead end, but it's not. All that's left to do is retrieve our data.
root> cat 7 >/var/log/recovered-syslog.log
root> head -4 /var/log/recovered-syslog.log
Jun 10 16:17:39 hostname syslog-ng: syslog-ng version 1.6.0rc3 starting
Jun 10 16:17:39 hostname syslog-ng: syslog-ng startup succeeded
Jun 10 16:17:39 hostname syslog-ng: klogd startup succeeded
root> tail -1 /var/log/recovered-syslog.log
Jun 12 14:17:18 hostname sendmail[xxxx]: ...
That's it, now I can archive the log file and know that I haven't lost any of today's logs.
This was done on a modified RedHat 7.3 system running the 2.4.19aa1 (Andrea Arcangeli) kernel.
lsof(8) is not strictly necessary, but makes things a lot easier.
If I remember correctly, this will also work on 2.2 kernels. A quick check on my workstation running 2.5.70 shows that this hack will work there, too. This means it should still apply for the 2.6 series once it arrives.
© 2007 O'Reilly Media, Inc.
| Customer Service:
| Book issues:
All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners.