linux file observability
In linux,“everything” is a file1. Regardless if its a regular file, directory, socket or even a device. Know how to observe open files, attached to a process, can become quite handy for debugging a linux host.
The first theoretical part will introduce file descriptors while the second part will give some practical examples.
Files and file descriptors
Whenever a program opens a file via a syscall from the open() family, the kernel returns a reference number that is unique for that process. This number is bettern known as file descriptor or fd for short. The program can act on that file (read, write, close,…) by using the fd from the open call.
Imagine a program like this:
// gcc example.c -o example
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <fcntl.h>
int main(int arc, char *argv[]){
open("example.c", 0);
while(1){sleep(1000);}
}
This program will open a file (itself for simplicity) and goes into an infinite loop in order to stay alive for observability.
On open the system creates a procfs entry under /proc/<pid>/fds/<fd> . Listing the proc dir for our process3 looks something like this:
$ ls -la /proc/1234/fd
total 0
dr-x------ 2 raphael raphael 0 Jun 29 21:11 .
dr-xr-xr-x 9 raphael raphael 0 Jun 29 21:11 ..
lrwx------ 1 raphael raphael 64 Jun 29 21:11 0 -> /dev/pts/0
lrwx------ 1 raphael raphael 64 Jun 29 21:11 1 -> /dev/pts/0
lrwx------ 1 raphael raphael 64 Jun 29 21:11 2 -> /dev/pts/0
lr-x------ 1 raphael raphael 64 Jun 29 21:11 3 -> /tmp/example.c
Observe
Wich process opened the file?
lsof got you covered:
$ lsof /tmp/example
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
example 444 raphael txt REG 0,40 23656 26759037 /tmp/example
Which files do a process holding?
It’s lsof again. The opened example.c can be seen at last with fd 3 linking to the original file. The other three fd’s 0 , 1 and 2 are the three pts devices for stdin, stdout stderr that get automatically created by the OS for any new process.
$ lsof -p 444
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
example 444 raphael cwd DIR 8,32 4096 267422 /tmp/
example 444 raphael rtd DIR 8,32 4096 2 /
example 444 raphael txt REG 8,32 16088 46778 /tmp/example
example 444 raphael mem REG 8,32 2029592 142704 /usr/lib/...
example 444 raphael mem REG 8,32 191504 142576 /usr/lib/...
example 444 raphael 0u CHR 136,0 0t0 3 /dev/pts/0
example 444 raphael 1u CHR 136,0 0t0 3 /dev/pts/0
example 444 raphael 2u CHR 136,0 0t0 3 /dev/pts/0
example 444 raphael 3r REG 8,32 196 46772 /tmp/example.c
Is my server already listening for incoming connections?
Let’s spin up a tcp server waiting for incoming connections using ncat:
$ ncat -l 8080
Using lsof again, we can observe that ncat is even doing dual-stack waiting for incoming ipv4 and ipv6 connections, as one can see in the forelast entries.
$ lsof -p 532581
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ncat 532581 raphael cwd DIR 0,40 450 27369190 /tmp/example
ncat 532581 raphael rtd DIR 0,38 206 256 /
ncat 532581 raphael txt REG 0,38 435520 9123417 /usr/bin/ncat
ncat 532581 raphael mem REG 0,36 9123417 /usr/bin/ncat (path dev=0,38)
ncat 532581 raphael mem REG 0,36 9000771 /usr/lib64/...
ncat 532581 raphael 0u CHR 136,1 0t0 4 /dev/pts/1
ncat 532581 raphael 1u CHR 136,1 0t0 4 /dev/pts/1
ncat 532581 raphael 2u CHR 136,1 0t0 4 /dev/pts/1
ncat 532581 raphael 3u IPv6 9561146 0t0 TCP *:webcache (LISTEN)
ncat 532581 raphael 4u IPv4 9561147 0t0 TCP *:webcache (LISTEN)
ncat 532581 raphael 6u unix 0x0000000014cd383f 0t0 31426 type=STREAM (CONNECTED)
Let’s go one step further and establish a connection to ncat by browsing to localhost:8080. We can see that ncat
closed the unused listener and from which address+port the connection is coming from:
$ lsof -p 532581
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ncat 532581 raphael cwd DIR 0,40 450 27369190 /tmp/example
ncat 532581 raphael rtd DIR 0,38 206 256 /
ncat 532581 raphael txt REG 0,38 435520 9123417 /usr/bin/ncat
ncat 532581 raphael mem REG 0,36 9123417 /usr/bin/ncat (path dev=0,38)
ncat 532581 raphael mem REG 0,36 9000771 /usr/lib64/...
ncat 532581 raphael 0u CHR 136,1 0t0 4 /dev/pts/1
ncat 532581 raphael 1u CHR 136,1 0t0 4 /dev/pts/1
ncat 532581 raphael 2u CHR 136,1 0t0 4 /dev/pts/1
ncat 532581 raphael 5u IPv4 9557641 0t0 TCP localhost:webcache->localhost:32886 (ESTABLISHED)
ncat 532581 raphael 6u unix 0x0000000014cd383f 0t0 31426 type=STREAM (CONNECTED)
Does my process still hold fds to deleted files?
Let’s create a file and immediately unlink it but keep it open:
// gcc ulink.c -o ulink
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
int main(int argc, char* argv[]) {
open("test.log", O_CREAT);
unlink("test.log");
while(1){sleep(1000);}
return 0;
}
The file is not existent, but can be observed via lsof:
$ lsof -p 635
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ulink 635 raphael cwd DIR 8,32 4096 267422 /tmp/example
ulink 635 raphael rtd DIR 8,32 4096 2 /
ulink 635 raphael txt REG 8,32 16168 50490 /tmp/example
ulink 635 raphael mem REG 8,32 2029592 142704 /usr/lib/...
ulink 635 raphael 0u CHR 136,0 0t0 3 /dev/pts/0
ulink 635 raphael 1u CHR 136,0 0t0 3 /dev/pts/0
ulink 635 raphael 2u CHR 136,0 0t0 3 /dev/pts/0
ulink 635 raphael 3r REG 8,32 0 40397 /tmp/test.log (deleted)
- Rabbit hole warning! source based on wikipedia
- Yes, I am looking at you PHP!
- use “prgrep -f example” to get the pid