Resolving problems unmounting partitions
Identify the processes that are using mounted file systems
All computer operating systems verify whether the file systems that they mount at boot time are consistent, meaning that there are no errors in their internal data structures and the associated storage that they map to. UNIX, Linux®, and other UNIX-like operating systems take a clever approach to determining whether the consistency of a file system needs to be verified (typically by using the
fast command). When these systems mount a file system, they set a value in the file system header that marks the file system as
DIRTY, meaning that it is in use and may be transiently inconsistent as updates are being written to it. When file systems are unmounted as part of a system shutdown, they are marked as
CLEAN. When the system reboots, only file systems that are still marked as
DIRTY need to be checked for consistency.
File systems are automatically unmounted as part of the system shutdown process, which usually occurs after all non-system processes have been terminated. Regardless, unmounting a file system can still fail with the traditional message:
$ sudo umount /mnt/NAS umount: /mnt/NAS: device is busy
In this case,
busy means that a process is writing to or running from that file system. The fact that you cannot unmount a file system in either of these cases is one of the basic rules of computer systems. If this were not the case, you could unmount a file system while some process is writing to a file that it contains, which could leave the file in an inconsistent state while the file system itself is marked as
The standard Linux version of the
umount command includes a lazy unmount option,
-l, to help unmount file systems that are in use. This command requires Linux kernel version 2.4.11 or greater, which isn’t much of a problem today. Executing
umount -l /name/of/file system detaches the specified file system from the system’s directory hierarchy so that no new processes can use the file system and then unmounts the file system when all of the processes that were accessing it terminate. This can be handy but is not exactly what you want to use when you need to unmount a file system now.
If you need to unmount a file system now, and that file system reports that it is busy, you still have some options. If you are the only user of a system, terminating the processes that are preventing you from unmounting a file system is as easy as looking through all your windows for suspended or background processes that are writing to the partition in question or using it as their current working directory and terminating them. However, on multi-user systems with many local and remote users this approach isn’t practical. Luckily, the open source community offers some commands that make it easy to identify and terminate such processes.
Locating open files with lsof
lsof (list open files) command displays a list of all open files and the processes associated with them on a specific file system, directory, or device. The
lsof command is available for most UNIX and UNIX-like systems, including IBM® AIX®, Berkeley Software Distribution (BSD®), Hewlett Packard UNIX (HP-UX®), Linux, and Solaris®. See resources on the right for information about obtaining
lsof for your system.
By default, the
lsof command lists all files, shared libraries, and directories that are currently open and provides as much information as possible about each of them. The output from this is huge, even on a lightly loaded system, so you typically either supply the name of a directory as a command-line argument or use a pipe to filter its output. For example, suppose that you want to unmount a file system that is mounted on the /opt2 directory. To see all of the processes associated with the /opt2 directory, you execute a command such as the one shown in Listing 1.
Listing 1. Processes associated with a mounted file system
$ lsof /opt2 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME bash 23334 wvh cwd DIR 8,17 4096 2 /opt2 more 23402 wvh cwd DIR 8,17 4096 2 /opt2 more 23402 wvh 3r REG 8,17 10095 264 /opt2/resume.txt
You need to terminate all of these processes before you can unmount the /opt2 partition. Because none of the processes in this listing can be writing any files, you could use the
kill command to terminate the process IDs (PIDs) that are listed in the second column and then unmount the partition with no problems. Note that PID 23402 is associated with the last two lines—the first line indicates that the
more command is running with a current working directory (
cwd) of /opt2, and the second indicates that the
more command has the /opt2/resume.txt file open.
However, suppose that the output of the
lsof command looks like Listing 2.
Listing 2. A different set of processes associated with a mounted file system
$ lsof /opt2 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME bash 23334 wvh cwd DIR 8,17 4096 2 /opt2 more 23402 wvh cwd DIR 8,17 4096 2 /opt2 more 23402 wvh 3r REG 8,17 10095 264 /opt2/resume.txt bash 21343 djf cwd DIR 8,17 4096 2 /opt2 emacs 21405 djf cwd DIR 8,17 4096 2 /opt2
The first three commands associated with the /opt2 directory are the same, but the last two are being run by another user. Of these, the
emacs command is designed for editing files, so you might want to have the user listed in the
USER column save and exit before you terminate that process.
Customizing lsof output
The previous section showed how to identify open files and directories on a local device, but you can just as easily get the same information about a mounted remote file system.
To provide a consistent set of examples for this article, all of the command and output examples refer to mounted partitions from the system shown in Listing 3.
Listing 3. File systems used in this article
$ df Filesystem 1K‑blocks Used Available Use% Mounted on /dev/sda1 230528596 201462232 17356188 93% / /dev/sdb1 240362656 12533532 215619324 6% /opt2 //nas.vonhagen.org/writing 100790048 75945920 197241926 80% /mnt/NAS 192.168.6.166:/mnt/disk1 714854640 386972432 291569696 58% /mnt/yellowmachine
As shown in Listing 3, /mnt/NAS is the mountpoint for a Samba share called writing that is shared from the device nas.vonhagen.org. Specifying the name of the mountpoint as an argument to the
lsof command creates output similar to Listing 2 but specific to that device and directory, as shown in Listing 4.
Listing 4. Processes associated with a remote file system
$ lsof /mnt/NAS COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME bash 23236 wvh cwd DIR 0,27 4096 6406145 /mnt/NAS/writing \ (nas.vonhagen.org:/writing)
lsof command also provides options that enable you to restrict its output to reporting open files and directories on specific types of devices. For example, as shown in Listing 3, the /mnt/yellowmachine directory is a mountpoint for a Network File System (NFS) mount of the /mnt/disk1 directory on the 192.168.6.166 device. You can easily supply the name of the mountpoint for this device as an argument to the
lsof command, as shown in Listing 5.
Listing 5. Processes associated with a remote NFS file system
$ lsof /mnt/yellowmachine COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME bash 23334 wvh cwd DIR 0,23 4096 2 /mnt/yellowmachine \ (192.168.6.166:/mnt/disk1)
You can also use the
-N option to list only files and directories that are in use on NFS-mounted devices, as shown in Listing 6.
Listing 6. Processes associated with all mounted NFS partitions
$ lsof ‑N COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME bash 23334 wvh cwd DIR 0,23 4096 2 /mnt/yellowmachine (192.168.6.166:/mnt/disk1)
lsof command has many more options that can help you identify open files and directories on different types of file systems, as well as the processes that have network sockets open, who is using specific binaries, and more. The downside of the
lsof command is that you always have to either contact users and ask them to terminate certain processes or manually terminate them yourself. The
fuser command is a more cryptic but also more powerful command that can do much of your process termination work for you when run as the root user.
Finding user processes with fuser
fuser (find user processes) command is another open source application that can help identify processes that are preventing you from unmounting file systems. The
fuser command finds user processes that are associated with whatever files, directories, or file system mountpoints that you supply as command-line arguments. This article focuses on using
fuser with file system mountpoints. For more generic information about the
fuser command, see its online reference information. The
fuser command requires that your system supports the /proc file system. Therefore, it’s available for all Linux distributions and FreeBSD systems. See resources on the right for information about obtaining the source code for the
As with the
lsof command, supplying the name of a file system mountpoint as a command-line argument is the simplest way to use the
fuser command to identify processes that are preventing you from unmounting a file system:
$ fuser /mnt/yellowmachine /mnt/yellowmachine: 23334c 23697c
The output of the
fuser command simply identifies the PIDs of processes that are using the specified mountpoint. Each PID is followed by a single letter that identifies the way in which the process associated with that PID is using the specified mountpoint. The most common of these is the letter
c, shown in the previous example, which indicates the specified process is using a directory on that file system as its current working directory.
Unfortunately, the default output of the
fuser command isn’t end-user friendly, even by Linux standards. The
fuser command provides a
-v option that adds some output that is similar to the output of the standard
ps command to the output of the
fuser command, as shown in Listing 7.
Listing 7. User processes on a mounted NFS file system
$ fuser ‑v /mnt/yellowmachine USER PID ACCESS COMMAND /mnt/yellowmachine: wvh 23334 ..c.. bash wvh 23697 ..c.. emacs
This is handier because it at least identifies what the processes are. After you obtain the PID information from the
fuser command, you can always use a combination of the standard
egrep commands to get as much detail as possible about the processes before terminating them, as shown in Listing 8.
Listing 8. Search for specific processes on a system
#ps alxww |egrep '23334|23697' 4 1000 23334 23332 20 0 18148 2076 wait Ss pts/13 0:00 ‑bash 0 1000 23697 23334 20 0 75964 12352 poll_s S+ pts/13 0:00 emacs ‑nw file2.txt 0 0 23703 23665 20 0 6060 632 ‑ R+ pts/16 0:00 egrep 23334|23697
You can then use the standard
kill command to terminate the specified processes manually or, as explained in the next section, use some of the advanced capabilities of the
fuser command to terminate them automatically.
Terminating processes with fuser
-k option automatically terminates processes that it detects are using a mountpoint that you specify as an argument. You must, of course, execute the
fuser command as root to be able to terminate processes that may be owned by other users, as shown in Listing 9.
Listing 9. Terminating processes associated with a mounted NFS file system
#fuser ‑k /mnt/yellowmachine /mnt/yellowmachine: 23334c 23697c Could not kill process 23697: No such process
In this case, the second process (emacs) was a child of the first (the bash shell), and therefore terminated when the first was killed by the
If you want to specify the name of an underlying physical device rather than simply the mountpoint for the file system that it contains, you must also specify the
-m option, as shown in Listing 10.
Listing 10. Process listings for mountpoint and devices
#fuser ‑v /opt2 USER PID ACCESS COMMAND /opt2: wvh 23712 ..c.. bash wvh 23753 ..c.. emacs #fuser ‑v /dev/sdb1 #fuser ‑vm /dev/sdb1 USER PID ACCESS COMMAND /dev/sdb1: wvh 23712 ..c.. bash wvh 23753 ..c.. emacs
The first command returns the output that you would expect because it references the mountpoint for a file system. The second command shows that you cannot directly query the underlying device by using the standard
fuser options. The third illustrates that the
-m option enables you to specify a device directly. You could add the
-k option to either of the first or third commands in this example to terminate the processes in the file system that is located on the /dev/sdb1 device.
At some point, every Linux or UNIX systems administrator needs to unmount a partition in response to some emergency or simply to remove a device such as a mounted CD-ROM or DVD. When the system won’t let you unmount a device because that device is busy, examining every process on the system is both irritating and slow. The
fuser commands make it easy to identify the processes that are preventing you from unmounting a file system. The
fuser command even terminates them for you if you’re in a hurry.