Learn Linux, 101: Maintain the integrity of filesystems

Overview

In this tutorial, learn to:

• Verify the integrity of filesystems
• Monitor free space and inodes
• Repair simple filesystem problems

This tutorial covers standard and journaling (also called journaled) filesystems with an emphasis on ext2 (standard filesystem) and ext3 (journaling filesystem), but tools for other filesystems are mentioned too.

Why check filesystems?

In cases when your system crashes or loses power, Linux may not be able to cleanly unmount your filesystems. Thus, your filesystems may be left in an inconsistent state, with some changes completed and some not. Operating with a damaged filesystem is not a good idea as you are likely to further compound any existing errors.

Other problems may arise for filesystems running out of space of having sufficient filesystem management space (inodes) to continue operating. We’ll introduce you to some tools to help you manage such problems.

This tutorial helps you prepare for Objective 104.2 in Topic 104 of the Linux Server Professional (LPIC-1) exam 101. The objective has a weight of 2.

Prerequisites

To get the most from the tutorials in this series, you should have a basic knowledge of Linux and a working Linux system on which you can practice the commands covered in this tutorial. Sometimes different versions of a program will format output differently, so your results may not always look exactly like the listings and figures shown here. Most examples in this tutorial use CentOS 6, with a 2.6.32 kernel. Your results on other systems may differ.

You should also be familiar with the material in our tutorial “Learn Linux 101: Create partitions and filesystems.”

Filesystem checking tools

The main tool for checking filesystems is fsck, which, like mkfs, is really a front end to filesystem-checking routines for the various filesystem types. Some of the underlying check routines are shown in Listing 1.

[ian@attic4‑cent ~]$ ls /sbin/fsck
/sbin/btrfsck  /sbin/fsck         /sbin/fsck.ext3     /sbin/fsck.msdos
/sbin/dosfsck  /sbin/fsck.cramfs  /sbin/fsck.ext4     /sbin/fsck.vfat
/sbin/e2fsck   /sbin/fsck.ext2    /sbin/fsck.ext4dev  /sbin/fsck.xfs

You may be surprised to learn that several of these files are hard links to just one file as shown in Listing 2. Remember that these programs may be used so early in the boot process that the filesystem may not be mounted and symbolic link support may not yet be available. See our tutorial Learn Linux, 101: Create and change hard and symbolic links for more information about hard and symbolic links.

[ian@attic4‑cent ~]$ find /sbin ‑samefile /sbin/e2fsck
/sbin/fsck.ext3
/sbin/fsck.ext4
/sbin/e2fsck
/sbin/fsck.ext4dev
/sbin/fsck.ext2

The system boot process uses fsck with the -A option to check the root filesystem and any other filesystems that are specified for checking in the /etc/fstab control file. If the filesystem was not cleanly unmounted, a consistency check is performed and repairs are made, if they can be done safely. This is controlled by the pass (or passno) field (the sixth field) of the /etc/fstab entry. Filesystems with pass set to 0 are not checked at boot time. The root filesystem has a pass value of 1 and is checked first. Other filesystems will usually have a pass value of 2 (or higher), indicating the order in which they should be checked.

Multiple fsck operations can run in parallel if the system determines it is advantageous, so different filesystems are allowed to have the same pass value, as is the case for the /grubfile and /home/ian/data filesystems shown in Listing 3. Note that fsck will avoid running multiple filesystem checks on the same physical disk. To learn more about the layout of /etc/fstab, check the man pages for fstab.

filesystem              mount point                               type    options      dump pass
UUID=2f60a3b4‑ef6c‑4d4c‑9ef4‑50d7f75124a2 /                       ext3    defaults        1 1
UUID=3c3de27e‑779a‑44d5‑ad7a‑61c5fd03d9e7 /grubfile               ext3    defaults        1 2
UUID=158d605e‑2591‑4749‑bf59‑5e92e1b1c01d swap                    swap    defaults        0 0
tmpfs                                     /dev/shm                tmpfs   defaults        0 0
devpts                                    /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                                     /sys                    sysfs   defaults        0 0
proc                                      /proc                   proc    defaults        0 0
UUID=4c962b67‑c646‑467f‑96fb‑cbbd6de40140 /home/ian/data          ext4    defaults        1 2
UUID=0998d33c‑3398‑463d‑b0e3‑7c13ca0c675f /home/ian/research      ext3    defaults        1 2
UUID=e3be4658‑b79b‑470d‑82fe‑bb434bcdcc2f /home/ian/pictures      ext4    defaults        1 2

Some journaling filesystems, such as ReiserFS and XFS, might have a pass value of 0 because the journaling code, rather than fsck, does the filesystem consistency check and repair. On the other hand, some filesystems, such as /proc, are built at initialization time and therefore do not need to be checked.

You can check filesystems after the system is booted. You will need root authority, and the filesystem you want to check should be unmounted first. Listing 4 shows how to check two of our filesystems, using the device name, label, or UUID. You can use the blkid command to find the device given a label or UUID, and the label and UUID, given the device.

[root@attic4‑cent ~]##Find label and UUID for /dev/sdc4
[root@attic4‑cent ~]#blkid /dev/sdc4
/dev/sdc4: LABEL="IAN‑GPT‑EXT4" UUID="f69e0b28‑beda‑4255‑ad5a‑4d73672ac9e4" TYPE="ext4" 
[root@attic4‑cent ~]##Check /dev/sdc4
[root@attic4‑cent ~]#fsck /dev/sdc4
fsck from util‑linux‑ng 2.17.2
e2fsck 1.41.12 (17‑May‑2010)
IAN‑GPT‑EXT4: clean, 11/2424832 files, 197218/9681152 blocks
[root@attic4‑cent ~]##Check it by label using fsck.ext4
[root@attic4‑cent ~]#fsck.ext4 LABEL=IAN‑GPT‑EXT4
e2fsck 1.41.12 (17‑May‑2010)
IAN‑GPT‑EXT4: clean, 11/2424832 files, 197218/9681152 blocks
[root@attic4‑cent ~]##Check it by UUID using e2fsck
[root@attic4‑cent ~]#e2fsck UUID=f69e0b28‑beda‑4255‑ad5a‑4d73672ac9e4
e2fsck 1.41.12 (17‑May‑2010)
IAN‑GPT‑EXT4: clean, 11/2424832 files, 197218/9681152 blocks
[root@attic4‑cent ~]##Finally check the small vfat partition /dev/sda3
[root@attic4‑cent ~]#fsck /dev/sda3
fsck from util‑linux‑ng 2.17.2
dosfsck 3.0.9, 31 Jan 2010, FAT32, LFN
/dev/sda3: 0 files, 0/1265 clusters

If you attempt to check a mounted filesystem, you may see a warning. More recent versions of fsck will abort the check as Listing 5 shows where we try to check our root filesystem. If the check does not abort, heed the warning and do not do it!

[root@attic4‑cent ~]#df /
Filesystem     1K‑blocks    Used Available Use% Mounted on
/dev/sda11      79040416 7444796  67580568  10% /
[root@attic4‑cent ~]#fsck /dev/sda11
fsck from util‑linux‑ng 2.17.2
e2fsck 1.41.12 (17‑May‑2010)
/dev/sda11 is mounted.
e2fsck: Cannot continue, aborting.

It is also a good idea to let fsck figure out which check to run on a filesystem. Running the wrong check can corrupt the filesystem. If you want to see what fsck would do for a given filesystem or set of filesystems, use the -N option as shown in Listing 6.

[root@attic4‑cent ~]#fsck ‑N /dev/sda11 /dev/sdb*
fsck from util‑linux‑ng 2.17.2
[/sbin/fsck.ext3 (1) ‑‑ /] fsck.ext3 /dev/sda11 
[/sbin/fsck.ext2 (2) ‑‑ /dev/sdb] fsck.ext2 /dev/sdb 
[/sbin/fsck.ext3 (3) ‑‑ /dev/sdb1] fsck.ext3 /dev/sdb1 
[/sbin/fsck.ext4 (4) ‑‑ /home/ian/data] fsck.ext4 /dev/sdb2 
[/sbin/fsck.ext3 (5) ‑‑ /home/ian/research] fsck.ext3 /dev/sdb3 
[/sbin/fsck.ext4 (6) ‑‑ /dev/sdb4] fsck.ext4 /dev/sdb4 

So far, we have checked ext and vfat filesystems. Let’s now check the XFS filesystem on /dev/sdb3. As you can see in Listing 7, the fsck command simply tells us that we should use the xfs_check command. If there are no errors, then xfs_check does not display any output. There is a -v option for verbose output, but it is much too verbose for a simple check.

[root@attic4‑cent ~]#fsck ‑N /dev/sdc3
fsck from util‑linux‑ng 2.17.2
[/sbin/fsck.xfs (1) ‑‑ /dev/sdc3] fsck.xfs /dev/sdc3 
[root@attic4‑cent ~]#fsck /dev/sdc3
fsck from util‑linux‑ng 2.17.2
If you wish to check the consistency of an XFS filesystem or
repair a damaged filesystem, see xfs_check(8) and xfs_repair(8).
[root@attic4‑cent ~]#xfs_check /dev/sdc3

Monitoring free space

On a storage device, a file or directory is contained in a collection of blocks. Information about a file is contained in an inode, which records information such as who the owner is, when the file was last accessed, how large it is, whether it is a directory, and who can read from or write to it. The inode number is also known as the file serial number and is unique within a particular filesystem. See our tutorial Learn Linux, 101: File and directory management for more information on files and directories.

Data blocks and inodes each take space on a filesystem, so you need to monitor the space usage to ensure that your filesystems have space for growth.

The df command

The df command displays information about mounted filesystems. If you add the -T option, the filesystem type is included in the display; otherwise, it is not. The output from df for the CentOS 6 system that we used above is shown in Listing 8. For additional interest, we have created a mount point at /mnt/btrfs-test and mounted our btrfs filesystem (/dev/sdc5) over it. We have done the same for our small vfat partition on /dev/sda3.

[root@attic4‑cent ~]#mkdir /mnt/btrfs‑test
[root@attic4‑cent ~]#mount /dev/sdc5 /mnt/btrfs‑test
[[root@attic4‑cent ~]#mkdir /mnt/vfat‑test
[root@attic4‑cent ~]#mount /dev/sda3 /mnt/vfat‑test
[root@attic4‑cent ~]#df ‑T
Filesystem     Type  1K‑blocks      Used Available Use% Mounted on
/dev/sda11     ext3   79040416   7444800  67580564  10% /
tmpfs          tmpfs   1961548       232   1961316   1% /dev/shm
/dev/sda1      ext3     988420    116067    821349  13% /grubfile
/dev/sdb2      ext4  124077136  49155856  68611636  42% /home/ian/data
/dev/sdb3      ext3   60458064  30808664  26578276  54% /home/ian/research
/dev/sdc1      ext4  467126880 134497524 308894012  31% /home/ian/pictures
/dev/sda5      ext4   71168700  31178752  36368096  47% /mnt/sda5
/dev/sdc5      btrfs  39062528        56  36936704   1% /mnt/btrfs‑test
/dev/sda3      vfat       2530         0      2530   0% /mnt/vfat‑test

Notice that the output includes the total number of blocks as well as the number used and free (available). Also notice the filesystem, such as ext3 for our root filesystem on /dev/sda11, and its mount point: /. The tmpfs entry is for a virtual memory filesystem. These exist only in RAM or swap space and are created when mounted without need for a mkfs command.

For specific information on inode usage, use the -i option on the df command. You can exclude certain filesystem types using the -x option, or restrict information to just certain filesystem types using the -t option. Use these multiple times if necessary. See the examples in Listing 9.

[root@attic4‑cent ~]#df ‑i ‑x tmpfs
Filesystem       Inodes  IUsed    IFree IUse% Mounted on
/dev/sda11      5021696 236498  4785198    5% /
/dev/sda1        251000    537   250463    1% /grubfile
/dev/sdb2       7904304 125115  7779189    2% /home/ian/data
/dev/sdb3       3842048  40341  3801707    2% /home/ian/research
/dev/sdc1      29671424  48944 29622480    1% /home/ian/pictures
/dev/sda5       4530176 245603  4284573    6% /mnt/sda5
/dev/sdc5             0      0        0     ‑ /mnt/btrfs‑test
/dev/sda3             0      0        0     ‑ /mnt/vfat‑test
[root@attic4‑cent ~]#df ‑iT ‑t ext4 ‑t vfat
Filesystem     Type   Inodes  IUsed    IFree IUse% Mounted on
/dev/sdb2      ext4  7904304 125115  7779189    2% /home/ian/data
/dev/sdc1      ext4 29671424  48944 29622480    1% /home/ian/pictures
/dev/sda5      ext4  4530176 245603  4284573    6% /mnt/sda5
/dev/sda3      vfat        0      0        0     ‑ /mnt/vfat‑test

You may not be surprised to see that the FAT32 filesystem does not have inodes. You may be more surprised to see no inode information for the btrfs filesystem. Btrfs (and also ReiserFS) keep inode information in dynamically allocated structures so there are no special inode blocks as there are for ext2, ext3 or ext4.

There are several other options you may use with df to limit the display to local filesystems or control the format of output. For example, use the -h option to display human readable sizes, such as 1K for 1024, or use the -H (or --si) option to get sizes in powers of 10 (1K=1000).

If you aren’t sure which filesystem a particular part of your directory tree lives on, you can give the df command a parameter of a directory name or even a filename as shown in Listing 10.

[root@attic4‑cent ~]#df ‑‑si ~ian/index.html
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda11       81G  7.7G   70G  10% /

The tune2fs command

The ext family of filesystems also has a utility called tune2fs, which can be used to inspect information about the block count as well as information about whether the filesystem is journaled (ext3 or ext4) or not (ext2). The command can also be used to set many parameters or convert an ext2 filesystem to ext3 by adding a journal. Listing 11 shows the output for a newly created ext4 filesystem using the -l option to simply display the existing information.

[root@attic4‑cent ~]#tune2fs ‑l /dev/sdc4
tune2fs 1.41.12 (17‑May‑2010)
Filesystem volume name:   IAN‑GPT‑EXT4
Last mounted on:          <not available>
Filesystem UUID:          f69e0b28‑beda‑4255‑ad5a‑4d73672ac9e4
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype extent flex_bg 
sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         signed_directory_hash 
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              2424832
Block count:              9681152
Reserved block count:     484057
Free blocks:              9483934
Free inodes:              2424821
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      1021
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
RAID stride:              1
Flex block group size:    16
Filesystem created:       Wed Aug  5 10:58:58 2015
Last mount time:          n/a
Last write time:          Thu Aug  6 10:04:27 2015
Mount count:              0
Maximum mount count:      23
Last checked:             Wed Aug  5 10:58:58 2015
Check interval:           15552000 (6 months)
Next check after:         Mon Feb  1 09:58:58 2016
Lifetime writes:          725 MB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:              256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      cb683b6a‑2ef7‑4588‑954a‑d163b0219652
Journal backup:           inode blocks

xfs_info

For XFS filesystems you can display the same information that mkfs.xfs displayed when the filesystem was created using the xfs_info as shown in Listing 12. You need to use xfs_info on a mounted filesystem.

[root@attic4‑cent ~]#mkdir /mnt/xfs‑test
[root@attic4‑cent ~]#mount /dev/sdc3 /mnt/xfs‑test
[root@attic4‑cent ~]#xfs_info /mnt/xfs‑test
meta‑data=/dev/sdc3              isize=512    agcount=16, agsize=655360 blks
         =                       sectsz=4096  attr=2, projid32bit=0
data     =                       bsize=4096   blocks=10485760, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii‑ci=0
log      =internal               bsize=4096   blocks=5120, version=2
         =                       sectsz=4096  sunit=1 blks, lazy‑count=1

The du command

The df command gives information about a whole filesystem. Sometimes you might want to know how much space is used by your home directory, or how big a partition to use if you wanted to move /usr to its own filesystem. To answer this kind of question, use the du command.

The du command displays information about the filename (or filenames) given as parameters. If a directory name is given, then du recurses and calculates sizes for every file and subdirectory of the given directory. The result can be a lot of output. Fortunately, you can use the -s option to request just a summary for a directory. If you use du to get information for multiple directories, then you can add the -c option to get a grand total. You can also control output format using the same set of size options (-h, -H, --si, and so on) that are used for df. Listing 13 shows two views of the home directory of a newly created user who has logged in once and created an index.html file.

[testuser@attic4‑cent ~]$ du ‑hc *
4.0K    Desktop
4.0K    Documents
4.0K    Downloads
16K    index.html
4.0K    Music
4.0K    Pictures
4.0K    Public
4.0K    Templates
4.0K    Videos
48K    total
[testuser@attic4‑cent ~]$ du ‑hs .
980K    .

The reason for the difference between the 48K total from du -c `du-cand the 980K summary from du -sdu-s` is that the latter includes the entries starting with a dot, such as .bashrc, while the former does not.

One other thing to note about du is that you must be able to read the directories that you are running it against.

So now, let’s use du to display the total space used by the /usr tree and each of its first-level subdirectories. The result is shown in Listing 14. Use root authority to make sure you have appropriate access permissions.

[root@attic4‑cent ~]#du ‑shc /usr/*
257M    /usr/bin
4.0K    /usr/etc
4.0K    /usr/games
132M    /usr/include
383M    /usr/lib
1.5G    /usr/lib64
62M        /usr/libexec
136K    /usr/local
67M        /usr/sbin
2.4G    /usr/share
98M        /usr/src
0        /usr/tmp
4.8G    total

Repairing filesystems

Occasionally, very occasionally we hope, the worst will happen and you will need to repair a filesystem because of a crash or other failure to unmount cleanly. The fsck command that you saw above can repair filesystems as well as check them. Usually the automatic boot-time check will fix the problems and you can proceed.

If the automatic boot-time check of filesystems is unable to restore consistency, you are usually dumped into a single user shell with some instructions to run fsck manually. For an ext2 filesystem, which is not journaled, you may be presented with a series of requests asking you to confirm proposed actions to fix particular blocks on the filesystem. You should generally allow fsck to attempt to fix problems, by responding y (for yes). When the system reboots, check for any missing data or files.

If you suspect corruption, or want to run a check manually, most of the checking programs require the filesystem to be unmounted, or at least mounted read-only. Because you can’t unmount the root filesystem on a running system, the best you can do is drop to single user mode (using telinit 1telinit1) and then remount the root filesystem read-only, at which time you should be able to perform a consistency check. A better way to check a filesystem is to boot a recovery system, such as a live CD or a USB memory key, and perform the check of your unmounted filesystems from that.

If fsck cannot fix the problem, you do have some other tools available, although you will generally need advanced knowledge of the filesystem layout to successfully fix it.

Why journal?

An fsck scan of an ext2 disk can take quite a while to complete, because the internal data structure (or metadata) of the filesystem must be scanned completely. As filesystems get larger and larger, this takes longer and longer, even though disks also keep getting faster, so a full check may take one or more hours.

This problem was the impetus for journaled, or journaling, filesystems. Journaled filesystems keep a log of recent changes to the filesystem metadata. After a crash, the filesystem driver inspects the log in order to determine which recently changed parts of the filesystem may possibly have errors. With this design change, checking a journaled filesystem for consistency typically takes just a matter of seconds, regardless of filesystem size. Furthermore, the filesystem driver will usually check the filesystem on mounting, so an external fsck check is generally not required. In fact, for the xfs filesystem, fsck does nothing!

If you do run a manual check of a filesystem, check the man pages for the appropriate fsck command (fsck.ext3, e2fsck, xfs_check, and so on) to determine the appropriate parameters. The -p option, when used with ext2, ext3, or ext4 filesystems will cause fsck to automatically fix all problems that can be safely fixed. This is, in fact, what happens at boot time.

We’ll illustrate the use of e2fsck and xfs_check by first running e2fsck on an empty XFS filesystem and then using xfs_check to fix it. Remember we suggested that you use the fsck front end to be sure you are using the right checker, and we warned you that failure to do so may result in filesystem corruption.

In Listing 15, we start running e2fsck against /dev/sda8, which contains an XFS filesystem. After a few interactions we use ctrl-Break to break out, but it is too late. Warning: Do NOT do this unless you are willing to destroy your filesystem.

[root@attic4‑cent ~]#xfs_check ‑s /dev/sdc3
[root@attic4‑cent ~]#e2fsck /dev/sdc3
e2fsck 1.41.12 (17‑May‑2010)
e2fsck: Group descriptors look bad... trying backup blocks...
/dev/sdc3 was not cleanly unmounted, check forced.
Resize inode not valid.  Recreate<y>? yes

Pass 1: Checking inodes, blocks, and sizes
Deleted inode 163841 has zero dtime.  Fix<y>? ctrl‑Break

/dev/sdc3: e2fsck canceled.

/dev/sdc3: * FILE SYSTEM WAS MODIFIED *

Even if you broke out at the first prompt, your XFS filesystem would still have been corrupted. Repeat after me. Do NOT do this unless you are willing to destroy your filesystem.

Now let’s use xfs_check to check the XFS filesystem. The xfs_check command is quite verbose, but it has a -s option which reports only serious errors. The output is shown in Listing 16.

[root@attic4‑cent ~]#xfs_check ‑s /dev/sdc3
xfs_check: cannot init perag data (117)
blocks 17039360/3..3 claimed by block 17039360/0
can't seek in filesystem at bb 89335319756824
can't read agfl block for ag 17039360
can't seek in filesystem at bb 89335588200448
can't seek in filesystem at bb 89335319756800
can't seek in filesystem at bb 89335319756800

You should use xfs_repair to repair an XFS filesystem. Like xfs_check, it is quite verbose, and it does not have an -s option. If you’d like just to see what needs repair without actually repairing it, use xfs_repair -n. Listing 17 shows part of the output from xfs_repair. As you can see, running the wrong filesystem check can do a lot of damage very quickly.

[root@attic4‑cent ~]#xfs_repair /dev/sdc3
Phase 1 ‑ find and verify superblock...
Phase 2 ‑ using internal log
        ‑ zero log...
        ‑ scan filesystem freespace and inode maps...
bad magic #0x1060000 for agf 0
bad version #33947648 for agf 0
bad sequence #17039360 for agf 0
bad length ‑143002337 for agf 0, should be 655360
flfirst 25427968 in agf 0 too large (max = 1024)
fllast ‑42401760 in agf 0 too large (max = 1024)
bad magic #0x80024000 for agi 0
bad version #‑2130558976 for agi 0
bad sequence #16384 for agi 0
bad length #‑25362400 for agi 0, should be 655360
reset bad agf for ag 0
reset bad agi for ag 0
freeblk count 1 != flcount 1024 in ag 0
bad agbno 33555456 for btbno root, agno 0
bad agbno 0 for btbcnt root, agno 0
bad agbno 0 for inobt root, agno 0
agi_count 1024, counted 0 in ag 0
agi unlinked bucket 0 is 8404992 in ag 0 (inode=8404992)
agi unlinked bucket 1 is 4269604896 in ag 0 (inode=4269604896)
agi unlinked bucket 2 is 1024 in ag 0 (inode=1024)
...
agi unlinked bucket 62 is 2307015680 in ag 0 (inode=2307015680)
agi unlinked bucket 63 is 2323792896 in ag 0 (inode=2323792896)
sb_icount 64, counted 0
sb_ifree 61, counted 0
sb_fdblocks 10480520, counted 9825176
root inode chunk not found
Phase 3 ‑ for each AG...
        ‑ scan and clear agi unlinked lists...
        ‑ process known inodes and perform inode discovery...
        ‑ agno = 0
        ‑ agno = 1
        ‑ agno = 2
...
        ‑ agno = 15
        ‑ process newly discovered inodes...
Phase 4 ‑ check for duplicate blocks...
        ‑ setting up duplicate extent list...
        ‑ check for inodes claiming duplicate blocks...
        ‑ agno = 1
        ‑ agno = 0
        ‑ agno = 2
...
        ‑ agno = 15
Phase 5 ‑ rebuild AG headers and trees...
        ‑ reset superblock...
Phase 6 ‑ check inode connectivity...
        ‑ resetting contents of realtime bitmap and summary inodes
        ‑ traversing filesystem ...
        ‑ traversal finished ...
        ‑ moving disconnected inodes to lost+found ...
Phase 7 ‑ verify and correct link counts...
done

Superblocks

You may be wondering how all these checking and repairing tools know where to start. Linux and UNIX filesystems usually have a superblock, which describes the filesystem metadata, or data describing the filesystem itself. This is usually stored at a known location, frequently at or near the beginning of the filesystem, and replicated at other well-known locations. You can use the -n option of mke2fs to display the superblock locations for an existing filesystem. If you specified parameters such as the bytes per inode ratio, you should invoke mke2fs with the same parameters when you use the -n option. Listing 18 shows the location of the superblocks on /dev/sda5. Note that the filesystem must not be mounted when you do this.

[root@attic4‑cent ~]#mke2fs ‑n /dev/sda5
mke2fs 1.41.12 (17‑May‑2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
4530176 inodes, 18109263 blocks
905463 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
553 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
    4096000, 7962624, 11239424

Advanced tools

There are several more advanced tools that you can use to examine or repair a filesystem. Check the man pages for the correct usage and the Linux Documentation Project (see resources on the right) for how-to information. Almost all of these commands require a filesystem to be unmounted, although some functions can be used on filesystems that are mounted read-only. A few of the commands are described below.

You should always back up your filesystem before attempting any repairs.

Tools for ext2 and ext3 filesystems

tune2fs
Adjusts parameters on ext2 and ext3 filesystems. Use this to add a journal to an ext2 system, making it an ext3 system, as well as display or set the maximum number of mounts before a check is forced. You can also assign a label and set or disable various optional features.

dumpe2fs
Prints the super block and block group descriptor information for an ext2 or ext3 filesystem.

debugfs
Is an interactive filesystem debugger. Use it to examine or change the state of an ext2 or ext3 filesystem.

Tools for Reiserfs filesystems

reiserfstune
Displays and adjusts parameters on ReiserFS filesystems.

debugreiserfs
Performs similar functions to dumpe2fs and debugfs for ReiserFS filesystems.

Tools for XFS filesystems

xfs_info
Displays XFS filesystem information.

xfs_growfs
Expands an XFS filesystem (assuming another partition is available).

xfs_admin
Changes the parameters of an XFS filesystem.

xfs_repair
Repairs an XFS filesystem when the mount checks are not sufficient to repair the system.

xfs_db
Examines or debugs an XFS filesystem.

Tools for btrfs filesystems

Note: you may have to search online for the btrfs man pages.

btrfs
Displays many aspects of btrfs filesystem information

btrfsck
Check btrfs filesystems

btrfs-find-root
Finds the block that is the root of the btrfs filesystem

btrfs-debug-tree
Displays btrfs internal metadata

btrfstune
Tune various btrfs filesystem parameters, and enables or disables some extended features

btrfs-restore
Attempt to restore files from a damaged btrfs filesystem

We will wrap up our tools review with an illustration of the debugfs command, which allows you to explore the inner workings of an ext family filesystem. By default, it opens the filesystem in read-only mode. It does have many commands that allow you to attempt undeletion of files or directories, as well as other operations that require write access, so you will specifically have to enable write access with the -w option. Use it with extreme care.

Listing 19 shows how to open the root filesystem on my system; navigate to my home directory; display information, including the inode number, about a file called index.html; and finally, map that inode number back to the pathname of the file. Use ? to get a list of available commands while in debugfs. In this example, the filesystem is mounted, which is alright for inspecting. Do not attempt repair on a mounted filesystem.

[root@attic4‑cent ~]#debugfs /dev/sda11
debugfs 1.41.12 (17‑May‑2010)
debugfs:  cd home/ian
debugfs:  pwd
[pwd]   INODE: 3825697  PATH: /home/ian
[root]  INODE:      2  PATH: /
debugfs:  stat index.html
Inode: 3832472   Type: regular    Mode:  0664   Flags: 0x0
Generation: 301826695    Version: 0x00000000
User:  1000   Group:  1000   Size: 13430
File ACL: 0    Directory ACL: 0
Links: 1   Blockcount: 32
Fragment:  Address: 0    Number: 0    Size: 0
ctime: 0x55c37766 ‑‑ Thu Aug  6 11:04:06 2015
atime: 0x55c37c42 ‑‑ Thu Aug  6 11:24:50 2015
mtime: 0x55940f92 ‑‑ Wed Jul  1 12:04:34 2015
Size of extra inode fields: 4
Extended attributes stored in inode body: 
  selinux = "unconfined_u:object_r:user_home_t:s0\000" (37)
BLOCKS:
(0‑1):15323651‑15323652, (2):15323686, (3):15342497
TOTAL: 4

debugfs:  ncheck 3832472
Inode    Pathname
3832472    /home/ian/index.html
debugfs:  q

Conclusion

We’ve covered many tools you can use for checking, modifying, and repairing your filesystems. Remember to always use extreme care when using the tools discussed in this tutorial or any other tools. Data loss may be only a keystroke away.