Linux file systems explained: ext4 and beyond

Today I will take you through the history of ext4, including its differences from ext3 and other previous file systems.

Most modern Linux distributions default to the ext 4 filesystem, just as older Linux distributions defaulted to ext3, ext2, and - if you go back far enough - ext.
If you are new to Linux or new to file systems in general, you might be wondering what ext4 brings to the table that ext3 does not. Given news coverage of alternative file systems such as btrfs, XFS, and ZFS, you might also wonder whether ext4 is still under active development.
We can't cover everything there is to know about file systems in a single article, but we'll try to give you an idea of ​​the history of Linux's default file system, where it stands, and what to expect.
I drew heavily from various ext filesystem articles and my own experiences in writing this overview.

A brief history of ext

MINIX file system Before ext, the MINIX file system was used. If you're not familiar with the history of Linux, MINIX was a very small Unix-like system for the IBM PC/AT microcomputer. Andrew Tannenbaum developed it for teaching purposes and released the source code (in printed format!) in 1987.

IBM PC/AT mid-1980s, MBlairMartin, CC BY-SA 4.0

Although you can peruse the source code of MINIX, it is not actually free and open source software (FOSS). The publisher of Tannebaum's book requires you to pay a $69 license fee to run MINIX, and this fee is included in the price of the book. Still, it was very cheap for the time, and use of MINIX grew rapidly, quickly outgrowing Tannebaum's original intention of using it to teach operating system coding. Throughout the 1990s, you could find MINIX installations very popular in universities around the world. At this time, the young Linus Torvalds used MINIX to develop the original Linux kernel, which was first announced in 1991 and then released under the GPL open source agreement in December 1992.
But wait, this is a file system article, isn't it? Yes, MINIX had its own file system, and early Linux versions depended on it. Like MINIX, Linux's file system is as small as a toy - the MINIX file system can handle file names of up to 14 characters and can only handle 64MB of storage space. By 1991, the typical hard drive size was 40-140 MB. Clearly, Linux needs a better file system.

ext

While Linus was developing the fledgling Linux kernel, Rémy Card was working on the first generation of the ext file system. The ext file system was first implemented and released in 1992 -- just one year after Linux was first released! -- ext solves the worst problems of the MINIX file system.
ext in 1992 used the new Virtual File System (VFS) abstraction layer in the Linux kernel. Unlike the previous MINIX file system, ext can handle up to 2 GB of storage and handle file names of 255 characters.
But ext didn't dominate for long, mainly due to its primitive timestamps (only one timestamp per file, rather than the inode, last file access time, and last file modification time timestamps we are familiar with today). Just a year later, ext2 replaced it.

ext2

Rémy quickly realized the limitations of ext, so a year later he designed ext2 as its replacement. While ext still has its roots in "toy" operating systems, ext2 was designed from the ground up to be a commercial-grade filesystem, following the design principles of BSD's Berkeley filesystem.
Ext2 offered maximum file sizes in the GB range and file system sizes in the TB range, firmly cementing its place in the file system big leagues in the 1990s. Soon it was widely used, both in the Linux kernel and eventually in MINIX, and with third-party modules it was made available for MacOS and Windows.
But there are still some problems to be solved: the ext2 file system, like most file systems of the 1990s, is susceptible to catastrophic data corruption if the system crashes or loses power while data is being written to disk. Over time, they also suffered severe performance losses due to fragmentation (a single file is stored in multiple locations, physically spread across the spinning disks).
Despite these problems, ext2 is still used in some special cases today -- most commonly, as a file system format for portable USB drives.

ext3

In 1998, six years after ext2 was adopted, Stephen Tweedie announced that he was working on improving ext2. This became ext3, and was adopted into the mainline Linux kernel in November 2001 in kernel version 2.4.15.

Packard Bell computer from the mid-1990s, Spacekid, CC0

For the most part, ext2 worked well enough with Linux distributions, but like FAT, FAT32, HFS, and other filesystems of the time—it was prone to catastrophic corruption in the event of a power outage. If power is lost while data is being written to the file system, it may be left in what is called an inconsistent state—something that is half done and half not done. This can result in the loss or corruption of a large number of files that are unrelated to the file being saved or even render the entire file system unmountable.
ext3 and other file systems from the late 1990s, such as Microsoft's NTFS, use journaling to address this problem. The journal is a special allocated area on disk where writes are stored within a transaction; if that transaction completes the disk write, the data in the journal is committed to the file system itself. If the system crashes before this operation is committed, the restarted system recognizes it as an incomplete transaction and rolls it back as if it never happened. This means that files in progress may still be lost, but the file system itself remains consistent and all other data is safe.
Three levels of logging are implemented in the Linux kernel using the ext3 file system: journal, ordered, and writeback.
Journaling is the lowest risk mode, where data and metadata are written to a journal before being committed to the file system. This guarantees that the files being written are consistent with the overall file system, but it significantly reduces performance.
Sequential is the default mode for most Linux distributions; sequential mode writes metadata to a journal and commits data directly to the file system. As the name implies, the order of operations here is fixed: first, the metadata is committed to the log; second, the data is written to the file system, and then the associated metadata in the log is updated to the file system. This ensures that in the event of a crash, the metadata associated with the incomplete writes is still in the journal, and the file system can clean up the incomplete write transactions when rolling back the journal. In sequential mode, a system crash may result in errors for files that were actively written during the crash, but the file system itself -- and files that were not actively written -- are guaranteed to be safe.
Writeback is the third mode - and the least secure logging mode. In write-back mode, like sequential mode, metadata is logged, but data is not. Unlike sequential mode, both metadata and data can be written in any order that facilitates optimal performance. This can significantly improve performance, but is much less secure. Although write-back mode still guarantees the safety of the file system itself, files written before a crash or corruption can be easily lost or corrupted.
Like ext2 before it, ext3 uses 16-bit internal addressing. This means that the maximum file size that can be handled by ext3 with a 4K block size is 2 TiB in a file system with a maximum size of 16 TiB.

ext4

Ext4 was released by Theodore Ts'o (the main developer of ext3 at the time) in 2006, and was added to the Linux mainline two years later in the 2.6.28 kernel version.
Ts'o describes ext4 as an interim technology that significantly extends ext3 but still relies on older technology. He predicts that ext4 will eventually be replaced by a true next-generation file system.

Dell Precision 380 Workstation, Lance Fisher, CC BY-SA 2.0

Ext4 is very similar in functionality to ext3, but supports larger file systems, has improved resistance to fragmentation, has higher performance, and has better timestamps.

ext4 vs ext3

There are some very specific differences between ext3 and ext4, which we will focus on here.
Backward Compatibility
Ext4 was specifically designed to be as backwards compatible with ext3 as possible. This not only allows ext3 filesystems to be upgraded in-place to ext4; it also allows the ext4 driver to automatically mount ext3 filesystems in ext3 mode, thus eliminating the need to maintain two separate code bases.

Large file system

The ext3 filesystem uses 32-bit addressing, which limits it to supporting only 2 TiB file sizes and 16 TiB filesystem sizes (this assumes a 4 KiB block size, some ext3 filesystems use smaller block sizes, thus limiting it further)
Ext4 uses 48 bits of internal addressing and can theoretically allocate files up to 16 TiB in size on the file system, with file systems up to 1,000,000 TiB (1 EiB) in size. In early ext4 implementations some userspace programs still limited it to a maximum filesystem size of 16 TiB, but as of 2011, e2fsprogs directly supports ext4 filesystems larger than 16 TiB. For example, Red Hat Enterprise Linux contractually supports only ext4 file systems up to 50 TiB, and recommends that ext4 volumes be no larger than 100 TiB.

Improved allocation method

ext4 makes a number of improvements to the way storage blocks are allocated before they are written to disk, which can significantly improve read and write performance.

Range section

An extent is a series of contiguous physical blocks (up to 128 MiB, assuming a 4 KiB block size) that can be reserved and addressed at once. Using extents can reduce the number of inodes required for a given file and significantly reduce fragmentation and improve performance when writing large files.

Multi-block allocation

ext3 calls the block allocator once for each newly allocated block. When multiple writers open the allocator at the same time, it is easy to cause severe fragmentation. However, ext4 uses delayed allocation, which allows it to coalesce writes and make better decisions about how to allocate blocks for writes that have not yet been committed.

Persistent preallocation

When pre-allocating disk space for a file, most file systems must write zeros to the file's blocks when it is created. ext4 allows the use of fallocate() instead, which guarantees the availability of space (and attempts to find contiguous space for it) without needing to write to it first. This significantly improves the performance of writing and future reading of written data for streaming and database applications.

Delayed Allocation

This is an intriguing and controversial feature. Delayed allocation allows ext4 to wait to allocate the actual blocks where the data will be written until it is ready to commit the data to disk. (In contrast, ext3 allocates blocks immediately, even if data is still being written to the write cache.)
Delaying allocation of blocks as data accumulates in the cache allows the file system to make better choices about how to allocate blocks, reducing fragmentation (on writes, and later on reads) and significantly improving performance. Unfortunately, however, it increases the likelihood of data loss for programs that have not specifically called the fsync() method (when the programmer wants to ensure that data is completely flushed to disk).
Suppose a program completely rewrites a file:

fd=open("file", O_TRUNC); write(fd, data); close(fd);

With older file systems, close(fd); is sufficient to ensure that the contents of file are flushed to disk. Even though writes are not strictly speaking transactional, there is a small risk of losing data if a crash occurs after the file has been closed.
If the write is unsuccessful (due to a bug in the program, an error on the disk, a power outage, etc.), both the original and the newer version of the file may lose data or become corrupted. If other processes access the file while it is being written, they will see the corrupted version. If some other process has the file open and does not want its contents to change -- for example, a shared library that is mapped into multiple running programs. These processes may crash.
To avoid these problems, some programmers avoid using O_TRUNC entirely. Instead, they might write to a new file, close it, and then rename it to the old file name:

fd=open("newfile"); write(fd, data); close(fd); rename("newfile", "file");

On file systems without delayed allocation, this is sufficient to avoid the potential corruption and crash problems listed above: since rename() is an atomic operation, it will not be interrupted by a crash; and running programs will continue to refer to the old file. Now the unlinked version of file only needs an open file handle to the file. But because ext4's delayed allocation causes writes to be delayed and reordered, rename("newfile", "file") can be executed before the contents of newfile are actually written to disk, which causes the problem of getting a bad version of file again in parallel.
To mitigate this, the Linux kernel (since version 2.6.30) attempts to detect these common code cases and force an immediate allocation. This will reduce but not prevent the possibility of data loss -- and it won't help with new files. If you are a developer, please note: the only way to guarantee that data is written to disk immediately is to call fsync() correctly.

Unlimited subdirectories

ext3 is limited to 32,000 subdirectories; ext4 allows an unlimited number of subdirectories. Starting with kernel version 2.6.23, ext4 uses HTree indexes to reduce performance loss for large numbers of subdirectories.

Log verification

Ext3 does not perform checksum logging, which poses problems for disks outside the kernel's direct control or for controller devices with their own cache. If the controller or disks with their own cache get out of order with writes, it is possible that the order of ext3's journal transactions will be disrupted, potentially corrupting files that were written during (or some time before) the crash.
In theory, this problem can be solved using write barriers - when mounting the filesystem, you set barrier=1 in the mount options, and the device will faithfully perform fsync all the way down to the underlying hardware. In practice, it has been found that storage devices and controllers often do not honor write barriers - improving performance (and performance benchmarks compared to competitors), but increasing the likelihood of data corruption that should be prevented.
Checksumming the journal allows a file system to be realized when it is first mounted after a crash that some of its entries are invalid or out of order. This thus avoids errors that roll back partial or out-of-order log entries and further corrupt the file system - even if some storage devices fake or do not obey write barriers.

Fast file system check

Under ext3, when fsck is invoked it checks the entire filesystem -- including deleted or empty files. In contrast, ext4 marks unallocated blocks and sectors in the inode table, allowing fsck to skip them entirely. This greatly reduces the time it takes to run fsck on most filesystems and is implemented in kernel 2.6.24.

Improved timestamps

ext3 provides timestamps with a granularity of one second. While sufficient for most purposes, mission-critical applications often require tighter timing control. By providing nanosecond timestamps, ext4 makes it useful for enterprise, scientific, and mission-critical applications.
The ext3 file system also does not provide enough bits to store dates after January 18, 2038. ext4 adds two bits here, extending the Unix epoch by 408 years. If you're reading this in 2446 AD, there's a good chance you've moved on to a better file system - which will allow me to sleep peacefully after my death if you're still measuring time since 00:00 (UTC), 1 January 1970.

Online Defragmentation

Neither ext2 nor ext3 directly supports online defragmentation -- that is, defragmenting the file system while it is mounted. ext2 has an included utility called e2defrag, which does what its name implies -- it needs to be run offline while the filesystem is not mounted. (Obviously, this is very problematic for a root filesystem.) The situation is even worse on ext3 -- while ext3 is less susceptible to severe fragmentation than ext2, running e2defrag on an ext3 filesystem can result in catastrophic corruption and data loss.
Although ext3 was originally thought to be "immune to fragmentation", use of massively parallel write processes to the same file (such as BitTorrent) clearly shows that this is not entirely the case. Some userspace tricks and workarounds, such as Shake, address this problem in one way or another - but they are slower and less satisfactory in various ways than a true, filesystem-aware, kernel-level defragmentation process.
ext4 solves this problem with e4defrag, an online, kernel-mode, filesystem-aware, block- and extent-level defragmentation utility.

Ongoing ext4 development

ext4, as the plague-infected in Monty Python once said, "I'm not dead yet!" Although its lead developer considers it only a stopgap measure on the way to a true next-generation filesystem, no likely candidate will be ready (due to technical or licensing issues) to be deployed as a root filesystem for some time.
There are still some key features to be developed in future versions of ext4, including metadata checksums, first-class quota support, and large allocation blocks.

Metadata Checksum

Since ext4 has redundant superblocks, this provides a way for the file system to verify the metadata therein and determine for itself whether the primary superblock is corrupt and needs to use a spare block. It is possible to recover from a corrupt superblock without checksums - but the user first needs to realize that it is corrupt and then try to manually mount the filesystem using an alternate method. Since mounting a filesystem read-write with a corrupted primary superblock can, in some cases, cause further damage that even an experienced user cannot avoid, this is not a perfect solution either!
Compared to the extremely strong per-block checksums provided by next-generation file systems such as Btrfs or ZFS, ext4's metadata checksums are very weak. But it's better than nothing. Although verifying everything sounds easy! -- In fact, there are some significant challenges in coupling checksums with a file system; see the design document for details.

First-class quota support

Wait, quotas? ! We've had these since the day ext2 came out! Yes, but they’ve always been something that was added as an afterthought, and they’ve always been silly. It's probably not worth going into detail here, but the design document lays out the way quotas will be moved from userspace into the kernel and will be enforced more correctly and efficiently.

Large allocation blocks

Over time, those pesky storage systems just keep getting bigger and bigger. Since some SSDs already use 8K hardware block size, ext4's current limitation of 4K blocks is becoming increasingly restrictive. Larger storage blocks can significantly reduce fragmentation and improve performance, at the expense of increased "slack" space (the space left over when you only need part of a block to store a file or the last block of a file). You can find detailed instructions in the design document.

Practical limitations of ext4

ext4 is a robust, stable file system. Most people should be using it as their root filesystem these days, but it can't handle all needs. Let’s briefly talk about some of the things you shouldn’t expect — now or possibly in the future:

While ext4 can handle data up to 1 EiB in size (equivalent to 1,000,000 TiB), you really shouldn't try to do so. Besides being able to remember the addresses of more blocks, there are also issues of scale. And now ext4 won't handle (and probably never will) more than 50-100 TiB of data.

Ext4 is also not sufficient to ensure data integrity. While journaling was a major advancement back in the day with ext3, it did not cover many common causes of data corruption. If the data has already been corrupted on disk -- due to faulty hardware, the effects of cosmic rays (yes, really), or simply data decay over time -- ext4 cannot detect or repair this corruption.

Based on the above two points, ext4 is just a pure file system, not a storage volume manager. This means that even if you have multiple disks -- that is, parity or redundancy -- you could theoretically recover corrupted data from ext4, but there's no way to know if using it to your advantage would be helpful. While it is theoretically possible to separate the file system and storage volume management system in different layers without losing automatic corruption detection and repair capabilities, this is not the way current storage systems are designed, and it would pose significant challenges for new designs.

Alternative file systems

Before we begin, a word of warning: be very careful, no alternate filesystems are built-in and directly supported as part of the mainline kernel!

Even if a filesystem is safe, using it as a root filesystem can be very scary if something goes wrong during a kernel upgrade. If you don't have a good reason to use alternative media booting via a chroot, be patient with kernel modules, grub configuration and DKMS... don't remove reserved root files on a critical system.

There may be good reasons to use a filesystem that your distribution doesn't directly support - but if you do, I strongly recommend that you install it after your system is up and running. (For example, you might have an ext4 root filesystem, but store most of your data in a ZFS or Btrfs pool.)

XFS

XFS has the same status as non-ext file systems in mainline Linux. It is a 64-bit journaling filesystem that has been built into the Linux kernel since 2001, providing high performance for large filesystems and high concurrency (ie, a large number of processes writing to the filesystem at once).
Starting with RHEL 7, XFS is the default file system for Red Hat Enterprise Linux. It still has some drawbacks for home or small business users -- most notably, resizing an existing XFS filesystem is such a pain that it doesn't make sense to just create another one and copy the data.
While XFS is stable and performant, there aren't enough specific end-use differences between it and ext4 to recommend using it anywhere other than the default (like RHEL7) unless it solves a specific problem with ext4, such as filesystems larger than 50 TiB.
XFS is not in any way a "next generation" file system to ZFS, Btrfs, or even WAFL (a proprietary SAN file system). Just like ext4, it should be seen as a stopgap measure until a better way comes along.

ZFS

Developed by Sun Microsystems, ZFS is named after the zettabyte -- the equivalent of 1 trillion gigabytes -- because it can theoretically address very large storage systems.
As a true next-generation file system, ZFS offers volume management (the ability to handle multiple separate storage devices in a single file system), block-level cryptographic checksums (allowing data corruption to be detected with extremely high accuracy), automatic corruption repair (where redundant or parity storage is available), fast asynchronous incremental replication, inline compression, and much more.
From a Linux user's perspective, the biggest problem with ZFS is the licensing issue. The ZFS license is the CDDL, a semi-permissive license that conflicts with the GPL. There is a lot of controversy about the significance of using ZFS in the Linux kernel, with arguments ranging from "it's a GPL violation" to "it's a CDDL violation" to "it's totally fine, it hasn't been tested in court." Most notably, Canonical has had the ZFS code inlined in its default kernel since 2016, and there have been no legal challenges so far.
At this point, even as an avid ZFS user, I do not recommend using ZFS as a Linux root filesystem. If you want to take advantage of ZFS on Linux, set up a small root filesystem with ext4, then use ZFS for the rest of your storage, putting data, applications, and whatnot on it -- but keep the root partition on ext4 until your distribution explicitly supports ZFS roots.

Btrfs

Btrfs — short for B-Tree Filesystem, often pronounced "butter" — was released by Chris Mason in 2007 while he was at Oracle. Btrfs aims to have most of the same goals as ZFS, providing multiple device management, per-block checksums, asynchronous replication, inline compression, and more.
As of 2018, Btrfs is fairly stable and can be used as a standard single-disk filesystem, but should probably not be relied upon for volume managers. It has severe performance issues compared to ext4, XFS, or ZFS in many common use cases, and its next-generation features -- replication, multi-disk topologies, and snapshot management -- can be so numerous that the consequences can range from catastrophic performance degradation to actual data loss.
The continued status of Btrfs is controversial; SUSE Enterprise Linux adopted it as the default file system in 2015, while Red Hat announced in 2017 that it would no longer support Btrfs starting with RHEL 7.4. It may be worth noting that the product supports Btrfs deployments used as a single-disk file system, rather than a multi-disk volume manager like in ZFS. Even Synology uses Btrfs in its storage devices, but it is layered on top of traditional Linux kernel RAID (mdraid) to manage the disks.

Below we briefly list the introduction, features and advantages of ext~ext4:

Summarize

The above is the full content of this article. I hope that the content of this article will have certain reference learning value for your study or work. Thank you for your support of 123WORDPRESS.COM. If you want to learn more about this, please check out the following links