Speeding Up WAL in Dolt: Why fdatasync Only Works With Zero-PaddingLongread · digest for October 24
Patent-style infographic blueprint of a WAL file layout with a 4 MiB zero buffer and fdatasync syscall.
Longread5 sections≈ 8 min

Speeding Up WAL in Dolt: Why fdatasync Only Works With Zero-Padding

Every backend engineer eventually runs into the same fundamental wall: database transaction commit latency. Your app can be written in a blazing-fast compiled language, your codebase can have zero extra allocations, and your network stack can fly on eBPF. Yet the moment a client sends COMMIT, the program has to freeze and wait until the bytes physically settle on non-volatile storage.

That durability guarantee (the D in ACID) is handled by the classic Write-Ahead Log, or WAL. And almost everywhere under the hood, you'll find the familiar fsync(2) system call.

In the cloud era, though, that call turns into a bottleneck. On a local NVMe drive, flushing the cache takes microseconds; on a network-attached block volume like AWS EBS gp3, every round-trip to storage jumps to around 1 millisecond. Now imagine your database processing a sequential stream of small transactions: a thousand commits per second becomes a hard ceiling purely because of storage controller network latency.

That exact dilemma hit the engineers behind the versioned relational database Dolt, led by Aaron Son. They set out to optimize their log, and their first obvious fix turned into a great lesson in how Linux filesystems actually work.

The Naive Optimization Trap: fsync vs. fdatasync

If you open the Linux kernel man pages for system calls, the difference between fsync(2) and fdatasync(2) looks dead simple:

  • fsync flushes both the modified data pages and all file metadata: modification time (mtime), permissions, file size, and the extent map of disk blocks.
  • fdatasync promises to flush only the data pages, skipping metadata attributes unless they are strictly needed to read the data back later.

Sounds like an instant win, right? A database doesn't care about the WAL's mtime during a commit—if the server crashes and reboots, the WAL parser reads internal record headers and CRC checksums, not filesystem timestamps.

So the Dolt engineers swapped fsync for fdatasync on every commit. They ran the benchmarks and... saw zero difference. Sequential transaction latency stayed just as high.

Why? To see what broke, we have to drop down to the POSIX standard and filesystem drivers. Under POSIX, fdatasync must guarantee that the data can still be read after a power loss. Now think about how a log is normally written: the database opens the file in append mode and keeps writing new bytes to the very end.

Every append increases the file size (st_size)! And file size is critical metadata. If the filesystem doesn't record the new size in the inode, the OS will reboot thinking the newly written chunk is past the end of the file and return EOF. As a result, the Linux kernel is forced to run the exact same heavy metadata flush as a regular fsync.

Anatomy of Disk Round-Trips on Ext4

Let's peek under the hood of Ext4, the workhorse filesystem on most Linux servers.

When a process appends to a file and flushes it, the Ext4 driver and its JBD2 (Journaling Block Device) subsystem perform three sequential round-trips to the block device:

  1. Write the data: modified pages in RAM are sent to the disk.
  2. Write metadata to the journal: the driver locks the file structures and writes the updated file size and new disk extents into Ext4's circular journal buffer.
  3. Commit the journal transaction: a hardware flush barrier goes out to the drive to guarantee the metadata is persisted.

On a network-attached EBS volume, each step is a separate 1 ms network round-trip. One tiny transaction commit costs your app at least 3 milliseconds of latency!

XFS handles this slightly better: it can batch the metadata write and journal commit into two round-trips instead of three. Still, two round-trips is twice as slow as shipping only the payload data.

Why fallocate and ftruncate Didn't Help

Okay, if changing the file size is the culprit, why not preallocate space ahead of time? Say, call fallocate(2) or ftruncate(2) for 100 megabytes up front. Then the file size is already huge, so subsequent writes won't change it—right?

The Dolt engineers tested that hypothesis too. Again, no speedup.

The reason lies in a Linux kernel security mechanism. When you call fallocate, the filesystem allocates disk blocks but marks them as unwritten extents. That prevents a program from accidentally reading leftover data from another process before it writes its own bytes.

The first time your app writes real bytes into an unwritten block, the Linux kernel intercepts the operation and updates the inode's extent tree, flipping the block from "uninitialized" to "valid." And that is yet another metadata update—meaning another Ext4 journal lock and another flush barrier to storage!

The Counterintuitive Fix: Pre-Filling With Zeros

The only way to make the filesystem treat disk blocks as fully initialized is to physically write data into them.

Isometric diagram of pre-filling the WAL buffer with zeros and flushing data via fdatasync.

The Dolt team used a neat engineering trick:

  1. The database keeps a 4 MiB buffer filled with zeros (zero-padding).
  2. When the write cursor nears the end of the initialized region, the process issues a regular write(2) call to append 4 MiB of zeros to the file.
  3. Because zeros are physically written to storage, the filesystem creates real, initialized extents.
  4. Thousands of subsequent small transactions inside that 4 MiB window are no longer file extensions (append)—they become in-place overwrites of existing sectors!

When overwriting already allocated blocks, the file size never changes, the extent map stays untouched, and fdatasync finally behaves as advertised: it flushes only the modified data in one single network round-trip!

Go Implementation

Here is how that logic looks in pure Go using syscall.Fdatasync directly:

package wal

import (
    "os"
    "syscall"
)

// Allocate a 4 MiB zero buffer for pre-initializing the file
const zeroBlockSize = 4 * 1024 * 1024
var zeroBytes = make([]byte, zeroBlockSize)

type WALWriter struct {
    file          *os.File
    currentOffset int64
    allocatedEnd  int64
}

// WriteCommit atomically persists a transaction to the write-ahead log
func (w *WALWriter) WriteCommit(record []byte) error {
    recordLen := int64(len(record))

    // If the next transaction reaches the boundary of initialized blocks
    if w.currentOffset+recordLen > w.allocatedEnd {
        // Extend the file by physically writing 4 MiB of zeros
        if _, err := w.file.WriteAt(zeroBytes, w.currentOffset); err != nil {
            return err
        }
        w.allocatedEnd = w.currentOffset + zeroBlockSize
    }

    // Overwrite the zeroed region with actual commit data (in-place overwrite)
    if _, err := w.file.WriteAt(record, w.currentOffset); err != nil {
        return err
    }
    w.currentOffset += recordLen

    // fdatasync flushes only the data cache without metadata locks
    return syscall.Fdatasync(int(w.file.Fd()))
}

To verify I/O behavior, you can trace system calls in real time with strace:

# Monitor fdatasync syscall latency in real time
strace -f -T -e trace=fdatasync,write -p $(pgrep dolt) 2>&1 | grep fdatasync

In the terminal output, you can literally watch fdatasync latency drop from 3.2 ms down to a steady 0.9–1.1 ms per commit. In synthetic Sysbench OLTP benchmarks with short sequential transactions, total commit time fell by 10–40%.

The cost of optimization: Write Amplification

This trick isn't free magic. Pre-writing 4 MiB of zeros roughly doubles the physical byte volume sent to the drive (Write Amplification increases by ~2x). If your storage subsystem is already saturated at 100% throughput (MB/s) on a local SSD, zero-padding will actually slow you down. It shines specifically on network-attached storage with high round-trip latency (RTT) and plenty of spare bandwidth.

Also, the in-place overwrite trick does nothing on Copy-on-Write filesystems like Btrfs or OpenZFS, where overwriting a block physically allocates a new block and updates the metadata tree anyway. But for classic server setups on Ext4 and XFS, it's one of the sharpest, least obvious ways to beat disk latency.

Want a deep diveVotes help us choose what to break down in detail. This material has 0 votes.
Sources · 2