Apache Nessie: Version Control and Transaction Isolation in the Data LakehouseLongread · digest for October 22
Isometric 3D model of multi-layered Data Lakehouse data branches in Apache Nessie with commit history and zero-copy merging.
Longread5 sections≈ 6 min

Apache Nessie: Version Control and Transaction Isolation in the Data Lakehouse

If you've ever built a corporate data warehouse or a lake spanning hundreds of terabytes, you probably remember that cold sweat: it's 3 a.m., a complex ETL script crashed while updating the fifth data mart out of ten, and at 8 a.m. leadership opens the dashboards. Half the data is fresh, the other half is from yesterday, the financial numbers don't add up, and you're stuck frantically rolling back Parquet files in S3 by hand.

In classic relational databases, transactions save us: BEGIN TRANSACTION ... COMMIT. If something breaks, the database simply rolls back the state, and readers never see the half-baked mess. But when your data lives distributed across cloud object storage in Apache Parquet format, traditional transactions remained a pipe dream for a long time.

Table formats like Apache Iceberg and Delta Lake took a huge step forward by introducing snapshots and single-table isolation. Still, what if a pipeline updates ten interconnected tables at once? How do you guarantee analysts won't see a "torn" state?

That's where the open-source project Apache Nessie steps in, bringing the "Git-for-Data" concept to the Lakehouse architecture. Let's look at how it works under the hood and why it changes the game for data engineering.

The Dirty Read Problem in Data Lakes

To appreciate Nessie, think about how ETL isolation is usually hacked together:

  1. Double buffering (Staging Schemas): data is written to a temporary schema and then swapped in with clunky scripts or ALTER TABLE ... SWAP WITH. It's cumbersome, breaks permissions, and often isn't supported by distributed engines.
  2. Readiness flags: you write an is_ready = false column and require every analytical query to filter on it. One junior engineer forgets WHERE is_ready, and garbage shows up on executive charts.
  3. Writing straight to live: the riskiest strategy of all, where intermediate data is exposed to every reader mid-insert.

When Skillbox ran their corporate data warehouse migration to Trino, Iceberg, and S3, they hit this exact pain point trying to keep hundreds of analytical streams consistent. Nessie solves it by acting as a central metadata catalog with native support for branches and commits.

Under the Hood: Zero-Copy Branching

The core magic of Apache Nessie is that it never copies physical data files.

Diagram of Zero-Copy branching in Apache Nessie: an isolated branch with new commits references shared data files in object storage without duplication.

When you create a branch in Git, it doesn't clone your entire hard drive. Git just creates a lightweight pointer to a specific commit in the history graph. Nessie does the exact same thing—not for source code, but for Iceberg table metadata:

  • Catalog as a content tree: Nessie stores a tree of references to table manifests (metadata.json). Each table is identified by a ContentKey.
  • Zero-Copy Branch: running CREATE BRANCH etl_batch FROM main takes just a few milliseconds. The catalog database simply creates a named pointer to the current root snapshot. Terabytes and petabytes of Parquet files in your S3 bucket stay right where they are.
  • Isolated commits: while your ETL job writes new Parquet files and generates Iceberg metadata, all changes are tied exclusively to the etl_batch branch. Queries against main physically cannot see those commits.

Inside Nessie, commit history is organized as a directed acyclic graph (DAG). For persistent metadata storage, Nessie supports PostgreSQL, DynamoDB, RocksDB, or distributed in-memory backends.

In Practice: Branching and Atomic Merges in SQL

Let's see what working with Nessie looks like in Spark SQL or Trino. Thanks to Nessie's reference syntax, data version control takes just a handful of standard commands:

-- 1. Create an isolated branch for the nightly run before ETL starts
CREATE BRANCH etl_nightly_sales IN nessie FROM main;

-- 2. Switch the session to work on the new branch
USE REFERENCE etl_nightly_sales IN nessie;

-- 3. Run heavy transformations and update data marts
-- All writes go to S3, but in the catalog, changes are visible ONLY on our branch
INSERT INTO analytics.orders_summary
SELECT 
    customer_id, 
    date_trunc('day', order_date) AS order_day,
    sum(amount) AS total_spent
FROM raw_stage.orders
WHERE order_date >= '2026-10-21'
GROUP BY 1, 2;

-- 4. Update customer metrics in a neighboring table
UPDATE analytics.customer_profiles
SET loyalty_tier = 'Gold'
WHERE total_spent > 100000;

Notice what's happening here: BI dashboards are still pointed at main. Not a single analyst sees a half-populated orders_summary table or hits table-lock errors.

Once ETL finishes and passes data quality checks (say, via Great Expectations), it's time to publish:

-- 5. Switch back to the main branch and merge
USE REFERENCE main IN nessie;

-- Atomically merge multiple tables into production at once!
MERGE BRANCH etl_nightly_sales INTO main IN nessie;

The MERGE BRANCH operation executes atomically. The main branch pointer advances to the merge commit, and for every reader, the new versions of both tables appear simultaneously in the exact same millisecond.

Concurrent Write Conflicts and Merge Strategies

What happens if someone else writes data to main while our long-running ETL pipeline is busy crunching numbers?

Nessie uses Optimistic Concurrency Control (OCC). During a merge, it supports two core strategies:

  1. Fast-Forward Merge: if main hasn't moved forward since we created our working branch, the main pointer simply jumps to the tip of our branch. No extra commits are created.
  2. Three-Way Merge: if main has new commits, Nessie compares the common ancestor, the target branch, and the source branch. As long as the commits touched different tables (for instance, nightly ETL updated orders_summary while streaming updated clickstream), the merge succeeds automatically with zero conflicts.

If both processes tried to modify the exact same table, Nessie blocks the operation and throws a conflict error, protecting the lake from inconsistency.

Gotchas and Garbage Collection

It sounds ideal, but in production every clean architectural pattern has a cost. With Nessie, that cost is storage lifecycle management.

Because creating branches is practically free, engineers tend to spin up dozens of experimental branches and forget to delete them. Each branch holds references to specific Iceberg snapshots. As a result, Iceberg's garbage collector (expire_snapshots) can't clean up old data files from S3 because a forgotten two-month-old branch is still pointing to them.

To keep cloud storage bills from ballooning, teams should adopt two rules right away:

  • Set a TTL on temporary branches and automatically delete branches once ETL jobs finish.
  • Run Nessie GC (Garbage Collection) regularly to find files unreachable from any live branch or tag and safely purge them from S3.

At the end of the day, Apache Nessie turns a data lake from a wild west of Parquet files into a disciplined, predictable, transactional platform—where rolling back a mistake is a simple RESET REFERENCE instead of a sleepless night of manual recovery.

Want a deep diveVotes help us choose what to break down in detail. This material has 0 votes.
Sources · 3