A database controller sends a health check to a PostgreSQL primary and gets silence in return. At that exact second, trouble is already brewing behind the scenes. Applications can't write normally, and the control plane has to guess what happened. Did the server actually crash, or did the network between the controller and the primary just blip while the primary keeps serving clients? If you promote a replica too fast, you end up with two live primaries at once. Say hello to split-brain and diverging data.
PlanetScale engineer Shlomi Noach broke down how to prevent this in a November 5, 2026 post about unplanned failover in Neki, their sharded PostgreSQL system. His previous post covered planned switchovers, where every node is healthy, the successor is picked ahead of time, and routers buffer queries. During a sudden network cut or hardware failure, you don't have that luxury. The system has to figure out on the fly whether the primary is truly unreachable and which replica can safely take over.
Why a Silent Health Check Isn't Enough to Replace a Primary
Picture a standard Neki shard: primary P accepts writes, while replicas R1, R2, and R3 continuously stream and replay the WAL (Write-Ahead Log, where PostgreSQL records every change first). If the control node Admin loses contact with P, retrying the health check won't always help. Retries fix brief packet loss, but if Admin itself is isolated on the network, it will never get an answer even though the primary and replicas are still running fine.
Many high-availability systems rely on distributed leases. An agent on the primary renews a timed key in shared storage, and once the key expires, the server is declared dead. Neki skips leases for health checks. Instead, Admin queries the replicas through the pg_stat_wal_receiver system view.
-- Admin checks WAL reception status on every live replica
SELECT status,
written_lsn,
flushed_lsn,
last_msg_receipt_time
FROM pg_stat_wal_receiver;
The decision rule is straightforward. If Admin can't reach P, but at least one replica is still streaming from it, no failover happens. Most likely, Admin itself got cut off. If Admin can't reach P, at least one reachable replica reports a broken stream, and no other replica contradicts it, the primary is declared dead. And if neither P nor any replicas respond, the controller simply waits. Promoting someone blindly is a direct path to a second primary.
Synchronous replication provides the safety guarantee here. By default, Neki configures synchronous_standby_names with ANY 1 (R1, R2, R3) and synchronous_commit = on. The primary won't confirm a client commit until at least one replica flushes the WAL entry to disk. At time T, R1 might confirm; at T+1, R3 might. But if no replica can reach the primary, the primary physically cannot complete any write, even if it's still alive.
Four Failure Modes, One Signal: Catching Even a Frozen Postgres
For replica polling to react quickly, replicas need to notice a broken connection fast. And databases fail in several ways.
When a process crashes normally, the local PostgresManager (whose goroutine waits on process exit) immediately sends a PROCESS_EXITED event to Admin, though the controller still checks with the replicas. If a firewall sits between nodes and drops traffic, connections fail with an error right away.
Silent packet loss is worse: the socket goes quiet, yet the TCP connection looks healthy for a long time. The trickiest case, as Noach notes, is a hung PostgreSQL process. The server kernel still answers TCP packets, so the connection looks fine, while the database itself does nothing.
Replicas don't need to tell these scenarios apart. Neki tunes timeouts so that every failure ends in a broken replication stream. Short TCP keepalives and tcp_user_timeout catch packet loss, while wal_receiver_timeout cuts off a hung process. PostgreSQL defaults wal_receiver_timeout to 60 seconds, but Neki shrinks it to just a few seconds. If the primary stays silent longer than that, the replica drops the connection itself.
# On the primary (postgresql.conf): commit waits for WAL flush on 1 replica
synchronous_commit = on
synchronous_standby_names = 'ANY 1 (R1, R2, R3)'
# On replicas (postgresql.conf): fast disconnect if primary hangs
# PostgreSQL default is 60s; 5s here illustrates 'a few seconds'
wal_receiver_timeout = 5s
This collapses four very different failures into a single signal in pg_stat_wal_receiver, with no custom workarounds for each failure type.
The Double-Slash Trick: Blocking a 'Resurrected' Primary
Once Admin decides to fail over, a strict rule kicks in: assume the worst, because the old primary could wake up at any second. Before failover starts, P coming back is harmless—replicas just reconnect. But if it comes back mid-failover, replicas will reconnect to it on their own, WAL will flow again, and P will be able to confirm commits under ANY 1.
MySQL has STOP REPLICA IO_THREAD to halt log reception, but PostgreSQL physical replication has no equivalent switch. So Neki makes a reversible edit to primary_conninfo on all reachable replicas by prepending // to the primary hostname.
# Before failover: replica connected to primary pg-primary-p
primary_conninfo = 'host=pg-primary-p port=5432 application_name=R2'
# Failover starts: // prefix breaks host resolution, blocking the replica
# from reconnecting to a revived P and freezing its WAL position
primary_conninfo = 'host=//pg-primary-p port=5432 application_name=R2'
Neki uses the same trick during planned switchovers. It gives two benefits at once: a revived P is left with zero replicas and can't commit anything, and WAL positions across all nodes freeze in place for a fair comparison.
Picking the New Leader by LSN, Switching Over, and Retiring the Old Primary
Each replica's log position is tracked by its LSN (Log Sequence Number): receive_lsn shows how much WAL it fetched over the network, and replay_lsn shows how much it has already applied. Here is an example snapshot after freezing:
| Replica | Received (receive_lsn) | Replayed (replay_lsn) |
|---|---|---|
R1 | 120 | 120 |
R2 | 125 | 122 |
R3 | 123 | 123 |
The second replica fetched the most records over the network, but it's still replaying the tail of its local log. The third is slightly behind on receiving, yet it has applied everything it received. The first is simply behind. First, Admin looks for a node that has already replayed the latest log. None has, so it waits a short window for the receive leader to finish replaying its tail. If nobody catches up within that window, it picks the candidate with the highest receive_lsn, breaking ties with the highest replay_lsn.
Because WAL is strictly sequential, the highest receive_lsn guarantees the candidate holds all committed history, regardless of which replica confirmed individual commits. As a bonus, the remaining replicas already share a prefix with the new primary and switch cleanly to the new timeline. The one exception is if a replica undergoing maintenance happens to be ahead and cannot be promoted; it will later need a rewind or rebuild.
Next, Admin creates replication slots on the chosen node, promotes it in PostgreSQL, updates Routers, and repoints the other replicas. The first writes on the new primary will pause briefly: under ANY 1, it needs at least one replica connected.
After a crash, the old primary P is never automatically rejoined as a replica, because it might hold uncommitted writes. It is moved to TEARDOWN (outside traffic, replication, and monitoring), but its disk isn't wiped right away. Its logs, final WAL, and disk state are the best evidence for a postmortem.
The design is honest about its limits, too. If P and one replica get partitioned together, they can keep committing under ANY 1; cross-AZ acknowledgments reduce that risk at the cost of write latency. Automation also can't save you if the primary and the most up-to-date replica die at the exact same time, and in asynchronous mode, broken replica streams are only a strong signal, not a guarantee. Finally, Admin itself can go down—Shlomi Noach saved that breakdown for the next article.

