wellcake: A Kubernetes Operator for Managing Valkey Across Four TopologiesArticle · digest for November 11
Architectural isometric diagram of the wellcake operator managing four Valkey topologies in a Kubernetes cluster.
Article4 sections≈ 7 min

wellcake: A Kubernetes Operator for Managing Valkey Across Four Topologies

When you need to deploy the in-memory database Valkey in Kubernetes, engineers usually choose between two extremes. The first path is renting a managed DBMS in the cloud: you get fault tolerance out of the box, but you overpay for memory and lose access to fine-grained tuning. The second path is running the database inside your own cluster via a StatefulSet, surrounding it with replica failover scripts, and flinching at midnight alerts whenever the primary restarts.

I was drawn to the open-source wellcake project under the Apache-2.0 license, which was highlighted in the DevOps&SRE Library channel. Its authors eliminated the choice between expensive cloud rentals and brittle manifests: the Go-based valkey-operator controller moves Valkey operational rules—quorum control, leader failover, resharding, and backups—directly into the Kubernetes reconciliation loop. Let's look at why basic orchestrator primitives fall short for in-memory databases and how this automation works.

Why a Basic StatefulSet Is Blind to Valkey Failures and How the Operator Works

After the Redis license change, the industry rallied behind the open Valkey fork under the Linux Foundation (other RESP-compatible solutions are evolving in parallel—we recently covered KiviDB as a multithreaded Rust alternative to Redis and Valkey). Yet the old operational pains moved into Kubernetes right alongside the database.

You might ask: what is wrong with a standard StatefulSet if it gives pods stable names and disks? To the core Kubernetes controller, a database container looks no different from a web server. It is like a building superintendent who checks whether the lights are on in each office, without knowing who holds the master ledger. If the write node goes down, the orchestrator spins up a fresh container with the same network name, while replicas without an election command keep waiting for the old leader or sync with an empty node, wiping data.

To make the cluster understand database logic, you use the operator pattern—a controller with Custom Resource Definitions (CRDs). In wellcake, there are two: ValkeyCluster defines the topology and disks, while ValkeyACL configures user permissions via the idempotent ACL SETUSER ... reset command.

The controller blocks common mistakes out of the box: it forbids temporary emptyDir folders and requires PersistentVolumeClaim volumes so that kubelet does not evict a database pod when disk space runs low. The maxmemory limit is automatically set to 60% of the container's memory limit (resources.limits.memory). Instead of manually tuning dozens of parameters, two profiles are available:

  • Cache — a fast cache with allkeys-lru eviction, no disk persistence, and a 1Gi volume;
  • Durable — a store with a noeviction policy, RDB snapshots, an AOF log, and a 10Gi volume.

Four Topologies in One Resource: From a Single Node to a Sharded Cluster

The key design win of wellcake is unifying four Valkey deployment schemes inside the ValkeyCluster resource. The operator is installed from the oci://ghcr.io/melancholictheory/charts/valkey-operator registry into the valkey-system namespace.

Let's walk through an end-to-end scenario: we will spin up a fault-tolerant cache in the apps namespace via the valkey-cluster chart with topology=Replication, profile=Cache, and replicas=3. Depending on the topology, the controller activates its own protection mechanics:

  1. Standalone. A single pod without replication for test environments and ephemeral caches.
  2. Replication. The setup from our example: a primary node (primary) and two replicas connected via REPLICAOF. Every fifteen seconds, the operator runs a TCP probe on the nodes. If the primary goes silent, the controller compares replication offsets (offset), promotes the replica with the freshest data to leader, and protects the database from split-brain (split-brain) when pod-0 returns. At the same time, the operator updates the valkey.wellcake.io/role label: the <cluster>-primary service routes writes strictly to the leader, while <cluster>-replicas distributes reads across replicas.
  3. Sentinel. For applications whose clients query Sentinel directly, the operator runs a separate StatefulSet and service on port 26379 (26380 with TLS), where a quorum of arbiters decides leader changes.
  4. Cluster. Keys are split into hash slots across masters linked by the gossip protocol. The operator assembles the cluster via valkey-cli --cluster create and automatically rebalances slots when scaling (autoReshard). On Valkey 9.1+, Atomic Slot Migration (ASM, ADR 0001) transfers a slot as a single transaction with no stuck keys on network failure. The operator also rescues the cluster if more than half of the masters are lost at once (ADR 0006) and gossip loses quorum: it isolates dead nodes and executes CLUSTER FAILOVER TAKEOVER on surviving replicas (automatically for Cache, or via the valkey.wellcake.io/quorum-takeover annotation for Durable).

Seamless Operations: Proactive Rollout, Secret Rotation, and S3 Backups

Let's continue with our Replication cluster example: suppose we updated the container image or changed memory limits.

During a standard update, Kubernetes immediately terminates the primary pod: while the system detects the failure and reroutes traffic, writes stall for 15–20 seconds. In wellcake, proactive rollout (ADR 0004) is enabled with the valkey.wellcake.io/proactive-rollout: "true" annotation, switching the StatefulSet to OnDelete mode. The operator updates the replicas one by one and, before stopping the primary, hands over leadership (replica promote, CLUSTER FAILOVER, or SENTINEL FAILOVER) to an already updated replica. The old primary restarts as a regular replica, cutting downtime virtually to zero. On Valkey 9.0+ images, the shutdown-on-sigterm failover directive is also enabled.

Operational Annotations and the kubectl valkey Plugin

Managing the cluster is convenient via built-in toggles and the CLI extension:

  • valkey.wellcake.io/proactive-rollout: "true" — zero-downtime updates with proactive primary role handover;
  • valkey.wellcake.io/rotate-password — on-the-fly password rotation via ACL SETUSER without pod restarts;
  • kubectl valkey plugin — quick status checks (status), backups (backup), resharding (reshard), primary failover (failover), and cluster pausing (hibernate).

For security, replicas connect to the primary under a dedicated replicator account (+psync +replconf +ping) with no access to keys. Updated TLS certificates are picked up on the fly via CONFIG SET tls-cert-file (plus automatic checks with tls-auto-reload-interval 3600). S3 backups run on a CronJob schedule via valkey-cli --rdb with restoration from spec.restoreFrom, while spec.replicateFrom sets up asynchronous cross-region replication with certificate bundling (replicateFrom.caSecret) for mutual TLS.

Where wellcake Saves Team Resources and When to Hold Off

Controllers like wellcake show how automation is moving out of cloud consoles into the open ecosystem (we discussed the growth of such solutions in our review of six Kubernetes platform engineering tools). Still, every tool has its operating conditions and limits.

The operator is a great fit for teams running their own Kubernetes clusters on bare-metal or across multiple clouds that want to manage a Valkey fleet declaratively via GitOps, separate read and write traffic across services, and automate resharding alongside S3 snapshots.

You should avoid adopting it in two cases:

  1. Mini-projects and disposable environments. For a small service where losing the cache on restart goes unnoticed, installing an operator and CRDs is unnecessary complexity; a single container is enough.
  2. Mission-critical financial systems with strict SLAs. The authors explicitly mark the project status as early / experimental: APIs are at v1alpha1 and v1beta1, and validation webhooks are disabled by default (--set webhooks.enabled=true). You should battle-test the operator in a staging environment before running it in production.

Moving to Valkey in Kubernetes no longer forces you to write fragile failover scripts or tolerate write pauses during upgrades: wellcake packs four topologies, seamless failover, and S3 backups into a single resource. If you want to explore the controller source code, read the notes in docs/adr/, or try the kubectl valkey plugin, check out the documentation and example manifests in the project repository.

Want a deep diveVotes help us choose what to break down in detail. This material has 0 votes.
Sources · 2