If your service processes bank card payments or subscriptions, the very phrase "passing the annual audit" sends a very understandable chill down your spine. The PCI DSS (Payment Card Industry Data Security Standard) payment security standard and SOC 2 regulations leave no room for improvisation: any external auditor demands a detailed answer to who exactly, when, on what basis, and which specific line of code was changed on a production server.
If deployments at your company are still set up as a hand-rolled bash script called from a CI/CD pipeline — or, even more dangerously, via admin privileges held by an on-call engineer with direct kubectl apply -f access — passing the check honestly is practically impossible. Someone pushed an urgent hotfix at midnight right in the cluster console, configuration drifted from the repository (production drift), and a month later nobody on the team understands why a production container behaves differently from staging.
Ayush Agarwal, CTO of the international payments platform Dodo Payments, published a detailed breakdown of how their engineering team rebuilt change delivery around GitOps and the ArgoCD orchestrator. They managed not only to guarantee 100% audit transparency, but also to drive deployment failures down to zero.
Zero kubectl: Git as the single source of truth
The first and toughest architectural decision was completely revoking direct administrative access to the production Kubernetes cluster from all human developers and DevOps engineers. No client certificates on personal laptops, no emergency SSH tunnels.
The entire infrastructure was split into two isolated repositories:
- Application repository (
app-repo): holds only the payment services' source code, unit tests, and Docker image build pipelines. When a Pull Request is merged, the build system pushes a tested image to the container registry with a unique commit tag. - Infrastructure repository (
infra-gitops-repo): describes the canonical state of the entire platform as declarative manifests. The repository is strictly split into environment branches (staging, andmainfor production).
The only entity with permissions to modify resources in the cluster became the service account of the ArgoCD controller itself. An engineer edits a manifest, opens a Pull Request, passes mandatory code review with approvals from two senior developers, and runs automated security checks. Once the commit is merged into main, ArgoCD reads the changes on its own and reconciles the cluster to the described state.
If anyone inside the infrastructure tries to manually change the replica count or edit a pod's environment variable, ArgoCD detects the drift and overwrites the resource with the canonical value from Git within seconds (the Self-Heal mechanism).
ArgoCD Application declaration and automated auditing
To manage a service in the cluster, an Application manifest is deployed. Below is the actual configuration file for the core payment engine:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: payment-core
namespace: argocd
spec:
project: default
source:
repoURL: 'https://github.com/dodopayments/infra-gitops-repo.git'
targetRevision: main
path: environments/production/payment-core
destination:
server: 'https://kubernetes.default.svc'
namespace: payments
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
Look at the syncPolicy.automated block: prune: true removes obsolete resources from the cluster if they were deleted from Git, while selfHeal: true immediately wipes out any unauthorized manual drift. For a PCI DSS auditor, the Git commit log becomes exhaustive evidence: every configuration point is signed with the commit author's cryptographic GPG key.
Database proxy sidecar and two-phase graceful shutdown
In payment services, the most dangerous situation during a new version rollout is losing a transaction while a pod restarts. Imagine: a user pays for a subscription, the card is charged, the payment gateway sends a webhook, and at that exact moment Kubernetes kills the old service pod with a SIGTERM signal to rotate images. The database connection drops, the transaction rolls back, and the money hangs in limbo.
The Dodo Payments team solved this using a combination of two proven patterns:
- Database Proxy Sidecar: alongside the application container in the same pod runs a lightweight database proxy sidecar. The application talks to it over a local socket, while the proxy maintains a persistent connection pool to the database, isolating brief network blips.
- Two-phase
preStopdelay: by default, Kubernetes sends SIGTERM to a container at the same time it removes its IP address from the service's Endpoints list. Because iptables updates are asynchronous, the load balancer still manages to send a few more requests to the already dying container.
To eliminate this completely, a 15-second lifecycle pause was added to the Deployment manifest:
apiVersion: apps/v1
kind: Deployment
metadata:
name: payment-service
namespace: payments
spec:
replicas: 3
template:
spec:
containers:
- name: payment-api
image: dodopayments/payment-core:v2.14.0
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 15"]
- name: db-proxy-sidecar
image: dodopayments/db-pool-proxy:v1.2.0
While sleep 15 runs, the external Ingress controller is guaranteed to remove the node from the load-balancing pool, new requests are routed to neighboring replicas, and the container calmly finishes active financial operations before closing connections.
Managing secrets without compromising Git
One of the biggest fears when moving to GitOps is accidentally leaking private encryption keys and payment gateway tokens into a public or internal repository.
At Dodo Payments, this was solved by pairing External Secrets Operator with a corporate HashiCorp Vault. Only references to keys (ExternalSecret manifests) describing where to fetch a token are stored in Git. ArgoCD syncs these references, and the operator inside the secured cluster perimeter pulls the actual values from Vault directly into Kubernetes Secrets in memory.
Combined with perimeter monitoring tools such as a specialized Kubernetes security console, which we wrote about earlier, the platform gets solid protection against secret injection.
If your infrastructure repository has hundreds of manifests, never set the Git polling interval shorter than three minutes: GitHub will quickly rate-limit your requests. Set up Webhook notifications from the repository to the ArgoCD server on every successful push — syncs will trigger instantaneously without parasitic load.
Results of the migration
The outcome of this architectural overhaul speaks for itself. The PCI DSS certification audit passed without a single finding on the delivery pipeline, 502 Bad Gateway network errors during daily releases dropped to zero, and full disaster-recovery provisioning of the entire infrastructure from scratch in a backup region now takes just minutes.

