OpenShift Disaster Recovery: RPO, RTO, and Tested Restore Paths

etcd holds all Kubernetes and OpenShift object state. Snapshot etcd on a schedule — typically hourly for aggressive RPO, daily for lower tiers — using oc debug node commands or automated CronJobs documented by Red Hat. Store snapshots in object storage with versioning, encryption, and cross-region replication independent of the cluster being protected.

Restore drills prove snapshot integrity. Quarterly, rebuild a lab cluster from etcd snapshot following Red Hat restore procedures — not just mount the backup file. Measure wall-clock time from incident declaration to functional API server; that is your real RTO, not the slide deck estimate.

Defragment and monitor etcd db size proactively. Alarm conditions on etcd database size and leader election churn precede many corruption incidents. Pair backups with must-gather baselines for support escalation.

Encrypt etcd snapshots at rest and restrict object storage IAM to backup automation only — ransomware targets backup buckets. Immutable storage or WORM policies add defense when attackers gain cloud console access.

Active-passive regional pairs keep a warm standby cluster with GitOps-synced manifests and replicated container images. DNS or global load balancers swing traffic when the primary region fails health checks. RPO includes Git merge lag and image replication delay — measure both.

Active-active across regions is harder: split-brain data, conflict resolution, and regulatory data residency. Most BFSI workloads stay active-passive with synchronous DB replication limited to metro distance. OpenShift routes and external DNS TTLs affect how fast clients reach the survivor cluster.

ROSA and hyperscaler managed control planes simplify regional DR for the masters; you still own worker capacity, PVC replication, and stateful service design. Document which components Red Hat restores vs customer responsibility in support contracts.

Global load balancer health checks should hit application routes, not only API server — a cluster can answer 6443 while all ingress is broken. OpenShift disaster recovery validation includes end-user transaction paths.

Alert on backup job failures, snapshot age beyond RPO SLA, and object storage lifecycle expiring old backups too aggressively. Prometheus metrics from OADP and etcd backup CronJobs belong on platform dashboards reviewed weekly.

After failover, compare golden signals to pre-incident baselines — API latency, error rates, queue depth. Monitoring confirms recovery; users confirm correctness. Our OpenShift monitoring article details the metrics to watch during and after DR events.

OpenShift disaster recovery maturity is binary until tested: you either restored on deadline or you did not. Invest in automation and drills proportional to business impact — tier-one payment rails demand tighter RPO than internal staging clusters.

Automate backup verification jobs that restore a single namespace to a sandbox weekly — silent backup corruption is worse than obvious backup failure alerts.

Operators for PostgreSQL, MongoDB, and Kafka often provide native DR — prefer vendor-supported replication over DIY PVC copy when data correctness matters.

Synchronous replication across metros trades RPO for latency — measure application impact before promising zero data loss.

Git-declared state recovers quickly; databases recover slowly — DR runbooks should sequence platform restore before application scale-up and DB promotion.

Tabletop exercises without touching production build muscle memory — quarterly tabletops plus annual full restore drills balance risk and effort.