Insights · OpenShift
OpenShift Upgrade Planning for Zero-Surprise Cluster Updates
Overview
OpenShift upgrade planning is the discipline of moving the Cluster Version Operator (CVO) from one minor release to the next without breaking operators, storage, or workloads that assumed the old API surface. Unlike ad-hoc kubectl version bumps, OCP upgrades orchestrate control-plane static pods, machine config pools, and hundreds of cluster operators in a defined graph published by Red Hat.
Teams that treat upgrades as a quarterly calendar event — with runbooks, lab rehearsal, and stakeholder sign-off — stay current on security patches and retain support entitlements. Teams that defer until they are four minors behind pay compound interest: deprecated APIs disappear, OperatorHub bundles require jump upgrades, and etcd schema migrations offer no casual undo button.
This guide focuses on self-managed OCP; ROSA and ARO automate portions of the control plane but still require worker pool and operator compatibility review. Use it alongside your change-management process and Red Hat compatibility matrix before you patch channel spec in ClusterVersion.
Need help implementing this?
Talk to engineers who deploy these patterns on OpenShift in production—not generic advisory decks.
Upgrade Graph, Channels, and Release Acceptance
The CVO reads upgrade graph metadata from release images hosted on quay.io (or your mirror). Channels — fast, stable, eus — map to recommended edges; pinning to an explicit version via spec.desiredUpdate bypasses channel float when you need determinism. Extended Update Support (EUS) channels buy time for slow-moving tenants but require proactive planning before EUS windows end.
Before changing channel or desired version, run oc adm upgrade recommend or consult the Red Hat upgrade path documentation for your current version. Skipping intermediate minors is rarely supported. Download release image signatures and verify against Red Hat GPG keys in disconnected environments — a tampered release image is a cluster-wide compromise.
Acceptance testing belongs in a lab cluster with representative operators (logging, service mesh, ODF, GitOps) and sample workloads. Record operator versions from ClusterServiceVersion objects and compare against the Red Hat Operator Compatibility Matrix. If a certified operator lacks support on the target OCP version, upgrade the operator first or defer the platform bump.
Build an upgrade calendar shared with application owners: blackout periods, regulatory freeze windows, and dependency milestones. Communicate expected maintenance duration from lab rehearsal data, not optimism. Stakeholders approve windows when they trust estimates backed by prior runs.
Pre-Flight Checks: etcd, Machines, and Deprecated APIs
etcd health gates every upgrade. Run oc etcd health-check and inspect etcd metrics for db size, leader changes, and fsync duration. Defragment etcd during a maintenance window if db size approaches quota — never defragment all members simultaneously. Ensure backup snapshots complete successfully within 24 hours of the upgrade window.
Machine Config Pools (MCP) must show UPDATED=True with zero unavailable nodes before and during upgrade. Paused pools block cluster completion — document any intentional pause for golden-image testing. Review oc get apirequestcounts for deprecated API usage; remove or migrate workloads calling removed beta APIs before they break on the new kube-apiserver.
PodDisruptionBudgets and cluster autoscaling settings affect node drain during upgrade. Verify PDBs allow at least one disruption for stateless tiers or plan manual scale-up. For workloads using CSI volumes, confirm driver compatibility with the target OCP release — storage is the most common long-tail upgrade blocker after custom operators.
Clear Alertmanager silences and maintenance mode entries that hide real problems during upgrade. Pre-scale critical Deployments if drain eviction is slow. Export current ClusterVersion and operator versions to a change ticket for audit comparison post-upgrade.
Executing the Upgrade and Monitoring Progress
Open a maintenance window with observability dashboards focused on API latency, etcd, ingress error rates, and operator Degraded conditions. Start the upgrade with oc adm upgrade --to-image or by updating ClusterVersion spec; watch oc adm upgrade. The CVO progresses through Downloading, Progressing, and Complete phases — each cluster operator upgrades in dependency order.
If the upgrade stalls, inspect ClusterVersion conditions and oc get clusteroperators for Degraded=True. Common stalls include machine config drain timeouts, unavailable ingress, or third-party webhooks rejecting updated resources. Red Hat KB solutions and must-gather bundles accelerate support cases — capture them before rolling back.
Worker-only upgrades follow a similar MCP-driven process after the control plane completes. Align RHEL CoreOS machine config with worker OS expectations when mixing openShift virtualization workers or RHEL compute nodes. GPU and specialized hardware nodes may need separate MCP labels — upgrade them in waves after the general pool validates.
Assign an upgrade commander with authority to pause or rollback — committees deciding mid-incident extend outages. Keep war-room bridges open until smoke tests pass and error budgets recover. Document actual vs planned duration for continuous improvement.
Rollback, Recovery, and Post-Upgrade Validation
OpenShift supports rollback to the previous minor only under specific conditions while the prior release image remains available. etcd restore from snapshot is the disaster path when rollback is impossible — rehearse this in a lab, not during production panic. Document RPO and RTO with stakeholders; upgrades do not eliminate the need for etcd backups.
Post-upgrade validation mirrors install handover: all cluster operators Available, sample app route healthy, logging and monitoring agents current, and authentication flows intact. Re-run conformance or workload-specific smoke tests. Update internal documentation for new default APIs, removed features, and changed SCC defaults.
Feed lessons learned into the next OpenShift upgrade planning cycle: operator lag, hidden deprecated API consumers, and maintenance window duration estimates. Clusters on a current stable channel with quarterly rehearsals rarely surprise anyone; deferred upgrades always do.
Automate pre-flight checks where possible — scripts querying apirequestcounts, MCP status, and etcd alarms should run nightly, not only before upgrades. Surprises discovered Friday before Monday upgrade get rescheduled without shame; surprises Monday morning do not.
Coordinating Upgrades Across Multiple Clusters
Fleet upgrades stagger by wave: lab, dev, staging, prod region A, prod region B. GitOps repos pin ClusterVersion desired versions per cluster folder — promote version bumps via PR after prior wave soaks seven days. ACM policies can alert when any cluster drifts off approved version.
Shared services — identity, logging, registry mirrors — must upgrade compatibly with spoke clusters. Upgrade hub GitOps before spokes if hub runs newer operators spokes depend on. Document cross-cluster dependencies in architecture diagrams operators actually read.
OpenShift upgrade planning at fleet scale is program management as much as engineering — track operator certification lag per cluster, not just Kubernetes version. One straggler cluster on an old minor blocks organization-wide feature adoption.
ROSA and ARO control planes upgrade on Red Hat schedules — subscribe to release notes and validate worker pool compatibility before auto-upgrade windows. Managed control plane does not eliminate application testing.
Upgrade Planning for ROSA, ARO, and Managed Control Planes
ROSA documents upgrade policies per cluster — configure upgrade gates, maintenance windows, and notification channels in OCm. Worker pools and machine types must support target versions before approval.
ARO integrates with Azure maintenance — align AKS-adjacent dependencies and private link endpoints during upgrade rehearsal. Identity and DNS cutovers affect workloads even when control plane is managed.
Self-managed and managed clusters in one fleet need a unified calendar — developers do not care which API upgraded first; they care that integrations still work Monday morning.
Document emergency stop procedures — pausing CVO, freezing MCP — when upgrade regression threatens revenue. Knowing how to halt safely matters as much as knowing how to start.
Stakeholder Communication During OpenShift Upgrade Planning
Application owners need notice for deprecated API removals — publish dashboard from apirequestcounts monthly so teams fix clients before upgrade weekend.
Change advisory boards approve maintenance windows with explicit rollback criteria — if rollback triggers, who decides and within what time bound.
Post-upgrade reports to leadership summarize duration, incidents, and deferred items — builds trust for next quarter's upgrade slot.
OpenShift upgrade planning without communication is change imposed — imposed changes get blamed for unrelated outages for months afterward.
Explore further
Related services
- ServiceOpenShift Upgrade Services
Related technology
- TechnologyRed Hat
Related reading
- InsightOpenShift Installation Guide
Need help with OpenShift?
Talk to engineers who implement these patterns in production—not generic advisory decks.