A Kubernetes upgrade is safer when it is treated as a compatibility project rather than a single maintenance command. This reusable checklist covers version planning, deprecated APIs, workload testing, backups, admission controls, node replacement, rollback limits, and post-upgrade validation so your team can plan a cluster change without relying on assumptions.
Overview
A Kubernetes version upgrade can affect more than the control plane. Workloads, admission webhooks, ingress controllers, storage integrations, observability agents, policy engines, CI/CD tools, and node images may all depend on particular API behavior or supported version ranges.
Start by separating the upgrade into three related activities:
- Control-plane upgrade: moving the cluster management components to a newer Kubernetes version.
- Node and runtime change: replacing or upgrading worker nodes, operating system images, container runtimes, or machine types.
- Application compatibility work: updating manifests, controllers, charts, operators, and delivery workflows that interact with the cluster.
Managed Kubernetes providers usually define their own upgrade paths, maintenance windows, node-pool procedures, and supported-version rules. Confirm those details in the provider's current documentation before scheduling work. For self-managed clusters, document the control-plane topology, certificate process, datastore backup method, networking, storage, and node replacement procedure.
A useful upgrade plan has an explicit target version, a test environment that resembles production, a list of known compatibility risks, a recovery procedure, and named owners for each validation step. If your deployments are managed through Git, review the repository and environment structure alongside the cluster change. The GitOps workflow guide can help teams define what belongs in source control and how environment changes are promoted.
Checklist by scenario
Before any Kubernetes version upgrade
- Record the current state. Capture the Kubernetes version, provider or distribution, node pools, operating system and runtime versions, cluster add-ons, ingress implementation, storage classes, custom resource definitions, admission webhooks, and critical workloads.
- Choose the target version and path. Check the supported upgrade sequence for your distribution or managed service. Do not assume that a direct jump between arbitrary minor versions is allowed. Record required intermediate versions, maintenance constraints, and any provider-specific prerequisites.
- Inventory deprecated APIs. Search application repositories, Helm values, rendered manifests, operators, and generated YAML for APIs that may be removed or changed in the target version. Review both active resources and templates that are not currently deployed. Pay particular attention to custom controllers and older chart dependencies.
- Build a compatibility matrix. Use a table like this before testing:
Component Current version Target compatibility Owner Test result Action Ingress controller Recorded value Confirm in vendor or project documentation Team Pass, fail, or pending Upgrade, replace, or accept Policy engine Recorded value Confirm supported Kubernetes range Team Pass, fail, or pending Update rules or version Storage integration Recorded value Confirm driver and API compatibility Team Pass, fail, or pending Test provisioning and attach - Define success criteria. Specify acceptable API error rates, workload readiness, deployment success, ingress behavior, storage operations, background jobs, alerts, and recovery time. A vague “cluster is healthy” is not enough.
- Schedule a reversible window. Avoid starting immediately before a major release, billing process, data migration, or staffing gap. Identify a change owner, an observer, and a decision-maker who can stop or postpone the work.
When upgrading a development or staging cluster
- Take the same backup and inventory steps used for production, even if the data is noncritical. Staging should test the procedure, not merely the new version.
- Deploy representative workloads, including stateful services, jobs, autoscaling workloads, network policies, and applications with admission or sidecar dependencies.
- Run the normal CI/CD path from manifest generation through deployment. If you use automated maintenance, review how dependency changes are proposed and tested; the dependency update tools comparison provides useful context for that workflow.
- Exercise failure cases: drain a node, restart a pod, roll back a deployment, provision storage, rotate a secret, and test ingress from outside the cluster.
- Record every manual workaround. A workaround that exists only in a staging operator's memory is a production risk.
When upgrading production
- Confirm that the tested target and tested configuration match the production plan.
- Verify recent backups and perform a restore test when the cluster stores critical state or cluster configuration.
- Freeze unrelated infrastructure and application changes for the upgrade window.
- Check disruption budgets, replica counts, topology rules, capacity headroom, and node drain behavior.
- Upgrade according to the provider or distribution procedure, observing each control-plane and node-pool step rather than treating the process as one opaque action.
- Use a canary node pool or a small, representative workload where the platform supports that approach. Watch scheduling, readiness, networking, storage attachment, and application errors before expanding.
- Record the exact versions, timestamps, commands, decisions, and observed failures in the change record.
What to double-check
Backups and rollback
Define what “rollback” means before the upgrade begins. A node-pool change may be reversible by retaining the previous pool temporarily, while a control-plane change may not offer a simple downgrade path. If a downgrade is unsupported, the recovery plan may instead be restoring workloads and data into a known-good cluster or rebuilding affected components.
Verify that backups include the data your recovery plan depends on. Test access credentials, encryption keys, retention, and restore permissions. For stateful applications, validate application-level backup and restore rather than assuming that a cluster configuration backup contains database data.
Admission controllers and policy
Review validating and mutating webhooks, policy engines, image verification, namespace rules, and security controls. An unavailable webhook can block deployments, while a changed policy can reject manifests that previously worked. Confirm webhook service endpoints, certificates, failure behavior, timeouts, and namespace selectors. Test both an allowed deployment and an intentionally rejected deployment.
Workloads and platform services
Check more than pod status. Validate:
- Deployments, StatefulSets, DaemonSets, Jobs, and scheduled jobs.
- Readiness, liveness, and startup probes.
- Horizontal or vertical autoscaling behavior.
- Ingress, service discovery, DNS, network policies, and external load balancing.
- Persistent volume provisioning, mounting, resizing, and recovery.
- Secrets, certificates, identity integration, and cloud permissions.
- Logs, metrics, traces, alerts, and dashboards.
- Operators and custom resources, including reconciliation after a controller restart.
Use your release process as part of the test. Teams that need a broader framework for deployment approvals, canaries, and recovery decisions can also consult this guide to release management tools.
Post-upgrade validation
- Confirm all control-plane components and nodes report the intended versions and healthy conditions.
- Check cluster events for scheduling, admission, authentication, networking, and storage errors.
- Run a smoke test that covers the most important user journey, not just a health endpoint.
- Deploy a small change through the normal delivery workflow.
- Inspect error rates, latency, queue depth, saturation, restart counts, and business-level signals for an agreed observation period.
- Close the change only after owners confirm application behavior and monitoring coverage.
Common mistakes
- Checking only the Kubernetes version: The cluster may be on the target version while an ingress controller, operator, CSI driver, or policy tool remains incompatible.
- Searching only live resources: Deprecated APIs can remain in Helm charts, overlays, examples, or generated templates and fail during the next deployment.
- Assuming pod readiness proves application health: A pod can be ready while external routing, authentication, storage, or a downstream dependency is failing.
- Relying on an untested backup: A successful backup job does not prove that the required data can be restored by the available team.
- Ignoring disruption settings: Strict disruption budgets, low replica counts, or restrictive topology rules can prevent safe node draining.
- Changing too many variables at once: Combining a Kubernetes upgrade with a networking migration, major chart update, and database change makes diagnosis harder.
- Calling every failure a rollback: Define the technical recovery action in advance, including whether the old node pool, application version, data snapshot, or separate cluster is involved.
When to revisit
Revisit this Kubernetes upgrade checklist before every minor-version change, node-pool replacement, operating system refresh, control-plane maintenance window, or major platform add-on upgrade. Refresh the compatibility matrix whenever your provider changes supported-version rules, Kubernetes publishes a new release, a dependency drops an API, or your organization adopts a new ingress, policy, storage, observability, or delivery component.
Make the checklist part of normal platform operations. Store it with the infrastructure repository or internal documentation, assign an owner, and link the completed change record to the tested manifests and recovery notes. Review it during seasonal planning cycles and after incidents, especially when an upgrade exposes a gap in monitoring, backup restoration, or application testing.
The practical next step is to create a copy of the matrix, fill in your current cluster and add-on versions, and schedule a staging rehearsal. Only move to production after the target path, deprecated API review, backup test, workload validation, admission-controller review, and recovery decision all have named owners and recorded results.
