TL;DR
Kubernetes becomes a platform product when application teams can depend on a defined service, not simply access a cluster. That requires supported workload patterns, measurable service-level objectives, evidence-based upgrade gates, capacity reserved for disruption, and tested recovery of the complete service. The practical starting point is one production service profile with clear ownership and acceptance criteria. Expand the platform only as its operating evidence supports the promises being made.
Introduction
Deploying Kubernetes and operating a dependable Kubernetes service are different responsibilities. The first establishes a technical environment. The second establishes what application teams can expect from that environment when they deploy, scale, change, and recover their workloads.
Consider an illustrative maintenance window. A production cluster reports healthy nodes, average utilization looks acceptable, and the platform team begins a worker-node upgrade. The first drain completes, but a replacement application Pod remains pending. Its resource and placement requirements cannot be satisfied by the remaining eligible nodes. Another application has a restrictive disruption budget, and the team cannot proceed without either restoring capacity or accepting a service interruption.
The upgrade procedure may be correct. The operating assumptions are not.
This is where Kubernetes as a platform product becomes a useful model. Instead of asking only whether the cluster is healthy, the organization asks whether a supported business service can continue through the changes and failures the platform claims to accommodate.
SLOs, upgrade gates, capacity, and recovery should therefore be designed together. A service objective without capacity is an unfunded promise. An upgrade gate without a recovery path is an incomplete change decision. A backup without application validation is recovery material, not evidence of a recovered service.
Define the Platform Boundary Before the Service Promise
The CNCF Platforms White Paper describes a platform in terms of the capabilities and experiences it provides to its users. That is a useful boundary for Kubernetes operations: the product is the supported way to consume the platform, not the collection of components installed underneath it.
A platform offering might include namespace onboarding, workload identity, approved deployment templates, network policy, storage classes, observability, and a supported release lifecycle. It does not have to implement every underlying service itself. It does need to explain which capabilities it owns and how their dependencies are supported.
The operating shift is straightforward, but it changes what the team must demonstrate.
| Operating concern | Cluster-focused view | Platform-product view |
|---|---|---|
| Service health | Nodes and control-plane components are responding | Supported consumer journeys meet agreed objectives |
| Upgrades | The upgrade job completed | Representative workloads passed release acceptance |
| Capacity | Current utilization is acceptable | Workloads fit and perform under the promised disruption scenario |
| Recovery | Backups completed successfully | Service, data, and access were restored and validated |
The model below puts the service contract above the four operating disciplines. Notice that all four feed the same evidence review. They should not produce four disconnected definitions of production readiness.
Scope and Assumptions
This article assumes a platform team supporting multiple application teams in production. The model applies to managed and self-managed Kubernetes, but the supported upgrade procedures and recovery responsibilities remain specific to the distribution, provider, and installed integrations.
The numerical examples are illustrative planning inputs, not tested results or vendor guarantees. The article defines an operating model, not a universal cluster-upgrade or disaster-recovery runbook.
Keep application ownership explicit. The platform team should own the supported platform experience and coordinate its dependencies. Application owners should own business correctness, application-specific resilience, and acceptance of recovered data. A platform product owner should be accountable for the service promise, roadmap, and funding decisions.
Build Service Profiles That Application Teams Can Use
A generic production tier hides too many differences. A replicated web service, a stateful database, and a GPU-backed inference service do not have interchangeable placement, startup, disruption, or recovery requirements.
Start with a small number of supported profiles. For each profile, publish the deployment path, required configuration, support hours, lifecycle policy, capacity commitment, recovery scope, and owner. Include working examples and an exception process with an accountable approver and expiration date.
For a stateless application profile, onboarding might require resource requests, meaningful readiness checks, an approved image source, multiple replicas where availability requires them, and a demonstrated ability to tolerate worker replacement. Stateful applications need additional decisions about data consistency, backup methods, storage topology, and restore validation.
Choose the profile by the workload’s requirements: acceptable interruption, maximum data loss, placement restrictions, required capacity, and operating support. Do not assign the most demanding service tier simply because a team selected “production” in a portal.
The contract must also distinguish maintenance protection from failure protection. Surviving one failed node is not the same commitment as surviving an entire zone while another node is being upgraded. Those promises require different capacity and validation.
The service profile is where architecture requirements become operating obligations.
Measure the Platform Journeys That Matter
A service-level indicator measures an outcome. A service-level objective sets the target for that outcome over a defined period. Google’s Implementing SLOs guidance recommends expressing many indicators as good events divided by total eligible events, with the measurement tied to the user’s experience.
For a Kubernetes platform, start with a few journeys that represent the service being offered. The following targets are examples for discussion and baselining, not recommended defaults.
| Consumer journey | Example objective over 30 days | Measurement boundary |
|---|---|---|
| Deploy a supported application | 99% of eligible submissions become application-ready within five minutes | Submission through admission, scheduling, startup, and the agreed readiness test |
| Reach the platform’s reference service | 99.9% of eligible transactions succeed within one second | The supported DNS, traffic-routing, and service path |
| Provision persistent storage | 99% of eligible new-volume requests become usable within two minutes | An approved storage class, size range, and topology; excludes data restoration |
Define eligibility before collecting results. An invalid manifest can be excluded under a published rule. A valid submission rejected by broken admission infrastructure should not disappear from the denominator simply because Kubernetes never accepted it.
Likewise, define what application-ready means. A running container is not necessarily a usable service. The acceptance test might require a successful reference transaction or another workload-appropriate signal.
Keep Platform Evidence and Application Evidence Distinct
A reference workload helps test the platform path under controlled conditions. It does not establish the correctness or availability of every hosted application. Keep its results identifiable, and use application telemetry to validate actual business outcomes.
Measure important service profiles separately. A large number of successful lightweight deployments should not conceal repeated failures in a smaller stateful tier. Missing telemetry should trigger investigation, not silently count as success.
Make the Error Budget Affect Change Decisions
An error budget represents the unreliability permitted by the SLO. Google’s Example Error Budget Policy connects budget exhaustion to a pause in ordinary releases while retaining a path for critical incident work and security fixes.
For the platform, establish that policy before an upgrade becomes contentious. Define when reliability work takes priority, when promotion must stop, and who may approve a bounded emergency exception. Keep planned-maintenance treatment explicit rather than changing the calculation after a difficult maintenance window.
Security and data integrity require their own acceptance conditions. Unauthorized access does not become acceptable because an availability budget remains.
Treat Kubernetes Upgrades as Platform Releases
A platform release includes the components whose behavior applications depend on, not only the Kubernetes version. Depending on the environment, that includes node images, the container runtime, networking, storage integration, admission components, traffic management, and critical operators.
Kubernetes’ Version Skew Policy defines supported relationships between core components and notes that deployment tools can impose additional restrictions. Use that policy together with the support requirements for the actual distribution and integrations. An upstream-compatible combination is not automatically an approved enterprise platform release.
The Deprecated API Migration Guide also recommends using client warnings, metrics, and audit information to identify deprecated API use. Inspect rendered manifests and live clients. A controller can depend on an API that is absent from the application team’s repository.
Require Evidence at Each Promotion Gate
A useful gate names the evidence, the decision owner, and the condition that prevents progression.
| Gate | Evidence required | Reason to hold |
|---|---|---|
| Compatibility | Supported component combination, upgrade order, and API-use assessment | An essential integration or client lacks a validated path |
| Disruption and capacity | Drain rehearsal, workload placement, startup behavior, and scenario headroom | A required workload cannot relocate or maintain its service objective |
| Canary acceptance | Representative workload results, service indicators, and security tests | A regression appears or the observation period provides insufficient evidence |
| Recovery readiness | Accessible recovery artifacts, a tested procedure, and a named decision owner | The fallback depends on an untested or unavailable dependency |
These gates should stop the next unit of change, not merely produce a warning that operators are expected to ignore. Record the reason for a hold and the evidence required to clear it.
Understand the Limits of PodDisruptionBudgets
Kubernetes’ Disruptions documentation explains that a PodDisruptionBudget limits supported voluntary evictions. It cannot prevent involuntary failures, and direct Pod deletion can bypass it. Deployment and StatefulSet rolling updates use their own workload-controller settings rather than being constrained by the PDB.
A three-replica application with minAvailable: 2 may allow one eviction while all replicas are healthy. That permission is not proof that a replacement can be scheduled, become ready quickly enough, or handle the required load.
Review disruption budgets alongside rollout configuration, placement, termination behavior, and available capacity. When a drain blocks, determine which service constraint it has exposed. Bypassing the protection is a separate risk decision, not the routine completion step.
Match the Canary to the Change Boundary
Google’s Canarying Releases guidance emphasizes representative traffic, meaningful observation, and awareness of shared failure domains. Apply that principle to the platform’s actual change boundary.
A worker-pool canary provides evidence about a worker-level change. It does not isolate the rest of the cluster from a cluster-wide admission change or control-plane upgrade. Use a representative test cluster for changes whose effects cannot be isolated within a production cluster.
Define the fallback precisely. Replacing a worker pool, reverting an application release, restoring control-plane state, and rebuilding into another cluster are different operations. Do not label a plan “rollback” unless the exact reversal is supported and validated. Where reversal is unavailable, name the forward-fix or rebuild-and-restore path before approving the release.
Plan Capacity for the Failure You Promise to Survive
Kubernetes’ resource-management documentation makes an important distinction: scheduling uses resource requests, not the current utilization graph. A Pod can remain pending despite low observed CPU or memory use when no eligible node satisfies its requests.
Capacity planning therefore needs two tests. First, do the declared workloads fit under the scenario? Second, can those workloads meet their performance objectives under representative load? Understated requests may make the first test look better without improving the second.
Calculate Disruption Headroom Explicitly
Consider twelve workers distributed evenly across three zones. Assume each worker provides 24 CPU units and 96 GiB of workload-usable memory after system reservations and platform overhead.
The service profile promises to accommodate one lost zone while one additional worker in a surviving zone is unavailable for maintenance. The simplified calculation is:
The cluster has comfortable aggregate capacity during normal operation. It fails the combined disruption scenario before a safety margin is added.
The choices are concrete: fund additional capacity, reduce admitted demand, reduce concurrent surge, or revise the maintenance-time commitment. Stopping the next drain does not recover capacity already unavailable.
Test Placement, Not Just Totals
Passing the arithmetic is necessary, not sufficient. Validate per-node fit, node affinity, storage accessibility, specialized hardware requirements, and the placement of critical replicas. Also validate that logical failure-domain labels reflect the underlying infrastructure rather than assuming labels establish independence.
Kubernetes’ Node Autoscaling documentation identifies provisioning limits, incompatible configurations, and unavailable provider capacity as reasons expansion may not make every Pod schedulable. Treat autoscaling as a measured response mechanism, not a capacity reservation.
For each profile, test how long newly provisioned capacity takes to become useful and what happens while workloads wait. Include recovery work in the demand model rather than assuming it can consume resources already committed to production.
Reserved headroom should have a named purpose in the cost model. Removing it is a change to the resilience commitment, not simply a utilization improvement.
Recover the Service in Dependency Order
Recovery needs to distinguish Kubernetes object state from application data and the dependencies required to use both.
Kubernetes’ etcd guidance describes backing up the cluster’s object state. That does not make an etcd snapshot a backup of application data stored in persistent volumes or external databases. Velero’s How Velero Works documentation separately describes object backups and persistent-volume protection, and notes that cluster backups are not strictly atomic.
For a stateful application, define how the selected protection method produces an acceptable consistency point. A successful snapshot job does not answer that question on its own.
Choose the Recovery Method Before the Incident
For a self-managed platform, the supported path may restore the existing control-plane state or rebuild a replacement cluster and restore applications into it. Those paths have different dependencies and procedures. Use the supported recovery process for the distribution rather than mixing steps during the outage.
For a managed platform, establish which control-plane recovery actions belong to the provider and which application, data, and configuration protections belong to the customer. Document what the team can execute when the usual management interface is unavailable.
The workflow below illustrates service recovery rather than a product-specific restore sequence. Its main boundary is that traffic returns after validation, not immediately after the platform starts responding.
Use the dependency map to determine which activities can run in parallel. Required controllers and resource definitions must be available for their dependent applications, but that does not justify enabling every reconciliation loop at once. Control when deployment automation, scheduled jobs, and external writes resume.
Recover Access and Trust Alongside the Platform
Kubernetes’ encryption-at-rest guidance warns about losing the keys required to decrypt protected resources. Include key access, emergency credentials, certificates, image access, and recovery documentation in the recovery design.
Keep the necessary recovery capability outside the failure boundary it must survive. A replacement cluster is not useful when its only backup credentials or deployment artifacts remain trapped in the unavailable environment.
Retain recovery points appropriate to corruption and compromise scenarios as well as infrastructure loss. Restore the intended application and configuration version, then reconcile deliberately. Automatically deploying the newest configuration may reintroduce the condition being recovered from.
Stop the Recovery Clock at Accepted Service
Define the recovery time objective against a usable service outcome. Measure the exercise from the agreed disruption point through detection, decisions, restoration, validation, and traffic return. Define the recovery point objective against recoverable application data, not simply the timestamp of a completed backup job.
Require application-specific acceptance. An authorized user should complete a representative transaction; data checks should confirm the expected state; prohibited access should remain prohibited; and resumed processing should not create unintended duplicate effects.
Record the scenario, recovery point, configuration version, elapsed time, validation results, and application owner’s acceptance. A successful exercise is evidence for that tested scenario, not proof of every possible failure.
Make the Operating Contract Part of Delivery
The operating model becomes useful when it changes routine decisions. Keep service profiles and release criteria under version control, connect evidence to change records, and make the current service limits visible to application teams.
The following is an illustrative team-owned contract, not a Kubernetes API object or a deployable policy. It shows the decisions a delivery workflow should consume.
platform_service_profile:
name: production-web
accountable_owner: platform-product-owner
operational_owner: platform-engineering
deployment_objective:
success_ratio: 0.99
measurement_window_days: 30
application_ready_within_seconds: 300
upgrade_controls:
compatibility_evidence_required: true
representative_canary_required: true
missing_evidence_action: hold
recovery_method_required: true
capacity_commitment:
scenario: one_zone_lost_plus_one_worker_in_maintenance
placement_validation_required: true
load_validation_required: true
recovery_acceptance:
application_data_validation_required: true
access_control_validation_required: true
application_owner_approval_required: true
Replace the profile, owners, thresholds, and failure scenario with approved requirements. Connect each required check to actual test results and define the recovery objectives in the workload’s recovery record. A Boolean field is not evidence that a test passed.
Successful integration means a release can be held when required evidence is absent or unacceptable, with a clear owner for resolving the issue. Avoid building a second policy system that drifts from the pipeline, service catalog, and incident procedures it is supposed to describe.
Start With One Profile and One Complete Exercise
Choose a production workload profile with engaged application owners. Baseline its deployment and runtime journeys, rehearse a worker change, test the promised capacity scenario, and complete a timed recovery exercise. Use the findings to revise the service contract before onboarding more workloads under the same promise.
Review the results as a product team. Reliability is part of the discussion, but so are onboarding delays, recurring exceptions, manual support work, lifecycle exposure, and the cost of the service tier.
The review should produce decisions: add headroom, repair a deployment path, narrow a promise, retire an unsupported integration, or repeat recovery validation after a material change. More dashboards are not the default answer to a gap in ownership or a missing test.
Conclusion
Operating Kubernetes as a platform product means taking responsibility for a dependable way to deploy and run applications, with clear boundaries around what the platform supports. The cluster remains important, but its health is only one part of the service evidence.
SLOs define what consumers should experience. Upgrade gates control when changes can progress. Capacity makes the disruption commitment feasible. Recovery demonstrates how the service returns when prevention is insufficient. These decisions must agree with one another and with what the organization is prepared to fund and operate.
Start with one supported workload profile and prove its complete lifecycle: deploy it, change the platform underneath it, test the promised failure scenario, and recover it with application-owner acceptance. Then use the evidence to improve the offering.
The platform is ready for production when its service promises are explicit and its operating evidence supports them, not simply when its nodes report Ready.
External References
- CNCF TAG App Delivery: CNCF Platforms White Paper
Canonical URL: https://tag-app-delivery.cncf.io/whitepapers/platforms/ - Google SRE: Implementing SLOs
Canonical URL: https://sre.google/workbook/implementing-slos/ - Google SRE: Example Error Budget Policy
Canonical URL: https://sre.google/workbook/error-budget-policy/ - Kubernetes: Version Skew Policy
Canonical URL: https://kubernetes.io/releases/version-skew-policy/ - Kubernetes: Deprecated API Migration Guide
Canonical URL: https://kubernetes.io/docs/reference/using-api/deprecation-guide/ - Kubernetes: Disruptions
Canonical URL: https://kubernetes.io/docs/concepts/workloads/pods/disruptions/ - Google SRE: Canarying Releases
Canonical URL: https://sre.google/workbook/canarying-releases/ - Kubernetes: Resource Management for Pods and Containers
Canonical URL: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ - Kubernetes: Node Autoscaling
Canonical URL: https://kubernetes.io/docs/concepts/cluster-administration/node-autoscaling/ - Kubernetes: Operating etcd clusters for Kubernetes
Canonical URL: https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/ - Velero: How Velero Works
Canonical URL: https://velero.io/docs/v1.18/how-velero-works/ - Kubernetes: Encrypting Confidential Data at Rest
Canonical URL: https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/
TL;DR A migration-wave control room should authorize the next safe action, not simply report completed tasks. Build it around evidence-based gates, explicit...