OpenShift and Kubernetes: lifecycle and recovery responsibilities

Scope and applicability

This note applies to teams choosing between Red Hat OpenShift Container Platform and an upstream Kubernetes distribution, and to teams that already run one and need to write down who owns what. It covers OpenShift 4.x on the installed release and update channel, and upstream Kubernetes under the published version skew policy with whatever distribution packages it. It does not compare prices or feature lists. It compares lifecycle and recovery responsibilities, because that is where the two platforms differ for an operator.

In our layer model the cluster is T2. Below it, T1 provides the hosts, the hypervisor or cloud, the storage arrays and the network the cluster assumes exist. Above it, T3 workloads bring their own data, their own secrets and their own idea of what a restart costs. The platform team owns the cluster lifecycle, the infrastructure team owns what the cluster runs on, and the application team owns the application’s data and its restore. A distribution choice changes how much of the middle layer the vendor packages, not who owns the layers above and below.

This is an engineering reference, not a report of a customer platform and not a vendor support statement. Release numbers, Operator channels and the backup procedure cited here are illustrative; confirm every one against the documentation for the installed release and your subscription entitlement before a change window.

• Kubernetes version skew policy
• Red Hat OpenShift Container Platform life cycle
• Red Hat OpenShift Operator life cycles

Where the handoff seam breaks

Operators break first. On OpenShift the platform lifecycle is integrated, but every Operator installed through Operator Lifecycle Manager has its own channel, its own supported versions and its own maintainers. A cluster update that is supported for the platform can still leave an Operator without a compatible release, and the application that depends on that Operator finds out when its custom resources stop reconciling. On upstream distributions the same gap appears as add-ons the distribution did not package: the ingress controller, the storage driver and the certificate manager each follow their own release train.

Admission breaks the T3 seam. A cluster that tightens its pod security defaults during an update, or that adopts a restricted security context constraint for a namespace, rejects workloads that ran yesterday. The rejection is correct; the surprise is not. The check belongs before the update: evaluate every handed-over workload against the profile the target release will enforce, and give application owners the report while there is still time to change a manifest.

Storage breaks the T1 seam. The container storage interface driver, the hypervisor release and the storage array firmware form a compatibility triangle that no single team sees whole. A kubelet update that is supported by the distribution can be unsupported by the storage driver that mounts the volumes, and a volume that fails to attach is an application outage with a platform root cause.

Version skew breaks silently. The published policy allows the control plane and the kubelets to differ by a bounded number of minor versions, and allows clients to lag the API server. Teams read the bound as permission to drift and then update the control plane twice before any node moves. The policy is a limit, not a plan. The update order and the node pools it covers have to be written down.

• Red Hat OpenShift Operator life cycles
• Kubernetes version skew policy

Pre-flight and validation matrix

Run the matrix against the installed release in the agreed test environment, then against production during the pre-window checks. A row without an evidence reference is a failed row. The commands are read-only; the roles name who signs the evidence.

• OpenShift Container Platform 4.18: control plane backup and restore

Inspectable artifact: Kyverno policy in Audit mode

The policy below encodes two handoff rules for namespaces the platform team labels as T3: workloads run as non-root with no privilege escalation and declared limits, and every replicated or stateful workload has a PodDisruptionBudget. It starts in Audit mode with background scanning, so the policy report lists the existing workloads that would fail before anything is rejected. Switch to Enforce only when the report is clean and the application owners have signed it.

Kyverno is one admission controller; the same rules can be written for Gatekeeper or expressed through an OpenShift security context constraint. What matters is that the rules live in Git, produce a report before they enforce, and match the profile the target release will apply.

# t3-handoff-baseline.yaml -- Kyverno ClusterPolicy. Audit first; Enforce only after the
# PolicyReport is clean and the application owners have signed it.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: t3-handoff-baseline
  annotations:
    policies.example.invalid/owner: "REPLACE_WITH_PLATFORM_TEAM"
    policies.example.invalid/change: "REPLACE_WITH_CHANGE_ID"
spec:
  validationFailureAction: Audit   # switch to Enforce in a separate, reviewed change
  background: true                 # report existing workloads, not only new admissions
  rules:
    - name: restricted-pod-security
      match:
        any:
          - resources:
              kinds: [Deployment, StatefulSet, DaemonSet]
              namespaceSelector:
                matchLabels:
                  platform.example.invalid/tier: t3
      validate:
        message: >-
          Workloads handed to the platform run as non-root, drop all capabilities,
          forbid privilege escalation and declare resource limits. This is what the
          restricted admission profile will reject after the update.
        pattern:
          spec:
            template:
              spec:
                securityContext:
                  runAsNonRoot: true
                  seccompProfile:
                    type: RuntimeDefault
                containers:
                  - securityContext:
                      allowPrivilegeEscalation: false
                      capabilities:
                        drop: ["ALL"]
                    resources:
                      limits:
                        memory: "?*"
                        cpu: "?*"
    - name: disruption-budget-declared
      match:
        any:
          - resources:
              kinds: [Deployment, StatefulSet]
              namespaceSelector:
                matchLabels:
                  platform.example.invalid/tier: t3
      context:
        - name: budgets
          apiCall:
            urlPath: "/apis/policy/v1/namespaces/{{request.namespace}}/poddisruptionbudgets"
            jmesPath: "items[].metadata.name"
      validate:
        message: "A node drain during an update needs a PodDisruptionBudget for every replicated or stateful workload."
        deny:
          conditions:
            all:
              - key: "{{ budgets | length(@) }}"
                operator: Equals
                value: 0

• Kyverno: validate rules and policy reports

Rollback and recovery boundaries

Rolling an OpenShift cluster back to a previous minor version is not supported, and an etcd restore is a destructive last resort rather than a rollback mechanism. The cited 4.18 procedure requires a backup taken from the same z-stream release. Upstream distributions vary, but the honest position is the same: a control-plane backup protects cluster state, meaning API objects, and nothing else. Treat the control plane as something you restore to the same release, never as something you roll back.

Persistent application data lives on a different boundary. Persistent volumes, databases inside and outside the cluster, object storage and the secrets that unlock them are not inside an etcd snapshot. Restoring the control plane returns the PersistentVolumeClaim objects; it does not return the bytes they pointed at. Application owners need their own backup and a timed restore, and the platform team needs to know which storage classes support snapshots and which do not.

External dependencies are the third boundary: the identity provider, DNS, the certificate authority, the image registry and the Git repositories the cluster reconciles from. A perfect cluster restore that cannot authenticate users or pull images is still an outage. List them in the recovery plan with the team that owns each.

Write the recovery authority down: who can declare the update failed, which stop conditions trigger that call, when the vendor support case is opened, and how long each restore takes, measured rather than estimated. For OpenShift, that measurement is the only rollback you have.

• OpenShift Container Platform 4.18: control plane backup and restore
• Red Hat OpenShift Container Platform life cycle

How we approach it

Bring the installed release, the Operator list with channels, the storage driver versions and the list of handed-over workloads to the discussion. The Assessment Sprint produces the responsibility matrix, the validated pre-flight evidence and the recovery boundaries in writing. Update work and recovery rehearsals are scoped separately for the installed release.