GitOps configuration drift: detection, review and reconciliation

Operational context and boundary scope

This note applies to Kubernetes estates reconciled by Argo CD, including Red Hat OpenShift GitOps, where some applications sync automatically and others on demand. It covers drift in both directions: live resources that differ from Git because somebody changed them by hand, and Git that differs from live because a render step, a controller or an ignored field makes the comparison lie. Drift is not a dashboard colour. It is the moment the running cluster stops being the output of the pipeline that tested it.

The seam has three owners. The platform team, T2 in our layer model, owns Argo CD, its projects, the admission policies and the drift report. Application teams in T3 own the manifests in their repositories and the decision to retain or reverse a difference in their namespaces. Beneath both, T1 tooling such as storage driver configuration is the usual source of objects nobody declared. Between them sits the delivery pipeline: GitLab CI runs the deterministic gates, Trivy for vulnerabilities, Syft for the SBOM, Cosign for signatures, OPA or Kyverno for policy, and a human approves before Argo CD syncs. An edit that bypasses that path is drift even when it is correct.

The review described here runs under the read-only scope we onboard with (STC-ONB-01): Argo CD role:readonly for the application list and diffs, cluster-reader for live objects and PolicyReports, GitLab Reporter for merge request history. Nobody reconciles from the review seat. Reconciliation is a commit, a pipeline run and an approved sync, done by the people who own the namespace.

• Argo CD: automated sync policy
• GitLab: merge request pipelines and protected resources

Pre-window readiness and dependency matrix

Run the matrix before enabling any automated sync option on an application and again at every reconciliation review. A row without an evidence reference is a failed row; a failed row on an application means that application keeps manual sync until the row is green.

• Argo CD: diff customization
• SLSA: build provenance

Reference implementation artifact

The policy below is a Kyverno ClusterPolicy in Audit mode with two rules. The first reports objects in GitOps-managed namespaces that carry no Argo CD tracking marker, which is what a kubectl edit at two in the morning leaves behind. The second reports Pods whose images are referenced by a mutable tag rather than the digest the pipeline signed, because a retagged image is drift that no diff will show while the Application stays Synced. Neither rule blocks; the report makes the change visible by the morning review.

The exclusion keeps controller-created objects out of the report; extend it only with an owner and a reason, because every exclusion is a place drift can hide. Argo CD uses a label or an annotation as its tracking marker depending on the configured method, so the policy accepts either.

# gitops-drift-guard.yaml -- Kyverno ClusterPolicy in Audit mode. It reports and never blocks:
# an emergency kubectl edit must stay possible at 02:00; the report makes it visible by the morning review.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: gitops-drift-guard
  annotations:
    policies.example.invalid/owner: "REPLACE_WITH_PLATFORM_TEAM"
    policies.example.invalid/review-by: "REPLACE_WITH_REVIEW_DATE"
spec:
  validationFailureAction: Audit
  background: true                 # scan what already exists, not only new admissions
  rules:
    # Rule 1: everything in a GitOps-managed namespace carries the Argo CD tracking marker.
    - name: argo-tracking-marker-present
      match:
        any:
          - resources:
              kinds: [Deployment, StatefulSet, DaemonSet, ConfigMap, Service, NetworkPolicy]
              namespaceSelector:
                matchLabels:
                  gitops.example.invalid/managed-by: argocd
      exclude:
        any:
          - resources:
              selector:
                matchExpressions:
                  # Controllers create these; they are owned, not drifted. Extend only with an owner.
                  - {key: app.kubernetes.io/managed-by, operator: In, values: [Helm, olm, cert-manager]}
      validate:
        message: >-
          No Argo CD tracking marker. Record the object in the drift register with a
          retain-or-reverse decision, or extend the Application to cover this namespace.
        anyPattern:
          - metadata:
              annotations:
                argocd.argoproj.io/tracking-id: "?*"
          - metadata:
              labels:
                app.kubernetes.io/instance: "?*"
    # Rule 2: images are pinned by digest, so a retagged image cannot drift under a Synced application.
    - name: images-pinned-by-digest
      match:
        any:
          - resources:
              kinds: [Pod]
              namespaceSelector:
                matchLabels:
                  gitops.example.invalid/managed-by: argocd
      validate:
        message: >-
          Image referenced by a mutable tag. Reference the Cosign-verified digest the pipeline
          promoted (registry.example.invalid/REPLACE_WITH_IMAGE@sha256:...) and sync that revision.
        pattern:
          spec:
            containers:
              - image: "*@sha256:*"

• Kyverno: validate rules and policy reports
• Argo CD: diff customization

Execution sequence, blast radius and recovery

Reconcile in a sequence, not in a sweep. First produce the register from the read-only diff and the PolicyReport, and give every entry an owner and a decision. Retain means a commit: the emergency edit goes to a branch, through the gates, through approval and through a sync. Reverse means a sync of the current revision, one application at a time, watched by its owner. Self-heal stays off on an application until its register entries are empty, because self-heal with an open register is an automated rollback of a fix nobody reviewed.

The blast radius of a reconciliation is one Application. An Application that spans several namespaces, or that holds cluster-scoped objects beside namespaced ones, is split before it is reconciled, not after. Prune is never enabled during a reconciliation window; it is enabled per namespace in a planned window of its own, by the owner who has read the unmanaged-object report and knows what will disappear.

Declared state is the control-plane boundary. A Git revert followed by a sync restores the API objects to the previous revision, and that is a real, fast rollback for everything Git describes: Deployments, ConfigMaps, network policy, custom resources, and the restricted-v2 security context fields that an emergency edit tends to loosen. Every object outside Git and every ignored field is outside this rollback.

Persistent data is not restored by a sync. A reverted Deployment still points at the volumes it had, with whatever the new version wrote into them; a schema migration that ran as a Job does not run backwards; a secret rotated out of band keeps its new value. Application owners need their own backup and restore, and the register names which entries carry data changes. Controller-owned state is re-derived after a sync, sometimes slowly, and the plan says who waits for the Operator and how long.

Write the recovery authority down: who can revert a revision, which signals justify disabling self-heal during an incident, who records the emergency change and by when it must be committed or reversed. Reverting a correct fix to make the dashboard green is the drift this note is about.

• Argo CD: automated sync policy

Educational disclaimer

This is an engineering reference, not a report of a customer platform. Sync option names, tracking markers and the Kyverno rule syntax above follow the current stable Argo CD and Kyverno documentation and change between releases. Validate every field, command and gate against the installed releases of Argo CD or OpenShift GitOps, Kyverno and your pipeline tooling, and rehearse the revert in a test environment before a production reconciliation window.