AI-assisted NOC workflows: permissions, evaluation and fallback

Scope and applicability

This note applies to AI-assisted workflows in a network operations centre: alarm triage, ticket enrichment, runbook retrieval and the proposals an assistant hands to an operator. It covers three modes that should be chosen per workflow step, not per project: deterministic rules for validation and routing, model-assisted interpretation where a field has to be extracted or a request classified, and bounded autonomy where a system may select and execute an action inside an explicitly approved scope. It is model-agnostic. The controls are the same whichever model or hosting arrangement you use.

The seam is unusual because it crosses all three layers at once. The assistant reads from the manager’s fault management northbound interface and from the ticketing system in T1, runs as a workload on the cluster in T2, and is itself a T3 application with a reviewer interface. The NOC tooling team owns the workflow, the platform team owns the identities it runs under, and the network management team owns what it is allowed to read and, eventually, acknowledge.

This is an engineering reference, not a report of a customer deployment and not a product recommendation. The schema and the checks below describe a reference intake pipeline; the thresholds, the permitted actions and the reviewer roles are set per environment and written into the evaluation report before any access is widened.

• Anthropic: building effective agents and workflows

Where the handoff seam breaks

Permissions creep first. A service account created for reading alarms acquires a write role because a demo needed to acknowledge one, and six months later the assistant can clear alarms nobody reviewed. The control belongs in the connected systems, not in the prompt: the identity the assistant runs under has read scopes on the manager and the ticketing system, and the single write path, if any, goes through a separate identity that the approval service holds.

Untrusted text is the second seam. Alarm additional-text fields, ticket comments and runbook pages are inputs the model reads, and any of them can carry instructions. An alarm whose free-text field says to close all related tickets is data, not a command, and the evaluation set has to include that case so the pipeline proves it treats it as data. The same applies to tool output: a connector that returns an error page returns text, and text is not a result.

Duplicate and stale events are the third seam. The northbound interface resends on reconnect, the ticketing system fires a webhook per update, and a topology snapshot taken an hour ago describes a network that has since reconverged. Without idempotency keys and observation timestamps, the assistant proposes the same action twice and reasons about a neighbour that no longer exists.

Approval theatre is the quiet failure. A reviewer who sees two hundred proposals a shift approves them all, and the audit log records a human decision that did not happen. Measure review time per proposal in the pilot. If it collapses toward zero, the gate has stopped working, and the honest fix is fewer, better proposals rather than a faster button.

• MCP 2025-11-25: security best practices

Pre-flight and validation matrix

Run the matrix before the pilot on the evaluation set, then again on live-shaped cases before any widening of scope. A row without an evidence reference is a failed row. The commands are read-only; the roles name who signs the evidence.

Inspectable artifact: the proposal schema

The schema below is the contract between the assistant and the reviewer interface. Anything the model produces is validated against it before a person sees it, so a malformed or out-of-scope proposal is a counted rejection rather than a surprise. Three constraints carry most of the weight: the manager reference is a constant, so a proposal cannot point at a system the workflow was not approved for; the action kinds are an enumeration with no network write in it; and requiresApproval is a constant true, so no proposal can declare itself pre-approved.

Version the schema with the prompt revision it was evaluated against. A prompt change without a schema change is still a change to the pipeline and goes through the evaluation set again.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://schemas.example.invalid/noc/triage-proposal/v1.json",
  "title": "NOC triage proposal",
  "description": "What the assistant may hand to a reviewer. Anything outside this shape is rejected and counted before a person sees it.",
  "type": "object",
  "additionalProperties": false,
  "required": [
    "proposalId",
    "alarmRef",
    "classification",
    "proposedAction",
    "evidence",
    "requiresApproval",
    "model"
  ],
  "properties": {
    "proposalId": {
      "type": "string",
      "pattern": "^[A-Za-z0-9_-]{8,64}$"
    },
    "alarmRef": {
      "type": "object",
      "additionalProperties": false,
      "required": [
        "manager",
        "alarmId",
        "observedAt"
      ],
      "properties": {
        "manager": {
          "const": "enm-north.example.invalid"
        },
        "alarmId": {
          "type": "string",
          "maxLength": 128
        },
        "observedAt": {
          "type": "string",
          "format": "date-time"
        }
      }
    },
    "classification": {
      "enum": [
        "known-pattern",
        "needs-investigation",
        "duplicate",
        "outside-scope"
      ]
    },
    "proposedAction": {
      "type": "object",
      "additionalProperties": false,
      "required": [
        "kind",
        "target"
      ],
      "properties": {
        "kind": {
          "enum": [
            "annotate-ticket",
            "link-runbook",
            "suggest-escalation",
            "no-action"
          ]
        },
        "target": {
          "type": "string",
          "maxLength": 256
        },
        "parameters": {
          "type": "object",
          "maxProperties": 8
        }
      }
    },
    "evidence": {
      "type": "array",
      "minItems": 1,
      "maxItems": 10,
      "items": {
        "type": "object",
        "additionalProperties": false,
        "required": [
          "source",
          "ref"
        ],
        "properties": {
          "source": {
            "enum": [
              "alarm",
              "ticket",
              "runbook",
              "topology"
            ]
          },
          "ref": {
            "type": "string",
            "maxLength": 256
          }
        }
      }
    },
    "requiresApproval": {
      "const": true
    },
    "confidence": {
      "type": "number",
      "minimum": 0,
      "maximum": 1
    },
    "model": {
      "type": "object",
      "additionalProperties": false,
      "required": [
        "name",
        "promptRevision"
      ],
      "properties": {
        "name": {
          "type": "string",
          "maxLength": 64
        },
        "promptRevision": {
          "type": "string",
          "pattern": "^[0-9a-f]{7,40}$"
        }
      }
    }
  }
}

• Anthropic: building effective agents and workflows

Rollback and recovery boundaries

Turning the assistant off is a control-plane action and should be rehearsed: a feature flag that routes every case to the manual queue, revocation of its identities, and removal of its route. Measure how long it takes and who can do it at three in the morning. Rolling back a prompt or model revision is a redeployment of the previous, evaluated revision, with its own schema version.

Persistent data the assistant produced is a different boundary. Ticket annotations, runbook links it attached and any alarm acknowledgements the approval service executed persist in the ticketing system and the manager after the assistant is gone. They are not undone by turning it off. The recovery plan lists the record types the workflow may write, tags every write with the proposal identifier, and names who reviews those records if a revision is withdrawn.

Alarm data itself sits on the manager’s own recovery boundary. The assistant reads it and must never become its system of record. If the pipeline stores copies for evaluation, set a retention period and treat the copies as the sensitive data they are.

Write the recovery authority down: who can disable the assistant, on which stop conditions, who reviews its records afterwards, and how the manual baseline is re-established if the pilot is withdrawn.

• MCP 2025-11-25: security best practices

How we approach it

Bring the workflow, its manual baseline, the systems it would read and the actions you expect it to propose to the discussion. The Assessment Sprint produces the permission inventory, the evaluation set with its results and the recovery boundaries in writing. The integration, the reviewer interface and any widening of scope are scoped separately.