AI Agents and Kubernetes Remediation: Write Access Is the Easy Part

Do not let an agent run kubectl apply unless a separate piece of code, one the agent cannot edit, checks the result and reverts on failure. Granting write access takes minutes. The verification layer is the actual product, and most teams shipping agentic remediation today have built the first and skipped the second.

A successful tool call proves almost nothing

A September 2026 Cloud Native Now piece by Vasuki Uday Kiran Vudathala, "Write Access Is the Easy Part," makes the point in one line: an agent should never infer remediation success from tool-call success alone. It splits verification into four rungs: the request was accepted, state changed without duplicated effects, the desired cluster state was actually reached, and the service outcome recovered. Most agent frameworks stop at rung one because that is what the tool returns.

deployment.apps/api configured means the API server accepted your write. It says nothing about whether the new pods pass readiness, whether the HPA fought your replica change, or whether p99 latency got worse.

Scope the identity before you scope the prompt

Prompts are suggestions. RBAC is enforcement. Give the agent its own ServiceAccount, bound by a namespaced Role, with only the verbs the allowlisted actions need. No delete, no secrets, no ClusterRole.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: remediation-agent
  namespace: payments
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: remediation-agent
  namespace: payments
rules:
  - apiGroups: ["apps"]
    resources: ["deployments", "replicasets"]
    verbs: ["get", "list", "watch", "patch"]
  - apiGroups: [""]
    resources: ["pods", "pods/log", "events"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: remediation-agent
  namespace: payments
subjects:
  - kind: ServiceAccount
    name: remediation-agent
    namespace: payments
roleRef:
  kind: Role
  name: remediation-agent
  apiGroup: rbac.authorization.k8s.io

Prove the boundary with kubectl auth can-i --as, and put these checks in CI:

SA=system:serviceaccount:payments:remediation-agent
kubectl auth can-i patch deployments -n payments --as=$SA
yes
kubectl auth can-i delete pods -n payments --as=$SA
no
kubectl auth can-i get secrets -n payments --as=$SA
no
kubectl auth can-i patch deployments -n billing --as=$SA
no

If any "no" flips to "yes", the pipeline fails.

Preview every change twice

Before a write lands, run it through two gates. kubectl apply --dry-run=server sends the request through admission and validation on the API server without persisting it, so it catches what client-side checks miss: webhook rejections, immutable field changes, quota violations. Then kubectl diff shows what would change against live state. Its exit code is 0 for no differences, 1 when there are differences, and above 1 for errors, so a wrapper can require that the diff touches only the fields the action is allowed to touch.

The diff is also the thing a human reads when approval is required. We want destructive verbs (delete, scale to zero, anything touching PersistentVolumeClaims) to need a person, and we'd rather show that person a diff than a paragraph of model-generated reasoning. Our notes on AI agents for incident response reach the same rule: the agent prepares, a human confirms.

Define the verification per action, not per agent

Each allowlisted action gets its own postcondition, written by a human, in code the agent cannot modify. A memory-limit bump verifies with kubectl rollout status plus a check that restarts stopped climbing. Every action verifies against an independent service signal, such as error rate from Prometheus, because the article's fourth rung is exactly that: state can be perfect while users still see failures. This is the same idea as canary analysis that auto-aborts on bad metrics, applied to an agent's change instead of a release.

Here is the shape of the wrapper we'd run. The agent supplies a manifest; the wrapper owns everything after that.

#!/usr/bin/env bash
# verify-then-rollback.sh <namespace> <deployment> <manifest>
set -euo pipefail
NS=$1; DEP=$2; FILE=$3
PROM=${PROM_URL:-http://prometheus.monitoring:9090}
QUERY="sum(rate(http_requests_total{namespace=\"$NS\",code=~\"5..\"}[2m])) / sum(rate(http_requests_total{namespace=\"$NS\"}[2m]))"
MAX_ERR=0.02

rollback() {
  echo "verification failed, rolling back $DEP" | logger -t remediation-agent
  kubectl rollout undo "deployment/$DEP" -n "$NS"
  kubectl rollout status "deployment/$DEP" -n "$NS" --timeout=180s
  exit 1
}

kubectl apply --dry-run=server -f "$FILE" -n "$NS"
kubectl diff -f "$FILE" -n "$NS" || [ $? -eq 1 ]   # 1 = changes, >1 = error

kubectl apply -f "$FILE" -n "$NS"
echo "applied $FILE to $NS/$DEP" | logger -t remediation-agent

kubectl rollout status "deployment/$DEP" -n "$NS" --timeout=120s || rollback

sleep 120   # let the error-rate window fill with post-change traffic
ERR=$(curl -sG "$PROM/api/v1/query" --data-urlencode "query=$QUERY" \
  | jq -r '.data.result[0].value[1] // "0"')
awk -v e="$ERR" -v m="$MAX_ERR" 'BEGIN { exit !(e < m) }' || rollback
echo "verified: error ratio $ERR" | logger -t remediation-agent

Two details matter. kubectl rollout status blocks until the rollout completes and exits non-zero on timeout, which is what makes it usable as a gate. And kubectl rollout undo only reverts Deployment pod-template changes, so the allowlist must contain only actions this rollback can actually reverse.

Log the sequence, and cap the loop

The audit trail should record the request, the diff, the apply result, each verification, and the rollback if any, keyed by one operation ID. The article asks for durable operation identifiers so retries do not duplicate effects, and suggests reconstructing the sequence of observations, decisions, and actions as the next step for finding which intervention caused a regression.

Then limit the blast radius mechanically. One namespace per agent identity, one action per operation, and a cooldown after a rollback, so a flapping agent cannot alternate between two bad fixes. If you drain nodes as part of remediation, check PodDisruptionBudgets before node drains, because a drain is the fastest way to turn a small fix into an outage. Most of this is the containment thinking from our AI agent security post applied to a cluster.

The decision, concretely

• Should the agent hold a ClusterRole? No. Use a namespaced Role, prove it with kubectl auth can-i --as, and fail CI on any surprise "yes".
• Should the agent run kubectl apply directly? No. It hands a manifest to a wrapper that runs server-side dry-run, kubectl diff, apply, verify, and rollback.
• Is rollout status enough to call a fix successful? No. It proves state, not outcome. Add an independent error-rate or SLO check per action.
• Can the agent delete things on its own? No. Destructive verbs require human approval against a shown diff.

Questions people ask

Should an AI agent have cluster-admin or a ClusterRole in Kubernetes?

No. Give the agent its own ServiceAccount bound to a namespaced Role with only the verbs its allowlisted actions need, such as get, list, watch and patch on Deployments. Leave out delete, Secrets and any ClusterRole. Prove the boundary with kubectl auth can-i --as=system:serviceaccount:<ns>:<name> in CI, and fail the pipeline if any expected "no" turns into "yes".

How do you verify that an AI agent's Kubernetes fix actually worked?

Check four things, not one. The API server accepted the write, the change happened once, the cluster reached the desired state (kubectl rollout status returns success), and the service recovered according to an independent signal such as the 5xx ratio from Prometheus. A successful tool call only proves the first. Write each check as code the agent cannot modify, and roll back when any check fails.

What does kubectl diff exit code 1 mean?

Exit code 1 means kubectl diff found differences between the manifest and the live object, which is the expected result before a real change. Exit code 0 means no differences, and anything above 1 means kubectl or the diff program failed. A wrapper should treat 0 and 1 as success and stop on anything higher, as the || [ $? -eq 1 ] pattern in the script above does.

The call we'd make

Start with one namespace, three allowlisted actions, and a wrapper that rolls back on any failed check. Expand the allowlist only when an action has a written postcondition and a proven undo. An agent that can write is a demo; an agent whose every write is verified, logged, and reversible is something we'd let near production.

Sources

• Cloud Native Now: Write Access Is the Easy Part, The Verification Gap in Agentic Kubernetes Remediation
• Kubernetes Docs: Using RBAC Authorization
• kubectl reference: kubectl diff
• kubectl reference: kubectl rollout undo