AI Agents and Kubernetes Remediation: Write Access Is the Easy Part

Do not let an agent run kubectl apply unless a separate piece of code, one the agent cannot edit, checks the result and reverts on failure. Granting write access takes minutes. The verification layer is the actual product, and most teams shipping agentic remediation today have built the first and skipped the second.

A successful tool call proves almost nothing#

A recent Cloud Native Now piece by Vasuki Uday Kiran Vudathala, "Write Access Is the Easy Part," makes the point in one line: an agent should never infer remediation success from tool-call success alone. It splits verification into four rungs: the request was accepted, state changed without duplicated effects, the desired cluster state was actually reached, and the service outcome recovered. Most agent frameworks stop at rung one because that is what the tool returns.

deployment.apps/api configured means the API server accepted your write. It says nothing about whether the new pods pass readiness, whether the HPA fought your replica change, or whether p99 latency got worse.

Scope the identity before you scope the prompt#

Prompts are suggestions. RBAC is enforcement. Give the agent its own ServiceAccount, bound by a namespaced Role, with only the verbs the allowlisted actions need. No delete, no secrets, no ClusterRole.

# rbac/remediation-agent.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: remediation-agent
  namespace: payments
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: remediation-agent
  namespace: payments
rules:
  - apiGroups: ["apps"]
    resources: ["deployments", "replicasets"]
    verbs: ["get", "list", "watch", "patch"]
  - apiGroups: [""]
    resources: ["pods", "pods/log", "events"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: remediation-agent
  namespace: payments
subjects:
  - kind: ServiceAccount
    name: remediation-agent
    namespace: payments
roleRef:
  kind: Role
  name: remediation-agent
  apiGroup: rbac.authorization.k8s.io

Prove the boundary with kubectl auth can-i --as, and put these checks in CI:

$ SA=system:serviceaccount:payments:remediation-agent
$ kubectl auth can-i patch deployments -n payments --as=$SA
yes
$ kubectl auth can-i delete pods -n payments --as=$SA
no
$ kubectl auth can-i get secrets -n payments --as=$SA
no
$ kubectl auth can-i patch deployments -n billing --as=$SA
no

If any "no" flips to "yes", the pipeline fails.

Preview every change twice#

Before a write lands, run it through two gates. kubectl apply --dry-run=server sends the request through admission and validation on the API server without persisting it, so it catches what client-side checks miss: webhook rejections, immutable field changes, quota violations. Then kubectl diff shows what would change against live state. Its exit code is 0 for no differences, 1 when there are differences, and above 1 for errors, so a wrapper can require that the diff touches only the fields the action is allowed to touch.

The diff is also the thing a human reads when approval is required. We want destructive verbs (delete, scale to zero, anything touching PersistentVolumeClaims) to need a person, and we'd rather show that person a diff than a paragraph of model-generated reasoning. Our notes on AI agents for incident response reach the same rule: the agent prepares, a human confirms.

Define the verification per action, not per agent#

Each allowlisted action gets its own postcondition, written by a human, in code the agent cannot modify. A memory-limit bump verifies with kubectl rollout status plus a check that restarts stopped climbing. Every action verifies against an independent service signal, such as error rate from Prometheus, because the article's fourth rung is exactly that: state can be perfect while users still see failures.

Here is the shape of the wrapper we'd run. The agent supplies a manifest; the wrapper owns everything after that.

#!/usr/bin/env bash
# verify-then-rollback.sh <namespace> <deployment> <manifest>
set -euo pipefail
NS=$1; DEP=$2; FILE=$3
PROM=${PROM_URL:-http://prometheus.monitoring:9090}
QUERY="sum(rate(http_requests_total{namespace=\"$NS\",code=~\"5..\"}[2m])) / sum(rate(http_requests_total{namespace=\"$NS\"}[2m]))"
MAX_ERR=0.02

rollback() {
  echo "verification failed, rolling back $DEP" | logger -t remediation-agent
  kubectl rollout undo "deployment/$DEP" -n "$NS"
  kubectl rollout status "deployment/$DEP" -n "$NS" --timeout=180s
  exit 1
}

kubectl apply --dry-run=server -f "$FILE" -n "$NS"
kubectl diff -f "$FILE" -n "$NS" || [ $? -eq 1 ]   # 1 = changes, >1 = error

kubectl apply -f "$FILE" -n "$NS"
echo "applied $FILE to $NS/$DEP" | logger -t remediation-agent

kubectl rollout status "deployment/$DEP" -n "$NS" --timeout=120s || rollback

sleep 120   # let the error-rate window fill with post-change traffic
ERR=$(curl -sG "$PROM/api/v1/query" --data-urlencode "query=$QUERY" \
  | jq -r '.data.result[0].value[1] // "0"')
awk -v e="$ERR" -v m="$MAX_ERR" 'BEGIN { exit !(e < m) }' || rollback
echo "verified: error ratio $ERR" | logger -t remediation-agent

Two details matter. kubectl rollout status blocks until the rollout completes and exits non-zero on timeout, which is what makes it usable as a gate. And kubectl rollout undo only reverts Deployment pod-template changes, so the allowlist must contain only actions this rollback can actually reverse.

Log the sequence, and cap the loop#

The audit trail should record the request, the diff, the apply result, each verification, and the rollback if any, keyed by one operation ID. The article asks for durable operation identifiers so retries do not duplicate effects, and causal analysis to find which action caused a regression.

Then limit the blast radius mechanically. One namespace per agent identity, one action per operation, and a cooldown after a rollback, so a flapping agent cannot alternate between two bad fixes. If you drain nodes as part of remediation, check PodDisruptionBudgets before node drains, because a drain is the fastest way to turn a small fix into an outage. Most of this is the containment thinking from our AI agent security post applied to a cluster.

The decision, concretely#

• Should the agent hold a ClusterRole? No. Use a namespaced Role, prove it with kubectl auth can-i --as, and fail CI on any surprise "yes".
• Should the agent run kubectl apply directly? No. It hands a manifest to a wrapper that runs server-side dry-run, kubectl diff, apply, verify, and rollback.
• Is rollout status enough to call a fix successful? No. It proves state, not outcome. Add an independent error-rate or SLO check per action.
• Can the agent delete things on its own? No. Destructive verbs require human approval against a shown diff.

The call we'd make#

Start with one namespace, three allowlisted actions, and a wrapper that rolls back on any failed check. Expand the allowlist only when an action has a written postcondition and a proven undo. An agent that can write is a demo; an agent whose every write is verified, logged, and reversible is something we'd let near production.