Table of Contents
By Khimananda Oli | Last reviewed: November 2026
An AI agent that triages an incident and tells an engineer what's wrong is diagnosis. An AI agent that restarts the pod, rolls back the deploy, or fails over the database without a human clicking anything is auto-remediation — and it's a different problem with a different failure mode. A wrong diagnosis wastes someone's time. A wrong remediation action taken automatically on production can take down something that wasn't broken yet. The fix isn't more confidence in the model; it's tiering actions by blast radius and gating the risky ones behind a real approval mechanism, not a prompt that promises to be careful.
Why is auto-remediation a different problem than AI incident triage?
AI incident response and triage covers the part of this problem that's safe to automate aggressively: correlating alerts, pulling logs and metrics, matching symptoms to known runbooks, and producing a ranked set of likely causes for a human to read. That entire pipeline can hallucinate and the worst outcome is an engineer wastes two minutes reading a wrong hypothesis before moving on. Nothing in production changes state because the triage step was wrong.
Auto-remediation is the step after that — actually taking the action the runbook says to take, without a human in the loop for at least the common cases. Restart the crashlooping pod. Roll back the last deploy. Scale out the pool that's saturating. Fail over to the standby database. The gap between these two halves of the same incident-response pipeline is exactly where most teams stop: it is easy to trust an LLM to suggest "restart service X," and much harder to trust it to run the command, because now a wrong or premature action has a blast radius of its own, independent of whatever caused the original alert.
This matters because the two halves get conflated in vendor pitches constantly. "AI-powered self-healing infrastructure" sounds like one capability. In practice it's two systems with completely different risk profiles bolted together, and the auto-remediation half is the one that needs the engineering discipline this article covers — because it's the half that can turn a minor incident into an outage if you get it wrong.
How should remediation actions be tiered by risk?
The pattern that shows up repeatedly across teams doing this safely is a three-tier policy, keyed to what the action can break and how reversible it is — not to how confident the model is in its own diagnosis. Model confidence is a signal about the diagnosis; it says nothing about what happens if the action turns out to be wrong anyway.
| Tier | Example actions | Gate | Why this tier |
|---|---|---|---|
| Low risk | Restart a single crashlooping pod, clear a known-safe cache, rotate a log file | Auto-execute immediately | Reversible in seconds, blast radius is one instance, action is idempotent |
| Medium risk | Scale a pool up or down, restart a stateless service fleet-wide, rotate a credential | Notify + short delay before executing, cancellable | Reversible but affects capacity or many instances at once; a human should be able to veto before it lands |
| High risk | Database failover, rollback a stateful deploy, delete or drop a resource, anything touching customer data | Manual approval required, full diagnosis context attached | Hard or impossible to reverse cleanly; a wrong action here is its own incident |
The tiering decision belongs to whoever owns the service, made in advance and during a calm period — not improvised by the agent at incident time, and not left for the LLM to infer from the situation. An agent inferring "this looks low-risk" from the shape of an incident is exactly the judgment call you're trying to remove from the automated path. The tier is a property of the action type, decided once, reviewed periodically as the system changes (an action that was safe to auto-execute on a stateless service stops being safe the moment that service starts holding session state).
The gate for the high-risk tier is the part teams most often skip or build badly — either there's no real approval step and "high risk" just means a Slack message nobody reads before the action fires anyway, or the approval step is a custom webhook with no audit trail and no clean way to set who's allowed to approve. AWS Systems Manager's Automation service has a purpose-built action for exactly this, and its mechanics are worth copying even outside AWS if you're building your own approval gate.
How does a real approval gate work? (AWS Systems Manager's aws:approve action)
AWS Systems Manager Automation documents an action called aws:approve whose entire job is pausing an automation runbook for manual approval before it continues. It's a small, well-specified building block, and reading its actual parameters is a useful exercise even if you never touch AWS Automation, because it shows what a minimally complete approval gate needs to specify.
| Parameter | Required | What it controls |
|---|---|---|
Approvers | Yes | List of AWS-authenticated principals (IAM users, user ARNs, role ARNs, or assume-role ARNs) allowed to approve or reject the action. Maximum of 10. |
MinRequiredApprovals | No | Minimum number of approvals needed before the automation resumes. Defaults to 1 if not specified; can't exceed the number of listed approvers. |
NotificationArn | No | An SNS topic ARN the approval request is published to, so approvers actually get notified instead of having to poll for it. |
Message | No | Free-text context included in the notification — this is where the diagnosis output belongs, not a separate out-of-band message. |
Two details worth copying into your own design even if you're not using AWS: the default timeout for the step is 7 days with a hard maximum of 30 days, and the action's output includes an explicit ApprovalStatus (Approved, Rejected, or Waiting) plus an ApproverDecisions record of who decided what. That's the difference between a real approval gate and a notification that happens to precede an action anyway — a real gate blocks execution on an explicit decision, records who made it, and has a defined timeout behavior instead of waiting forever or firing by default. A homegrown Slack-button-based gate is fine, but it needs the same three properties: a hard block on unapproved execution, an audit trail of who approved what, and an explicit timeout policy — does the action get cancelled, or does it proceed after N hours? Choose that on purpose, don't let it be an accident of the implementation.
Note that AWS Systems Manager has a second, related construct — EnhancedApprovals — but that one is specific to Change Manager templates and adds per-approver-type minimums on top of the simple list; it's a different, heavier mechanism for change-management workflows, not a drop-in replacement for aws:approve in an incident-response runbook.
What should a remediation action log before and after it runs?
An auto-remediation system that works but can't be audited afterward is a liability the first time someone asks "why did this restart happen at 3am." At minimum, every remediation action — regardless of tier — should log: the triggering alert or diagnosis that proposed it, the risk tier it was classified under and why, who or what approved it (a named approver for high-risk, "auto" with the policy version for low-risk), the exact command or API call executed, and the observed outcome afterward (did the target metric recover, did a new alert fire). That last piece closes the loop back into whatever system produced the original diagnosis — if you're running the triage pipeline described in AI incident response and triage, the remediation outcome is exactly the feedback signal that should flow back to improve future diagnoses, not get thrown away once the incident closes.
{
"action_id": "remediate-7f2a91",
"triggered_by": "diagnosis-4e881c",
"action_type": "restart_pod",
"target": "checkout-api-7d9f8c-x2jkp",
"risk_tier": "low",
"policy_version": "2026-10-14",
"approval": {
"required": false,
"approved_by": "auto",
"approver_decisions": []
},
"executed_at": "2026-11-06T03:14:02Z",
"outcome": {
"target_metric_recovered": true,
"new_alert_fired": false,
"observed_after_seconds": 90
}
}Treat this record the same way you'd treat a distributed trace span — it should be queryable alongside your normal observability stack, not buried in a separate remediation tool's private log. If you're already instrumenting LLM calls with OpenTelemetry's GenAI semantic conventions, the remediation action is a natural child span of the diagnosis call that proposed it, which makes "show me every auto-remediation action taken last week and whether it worked" a single trace query instead of a cross-system investigation.
What guardrails does the agent proposing the action need, separate from the approval gate?
The approval gate controls whether an action executes. It doesn't control what the agent is capable of proposing in the first place, and that's a separate layer worth building even for low-risk, auto-executing actions. The same guardrail discipline covered in guardrails for autonomous AI agents applies directly here: an agent that can propose "restart the pod" should not, as a side effect of how its tools are wired, also be able to run arbitrary shell commands or touch resources outside the one service it's diagnosing. If you're giving an agent operational access to Linux hosts to execute remediation directly, building an AI agent for Linux server administration covers scoping that access correctly — the permission boundary on the tool is what makes the risk-tier policy enforceable at all; a risk tier is meaningless if the underlying credential can do more than the tier allows.
Two guardrails matter specifically for remediation, beyond general agent scoping. First, a hard action allowlist per risk tier — the agent should not be able to propose an action type that isn't on the pre-approved list for its tier, even if it reasons its way to a plausible-sounding new one. Second, a rate limit on auto-executed actions against a single target: a low-risk "restart" that's correct once becomes a crashloop-inducing denial-of-service against your own service if the agent keeps re-triggering it every time the same symptom reappears within minutes. Cap auto-execution to N attempts per target per time window, and escalate to the next tier up once that cap is hit — the fact that the cheap fix didn't actually fix anything is itself diagnostic information the human approving the escalation should see.
How do you test an auto-remediation system before it runs on real incidents?
The same way you'd test any system that changes production state under uncertainty: in a staging environment where you can deliberately break things, then in shadow mode against real production signals before giving it write access. Shadow mode means the agent runs its full diagnosis-to-proposed-action pipeline against live alerts, logs what it would have done and at what risk tier, and takes no action — a human compares the shadow log against what actually happened or what an on-call engineer would have done manually. Run this for long enough to cover your actual incident variety, not just a week of quiet traffic; the interesting failures in auto-remediation are the ones in the long tail of incident types, not the common crashloop you've already runbooked a hundred times.
Once a specific action type graduates out of shadow mode, start it in the medium-risk tier even if you eventually want it fully automated — the notify-and-delay gate gives you a cheap rollback window while you build confidence, and it's far easier to loosen a gate later than to explain after the fact why a fully automated action fired without one. The evaluation discipline from evaluating AI agents applies directly to grading shadow-mode output — did the agent propose the correct tool call, in the correct order, and would it have completed the actual task — before any of it touches a live approval gate.
If you want a risk-tiered auto-remediation pipeline designed against your own services — the tiering decisions, the approval gate, and the observability to prove it's working before you trust it with less supervision — my DevOps and cloud consulting services cover exactly this.
