Why your incident review process needs an upgrade for agentic AI.
On a Tuesday morning earlier this year, a claims-handling agent at a mid-sized insurer approved a settlement it should have escalated. It did everything its instructions told it to do. It looked up the policy in the right system, it cited the right precedents, it produced a confident summary, and it triggered the payment. The summary was wrong. The policy had been updated nine days earlier in a side system the agent did not know about, and the precedents it cited had been superseded by a regulatory change the team had communicated only in a SharePoint memo no one had vectorized. The agent did not malfunction. It worked exactly as designed against the data it could see. The data it could see was outdated.
The team did what every enterprise team does after a production incident. They opened an incident ticket, pulled the engineers into a room, and tried to run a post-mortem. Three days later, the post-mortem document was still empty. The traditional questions did not have answers. There was no stack trace. There was no log line that said "error." There was no code change to roll back. The agent had not crashed, it had reasoned, and the reasoning was the problem. The team's incident review process, which had served them well for fifteen years of conventional software, had nothing useful to say about a system whose behaviour was not deterministic, whose decision boundary was a probability distribution, and whose root cause lived in a SharePoint folder a thousand miles from the codebase.
This is the gap every enterprise running AI agents in production will encounter this year, and it is bigger than most leadership teams realise. The good news is that the gap can be closed. The bad news is that closing it requires retiring most of what your incident review template currently asks, and replacing it with a post-mortem playbook designed for systems that think, in the loose sense of the word, rather than systems that simply execute.
Why your existing post-mortem doesn't work on agents
The standard post-mortem template, refined across two decades of SRE practice, is built around a small set of assumptions. The system has a definable expected behaviour. The system either does or does not exhibit that behaviour. When it deviates, there is a state change somewhere in the stack that caused the deviation. The state change can be traced, the root cause identified, and the fix applied so the same incident does not recur.
Every single one of those assumptions degrades when the system in question is an AI agent. Expected behaviour is no longer a contract the system signed; it is a probability distribution shaped by a prompt, a model, retrieved context, and a tool ecosystem. Deviation is not a binary. Root cause is rarely a single state change, and far more often a confluence of small misalignments: a slightly stale retrieval index, a prompt that did not anticipate the edge case, a tool that returned data in an unexpected shape, a model checkpoint that quietly shifted between Tuesday and Thursday. And the fix, applied to one of those layers, does not guarantee the same incident will not recur, because the underlying system is non-deterministic and the next incident will look subtly different.
Enterprises that try to run their existing template against this kind of system end up with post-mortems that read like apologies. They identify a vague proximate cause, attach a corrective action that nobody believes will prevent recurrence, and quietly hope the agent behaves better next time. After enough of these, the team stops doing post-mortems at all. The incident becomes a Slack thread that fades, and the agent keeps running. This is the worst possible outcome and, in our experience, by far the most common one.
The six failure classes
A post-mortem playbook for agents starts by being honest about what can actually go wrong, because the failure surface is wider and stranger than most engineering teams expect. Across the engagements Apptad has run on production agentic systems in the last eighteen months, the failures fall into six recurring classes.
The first is the data class. The agent acted on data that was stale, incomplete, duplicated, or wrong, and the system has no way to know that. This is the most common class by a wide margin. The Tuesday morning incident above is a textbook data-class failure. The agent did not hallucinate; the data hallucinated for it.
The second is the prompt class. The agent received instructions that did not anticipate the situation it encountered, or that contained a subtle ambiguity the model resolved in a direction the designers did not intend. These failures often look like model failures and are routinely misdiagnosed as such. The model behaved exactly as it should given the prompt. The prompt was the bug.
The third is the model class. The model itself behaved poorly: a known weakness, a regression after a vendor update, a context-window overflow, a tokenisation edge case, a numerical stability issue. These failures are rare in aggregate but consequential when they happen, and they are also the failures most teams reach for first because they are the most legible. Most teams reach for them too often.
The fourth is the tool class. The agent called a tool, the tool returned a result, and the agent acted on it. The tool result was correct by the tool's own contract but wrong in context: a stale balance, a soft-deleted record returned as live, a timezone mismatch, a permission scope that returned redacted data the agent interpreted as complete. Tool-class failures are the most underrated category and the hardest to instrument because they cut across multiple systems your team does not own.
The fifth is the policy and governance class. The agent did exactly what it was allowed to do, and what it was allowed to do turned out to be wrong. This includes permission scopes set too broadly, kill switches that did not fire, escalation paths that did not exist, and approval thresholds that did not match the risk profile of the action. These failures are organisational, not technical, and a post-mortem that does not name them as such will misroute the corrective action every time.
The sixth is the user class. The user prompted the agent in a way that triggered a behaviour the system was not designed for. This includes prompt injection in the strict sense, but also the much more common case of a well-meaning user asking the agent to do something just outside its competence envelope, and the agent obliging. User-class failures shade quickly into governance failures and need to be treated as such rather than dismissed as "user error."
Most real incidents are not one of these classes. They are two or three of them, layered. The Tuesday morning incident was a data-class failure compounded by a tool-class failure (the side system was not surfaced through the canonical policy API) compounded by a governance-class failure (no one had set a kill switch on settlements that exceeded a certain dollar amount on policies updated in the previous fourteen days). A good post-mortem teases the layers apart. A bad one calls it a hallucination and moves on.
The sections your post-mortem document is missing
Once you accept the failure classes, the post-mortem document itself has to change. Most enterprise templates have sections for timeline, impact, root cause, contributing factors, and corrective actions. Those sections stay, but they are not sufficient for agentic systems. Four new sections need to sit alongside them.
The first is the trace section. Every agent action that produced the incident should be captured as a full trace: the prompt as it was rendered at runtime including retrieved context, the model version and parameters, the tool calls made in sequence with their inputs and outputs, the intermediate reasoning if available, and the final action. Without this, the post-mortem is speculation. Most teams in 2026 still do not capture this by default, which is the first thing to fix in any agent observability program.
The second is the data provenance section. For every piece of information the agent reasoned over, where did it come from, when was it last refreshed, what system of record produced it, and what was its lineage to the entity it described. This is the section where data-class failures surface, and it is the section enterprises with strong MDM and lineage practices write almost without effort, while enterprises without those practices struggle to write at all. The post-mortem is, in effect, a stress test of your data foundation.
The third is the decision-point section. Where in the trace did the agent make a decision that, if made differently, would have prevented the incident? Was it a retrieval decision, a tool-selection decision, a tool-input decision, a confidence-threshold decision, an escalation decision? This is the analogue of the "where could we have caught it" question in traditional post-mortems, and it is the section that drives the corrective actions that actually change agent behaviour. It also surfaces whether the right decision required information the agent did not have, which loops back to the data section in a productive way.
The fourth is the regression scenario section. Reduce the incident to a reproducible test case, articulated as a scripted input that the team can run against future versions of the agent. This is the artefact that prevents the incident from quietly recurring after the next prompt tweak or model update. Building a library of regression scenarios is, over time, the most valuable byproduct of running good agent post-mortems, and it is the thing teams without a post-mortem discipline never accumulate.
The reproducibility problem, and what to do about it
The single hardest part of an agent post-mortem is reproducibility. Most non-trivial agents will not produce the same output twice given the same input, because temperature is non-zero, retrieval indexes shift, models are updated by their vendors without notice, and tool dependencies have their own state. The team trying to reproduce the Tuesday morning incident may run the same prompt three times and get three different settlements, none of them the original.
The honest response to this is to lower the bar for what reproducibility means in this context. The goal is not pixel-perfect replay; the goal is behavioural confidence. Capture the inputs, the model version, the retrieval snapshot, and the tool outputs at the time of the incident, and replay against that frozen substrate. Run the test ten times rather than once and look at the distribution of outputs rather than the single point. If the agent reproduces the bad behaviour even 30% of the time against the frozen substrate, you have a real defect and a real regression scenario. If it reproduces it 0% of the time, you may be looking at a tail event whose root cause is upstream of the agent entirely. Either answer is useful. The mistake is treating reproducibility as binary.
The other half of the answer is to invest in evaluation infrastructure. A library of test cases that run on every prompt change, every model update, and every retrieval index refresh is the agentic equivalent of a regression test suite. It will not catch every incident, but it will catch most of the ones that matter, and it gives the post-mortem process a place to land its corrective actions where they will not be lost.
The cultural shift
The technical sections of the playbook are the easy part. The harder part is cultural. In a conventional incident, the engineer who pushed the code that broke production has a clear locus of accountability. In an agentic incident, the locus is diffuse. The prompt engineer wrote the prompt, the data team owned the retrieval source, the MLOps team chose the model, the platform team set the tool scopes, the policy team set the escalation thresholds, and the business owner approved the use case. When something goes wrong, every one of those roles can plausibly point at one of the others.
The teams that get this right import the blameless post-mortem tradition from SRE and adapt it for shared ownership. The post-mortem is a system review, not an individual review. The corrective actions are assigned to layers — the data layer, the prompt layer, the policy layer — and the layer owners are explicit. The CIO and CDO, or whichever pair shares accountability for the agent program, jointly sign off on the corrective actions. Without that joint sign-off, the post-mortem becomes a polite document that everyone reads and no one acts on. With it, the post-mortem becomes the meeting that actually changes how the agent runs in production.
There is a second cultural move worth naming. Most engineering teams treat post-mortems as a reaction to severity-one incidents. Agentic systems require a lower-severity threshold, because the cost of a bad agent decision is rarely catastrophic on its own and almost always cumulative across thousands of small decisions. The teams that run agent post-mortems only when the system is on fire learn nothing useful. The teams that sample agent decisions every week, find the worst ten, and run lightweight post-mortems on each of them learn at a rate their competitors cannot match.
The ninety-day setup
Standing up an agent post-mortem practice does not require a year. It requires ninety days of focused work, sequenced honestly.
The first thirty days are about instrumentation. Make sure every production agent emits a structured trace of every action, with prompt, model version, retrieval context, tool calls, and final action all captured against a stable trace identifier. Make sure data provenance is queryable for every retrieved chunk. Make sure tool calls log their inputs and outputs in a way the post-mortem author can read without spelunking through five vendor consoles. This is unglamorous work, and it is the single largest predictor of whether the rest of the program will succeed.
The second thirty days are about template and ritual. Adopt the four new sections — trace, data provenance, decision-point, regression scenario — alongside the conventional ones. Run the first post-mortem on a real incident, however small, and circulate the document widely. The first post-mortem will be uncomfortable because the new sections will not be writable cleanly, and the gaps in the instrumentation will be visible. That discomfort is the point; it tells you exactly what to fix in the instrumentation layer.
The third thirty days are about evaluation infrastructure and governance. Stand up a regression scenario library populated from the first month of post-mortems. Wire the library into the agent's CI/CD so prompt changes and model updates are gated by it. Set the cadence: a weekly sampling review for low-severity decisions, a within-48-hours post-mortem for material incidents, a monthly executive review of the corrective-action backlog. By the end of the quarter, the practice is no longer an aspiration. It is an operating cadence.
What this is really about
A post-mortem playbook for agents is not, ultimately, about agents. It is about the discipline a leadership team is willing to bring to a class of system that does not behave the way enterprise software has behaved for forty years. Agents are useful precisely because they reason; reasoning is useful precisely because it is not constrained to predefined paths; not being constrained to predefined paths means the agent will, eventually, make a decision its designers did not anticipate. That decision will sometimes be brilliant and sometimes be wrong. The institutions that build the discipline to learn from the wrong ones will pull steadily ahead of the institutions that do not. The institutions that do not will eventually pull their agents from production and conclude, incorrectly, that the technology was not ready.
Apptad partners with CIOs, CDOs, and AI program leaders to stand up agent observability, post-mortem practice, and the trusted data foundation underneath them — across Reltio, Informatica, STIBO, Salesforce, Databricks, Snowflake, and the open agent frameworks. If your production agents are running without a post-mortem discipline, the next incident is not a question of if. It is a question of how much you learn from it. That is the conversation worth having.
