Why Most Multi-Agent Systems Fail Even When Evaluation Passes

I keep running into some version of this failure wherever agents get chained together, in one shape or another. Take a support-ticket triage system, for example, that is three nodes deep. One classifies the incoming ticket, one pulls the customer's account history from an internal API, and another one drafts the resolution or escalation based on both.Then, it ships. It works well in the demo, which honestly, from my experience, doesn't tell you much. It works for the first few days in production too; again, that tells you slightly more but still not enough.Along the line, a complaint comes in about a canceled subscription refund. The account-history node calls the billing API, gets back a 200, and passes the payload downstream as if nothing happened.The payload is empty. No malformed data, no timeout, nothing that would even show up as a 500. Just an empty result set, formatted exactly as a valid response would be, because the account ID got messed up two steps upstream and the billing service quietly returned nothing for an account it couldn't match. The drafting node never sees an error. It sees a well-formed JSON object with no records in it, decides that means "no billing history," and writes a perfectly polite email explaining there's nothing to refund. It goes out, wrong refund decision and all. But still nobody catches it, because nothing about the process ever crashed. As far as the system's concerned, it did its job.I don't think this exact scenario needs to have happened to you specifically for it to be worth your time. If you've spent any real stretch of time around multi-agent systems in production, you've either already seen a version of this, or you're going to eventually.None of this is anecdotal, either. Datadog's 2026 State of AI Engineering report puts production failure rates for AI requests at around 5 percent, and only about 60 percent of that comes from the loud, capacity-driven failures you'd actually notice through an error code. The rest is closer to what happened above, a request that completes and still gets it wrong.One pipeline, three steps, one blind spot. Image by author.Why nothing catches itRun this pipeline through a standard evaluation suite, and it sails through. The final output reads well, is grammatically clean, and is professionally worded. A human skimming it for tone has no reason to flag anything, not unless they happen to go cross-reference the actual account, which kind of defeats the point of automating the check in the first place. If you score it against a rubric for resolution quality, it probably does well too. Clear, polite, and on topic.It passes because each of those checks looks at the same layer: the final text. None of them ask what happened between node two and node three. The account-history node didn't fail loudly; it failed by succeeding at returning the wrong thing, and succeeding is exactly what output-level eval is built to reward.This is the part I think is worth sitting with longer than feels natural, before jumping to a fix. Grading only the compiled output makes you structurally blind to intermediate states that look correct but aren't. That's not a gap you patch with a better prompt on the last node. It's a blind spot that comes baked into where you decided to look in the first place.Grading the UI instead of the application underneath itThere's an old comparison here that I keep coming back to. Nobody ships a compiled application and calls it tested because the login screen renders. You test the layer beneath it, the query that backs the login, the token it issues, and even the permission check it triggers along the way. The UI is the last place a bug shows itself, not the first place you'd think to look for one.Most production agent eval right now is UI-only testing bolted onto a system that doesn't even have a UI in the traditional sense. The final text response is the only thing getting graded, mostly because it's the only thing that's easy to grade. You can run it through a rubric, compare it line-by-line against a known-good answer, or have someone skim it over coffee.The tool calls, the JSON handoffs between nodes, the partial reasoning getting passed forward: none of that gets watched unless something crashes hard enough to leave a trace in a log somewhere.And the expensive failure mode was never the loud one. A 500, a server admitting outright that something broke, gets caught and escalated, because the system already expects that shape of failure and has some plan for it. The one that costs you is the 200, the response that says everything's fine, attached to a payload that's structurally fine and semantically garbage.An architecture for watching the middleSo the fix isn't another rubric bolted onto the end. It's moving some of the evaluation into the pipeline itself, right at the seams where one agent's output becomes another agent's input.I've been calling this an Intermediate State Eval architecture, mostly because it needed a name and that one stuck. The idea is a lightweight grader sitting between agent nodes, not waiting for the whole chain to finish. In the ticket-triage case we talked about earlier, that's a checkpoint between the account-history node and the drafting node, and its only job is asking whether the handoff looks plausible. Does the account ID in the payload actually match the one that was requested? Does this look like a real lookup result, or like the kind of default value a system quietly falls back to when it can't find what it's looking for?A watchdog grades each handoff before it's allowed to reach the next node, halting on anything implausible instead of letting it pass through. Image by author.You don't need a large model doing this judgment call, and honestly I'd argue against it. A small local model serving as a watchdog between nodes, the same basic idea behind using a model to judge another model's output, is enough to catch shape-level and plausibility-level problems, and keeping it local prevents the added latency and cost from becoming a second version of the exact problem you're trying to solve.Its verdict doesn't actually need to be clever. It just needs to be fast and binary: does this handoff look sane enough to move forward, or does it need to get flagged and stopped before the next node builds anything on top of it? That's the whole job, nothing more.Here's what that actually looks like in code. Start with the shape of the handoff itself. Defining it as a real schema with Pydantic instead of a loose dict is what makes the watchdog's job possible at all, since it gives the grader something concrete to check against instead of guessing at structure fresh on every call.The watchdog itself is just a small function, not some new service you need to stand up and maintain. It takes the outgoing payload, the request that produced it, and asks a local model one narrow question instead of an open-ended one. Keeping the question narrow is the whole trick, honestly, that's what keeps this cheap enough to actually sit on the critical path instead of becoming its own bottleneck.And yeah, the orchestration wiring is about as unglamorous as it gets; that's done on purpose, by the way. Somewhere in your pipeline you already have functions handling classification and drafting the final resolution. The only new piece here is the gate sitting between them, grading the handoff and raising instead of quietly letting a bad payload flow through to whatever writes the customer-facing response.None of this is fancy, and I'd be a little suspicious of anyone who dressed it up to sound like it was. It's a schema, one narrow grading call, and an exception where there used to be a silent pass-through. The value was never in the sophistication of any single piece. It's entirely in where you decided to put the check.The practical payoff is that failures stop happening in silence. Instead of a wrong email going out three steps downstream of the real problem, the pipeline stops right at the point of corruption, with the actual bad handoff attached to whatever alert fires. That turns a support escalation that costs you trust into a debugging session that costs you a few minutes.What this costs youNone of this is free, and I'd rather be upfront about that than sell you a strict improvement with no downside, because there isn't one, but three costs, specifically:Latency: Plain and simple. For a three-node pipeline, that's two extra inference calls sitting right on the critical path, and it adds up fast if you're building something where response time actually matters.A new failure surface: A miscalibrated watchdog starts rejecting perfectly good handoffs, trading silent corruption for a different annoyance, false halts that somebody now has to triage by hand. Getting that threshold right takes real iteration, not a set-it-once number.Judgment: Deciding where a handoff is actually worth grading, versus where you'd just be adding overhead for its own sake, depends entirely on how expensive being wrong is at that specific point.Not every node needs a watchdog sitting on it. The ones that sit right before something external-facing, an email going out or a refund actually firing, are the ones where the cost of catching a bad handoff clearly beats the cost of the extra hop. Everywhere else, you're probably just adding latency for the sake of feeling thorough.···Final thoughtsIf you're running a multi-agent pipeline today and want to try this without tearing the whole thing apart, don't start at the front of the chain. Trust me.Start at the last internal handoff before something external actually happens, right before an email sends, a record gets written, a decision gets acted on. That's the one boundary worth instrumenting first. Give it a week or two, see what the watchdog actually catches, and let that tell you whether it's worth pushing further back into the pipeline.You don't need the full architecture on day one to get something out of this. Next time something breaks in a pipeline you own, ask yourself whether your evals would have caught it before the output itself looked wrong. Most of the time, if you're honest, the answer's no. That gap is exactly where the riskiest handoff in your system has been sitting the whole time, waiting for someone to look.Your final output can lie to you. The trajectory can't.···Before you go!I write about the engineering decisions that decide whether an AI system holds up in production. You can subscribe to my newsletter if you'd like more of that.Connect With Me