On 22 May 2026, a team publishing on arXiv released Trajel, a dataset and evaluation framework built around a question most agent harnesses never ask. Not was the final answer right, but were the steps that produced it. Their abstract states the gap plainly: most hallucination benchmarks still evaluate only the final output, and as a result the most common failure modes go missing.

Arize's published field analysis of production agent traces supplies the concrete version. In its table of how agents interpret system signals, a database returning 200 OK with an empty result set becomes, in the agent's own words to the user, "The search worked perfectly. There is no data for this user." The root cause on that row is a guessed field name — the agent queried user_id when the schema required client_uuid. The query was valid. The response was valid. The output was fluent, well-formed and completely false.

Nothing in that output object is malformed. A schema validator passes it. A groundedness check passes it, because the summary is faithful to the observation it was given. The observation is what was wrong.

The answer is a self-report, not a receipt

The reason output verification keeps missing this class is architectural rather than a matter of insufficient checks.

A tool-calling agent's final message is generated by the same model, from the same context window, that produced the trajectory. It is a summary of a run written by the thing that did the run. When you validate it — schema conformance, cross-field assertions, tone, groundedness against retrieved context — you are validating the summary. You are checking whether the narrator is internally consistent.

Nothing in the output payload carries independent evidence that the tool call it describes returned what it claims. The observation step and the report of it collapse into one artefact, and the artefact is the one you are grading. Deterministic output checks remain worth running; they are cheap and they catch truncation, refusals and malformed payloads. What they cannot do is tell you the difference between an agent that found no rows and an agent that asked the wrong question.

This is the same structural property that makes agent postmortems hard, seen from the other end. There, the problem is that the run cannot be faithfully re-executed. Here, the problem is that the only artefact you kept is the one the run wrote about itself.

The error path is where false confidence gets manufactured

Arize's field analysis is worth reading as a catalogue of this failure rather than a list of bugs, because the pattern repeats at every status code. A 500 Internal Server Error is interpreted as "I successfully processed your request" — a backend crash masked by a polite completion. A 403 Forbidden becomes "I don't have access. I will try a different tool to get this data," a permissions boundary treated as a routing hint. A 404 Not Found becomes "The user must be new. I will attempt to create a record."

Every one of those produces a confident, well-formed final output. In several of them the agent is also attempting an action nobody authorised.

A practitioner writing on Hacker News named the same behaviour from the deployment side, listing "no hard veto layer" among the structural reasons autonomous agents stall before production: many agent systems "try another tool" or "fill in missing intent" instead of failing closed, which reads as resilience in a demo and as risk amplification in a real system. Two independent vantage points — one vendor's trace corpus, one engineer's deployment experience — describe the same mechanism. The agent's error-handling instinct is to produce an answer, and an answer is what gets verified.

The commercial consequence is that the failure never surfaces as a failure. It surfaces as a customer telling you their data is missing, weeks later, if at all.

Five ways a trajectory goes wrong before the output looks fine

Trajel gives the failure a taxonomy. Its authors annotate agent traces from AssetOpsBench with five hallucination types — factual, referential, logical, procedural and scope-based — evaluated at the level of individual Thought-Action-Observation steps rather than the final answer.

Three of their reported results matter for anyone designing verification.

The common failures are the ones existing benchmarks miss. That is the paper's stated headline: the most frequent modes originate in intermediate steps, which final-output evaluation does not observe.

Failures arrive in combination. Nearly half of the hallucinated trajectories in their data involve more than one type at once. A single output-level score cannot represent that. It collapses a compound failure into one number, and the number will usually be a passing one, because the compounding happens upstream of anything the score can see.

Accurate detectors still miss the subtle types. Automated detectors with high binary accuracy — good at answering "was there a hallucination" — still misclassify which kind. Binary quality gates on outputs inherit that limitation and add to it, since they are working from strictly less evidence.

Their conclusion is the one that bears on architecture: trajectory-aware detection significantly outperforms standard post-hoc verification. The evidence that distinguishes a good run from a lucky one is in the steps, and it is not recoverable from the reply.

The signal lives at the step

There is a longer-standing version of this result. In Let's Verify Step by Step, published in May 2023, Lightman and colleagues compared outcome supervision, which gives feedback on a final result, against process supervision, which gives feedback on each intermediate reasoning step. Process supervision significantly outperformed outcome supervision on the MATH dataset; their process-supervised model solved 78% of problems from a representative subset of the test set.

That work is about training reward models, not about verifying production runs, and the analogy should not be pushed past its evidence. What transfers is the shape of the finding: step-level signal carries information that the final label does not, and the gap is large enough to change outcomes.

Arize draws the operational form of the same line. Their guardrails-and-evals piece separates the two layers cleanly — an eval judges behaviour, a guardrail constrains it, and the guardrail is enforced at the code level. Their words: "A high-quality final answer does not prove that the agent followed an acceptable path." Among the questions they put to teams before increasing agent autonomy is whether the team can reconstruct the trajectory at all, because a final answer may hide retries, unnecessary tool calls, conflicting branches or policy violations.

That is the architectural conclusion. Verification of an output is terminal and advisory — by the time you have an output to check, the tool calls have happened, the record has been written, the money has moved. A check can change what happens next only if it runs before the next step does. Output validation is a quality signal. It is not a control.

How Waxell handles this

Waxell Observe is designed around exactly that granularity. Its product page describes what it records as "the full anatomy of an agent run — not just the output, but every decision that led to it": LLM calls with tokens, latency and cost; routing decisions with the options considered and the choice made; retrieval queries and relevance scores; tool calls with inputs, outputs and timing; and full execution trees with parent-child span relationships in OpenTelemetry. For the failure modes above, the load-bearing items are the tool call arguments and the raw observation — the two things a final-answer artefact does not preserve.

Capture is the precondition, not the control. The same page describes policies evaluating agent behaviour before execution, between steps and after completion, with structured feedback to the agent when one triggers: retry with adjusted parameters, escalate to a human, or halt. Two of the 50+ published policy categories map directly onto this post's failure modes. Quality covers output validation and quality gates — scoring outputs, flagging low-confidence responses, blocking inadequate results. Operations covers timeouts, retries and circuit breakers, described on the page as defining how agents fail "gracefully, with structure, not silently." That second one is the answer to the 500-becomes-success row: a defined failure path, rather than an agent improvising a polite one.

Observe auto-instruments Python agent frameworks in two lines of code, and what it sees is what runs inside the instrumented process. An agent making calls outside that process is outside its view, which is worth stating plainly when the whole argument is about trusting a record.

For workflows where the step itself is the risk — a payment, a clinical note, a production change — Waxell Runtime moves the check earlier still. Policies gate each step before it executes rather than evaluating it afterwards, using the same 50+ policy categories, with kill switches at the agent, workflow and session level and isolated execution per run. Runtime is the environment you build inside using the Waxell SDK decorators; agents already running on another Python framework stay with Observe.

One operational constraint belongs alongside the capability, since a record you cannot reach is not evidence. Waxell's published plan limits set trace retention at 14 days on Free, 30 on Team, 90 on Business and 365+ on Enterprise. The failures in this post surface through customer reports and reconciliation rather than through alerts, which is the slowest path there is. Set the retention window against that lag, not against your paging threshold.

FAQ

What is AI agent output verification?

AI agent output verification is the practice of checking an agent's final response before it reaches a user or a downstream system — schema conformance, cross-field assertions, groundedness against retrieved context, and quality scoring. It is a useful floor. Its structural limit is that the output is generated by the same process that produced the trajectory, so verifying it checks the agent's account of the run rather than the run.

Why does a schema-valid agent output still fail?

Because schema validity is a property of the payload's shape, not its provenance. An enum can collapse to a safe default, a numeric field can carry a stale value, and a summary can describe a tool call that returned nothing useful — all while conforming. Arize's field analysis documents the sharpest version: a database returning 200 OK with zero rows for a wrongly guessed field name, reported to the user as a successful, empty search.

What is a trajectory-level hallucination?

A hallucination that originates in an intermediate Thought-Action-Observation step rather than in the final answer. The Trajel framework, published in May 2026, classifies five types — factual, referential, logical, procedural and scope-based — over expert-annotated agent traces, and reports that nearly half of hallucinated trajectories involve more than one type at once. Its authors find that trajectory-aware detection significantly outperforms standard post-hoc verification.

Is a correct final answer evidence that the agent worked correctly?

Not on its own. Arize states it directly: a high-quality final answer does not prove that the agent followed an acceptable path. A run can reach the right result through retries, unnecessary tool calls, conflicting branches or a policy violation, and none of that is visible in the reply. The related research finding — that step-level supervision outperforms outcome-level supervision — points the same way.

What should an agent run record contain to make verification possible?

At minimum: tool names in order, the arguments passed at each step, raw tool outputs including error codes, the model calls with their parameters, decision points with the options considered, and parent-child relationships across sub-agents. Error responses matter most, because they are where a failed step gets converted into confident output text. A record built only from final answers cannot support any of these checks.

Where should the check actually run?

Wherever it can still change the outcome. A check on the finished output is advisory — the tool calls have already executed. Enforcement has to sit between steps, which is why evaluation and enforcement are separate layers: one judges behaviour after the fact, the other constrains it at execution time.

Sources

1. arXiv (Harshada Badave et al.), "Beyond Final Answers: Auditing Trajectory-Level Hallucinations in Multi-Agent Industrial Workflows", 22 May 2026, revised 26 May 2026.

2. Arize AI, "Why AI Agents Break: A Field Analysis of Production Failures", accessed 31 August 2026.

3. Arize AI, "AI agent guardrails vs. evals: How to build more reliable agent systems", accessed 31 August 2026.

4. arXiv (Hunter Lightman et al.), "Let's Verify Step by Step", 31 May 2023.

5. Hacker News, "Why autonomous AI agents fail in production", accessed 31 August 2026.

6. Waxell, "Waxell Observe — AI Agent Observability & Governance", accessed 31 August 2026.

7. Waxell, "Governed AI Agent Runtime & Execution", accessed 31 August 2026.

8. Waxell, "Pricing", accessed 31 August 2026.

The agent that tells you it finished is the same agent that decided it was finished. Keep the steps, not just the sentence.

Start free with Waxell Observe and one governed MCP upstream — pip install waxell, two lines to initialise, 10,000 traced executions a month on the Free tier. Create your workspace →

For workflows where a step must be gated before it runs, see Waxell Runtime, included on Business.