# Agent Run Outcome Labels: Tickets, Ledgers, Sign-Offs

> How to prove an AI agent run finished the job: ticket state, reconciled ledger lines, accepted root causes and human sign-off, with a field template.

Canonical: https://origindatapartners.com/guides/agent-run-outcome-labels
By: Origin Data Partners
Status: Published
Published: 2026-11-15
Updated: 2026-11-15
Sources checked: 2026-11-15

## Quick answer

An outcome label records whether an AI agent run finished the job and what proves it: a ticket that stayed closed, a ledger line that reconciled, a root cause accepted in the postmortem or a person's sign-off. Record the outcome, its source, when it was checked and by whom, and any correction a person made. A closed ticket alone is a start, not proof. Keep failed runs, labelled as failures.

## Key points

- Every run needs an outcome and the record that proves it.
- Wait long enough for reopens, reversals and repeat incidents before you label a run a success.
- Record what a person changed, and at which step.
- Failed and partial runs are worth keeping when they are labelled.
- An LLM judge can make a first pass; check it against runs people have labelled.

## How do you know an AI agent run actually succeeded?

You check it against a record outside the agent: the ticket system, the ledger, the postmortem or a person who signed off. The agent saying "done" is not that record.

Anyone who trains or tests a model on your runs needs to know which runs to copy and which to learn from as mistakes. Without an outcome, a run is a story with no ending. User approval is not enough on its own: a trace buyer's published essay (1 October 2026) warns that "naïve user approval is not always evidence of correctness". Benchmark builders work the same way: on one incident benchmark "Each submission is compared against a ground-truth set of root causes provided by IBM" ([Artificial Analysis](https://artificialanalysis.ai/articles/itbench-aa-launch)).


## What outcome fields should an agent trace dataset include, by function?

Each function has a system that already knows whether the work was done. Use it, and name it in the outcome_source field.

| Function | Success means | Source of truth | Wait before labelling |
| --- | --- | --- | --- |
| Support | Ticket solved and not reopened; customer did not come back on the same issue | Help desk ticket history | At least the reopen window your desk uses |
| Finance | Line reconciled, payment posted, no reversal or credit note | ERP or ledger | Until the period closes |
| Incidents | Root cause accepted in the postmortem, fix merged, alert cleared and not back | Postmortem, repository, alerting | Until the postmortem is signed off |
| Scheduling | Booking confirmed and kept | Calendar or booking system | Until the booked date has passed |
| Back office | Filing accepted, not returned or corrected | Filing system or the receiving body | Until acceptance is confirmed |

Help desks build in a reopen window. Zendesk's default automation is "Close ticket 4 days after status is set to solved" ([Zendesk help](https://support.zendesk.com/hc/en-us/articles/4408887712154-What-is-the-difference-between-a-solved-ticket-and-a-closed-ticket)). In Jira, Resolved means "A resolution has been taken, and it is awaiting verification by reporter", and Reopened means "This work item was once resolved, but the resolution was deemed incorrect" ([Atlassian docs](https://support.atlassian.com/jira-cloud-administration/docs/what-are-issue-statuses-priorities-and-resolutions/)). A label taken before those windows close can be wrong.


## Is a closed ticket enough to label an agent run as successful?

It is a start, not proof. Add a no-reopen window and a policy check, because a ticket can close with the wrong answer or with an action that broke a rule.

Vendors have met the same problem. Salesforce notes that under signal-based metrics a customer typing "thank you" "might count as a deflection whether the agent resolved their issue or not", and says "Session outcome metrics such as deflection and abandonment are moving from signal-based calculation to context-based evaluation" ([Salesforce](https://www.salesforce.com/blog/agentforce-observability/)). For a dataset, that means a deflection count is a signal and the outcome label needs its own check.


## How should human corrections be recorded?

Record what the person changed, at which step, and why if they said so. A run where a clerk corrected one amount and approved the rest is a different label from a run approved untouched.

Approval steps are common in finance. In one production payment agent, "A human, or a separate system, has to approve the release before the money moves" ([PYMNTS](https://www.pymnts.com/news/artificial-intelligence/2026/one-ai-agent-2395-payments-still-no-final-say/)). That approval record, with any edits, is the best outcome label the run has.


## How do you join agent traces to Zendesk, Jira, ServiceNow or ledger records?

Write the run ID onto the record the agent touched, and the record ID into the run. Then a join is a lookup, not a guess.

- Put the run ID in a ticket field, an internal note or a ledger memo when the agent acts.
- Give each action the agent takes an idempotent action ID, so a retried call does not look like two actions.
- Store timestamps with their time zone in both systems.
- Keep the record's status history, not just its current status, so a later reopen is visible.


## Should failed agent runs be kept in a training or evaluation set?

Yes, labelled as failures or partial runs. Failures show where agents go wrong and how they recover, which successful runs often do not.

Researchers who filter runs only by outcome note that "successful trajectories may lack error recovery or exhibit redundancy" ([TopoCurate, arXiv 2603.01714](https://arxiv.org/abs/2603.01714)). Benchmarks built from real runs are hard: one built from "252,557 sessions" found "Nine frontier LLMs achieve a mean pass rate of only 26.7%" ([TraceDance, arXiv 2609.33295](https://arxiv.org/abs/2609.33295)). Failed runs are where that difficulty shows.


## Template: outcome fields for each run

Add these six fields to each run. They sit alongside the fields of the [agent trace data card](https://origindatapartners.com/downloads/agent-trace-data-card.csv).

| Field | Values | Example |
| --- | --- | --- |
| outcome | success, partial, failure, unknown | success |
| outcome_source | The system or person that confirmed it | help desk ticket 88412, status history |
| outcome_checked_at | Date and time with time zone | 2026-10-21T09:00:00+01:00 |
| checker | human, system or model, and which | system: reopen check after 14 days |
| correction_steps | Step numbers a person changed, and what | step 37: refund amount changed from 120.00 to 80.00 |
| policy_check | pass, fail or not checked, and the rule | pass: refund under approval limit |

The example values are invented to show the format.


## Common mistakes with outcome labels

Each one makes a run look better or worse than it was.

- Labelling at the moment the agent finished instead of after the reopen or reversal window.
- Using deflection or a thank-you as the outcome.
- Dropping failed runs from the set.
- Not saying whether a person, a script or a model checked the outcome.
- Recording the approval but not the edits made before it.
- Joining runs to tickets by timestamp alone.


## Outcome-labelled runs as a data asset

Runs with a checked outcome and recorded corrections are what turn a log into training and evaluation data. They are also what buyers ask about first: did the run finish the job, and how do you know.

Origin helps you work out which outcome sources you already have, describe the labelled runs and build the data card. Nothing leaves the company at the start. See also [non-coding agent traces](https://origindatapartners.com/guides/license-non-coding-agent-traces) for what buyers ask for. We help you license your data. Origin Data Partners is an independent data partner: we tell you what your records could be worth and prepare the records, the terms and the license with you. The AI company that licenses them pays your company; a buyer may pay Origin when a deal closes.

Does your company hold records like these? [See what your data is worth](https://origindatapartners.com/check-fit).


## Questions and answers

### Who should confirm agent run outcomes, a person or a script?

Both where you can. A script can check ticket status, reconciliation or alert state at scale; a person checks the cases a script cannot judge. Record which one confirmed each run.

### Do failed agent runs have value for a buyer?

Yes, for evaluation and for learning recovery, if they are labelled as failures with the source of that label. Unlabelled failures are much less useful.

### How long after a run should we check its outcome?

Long enough for a reopen, a reversal or a repeat incident to show up. State the window you used in the data card, for example 14 days for tickets or the period close for ledgers.

### Can an LLM judge label agent run outcomes?

As a first pass, checked against a sample of runs people have labelled. Record that a model made the label, which model, and the prompt it used.

### Is a deflected support conversation a successful agent run?

Not on its own. A deflection means the customer did not reach a person; it does not show the issue was solved. Check for a repeat contact on the same issue before labelling it a success.

## Sources

- [Artificial Analysis: ITBench-AA, SRE incident tasks (27 May 2026)](https://artificialanalysis.ai/articles/itbench-aa-launch)
- [Zendesk help: the difference between a solved ticket and a closed ticket](https://support.zendesk.com/hc/en-us/articles/4408887712154-What-is-the-difference-between-a-solved-ticket-and-a-closed-ticket)
- [Atlassian docs: work item statuses, priorities and resolutions](https://support.atlassian.com/jira-cloud-administration/docs/what-are-issue-statuses-priorities-and-resolutions/)
- [Salesforce: Agentforce Observability and the Session Tracing Data Model (21 July 2026)](https://www.salesforce.com/blog/agentforce-observability/)
- [PYMNTS: one AI agent, 2,395 payments (21 August 2026)](https://www.pymnts.com/news/artificial-intelligence/2026/one-ai-agent-2395-payments-still-no-final-say/)
- [TopoCurate: interaction topology for tool-use agent training (arXiv 2603.01714)](https://arxiv.org/abs/2603.01714)
- [TraceDance: benchmarks from real-world agent deployment traces (arXiv 2609.33295)](https://arxiv.org/abs/2609.33295)

## Related reading

- [Agent Traces: Licensing Production AI Agent Runs (2026)](https://origindatapartners.com/guides/license-ai-agent-traces)
- [License data for AI evaluation: training, testing and holdouts](https://origindatapartners.com/guides/license-data-for-ai-evaluation)
- [What are RL environments? How AI labs build them and why they pay for company data](https://origindatapartners.com/guides/rl-environments-company-data)
- [Non-Coding Agent Traces: What Buyers Ask For (2026)](https://origindatapartners.com/guides/license-non-coding-agent-traces)
- [Long-Horizon Agent Traces: How to Keep the Middle](https://origindatapartners.com/guides/long-horizon-agent-traces)
- [/glossary/agent-trace](https://origindatapartners.com/glossary/agent-trace)
- [How to Log AI Agent Runs: Tool Calls, Reasoning, Outcomes](https://origindatapartners.com/guides/how-to-log-ai-agent-runs)
- [AI SRE Agent Traces: Incident Runs as Training Data](https://origindatapartners.com/guides/ai-sre-incident-agent-traces)
