Know a company with years of records? Refer it and earn up to $100K
Agent traces

Agent Run Outcome Labels: Tickets, Ledgers, Sign-Offs

How to prove an AI agent run finished the job: ticket state, reconciled ledger lines, accepted root causes and human sign-off, with a field template.

Fields per run
Sixoutcome, outcome_source, outcome_checked_at, checker, correction_steps, policy_check
Zendesk default
Closed 4 days after solvedA solved ticket can still reopen in that window
A laptop showing a code diff with a sprint board behind
Short answer

An outcome label records whether an AI agent run finished the job and what proves it: a ticket that stayed closed, a ledger line that reconciled, a root cause accepted in the postmortem or a person's sign-off. Record the outcome, its source, when it was checked and by whom, and any correction a person made. A closed ticket alone is a start, not proof. Keep failed runs, labelled as failures.

  • Every run needs an outcome and the record that proves it.
  • Wait long enough for reopens, reversals and repeat incidents before you label a run a success.
  • Record what a person changed, and at which step.
  • Failed and partial runs are worth keeping when they are labelled.
  • An LLM judge can make a first pass; check it against runs people have labelled.

How do you know an AI agent run actually succeeded?

You check it against a record outside the agent: the ticket system, the ledger, the postmortem or a person who signed off. The agent saying "done" is not that record.

Anyone who trains or tests a model on your runs needs to know which runs to copy and which to learn from as mistakes. Without an outcome, a run is a story with no ending. User approval is not enough on its own: a trace buyer's published essay (1 October 2026) warns that "naïve user approval is not always evidence of correctness". Benchmark builders work the same way: on one incident benchmark "Each submission is compared against a ground-truth set of root causes provided by IBM" (Artificial Analysis).

What outcome fields should an agent trace dataset include, by function?

Each function has a system that already knows whether the work was done. Use it, and name it in the outcome_source field.

FunctionSuccess meansSource of truthWait before labelling
SupportTicket solved and not reopened; customer did not come back on the same issueHelp desk ticket historyAt least the reopen window your desk uses
FinanceLine reconciled, payment posted, no reversal or credit noteERP or ledgerUntil the period closes
IncidentsRoot cause accepted in the postmortem, fix merged, alert cleared and not backPostmortem, repository, alertingUntil the postmortem is signed off
SchedulingBooking confirmed and keptCalendar or booking systemUntil the booked date has passed
Back officeFiling accepted, not returned or correctedFiling system or the receiving bodyUntil acceptance is confirmed

Help desks build in a reopen window. Zendesk's default automation is "Close ticket 4 days after status is set to solved" (Zendesk help). In Jira, Resolved means "A resolution has been taken, and it is awaiting verification by reporter", and Reopened means "This work item was once resolved, but the resolution was deemed incorrect" (Atlassian docs). A label taken before those windows close can be wrong.

Is a closed ticket enough to label an agent run as successful?

It is a start, not proof. Add a no-reopen window and a policy check, because a ticket can close with the wrong answer or with an action that broke a rule.

Vendors have met the same problem. Salesforce notes that under signal-based metrics a customer typing "thank you" "might count as a deflection whether the agent resolved their issue or not", and says "Session outcome metrics such as deflection and abandonment are moving from signal-based calculation to context-based evaluation" (Salesforce). For a dataset, that means a deflection count is a signal and the outcome label needs its own check.

How should human corrections be recorded?

Record what the person changed, at which step, and why if they said so. A run where a clerk corrected one amount and approved the rest is a different label from a run approved untouched.

Approval steps are common in finance. In one production payment agent, "A human, or a separate system, has to approve the release before the money moves" (PYMNTS). That approval record, with any edits, is the best outcome label the run has.

How do you join agent traces to Zendesk, Jira, ServiceNow or ledger records?

Write the run ID onto the record the agent touched, and the record ID into the run. Then a join is a lookup, not a guess.

  • Put the run ID in a ticket field, an internal note or a ledger memo when the agent acts.
  • Give each action the agent takes an idempotent action ID, so a retried call does not look like two actions.
  • Store timestamps with their time zone in both systems.
  • Keep the record's status history, not just its current status, so a later reopen is visible.

Should failed agent runs be kept in a training or evaluation set?

Yes, labelled as failures or partial runs. Failures show where agents go wrong and how they recover, which successful runs often do not.

Researchers who filter runs only by outcome note that "successful trajectories may lack error recovery or exhibit redundancy" (TopoCurate, arXiv 2603.01714). Benchmarks built from real runs are hard: one built from "252,557 sessions" found "Nine frontier LLMs achieve a mean pass rate of only 26.7%" (TraceDance, arXiv 2609.33295). Failed runs are where that difficulty shows.

Template: outcome fields for each run

Add these six fields to each run. They sit alongside the fields of the agent trace data card.

FieldValuesExample
outcomesuccess, partial, failure, unknownsuccess
outcome_sourceThe system or person that confirmed ithelp desk ticket 88412, status history
outcome_checked_atDate and time with time zone2026-10-21T09:00:00+01:00
checkerhuman, system or model, and whichsystem: reopen check after 14 days
correction_stepsStep numbers a person changed, and whatstep 37: refund amount changed from 120.00 to 80.00
policy_checkpass, fail or not checked, and the rulepass: refund under approval limit

The example values are invented to show the format.

Common mistakes with outcome labels

Each one makes a run look better or worse than it was.

  • Labelling at the moment the agent finished instead of after the reopen or reversal window.
  • Using deflection or a thank-you as the outcome.
  • Dropping failed runs from the set.
  • Not saying whether a person, a script or a model checked the outcome.
  • Recording the approval but not the edits made before it.
  • Joining runs to tickets by timestamp alone.

Outcome-labelled runs as a data asset

Runs with a checked outcome and recorded corrections are what turn a log into training and evaluation data. They are also what buyers ask about first: did the run finish the job, and how do you know.

Origin helps you work out which outcome sources you already have, describe the labelled runs and build the data card. Nothing leaves the company at the start. See also non-coding agent traces for what buyers ask for. We help you license your data. Origin Data Partners is an independent data partner: we tell you what your records could be worth and prepare the records, the terms and the license with you. The AI company that licenses them pays your company; a buyer may pay Origin when a deal closes.

Does your company hold records like these? See what your data is worth.

Common questions

Who should confirm agent run outcomes, a person or a script?

Both where you can. A script can check ticket status, reconciliation or alert state at scale; a person checks the cases a script cannot judge. Record which one confirmed each run.

Do failed agent runs have value for a buyer?

Yes, for evaluation and for learning recovery, if they are labelled as failures with the source of that label. Unlabelled failures are much less useful.

How long after a run should we check its outcome?

Long enough for a reopen, a reversal or a repeat incident to show up. State the window you used in the data card, for example 14 days for tickets or the period close for ledgers.

Can an LLM judge label agent run outcomes?

As a first pass, checked against a sample of runs people have labelled. Record that a model made the label, which model, and the prompt it used.

Is a deflected support conversation a successful agent run?

Not on its own. A deflection means the customer did not reach a person; it does not show the issue was solved. Check for a repeat contact on the same issue before labelling it a success.

Sources and further reading

What is your company’s data worth?

AI companies pay $20K to $5M for company records. Six questions, no files.