# How to Log AI Agent Runs: Tool Calls, Reasoning, Outcomes

> A logging spec for production AI agents: what to capture per run, content capture settings, retention, model and harness stamps, outcomes and redaction.

Canonical: https://origindatapartners.com/guides/how-to-log-ai-agent-runs
By: Origin Data Partners
Status: Published
Published: 2026-11-15
Updated: 2026-11-15
Sources checked: 2026-11-15

## Quick answer

To log AI agent runs well, record every step of every run: the task, each model call with its inputs and outputs, each tool call with full arguments and results, reasoning where the model returns it, the model, harness and prompt versions, human interventions and the final outcome. OpenTelemetry's GenAI conventions give the field names, but content capture is off by default, so turn it on deliberately and scrub secrets at the source.

## Key points

- A run you cannot replay step by step is a summary, not a log.
- OpenTelemetry GenAI instrumentations leave prompts and tool results out until you opt in.
- Stamp every run with model, harness, prompt and tool-schema versions.
- Keep a raw copy outside any tool whose plan deletes or hides old traces.

## What should you log for every AI agent run?

Enough to replay the run and grade it. That means the task as given, every step in order, every model call and tool call with full inputs and outputs, the reasoning the model returned, the versions that ran, any human step and the outcome.

OpenTelemetry's GenAI semantic conventions already name most of these fields. Use them even if you store runs elsewhere, so the log can move between tools. They are still marked as in development, so pin the version you follow.

| Record | Field (OpenTelemetry where one exists) | Why |
| --- | --- | --- |
| Run and session ID | gen_ai.conversation.id plus your run ID | Join steps, outcomes and corrections |
| Task | First input message | What the agent was asked to do |
| Model call inputs and outputs | gen_ai.input.messages, gen_ai.output.messages | The actual exchange, not a summary |
| Reasoning | Reasoning parts of the output messages | Why the agent chose each step |
| Tool calls | gen_ai.tool.name, gen_ai.tool.call.arguments, gen_ai.tool.call.result | What the agent did and what came back |
| Tokens | gen_ai.usage.input_tokens, gen_ai.usage.output_tokens | Length and cost per step |
| Versions | gen_ai.request.model, gen_ai.agent.version, your harness and prompt versions | Which system produced the run |
| Human steps | Your own event: who, what changed, at which step | Corrections are the most useful signal |
| Outcome | Your own field, with its source and check time | Whether the job was done |


## Does OpenTelemetry capture prompts and completions by default?

No. The conventions say instrumentations "SHOULD NOT capture them by default, but SHOULD provide an option for users to opt in" ([OpenTelemetry](https://raw.githubusercontent.com/open-telemetry/semantic-conventions-genai/main/docs/gen-ai/gen-ai-spans.md)). Without that switch you get timings and token counts, but no prompts, no answers and no tool results.

In the Python instrumentation the switch is OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT, which will "enable capture of prompts, completions, tool arguments, and return values" ([OpenTelemetry Python docs](https://opentelemetry-python-contrib.readthedocs.io/en/latest/instrumentation-genai/openai.html)). Tool results are separate opt-in attributes too: gen_ai.tool.call.arguments holds the "Parameters passed to the tool call." and gen_ai.tool.call.result holds "The result returned by the tool call (if any and if execution was successful)." ([OpenTelemetry](https://raw.githubusercontent.com/open-telemetry/semantic-conventions-genai/main/docs/gen-ai/execute-tool.md)).

Turn it on knowing what it brings in. The registry warns that the message attributes are "likely to contain sensitive information including user/PII data" ([OpenTelemetry registry](https://opentelemetry.io/docs/specs/semconv/registry/attributes/gen-ai/)).

Capture message content (OpenTelemetry Python):

```bash
export OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental
export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=span_only
```


## Which defaults capture more than you expect?

Some agent SDKs capture everything unless told otherwise. In the OpenAI Agents SDK, "By default, trace_include_sensitive_data is true", so model and tool inputs and outputs go into the spans and, by default, to OpenAI ([OpenAI Agents SDK docs](https://openai.github.io/openai-agents-python/tracing/)).

That is the record you want for later use, but it should go to a store you control and pass through a scrubber first. Scrub at the source: a filter in your own exporter that removes keys, tokens and passwords before anything is written, then a scan of what was written. A secret found later means rotating it, not just deleting the line.

The cost of skipping this is public. On 13 November 2026 Cursor disclosed that agent traces it had shared for model training were copied by an attacker, and "A small number of those traces contained credentials that had not been redacted" ([Cursor](https://cursor.com/blog/security-incident-update)).


## How long should agent traces be kept?

As long as you might need to replay, evaluate or explain a run, and no longer than your privacy notices and customer contracts allow. Decide that once, write it down, and make sure no tool deletes runs sooner without you knowing.

Tools differ. In LangSmith, "After the specified retention period, traces are no longer accessible in the tracing project UI or via the API" ([LangSmith docs](https://docs.langchain.com/langsmith/administration-overview)). "On Langfuse Cloud, each plan includes a data access window (Hobby: 30 days, Core: 90 days, Pro and Enterprise: 3 years)" ([Langfuse docs](https://langfuse.com/docs/administration/data-retention)). CloudWatch keeps logs until you set a period on the log group ([AWS docs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Working-with-log-groups-and-streams.html)).

Keep a raw copy of the runs that matter in storage you own, with sampling turned off for them. Sampling is fine for dashboards; it is not fine for the record.


## What versions should each run record?

Four: the model, the harness, the prompt and the tool schemas. The same model behaves differently in a different [agent harness](https://origindatapartners.com/glossary/agent-harness), so a run without the harness version cannot be compared with another.

OpenTelemetry gives gen_ai.request.model for the model and gen_ai.agent.version for "the version of the GenAI agent created during this operation" ([OpenTelemetry](https://raw.githubusercontent.com/open-telemetry/semantic-conventions-genai/main/docs/gen-ai/gen-ai-agent-spans.md)). Add your own attributes for the harness release, the prompt version and a hash of the tool definitions. Record reasoning effort or thinking settings too; output token counts "SHOULD include reasoning tokens and tokens of all modalities".


## How do you log the outcome and human corrections?

Write the run ID wherever the work lands, then record the outcome when it is known. For a support agent that is the ticket ID and its final state; for an invoice agent the ledger line; for an incident agent the postmortem.

Log corrections as their own events: who changed what, at which step, and why if they said. A run where a person fixed one tool argument and the agent then finished is worth more than either a clean success or a silent failure, because it shows where the agent went wrong.


## Template: the fields of one agent run record

One row per run, with steps stored as an ordered list. Field names are ours; map them to the gen_ai.* names in the first table where one exists.

| Field | Example (fictional) |
| --- | --- |
| run_id, session_id | run_8f2c, sess_1931 |
| started_at, ended_at (with time zone) | 2026-11-03T09:14:02Z, 2026-11-03T09:41:55Z |
| task | Reconcile October statement for vendor 4471 |
| model, model_version, reasoning_setting | open-weight model name, exact release, high |
| harness, harness_version, prompt_version, tool_schema_hash | in-house, 2.4.1, p-37, 9c1e… |
| steps[] | Each with type, input, output, tool name, arguments, result, tokens, timestamps |
| human_events[] | Step 14: analyst changed GL account 6100 to 6150 |
| outcome, outcome_source, outcome_checked_at | reconciled, ledger close report, 2026-11-10 |
| redaction_status | scrubbed at source; scan passed |


## Do your customer contracts let you keep agent logs?

For running and fixing the service, usually yes; for other uses, read them. Logs of an agent that works on customers' requests hold their content. Your terms, data processing agreements and privacy notice say what you may do with it beyond delivering the service.

If you want the option to use de-identified run data later, for evaluation, for training your own model or anything else, the place to say so is the contract and the notice, before the runs are recorded. This is general information, not legal advice; have counsel read your own terms.


## Common mistakes when logging agent runs

Each of these leaves a log that looks fine on a dashboard and fails when someone replays a run.

- Leaving content capture off, so spans hold timings and nothing else.
- Logging tool results as references to files that are later deleted.
- Sampling the runs you most need: long, failed or escalated ones.
- No harness or prompt version, so two runs cannot be compared.
- Recording "success" when the agent stopped, not when the work was checked.
- Logging secrets from environment dumps and tool outputs, then copying the log.


## Well-logged runs are also a data asset

A log built to this spec pays for itself in debugging and evaluation. It is also the record buyers ask for. Here is what buyers Origin works with asked for in November 2026: complete production runs with reasoning, tool calls and results, the model and harness stated, and a way to tell the job was done. If you already have six months of runs, tell us what you log. See [licensing agent traces](https://origindatapartners.com/guides/license-ai-agent-traces) and [outcome labels](https://origindatapartners.com/guides/agent-run-outcome-labels).

We help you license your data. Origin Data Partners is an independent data partner: we tell you what your records could be worth and prepare the records, the terms and the license with you. The AI company that licenses them pays your company; a buyer may pay Origin when a deal closes.

Does your company hold records like these? [See what your data is worth](https://origindatapartners.com/check-fit).


## Questions and answers

### Should we log full tool results even when they are large?

Yes, in the raw store. Keep the model's context lean with references if you need to, but the run record should hold the full result or a durable copy that will not expire before the run does.

### Is logging model reasoning a privacy risk?

It can be. Reasoning often repeats names, numbers and other details from the inputs, so treat it like any other content field: scrub it at the source and redact it the same way before any copy leaves.

### How do we stop secrets ending up in agent logs?

Filter at the source in your own exporter, before anything is written. Then scan what was written with a secret scanner, and rotate any credential you find. Deleting the line is not enough once a secret has been logged.

### Do we need customer consent to keep AI agent logs?

To operate and fix the service, your contracts usually cover it. For other uses, check your customer terms, data processing agreements and privacy notice. This is general information, not legal advice.

### What format should AI agent logs use?

OpenTelemetry GenAI spans for live tracing, plus a per-run JSON Lines export that keeps every step in order with its tool results inline. The spans let you change tools; the export lets anyone replay a run.

### Should we sample agent traces to cut costs?

Sample for dashboards if you must, never for the record. Keep every run that is long, failed, escalated or corrected, because those are the ones you will want to replay.

## Sources

- [OpenTelemetry GenAI semantic conventions: model spans and content capture](https://raw.githubusercontent.com/open-telemetry/semantic-conventions-genai/main/docs/gen-ai/gen-ai-spans.md)
- [OpenTelemetry GenAI semantic conventions: agent spans](https://raw.githubusercontent.com/open-telemetry/semantic-conventions-genai/main/docs/gen-ai/gen-ai-agent-spans.md)
- [OpenTelemetry GenAI semantic conventions: execute_tool span](https://raw.githubusercontent.com/open-telemetry/semantic-conventions-genai/main/docs/gen-ai/execute-tool.md)
- [OpenTelemetry attribute registry: gen_ai attributes](https://opentelemetry.io/docs/specs/semconv/registry/attributes/gen-ai/)
- [OpenTelemetry Python contrib docs: OpenAI instrumentation settings](https://opentelemetry-python-contrib.readthedocs.io/en/latest/instrumentation-genai/openai.html)
- [OpenAI Agents SDK (Python) docs: tracing](https://openai.github.io/openai-agents-python/tracing/)
- [LangSmith docs: administration overview (trace retention)](https://docs.langchain.com/langsmith/administration-overview)
- [Langfuse docs: data retention](https://langfuse.com/docs/administration/data-retention)
- [AWS docs: working with log groups and log retention in CloudWatch Logs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Working-with-log-groups-and-streams.html)
- [Cursor: security incident update (13 November 2026)](https://cursor.com/blog/security-incident-update)

## Related reading

- [Export Agent Traces: LangSmith, Langfuse, Phoenix and More](https://origindatapartners.com/guides/export-agent-traces)
- [PII Redaction for Agent Traces and LLM Logs: A Checklist](https://origindatapartners.com/guides/agent-trace-redaction-checklist)
- [Agent Traces: Licensing Production AI Agent Runs (2026)](https://origindatapartners.com/guides/license-ai-agent-traces)
- [Export Agent Runs: Agno, AgentCore, Agentforce, OpenAI SDK](https://origindatapartners.com/guides/export-agent-runs-from-agent-platforms)
- [Can an AI startup license its usage data and agent logs?](https://origindatapartners.com/guides/license-ai-product-usage-data)
- [/glossary/agent-harness](https://origindatapartners.com/glossary/agent-harness)
- [/refer/ai-agent-consultants-and-builders](https://origindatapartners.com/refer/ai-agent-consultants-and-builders)
- [Agent Run Outcome Labels: Tickets, Ledgers, Sign-Offs](https://origindatapartners.com/guides/agent-run-outcome-labels)
- [Long-Horizon Agent Traces: How to Keep the Middle](https://origindatapartners.com/guides/long-horizon-agent-traces)
