# AI SRE Agent Traces: Incident Runs as Training Data

> AI SRE agents that investigate incidents leave long runs with clear outcomes. What buyers want from them, how to export them and what to remove first.

Canonical: https://origindatapartners.com/guides/ai-sre-incident-agent-traces
By: Origin Data Partners
Status: Published
Published: 2026-11-15
Updated: 2026-11-15
Sources checked: 2026-11-15

## Quick answer

AI SRE agent traces are the full records of an agent investigating a production incident: the alert, every query of logs, metrics and traces, each command, the hypotheses it tried and dropped, and the root cause it reached. They are long, tool-heavy and checkable against the postmortem, which makes them some of the most useful agent runs a company holds. Secrets and customer data in the tool output must come out first.

## Key points

- An AI SRE run is a chain of hypotheses tested against real telemetry, ending in a root cause.
- The postmortem is the outcome label: accepted root cause, fix merged, alert cleared.
- Environment dumps and logs carry secrets, hostnames and customer IDs; scrub before any copy.
- Runs from an in-house agent are the company's; runs inside a vendor's product need the vendor's terms read.

## What is an AI SRE agent?

An AI SRE agent is software that investigates production incidents the way an on-call engineer does. It reads the alert, queries logs, metrics and traces, checks recent deploys, runs read-only commands and proposes a root cause and a fix.

Vendors describe the loop in the same terms. Datadog says its agent "iteratively forms hypotheses, gathers relevant telemetry, and uses data-based reasoning to help on-call engineers pinpoint root causes" ([Datadog docs](https://docs.datadoghq.com/bits_ai/bits_investigation/)). Many teams build their own on an agent framework with tools for their observability stack, ticketing and Kubernetes.


## What data does an AI SRE agent produce?

A long run with a clear shape: trigger, investigation, conclusion. Each step is a tool call and its result, and the reasoning between steps records why the agent looked where it did.

A public SRE benchmark shows the same inputs: each task "provides a Kubernetes incident snapshot containing alerts, events, traces, metrics, logs, and application topology" ([Artificial Analysis](https://artificialanalysis.ai/articles/itbench-aa-launch)). A production run adds what no benchmark has: your real systems, your real noise and the engineer who checked the answer.

- The alert or page, with its time and service.
- Queries and results: logs, metrics, traces, deploy history, config diffs.
- Commands run, such as kubectl describe or a database health check, with their output.
- Hypotheses the agent formed, tested and dropped.
- The proposed root cause, the suggested fix and what the engineer did with them.


## Are incident investigation agent logs useful beyond on-call?

Yes, because they are hard runs with checkable answers. Most agent runs are short; an incident run is long, uses many tools and ends in a claim that can be checked against what the team later found.

Current models still struggle with this work. Artificial Analysis and IBM launched ITBench-AA "starting with site reliability engineering tasks where frontier models score below 50%", and found that "longer trajectories do not translate to higher accuracy" ([Artificial Analysis](https://artificialanalysis.ai/articles/itbench-aa-launch)). Benchmarks built from real deployment sessions are hard too: "Nine frontier LLMs achieve a mean pass rate of only 26.7%" ([TraceDance](https://arxiv.org/abs/2609.33295)).


## How do you label an incident investigation as successful?

Against the postmortem, not the agent's own conclusion. A run succeeded if its root cause is the one the team accepted, its fix was merged or applied, and the alert cleared and stayed clear.

Record which check was made and when. Partial credit is real: an agent that narrowed the search to the right service but named the wrong cause still saved time, and that run is worth keeping with an honest label. The [outcome labels guide](https://origindatapartners.com/guides/agent-run-outcome-labels) gives field names.

| Check | Where it comes from | Label |
| --- | --- | --- |
| Root cause accepted | Postmortem or incident ticket | correct, partly correct, wrong |
| Fix applied | Merged change or runbook action | yes, no, different fix |
| Alert cleared and stayed clear | Monitoring history for an agreed window | yes, recurred |
| Time to resolution | Incident timeline | minutes, with and without the agent |
| Human steps | Run log | which step a person took over or corrected |


## What secrets and customer data appear in SRE agent traces?

More than in almost any other agent run, because the tools return raw system output. Environment variables, connection strings, tokens in headers, internal hostnames and IPs, customer IDs and emails in log lines, and payloads from failed requests all show up.

The lesson of 13 November 2026 is direct. Cursor disclosed that an attacker copied a bucket of coding-agent traces "which we were permitted to use for model training", and that "A small number of those traces contained credentials that had not been redacted". Then: "Starting with those credentials, the attacker reached select SaaS tools used by Cursor." ([Cursor](https://cursor.com/blog/security-incident-update)). Scan, remove and rotate before any copy leaves; the [redaction checklist](https://origindatapartners.com/guides/agent-trace-redaction-checklist) gives the order.

- Secrets first: keys, tokens, passwords, connection strings. Rotate every one you find.
- Customer content in log lines and request bodies.
- Personal data: names, emails, IPs tied to users.
- Internal hostnames, account IDs and network layout, replaced with consistent placeholders.


## Are logs from a vendor's AI SRE agent yours?

It depends on the vendor's terms. Runs from an agent you built and host on your own systems are your company's records. Runs inside a vendor's product sit in the vendor's system, under its agreement with you, and may be exportable only in part.

An AI SRE vendor holding many customers' runs has a different position: each customer's telemetry belongs to that customer, and its contract decides what the vendor may do with it. See the [vendor agent logs guide](https://origindatapartners.com/guides/vendor-ai-agent-logs-ownership). This page is general information, not legal advice.


## Worked example: a sample set of ten incident runs

A fictional platform team runs an in-house incident agent on an open-weight model. For a first description it picks ten runs from the last quarter: six where the agent named the accepted root cause, two partly right and two wrong.

For each it notes the incident type (bad deploy, saturation, dependency failure), the steps and tokens, the tools used, the postmortem verdict and the time to resolution. It scrubs one run fully as a test and writes down what the scrub removed. No logs leave the company at this stage.


## Common mistakes with AI SRE agent traces

Most of these cost value; the first one costs far more.

- Copying runs before scrubbing secrets from command output.
- Labelling a run by the agent's own conclusion instead of the postmortem.
- Keeping only the final summary, not the queries and results that led to it.
- Letting log retention delete the runs: CloudWatch, for one, says "Any data older than the current retention setting is deleted" ([AWS docs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Working-with-log-groups-and-streams.html)).
- Dropping failed investigations. They show where the agent went wrong.


## Incident runs are also a data asset

Incident runs are among the most valuable agent records because they are long, use real tools and end in a checkable answer. Incident investigation was among the non-coding work in demand. Here is what buyers Origin works with asked for in November 2026: real production runs, nothing cut from the middle, the model and harness stated and a verified outcome. See [non-coding agent traces](https://origindatapartners.com/guides/license-non-coding-agent-traces), [security incident data](https://origindatapartners.com/guides/license-security-incident-data-ai) and [licensing agent traces](https://origindatapartners.com/guides/license-ai-agent-traces).

We help you license your data. Origin Data Partners is an independent data partner: we tell you what your records could be worth and prepare the records, the terms and the license with you. The AI company that licenses them pays your company; a buyer may pay Origin when a deal closes.

Does your company hold records like these? [See what your data is worth](https://origindatapartners.com/check-fit).


## Questions and answers

### Do alert-triage runs count as incident agent traces?

Yes, if they record the steps and an outcome. Triage runs are shorter than full investigations, so they are worth more in volume and when they are tied to what the alert turned out to be.

### Do we need the postmortem to label AI SRE agent runs?

It is the best label. Without it, use the incident ticket's resolution, the merged fix and the alert history, and record which of those you used.

### What if the AI SRE agent only suggested commands and a human ran them?

Those runs still count. Log the suggestion, the command the engineer actually ran and its output, so the record shows where the person agreed with the agent and where they changed course.

### Are logs from a vendor AI SRE product ours to describe?

Read the vendor agreement first. Your telemetry is yours, but what you can export and how you may use the agent's output is set by the vendor's terms. This is general information, not legal advice.

### How many incident investigations are enough to start?

Ten described runs open a conversation. A first set is usually a few hundred long runs with outcomes, across several kinds of incident.

### Can an AI SRE vendor use its customers' incident runs?

Only as each customer contract allows. The telemetry and the incidents belong to each customer, so a vendor needs those terms, and often the customers' agreement, before any use beyond running the service.

## Sources

- [Artificial Analysis: ITBench-AA launch, SRE tasks](https://artificialanalysis.ai/articles/itbench-aa-launch)
- [Datadog docs: Bits Investigation](https://docs.datadoghq.com/bits_ai/bits_investigation/)
- [TraceDance: benchmarks from real-world agent deployment traces (arXiv 2609.33295)](https://arxiv.org/abs/2609.33295)
- [Cursor: security incident update (13 November 2026)](https://cursor.com/blog/security-incident-update)
- [AWS docs: working with log groups and log retention in CloudWatch Logs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Working-with-log-groups-and-streams.html)

## Related reading

- [Can a security company license alerts and incident data to AI?](https://origindatapartners.com/guides/license-security-incident-data-ai)
- [Cybersecurity Incident Data: A Guide for MSSPs and IR Firms](https://origindatapartners.com/guides/cybersecurity-incident-soc-case-data)
- [PII Redaction for Agent Traces and LLM Logs: A Checklist](https://origindatapartners.com/guides/agent-trace-redaction-checklist)
- [Agent Traces: Licensing Production AI Agent Runs (2026)](https://origindatapartners.com/guides/license-ai-agent-traces)
- [Vendor AI Agent Logs: Who Owns Them and What to Export](https://origindatapartners.com/guides/vendor-ai-agent-logs-ownership)
- [/glossary/agent-trace](https://origindatapartners.com/glossary/agent-trace)
- [Non-Coding Agent Traces: What Buyers Ask For (2026)](https://origindatapartners.com/guides/license-non-coding-agent-traces)
- [Agent Run Outcome Labels: Tickets, Ledgers, Sign-Offs](https://origindatapartners.com/guides/agent-run-outcome-labels)
- [Incident Postmortem Examples: Outage, Security and Template](https://origindatapartners.com/guides/incident-postmortem-examples)
- [Incident Timeline Template: Alert to Fix, On-Call Handoff](https://origindatapartners.com/guides/incident-timeline-template)
- [Config Drift: How to Detect It in Terraform and Networks](https://origindatapartners.com/guides/config-drift)
