# AI SOC and Autonomous SOC: Agents, Tests and Datasets

> What an AI SOC is, what autonomous SOC agents do at each tier, how they are evaluated, and which public datasets and benchmarks exist and how each was built.

Canonical: https://origindatapartners.com/guides/ai-soc-autonomous-soc
By: Origin Data Partners
Status: Published
Published: 2026-11-17
Updated: 2026-11-17
Sources checked: 2026-11-17

## Quick answer

An AI SOC is a security operations center where software agents built on language models do parts of the analyst work: triaging alerts, gathering evidence, writing up investigations and, in an autonomous SOC, taking some response actions without waiting for a person. They are tested on benchmarks such as ExCyTIn-Bench and on missed-attack and false-positive rates, and most public test data is lab-made.

## Key points

- AI SOC agents mostly work tier 1 and tier 2: triage and investigation. Containment is where autonomy is most limited.
- The key numbers are missed attacks, false positives and whether each step is recorded.
- Public datasets are mostly built in labs or test tenants, and some are years old.
- Run a pilot on your own alerts before trusting any vendor score.

## What is an AI SOC?

A security operations center (SOC) is the team, in-house or outsourced, that watches a company's systems, handles alerts and responds to incidents. NIST lists "outsourcing a security operations center [SOC] to a managed security services provider [MSSP]" as one common way companies staff incident handling ([NIST SP 800-61r3](https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-61r3.pdf)).

An AI SOC adds software agents built on large language models (LLMs) to that team. An agent here is a program that reads an alert, decides what to look up, runs queries against the SIEM (the system that collects security logs) and other tools, and writes a conclusion. "Agentic SOC" and "AI SOC analyst" mean much the same thing. An autonomous SOC goes one step further: the agents close alerts or start response actions on their own, and people review a sample instead of every case.

The pressure behind it is volume. The CORTEX paper opens by saying SOCs "are overwhelmed by tens of thousands of daily alerts, with only a small fraction corresponding to genuine attacks" ([CORTEX](https://arxiv.org/abs/2510.00311)).


## What do AI SOC agents do at each tier?

Many SOCs split analyst work into tiers: tier 1 sorts alerts, tier 2 investigates, tier 3 handles complex incidents and response. The table maps each tier to the outcomes NIST SP 800-61r3 lists for incident response, and to what an agent does there today ([NIST SP 800-61r3](https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-61r3.pdf)).

| Tier | NIST outcome | What the agent does | Where people stay involved |
| --- | --- | --- | --- |
| Tier 1: triage | "Incident reports are triaged and validated"; "Incidents are categorized and prioritized" | Reads each alert, enriches it with asset and user context, closes or escalates | Reviewing a sample of closed alerts |
| Tier 2: investigation | "Information is correlated from multiple sources"; analysis to establish "what has taken place during an incident and the root cause" | Queries logs across systems, follows the trail, drafts a timeline | Checking the conclusion before action |
| Tier 3: response | "Incidents are contained" | Proposes or runs actions such as isolating a host or disabling an account | Approving actions, most of all in an autonomous SOC |
| Every tier | "Actions performed during an investigation are recorded" | Logs each query, result and decision | Auditing the record |

The last row matters most for trust. If an agent cannot show which evidence it read and why it closed an alert, its decision cannot be checked.


## How are AI SOC agents evaluated?

By how often they miss real attacks, how often they raise false ones, and whether their investigations reach the right answer. The ALERT-BENCH study "replays enterprise telemetry through a live SIEM" and found that across 1,247 alerts from a multi-stage attack, "every approach missed at least 40.4% of attack-related alerts" before the authors' own design ([ALERT-BENCH paper](https://arxiv.org/abs/2610.10608)).

Investigation benchmarks score answers instead of verdicts. ExCyTIn-Bench asks questions whose answers sit several hops away in the logs, because "real-world security analysts must sift through a large number of heterogeneous security logs, follow multi-hop chains of evidence to investigate threats". Its authors report that "the best model so far can achieve a reward of 0.606" ([ExCyTIn-Bench](https://arxiv.org/abs/2507.14201)). Cybench tests the attacker side: it breaks each task into subtasks "for a more detailed evaluation" ([Cybench](https://arxiv.org/abs/2408.08926)).

- Recall on attacks: the share of real attack alerts the agent escalates.
- False positive rate: the share of benign alerts it escalates anyway.
- Investigation accuracy: whether its account of what happened matches the ground truth.
- Traceability: whether each step and source is recorded.
- Time and cost per alert, compared with the analyst baseline.


## Which public datasets and benchmarks exist for AI SOC work?

Each public set is built a different way, and that decides what it can test. The table lists the ones security teams and researchers cite most, with what each is built from according to its own paper or page.

| Name | What it is | Built from |
| --- | --- | --- |
| ExCyTIn-Bench | Threat investigation questions over logs | A controlled Azure tenant with 57 log tables from Microsoft Sentinel and related services; 7,542 generated questions ([paper](https://arxiv.org/abs/2507.14201)) |
| Cybench | Offensive tasks for agents | 40 professional Capture the Flag tasks from 4 competitions ([paper](https://arxiv.org/abs/2408.08926)) |
| CIC-IDS2017 | Labeled network traffic for intrusion detection | Five days of captures in July 2017, background traffic from 25 profiled users, attacks such as brute force, DoS, Heartbleed and botnet ([CIC](https://www.unb.ca/cic/datasets/ids-2017.html)) |
| UNSW-NB15 | Labeled network traffic and flow features | Generated in the Cyber Range Lab of UNSW Canberra, a mix of real normal activity and synthetic attacks; 100 GB of captures; nine attack types ([UNSW](https://research.unsw.edu.au/projects/unsw-nb15-dataset)) |
| Loghub | System logs for log analytics | 19 log datasets; some are production logs released by earlier studies, others come from systems in a lab ([paper](https://arxiv.org/abs/2008.06448), [repository](https://github.com/logpai/loghub)) |
| SIABENCH | Security incident analysis tasks | 25 deep-analysis scenarios and 135 alert-triage scenarios ([paper](https://arxiv.org/abs/2603.06422)) |
| CORTEX dataset | Alert triage investigations | SOC investigations from production environments with analyst steps and tool outputs ([paper](https://arxiv.org/abs/2510.00311)) |


## What should a security lead ask before buying an AI SOC tool?

Vendor scores are measured on the vendor's data. These questions bring the claim back to your environment.

- On which data was the reported accuracy measured: a public benchmark, a lab, or customer alerts?
- What share of real attacks did it miss, not only how many alerts it closed?
- Can every closed alert be traced to the queries the agent ran and the results it saw?
- Which actions can it take without approval, and can that list be set per action?
- How does it handle a search that returns nothing? ALERT-BENCH found attack alerts were more likely to be dismissed when searches came back empty.
- Where are our logs and alerts processed, and are they used to train anything?


## Worked example: scoring an AI SOC pilot on one week of alerts

An illustration, not a real company. A SaaS company runs an AI SOC agent in shadow mode for one week: it sees every alert and writes a verdict, but analysts still decide. Afterwards the team compares both against the week's confirmed incidents.

| Measure | Analysts | Agent |
| --- | --- | --- |
| Alerts handled | 4,200 | 4,200 |
| Confirmed attack alerts (ground truth) | 36 | 36 |
| Attack alerts escalated | 33 | 29 |
| Attack alerts missed | 3 | 7 |
| Benign alerts escalated | 410 | 260 |
| Median minutes per alert | 9 | 2 |
| Closed alerts with a full evidence trail | 61% | 100% |

Fictional numbers written for this page. The agent is faster and noisier in a good way, but it missed 7 of 36 attack alerts against the analysts' 3. The team keeps a person on every closure in two alert categories where the misses clustered, and repeats the test a month later.


## Common mistakes when adopting an AI SOC

Most early problems are about measurement, not the model.

- Judging the tool by alerts closed instead of attacks missed.
- Accepting a benchmark score built on lab data as a forecast for your own network.
- Granting response actions before the agent has run in shadow mode on your alerts.
- No record of what the agent queried, so a wrong closure cannot be traced.
- Treating an empty search result as proof that nothing happened.
- Feeding the agent logs with gaps or unsynchronized clocks, then blaming it for a broken timeline.

Source: [From Investigation Failures to Reliable SOC Agents: Understanding and Improving LLM-Based Alert Triage (arXiv 2610.10608)](https://arxiv.org/abs/2610.10608)

## Why do AI SOC builders look for real incident records?

Because the public sets are thin where SOC work is hardest. The SIABENCH authors point to "the lack of an LLM-ready dataset covering a wide spectrum of SIA tasks" ([SIABENCH](https://arxiv.org/abs/2603.06422)), and the SOC-bench authors write that "no systematic benchmark for evaluating coordinated multi-task blue team AI has been proposed in the literature" ([SOC-bench](https://arxiv.org/abs/2603.28998)). The network sets in the table were captured in a lab or test tenant, CIC-IDS2017 in 2017. The CORTEX team released investigations from production environments for exactly this reason.

AI data buyers have asked Origin for the records of companies that defend their own infrastructure: incidents joined to the change that fixed them (the ticket, the chat thread, the config diff or pull request, the outcome), attack and defense sequences that can be replayed in order, SOC tickets, runbooks, on-call records and postmortems from the same period, and configuration history with commits and reviews. They asked for the company's own incidents, not a client's unless the client has agreed, and for logs the company owns. The first step is a sample of two or three incidents end to end plus a slice of the config history. See [SOC case data](https://origindatapartners.com/guides/cybersecurity-incident-soc-case-data) for MSSP and MDR firms and the [hub page](https://origindatapartners.com/guides/license-infrastructure-incident-data-ai) for who else holds these records.

Origin is an independent data partner. Your company decides what is shared and signs any license itself, and a buyer may pay Origin when a deal closes.

- Personal data in logs and tickets: pseudonymize consistently. See [anonymize pcap files and logs](https://origindatapartners.com/guides/anonymize-pcap-files-logs).
- Secrets in configuration and tickets: rotate and remove them first. See [remove secrets from git history](https://origindatapartners.com/guides/remove-secrets-from-git-history).
- Client environments: stay out unless the client has agreed in writing.
- Shared threat intelligence: material received under the Traffic Light Protocol (TLP) or from a sharing group keeps its sharing limits.

Does your company hold records like these? [See what your data is worth](https://origindatapartners.com/check-fit).


## Questions and answers

### What is the difference between an AI SOC and an autonomous SOC?

An AI SOC uses language-model agents to help analysts with triage and investigation, while people still decide. An autonomous SOC lets the agents close alerts or start response actions on their own, with people reviewing samples and approving the riskiest steps. Most teams start with the first.

### What does an AI SOC analyst actually do?

An AI SOC analyst is a software agent that reads an alert, adds context about the user and the asset, queries logs in the SIEM and other tools, and writes a verdict with its evidence. It usually covers tier 1 triage and parts of tier 2 investigation.

### Can an AI SOC replace tier 1 analysts?

Not safely without testing. Research such as ALERT-BENCH found every studied agent approach missed at least 40.4% of attack-related alerts before its authors' fix. Run the agent in shadow mode on your own alerts, measure missed attacks, and keep people on closures where it misses most.

### What is ExCyTIn-Bench?

ExCyTIn-Bench is a benchmark that tests LLM agents on cyber threat investigation. It was built from a controlled Azure tenant with 57 log tables and holds 7,542 generated questions whose answers require following evidence across logs. Its authors report a best reward of 0.606, leaving room to improve.

### Are CIC-IDS2017 and UNSW-NB15 still good for testing AI SOC tools?

They are useful for network intrusion detection, but both were generated in labs. CIC-IDS2017 covers five days in July 2017 with profiled background users, and UNSW-NB15 mixes real normal traffic with synthetic attacks. Neither holds tickets, investigations or response actions, which is most of SOC work.

### How should you measure an AI SOC pilot?

Compare the agent with your analysts on the same alerts and the same confirmed incidents. Count attack alerts missed, benign alerts escalated, time per alert, and the share of closures with a full evidence trail. Missed attacks matter more than alerts closed.

### What data are AI SOC agents trained and tested on?

Mostly public benchmarks and lab-made sets: test tenants such as ExCyTIn-Bench, Capture the Flag tasks such as Cybench, lab network captures such as CIC-IDS2017 and UNSW-NB15, and log collections such as Loghub. Some researchers have released production SOC investigations, but these remain rare.

## Sources

- [NIST SP 800-61 Rev. 3: Incident Response Recommendations and Considerations for Cybersecurity Risk Management](https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-61r3.pdf)
- [Wu et al., ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigation (arXiv 2507.14201)](https://arxiv.org/abs/2507.14201)
- [Zhang et al., Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models (arXiv 2408.08926)](https://arxiv.org/abs/2408.08926)
- [Canadian Institute for Cybersecurity: Intrusion detection evaluation dataset (CIC-IDS2017)](https://www.unb.ca/cic/datasets/ids-2017.html)
- [UNSW Canberra: The UNSW-NB15 Dataset](https://research.unsw.edu.au/projects/unsw-nb15-dataset)
- [Zhu et al., Loghub: A Large Collection of System Log Datasets for AI-driven Log Analytics (arXiv 2008.06448)](https://arxiv.org/abs/2008.06448)
- [Loghub repository (GitHub)](https://github.com/logpai/loghub)
- [Before You Hand Over the Wheel: Evaluating LLMs for Security Incident Analysis (arXiv 2603.06422)](https://arxiv.org/abs/2603.06422)
- [Design principles for the Construction of a Benchmark Evaluating Security Operation Capabilities of Multi-agent AI Systems (arXiv 2603.28998)](https://arxiv.org/abs/2603.28998)
- [Wei et al., CORTEX: Collaborative LLM Agents for High-Stakes Alert Triage (arXiv 2510.00311)](https://arxiv.org/abs/2510.00311)
- [From Investigation Failures to Reliable SOC Agents: Understanding and Improving LLM-Based Alert Triage (arXiv 2610.10608)](https://arxiv.org/abs/2610.10608)

## Related reading

- [Cybersecurity Incident Data: A Guide for MSSPs and IR Firms](https://origindatapartners.com/guides/cybersecurity-incident-soc-case-data)
- [AI SRE Agent Traces: Incident Runs as Training Data](https://origindatapartners.com/guides/ai-sre-incident-agent-traces)
- [Can a security company license alerts and incident data to AI?](https://origindatapartners.com/guides/license-security-incident-data-ai)
- [License data for AI evaluation: training, testing and holdouts](https://origindatapartners.com/guides/license-data-for-ai-evaluation)
- [Runbook vs Playbook: Differences, Examples and a Template](https://origindatapartners.com/guides/runbook-vs-playbook)
- [Anonymize PCAP Files and Logs Without Breaking Analysis](https://origindatapartners.com/guides/anonymize-pcap-files-logs)
