All articles
November 10, 2026

How to Evaluate an AI SOC Platform When Every Vendor Says Agentic

“Agentic” is table stakes in every pitch. ASEF, a vendor-neutral evaluation framework, gives you a better test: what the platform delivers, how much it does on its own, and what evidence backs each claim.

TL;DR

  • “Agentic” no longer separates AI SOC vendors. Gartner calls the mislabeling “agent washing,” and its 2026 Hype Cycle for Security Operations places AI SOC agents at the Peak of Inflated Expectations, as reported by Cybersecurity Insiders.
  • Gartner predicts 70% of large SOCs will pilot AI agents by 2028, but only 15% will achieve measurable improvements without structured evaluation.
  • ASEF, the AI SecOps Evaluation Framework, gives you that structure. It is vendor-neutral and covers the full SecOps lifecycle, not only triage.
  • For every capability, ASEF records delivery, autonomy, and evidence separately. Roadmap, preview, and unknown items do not count as coverage.
  • Platform and Trust is checked in every zone. A limit written in a prompt is a request. A limit the platform enforces is a bound.
  • Run the PoC on your own alerts. That is how a vendor claim becomes validated evidence.

Why “Agentic” Stopped Being a Useful Buying Signal

Every AI SOC vendor now calls its product agentic. When one word covers alert summarizers, copilots, and systems that investigate and respond, it stops helping you build a shortlist.

The problem reaches beyond security. In June 2025, Gartner estimated only about 130 of the thousands of agentic AI vendors are “real.” It called the practice “agent washing,” and the estimate spans all industries, not SOC vendors alone.

The category is also young. Gartner’s Hype Cycle for Security Operations, 2026, places AI SOC agents at the Peak of Inflated Expectations, as reported by Cybersecurity Insiders. The same report puts market penetration at 1% to 5% and mainstream adoption 5 to 10 years out.

Pilots are coming anyway. As reported by SC Media, Gartner predicts that by 2028, 70% of large SOCs will pilot AI agents to augment Tier 1 and Tier 2 operations, but only 15% will achieve measurable improvements without structured evaluation. Cybersecurity Insiders describes the 70% figure as a Gartner Strategic Planning Assumption, so treat it as a prediction.

Adoption already runs ahead of governance. The global SANS 2026 SOC Report found 79% of SOCs use AI or ML tools, but only 36% have built them into a defined workflow.

So stop scoring vendors on how many agentic features they list. Score what each platform delivers, how much it does on its own, and what proof backs it. That is the job ASEF is built for.

What Is ASEF?

ASEF, the AI SecOps Evaluation Framework, is a vendor-neutral framework from SecOps Unpacked for evaluating AI-enabled security tools and MDR services across the security operations lifecycle.

It does not rank vendors. ASEF produces no universal best-vendor ranking and no blended maturity score. It shows what a platform covers, how that coverage is delivered, how autonomous it is, and what evidence backs each claim, within the scope you choose.

It also keeps technology and services apart. AI SOC platforms and MDR services are evaluated in separate tracks. For MDR, the provider’s people are assessed separately from what its technology automates.

Here is how to apply it in five steps.

Step 1: Map Your Scope to the ASEF Zones

ASEF organizes the SecOps lifecycle into zones on the SecOps AI Shift Map:

  • Data ingestion and processing: pipelines, ingestion, processing, and retention, including how the platform accesses your SIEM data.
  • Detection engineering and SecOps resilience: building, tuning, and testing detections.
  • Investigation and triage: hunting, investigating, and running the case to closure.
  • Response, remediation, and feedback loop: acting on the conclusion and feeding what you learned back.
  • Platform and Trust: cross-cutting, assessed in every zone.

Most AI SOC pitches focus on investigation and triage. Decide which zones you are buying for before the first demo, and evaluate every vendor on that same scope.

ASEF also treats feedback as cross-cutting. Investigation findings can improve detections without any remediation capability, so a feedback loop is not proof of remediation coverage.

Step 2: Record How Each Capability Is Delivered

A demo shows what is possible. Delivery tells you what you get. ASEF records each capability in one of seven delivery states: out of the box, configured, customer-built, partner-delivered, preview, roadmap, or not supported.

Preview, roadmap, and unknown status do not count as generally available coverage. Customer-built is not a gap, but it is engineering time that your team or a partner will spend. Record it so you compare like with like.

Questions to ask the vendor:

  • Is this capability out of the box, configured, or something we build?
  • Which parts of this demo are generally available today, and which are preview or roadmap?
  • Who builds and maintains the customer-built parts: our team, a partner, or yours?

Step 3: Score Autonomy on the ASEF Scale

ASEF scores autonomy separately from delivery, on five levels:

  • 0, Manual: a manual process. It does not mean the capability is missing.
  • 1C, System assists; human acts: the human performs the action with system assistance.
  • 1G, Human approves execution: the system proposes work and waits for human approval before executing.
  • 1A, System acts; human can intervene: the system acts inside policy, and a human can monitor and step in.
  • 2, Completes the defined workflow: the system completes a scoped workflow without human intervention. That does not mean the whole investigation.

Two rules keep this honest. Autonomy can come from automation, AI, or both, so deterministic steps count. And running read-only queries alone does not prove autonomous investigation.

Score autonomy per action, not per product. Isolating a host and enriching an IP address do not carry the same risk. The OWASP Top 10 for LLM Applications 2025, LLM06: Excessive Agency names three root causes: excessive functionality, excessive permissions, and excessive autonomy. It recommends human approval before high-impact actions, which is 1G on the ASEF scale.

Lower-risk actions can run at 1A if every action is logged and the agent can be stopped. NIST AI 600-1 (2024) calls for protocols so generative AI systems are “able to be deactivated when necessary.”

Questions to ask the vendor:

  • Which actions run at 1G, which run at 1A, and who sets those thresholds?
  • How do we stop a running agent, and what happens to its pending actions?
  • Which steps are deterministic automation, and which rely on model reasoning?

Step 4: Separate Evidence From Claims

ASEF keeps evidence types apart: a vendor claim or published RFI, documentation, a recorded demo, a live demo, and validation in your own environment. Each source should carry a section or timestamp, a product version, and a date. Missing information is unknown, not zero.

Published vendor claims are a starting point, not your PoC results. In an AI SOC, the claims that matter most sit behind each verdict. These tests turn them into validated evidence:

  • Query traceability: for any closed alert, an analyst should see every query the agent ran, against which source, and what came back.
  • Rerun the query: copy a query, run it in the source tool, and confirm the results match.
  • Reproducibility: in Cybersecurity Insiders, former Gartner analyst Rob Smith writes that “Auditability and reproducibility are different tests.” He recommends running the same alert twice, a week apart, on the same data.
  • Blind re-check: analysts review verdicts on your own alerts before seeing the agent’s conclusion, and you record disagreements.

Questions to ask the vendor:

  • Can an analyst see and rerun every query behind a verdict?
  • What happens when the same alert runs twice on the same data?
  • Can the agent return “inconclusive” and name the missing data, instead of forcing benign or malicious?

Step 5: Check Platform and Trust in Every Zone

Platform and Trust is the cross-cutting check in ASEF. It uses its own functional scale of None, Limited, and Full, separate from autonomy. Deployment checks cover hosting, tenancy, operating responsibility, and where telemetry, query results, and model inference are located. Deployment options alone do not raise the score.

The ASEF guide also suggests a question for every vendor: what happens when a decision is wrong, who can review or reverse it, and what gets logged? For agents that take action, three checks answer it.

Where the limit lives. A guardrail written into a prompt asks the model to behave. A harness is enforced by the platform, below the model, where the agent cannot see, negotiate, or reason around it. Scoped tools, isolated execution, resource limits, and circuit breakers are harness controls. A system prompt telling the agent never to isolate a domain controller is a guardrail. A guardrail is a request. A harness is a bound.

How the agent authenticates. The OWASP Top 10 for Agentic Applications (December 2025) names ASI02 Tool Misuse and ASI03 Identity & Privilege Abuse, and notes that “leaked credentials let them operate far beyond their intended scope.” Look for access scoped per action instead of one broadly privileged service account, secrets the model never sees, and authorization enforced in the downstream system. OWASP LLM06 recommends that last point directly.

What the record shows. Set the bar at human-readable logs of reasoning, evidence sources, decisions, and actions for every automated step, with approvals and overrides tied to the analyst’s identity. NIST AI 600-1 says to “Monitor and document instances where human operators or other systems override the GAI’s decisions.” ASEF also assesses how those corrections become controlled improvements, with approvals, versioning, and verification.

Questions to ask the vendor:

  • Is each action limit enforced by the platform or stated in a prompt?
  • What identity does the agent use for each action, and can the model ever see a credential?
  • Where are telemetry, query results, and model inference processed and stored?
  • Can we export a full incident’s reasoning, evidence, approvals, and overrides?

Your ASEF Evaluation Sheet

ASEF does not blend results into one score, and neither should you. Record each zone separately and compare vendors row by row on the same scope.

ASEF zone What to record PoC test
Data ingestion and processing Data access model (ingestion-based, federated/in-place, or hybrid), storage ownership, and where detection runs Query your SIEM or data lake through the platform and confirm what it reads in place versus copies
Detection engineering and SecOps resilience Delivery state for detection tuning and testing, and how investigation findings feed back into detections Follow one false positive from the PoC into a detection change, including approvals and versioning
Investigation and triage Delivery state, autonomy level, and evidence type for each investigation step Rerun three agent queries in the source tool, run one alert twice a week apart, and blind re-check verdicts
Response, remediation, and feedback loop Autonomy level per action, who sets the thresholds, and how a run is stopped Run one incident from alert through response on your own tools, with approvals on
Platform and Trust None, Limited, or Full, plus hosting, tenancy, and where telemetry, query results, and inference are located Ask the agent to act outside its scope, confirm the platform blocks it, then reconstruct the incident from the log alone

How to Run a PoC That Reveals the Difference

The ASEF PoC flow is short: choose your vendors and link your requirements, test the capabilities that matter to your team, assign your own scores and record the evidence, then compare saved evaluations on the same scope. Set success criteria before it starts, then run these tests:

  1. Use your own alerts and data. Replay a representative sample, including noisy and ambiguous alerts.
  2. Run a blind analyst re-check. Analysts review verdicts before seeing the agent’s conclusion, and you record disagreements.
  3. Track overrides. Log every analyst overrule by detection rule to find where the agent struggles.
  4. Rerun queries. Copy queries from closed investigations into the source tools and compare results. Run one alert twice, a week apart.
  5. Attempt an out-of-scope action. Ask the agent to act beyond its permissions, and confirm the platform blocks it.
  6. Review the audit log of a full incident. Reconstruct one incident end to end, including approvals and overrides.

Each test turns a vendor claim into validation in your own environment.

AI SOC vs. Agentic Security Operations: Why Lifecycle Scope Matters

ASEF evaluates across the full SecOps lifecycle because most security work does not end at a verdict. That matches how BlinkOps frames the market. The AI SOC and ASO definitions below are BlinkOps’ framing, not an industry standard.

AI SOC applies AI agents to the SOC queue: triage, enrichment, investigation, and case preparation. An AI SOC analyst agent collects evidence, correlates telemetry, and hands a documented verdict to a human. That work matters, but it usually ends at the edge of the SOC.

For a practitioner definition, see AI SOC Core Components | 2026 Edition on SecOps Unpacked. It defines an AI SOC as “a system that combines deterministic automation, machine learning, and large language models to autonomously enrich, investigate, and close security signals.” It breaks an AI SOC into four components: data ingestion, a knowledge or context graph, an investigation engine, and response with a feedback loop. Three of them map directly to ASEF zones, and the context graph sits in the Context capability group. Use them to check which parts of the lifecycle a vendor actually covers.

Agentic Security Operations (ASO) extends agents beyond the SOC to identity and access management (IAM), vulnerability management, and governance, risk, and compliance (GRC). An Agentic Security Operations Platform (ASOP) is what builds, runs, and governs those agents.

Deterministic workflows handle steps that must run the same way every time. Reasoning agents handle steps that need judgment. On the ASEF autonomy scale, both count.

That is why BlinkOps treats AI SOC as one capability of a broader platform rather than a category of its own. A triage tool solves one queue. A platform extends into adjacent domains without another buying cycle or renewal, which raises the stakes for credential scope and cross-domain control.

Where Blink Fits

Run Blink through ASEF the same way you run any other vendor. Start with the published profile, then look at four areas.

The SecOps Unpacked profile. Blink’s profile in the SecOps Unpacked directory places it in the Middle and Right zones of the SecOps AI Shift Map, with AI Agent Builder as the primary capability and AI SOC Capability, Automation, CIRM / Case Management, Threat Hunting, and Threat Intel as additional tags. It records SaaS, self-hosted, and hybrid deployment, US and EU data residency, and SOC 2 Type II, ISO 27001, GDPR, and ISO/IEC 42001. That gives you a head start on Platform and Trust before the first demo. In ASEF terms, it is a starting point, not PoC evidence.

The practitioner notes on the profile call it a build platform: pre-built solutions like Agentic SOC get you started, and the value comes from teams that build and own agents. They also point to the visibility: “Analysts trust output they can inspect, and you can defend a decision when leadership or compliance asks how it was made.”

Investigation and triage. Blink’s AI SOC runs on the Blink Agentic Security Operations Platform (ASOP). Blink logs every reasoning step, evidence source, decision, and action, so analysts can check each verdict instead of trusting a summary.

Response and autonomy. Agents act only through ability workflows you assign, so they never need direct access to your systems or credentials. Human-in-the-loop controls, customizable risk thresholds, and escalation paths let your team set autonomy per action and decide where the machine stops and the human starts. Analysts approve and execute remediation from the response copilot.

Platform and Trust. Reasoning agents and deterministic workflows run inside a platform-enforced harness: scoped tools, isolated execution, resource limits, and circuit breakers. Approval gates execute below the model, out of band, where a prompt injection cannot reach them. Any workflow an agent generates runs in Blink’s isolated execution engine and can call only the abilities already assigned to it.

Delivery and scope. With 30,000+ integrations, Blink stays vendor-neutral across the tools you already own, extending across the SOC, IAM, vulnerability management, cloud security, and GRC. BlinkOps’ AI as a Service model pairs the platform with Forward Deployed Engineers (FDEs), security engineers who build agentic solutions with your team for any use case. In ASEF terms, ask which of your use cases are out of the box, configured, or built with FDEs, and record each one.

The result is a force multiplier for the team you already have. Agents take on investigation and evidence collection, so analysts spend more time on detection engineering and threat hunting. Then test it: the PoC steps above apply to Blink as much as to anyone else.

Conclusion

“Agentic” tells you what a vendor wants to be called. It does not tell you what the platform delivers, how much it does on its own, or what evidence backs either.

ASEF gives you a vendor-neutral way to answer those questions. Map your scope to the zones, record delivery, autonomy, and evidence for each capability, check Platform and Trust everywhere, and confirm it all in a PoC on your own alerts.

When the evaluation holds, your team scales with agents it can trust. Analysts spend less time in the queue, and humans still decide where the machine stops.

See How Blink Governs AI SOC Agents

Blink’s AI SOC runs agents inside a platform-enforced harness and logs every decision.

Explore Blink’s AI SOC
Right Arrow

FAQ

What Is ASEF?

ASEF, the AI SecOps Evaluation Framework, is a vendor-neutral framework from SecOps Unpacked for evaluating AI-enabled security tools and MDR services across the SecOps lifecycle. It records delivery, autonomy, and evidence for each capability, checks Platform and Trust in every zone, and does not produce a single ranking or blended score.

How Do I Evaluate AI SOC Platforms?

Use ASEF. Map your scope to its zones, record how each capability is delivered, score autonomy per action, and separate vendor claims from evidence. Then run a PoC on your own alerts with blind re-checks, query reruns, an out-of-scope action attempt, and an audit log review.

What Questions Should I Ask an Agentic SOC Vendor?

Ask whether each capability is out of the box, configured, or customer-built, and what is still on the roadmap. Ask which actions need approval and who sets the thresholds. Ask whether you can rerun every query behind a verdict, whether action limits are enforced by the platform or written in a prompt, what identity the agent uses, and what happens when a decision is wrong.

What Is the Difference Between an AI SOC and Agentic Security Operations?

In BlinkOps’ framing, an AI SOC applies agents to alert triage and investigation. Agentic Security Operations (ASO) extends agents across the SOC, IAM, vulnerability management, and GRC, and an Agentic Security Operations Platform (ASOP) builds, runs, and governs them. Deterministic workflows and reasoning agents work together under the same controls.

What Criteria Matter When Buying AI for Security Operations?

In ASEF, three things matter for every capability: how it is delivered, how autonomous it is, and what evidence backs it. Platform and Trust is checked across all of them. If agents will take response actions, put the most weight on autonomy controls and Platform and Trust.

Should AI SOC Agents Be Allowed to Take Containment Actions Autonomously?

Only within limits the platform enforces. OWASP recommends human approval for high-impact actions, and isolating a host fits that description, which puts it at 1G on the ASEF autonomy scale. Lower-risk actions can run at 1A, with a human on the loop, if every action is logged and the agent can be stopped. That is controlled autonomy, not a fully autonomous SOC.

See it in your environment.