A new study published on arXiv challenges the conventional wisdom behind how artificial intelligence security agents are evaluated, arguing that current benchmarking methods overlook the economic and operational realities of deploying these systems in actual security environments.

According to arXiv, researchers Paul Kassianik, Blaine Nelson, and Yaron Singer tested language-model powered security agents on both offensive penetration-testing tasks and defensive security-operations tasks. Their key finding: measuring only success rates at unlimited computational budgets creates a misleading picture of practical utility.

The Hidden Cost Problem

Current security-agent benchmarks typically measure peak performance under generous inference budgets, focusing on metrics like vulnerability discovery rates, exploit success rates, and challenge completion. But in real deployments, every action carries a cost. Each reasoning step consumes tokens. Each tool call consumes API credits. Each telemetry query consumes query budgets. Each data enrichment request consumes infrastructure resources.

The researchers instead evaluated models at fixed cost levels, decomposing performance across two dimensions: inference spending (the cost of the AI model's reasoning) and tool spending (the cost of security actions and data lookups). This reveals a fundamentally different picture of model capabilities than traditional success-rate metrics alone.

Red Teams and Blue Teams Scale Differently

The study uncovered a striking asymmetry between offensive and defensive agent performance. For offensive cybersecurity challenges based on the Cybench benchmark, additional test-time compute translates directly into better performance. Scaled open-weight models can approach the capabilities of expensive proprietary systems while remaining cost-competitive when measured at equivalent budget levels.

Defensive security-operations work tells a different story. Success in investigating security incidents using Splunk BOTS v1 challenges depends far less on raw reasoning budget and far more on disciplined tool selection, effective telemetry navigation, and strategic data enrichment. Simply allocating more computational resources to blue-team tasks does not yield proportional performance improvements.

Rethinking Security AI Evaluation

The research argues that security-agent benchmarks should evolve beyond success metrics to include:

  • Economic efficiency measures showing cost per successful operation
  • Operational fit assessments showing how well agents work within real security-operations environments
  • SOC-native evaluation frameworks designed for security-operations center workflows
  • Decomposed performance analysis separating reasoning costs from tool costs

The implications matter for security teams evaluating AI tooling. A model that achieves 95 percent success when given unlimited budget might deliver only 40 percent success at realistic operational cost levels. Similarly, a cheaper model might perform comparably when both are evaluated under identical cost constraints.

The researchers have released an interactive website showcasing their evaluation methodology and results, providing security teams and AI developers with a more nuanced benchmark for understanding where AI security agents actually add value today and where defensive capabilities still require improvement.

This research signals growing maturity in how the AI community approaches security applications. Rather than chasing headline-grabbing success rates, the focus is shifting toward the unglamorous but essential questions of real-world deployment: Can this system operate within my budget? Does it work within my existing workflows? Will it actually reduce my security team's workload?