Security is a token budget problem: a guide to out-searching your attackers
Hacktober and (the first half of) Breachvember have been a clarifying period of almost daily security incidents affecting some of the internet’s most foundational infrastructure (PHP2Shell, Node.js, LiteLLM, Railway, eMudhra Certificates, OpenSSH etc.).
The whole industry is rapidly coming to terms with the fact that the threat model has changed radically. With the cost of finding and exploiting vulnerabilities trending towards zero for our adversaries, all of us are more exposed than ever. Defenders are having to rethink how they keep infrastructure secure.
Based on what we’ve learnt from our customers, many of whom are defensive cyber firms, and our own defensive work, we have some thoughts on how to spend tokens most effectively. This blog is a guide to proactive cyber defence: a framework for thinking about the problem, and a guide on how to spend your tokens to solve it.
We model vulnerability discovery as a search problem. It has a few characteristics that make it well suited to being treated as a search problem:
- The vulnerability is somewhere in a vast space of code paths and configurations; you can’t deduce the vulnerability from first principles.
- Vulnerabilities are verifiable: it is cheap and easy to confirm one when you find it, meaning false positives don’t matter much.
For an attacker, finding just one vulnerability is sufficient: success means just one correct hit, even if there are thousands of attempts that fail. So the only way for a defender to stay safe is to out-search their attacker. A defender has to find, and fix, the vulnerabilities their attacker could find, before the attacker does.This article is about proactive defence. Reactive defence isn’t best modelled as a search problem. Reactive defence is something you hope you’ll never need; proactive defence is something everyone needs.
How to out-search your adversaries? Use more tokens.
It has become clear that agentic systems solve search problems more efficiently than any other system we have invented. In fact, many of the most incredible scientific and mathematical discoveries that frontier models have made came in part from relentless and deep search of the possibility space. Vulnerability discovery is no different. To find vulnerabilities more reliably, we need to spend more tokens. The question is, how should we spend this token budget?
Run agents for longer? Run more of them? Run them more often? The answer is yes.
Run agents longer
This is the classic form of test-time scaling. AISI has demonstrated that on multi-step cyber-attack scenarios performance scales log-linearly with inference-time compute. Notably, they identified that there was no noticeable plateau when they stopped the test at 100M tokens.
Figure 1. Average number of steps completed on “The Last Ones”. Source: Measuring AI Agents’ Progress on Multi-Step Cyber Attack Scenarios, AISI.
A similar pattern shows up in a recent open-weights study on CTI-REALM, where DeepSeek-v4-flash’s score on a security detection task rose from 0.51 to 0.62 when its cap on agent turns was raised.
So in your quest to out-search your adversary, run every agent until it stops making progress. The bug you’re looking for might be a few steps away. And when an agent does get stuck, start a fresh one, which brings us to the next point.
Run more agents
Running more agents independently (or in a swarm) has been shown to be an excellent way to solve search problems. Let’s think about why this would be the case first, and then look at some real-world examples showing it works in practice too.
The chance of finding a vulnerability is approximately
where
- is the chance any single attempt succeeds
- is the number of attempts
Given how good models now are at finding vulnerabilities, we can assume is above 0. So the higher is, the more likely a vulnerability will be found. And, once you’ve found that first vulnerability, finding a second and a third vulnerability is also driven by how big is. The more attempts that you have, the more likely that one of your attempts will find a previously unknown vulnerability.
The interesting thing about this formula is that the chance of missing a vulnerability shrinks exponentially as grows. That means matters much less than you’d think. Run a few hundred attempts, and a model that succeeds 10% of the time and one that succeeds 30% of the time will both almost certainly find the bug. What matters is clearing the floor, then running at scale. This can be modelled with Bernoulli trial reasoning.
Figure 2. Bernoulli trial modelling for vulnerability scanning.
This makes sense intuitively, but it has also been shown time and time again to work in practice:
- Sean Heelan ran o3 100 times against the Linux kernel’s SMB implementation. It found a known benchmark bug in only 8 of 100 runs, falling to 1 in 100 when given more code, and a previously unknown remote zero-day (CVE-2025-37899) turned up in the output of those runs.
- Anthropic ran Mythos 1,000 times over OpenBSD. The most critical finding was a 27-year-old bug that let an attacker remotely crash any OpenBSD machine, and the same runs turned up several dozen other vulnerabilities. Anthropic’s lesson was that the models’ scalability lets them search essentially every important file, including ones a human would skip on the assumption someone must have checked already.
- Ramp ran 10,000 agents to find vulnerabilities in their core backend. Rather than giving them all generic instructions across the code base, each agent was given one endpoint to interrogate. They found thousands of low-severity issues and seven high-severity bugs.
We see this pattern with our own customers, who regularly run swarms with 2,000+ concurrent agents targeting the same core software.
Because vulnerability discovery behaves broadly like a series of independent trials, and false positives don’t matter because of how cheap it is to verify, what matters most is how many you can run.
Search continuously
Defenders need to use these tokens constantly. Your product surface and model capabilities are constantly changing. So if you search once a quarter, once a month, or even once a week, you are not out-searching your adversaries. Out-searching your adversaries requires constant paranoia, scanning, and pen testing. Take sshole. The bug had sat in OpenSSH for a decade, despite continuous fuzzing and static analysis, until Özgün Kültekin of Wakefield found it using AI agents. Public exploit code appeared within about four hours of the fix, and attackers were exploiting it in the wild within two days. OpenSSH’s maintainers now skip long embargoes because AI tools rediscover reported bugs within hours. Be scanning constantly.
Defence is a token budget problem
Put the three ways of spending tokens together and the picture is simple. Out-search your attacker and the odds tip decisively in your favour. Every extra step, every extra agent, every extra scan is more tokens, and every one of them raises your odds of finding the vulnerability first.
Most security teams have a fixed budget. So the deciding question becomes what each token costs, and how much search it buys you. Which brings us to this question: Which model should you spend these tokens on?
Does it matter which model you use?
Less than you’d think. Models need to clear a capability floor to be useful, but in our experience most models released in the last six months clear it. Once they do, the right question isn’t which model is smartest, but which finds the most vulnerabilities per dollar. Go back to our formula. If is the chance any single attempt succeeds, then:
A smarter model raises . A cheaper model raises attempts per dollar. You only care about the product.
On that measure, cheaper models often win. In Ramp’s tests, Kimi K2.6 found their high-severity bugs 37% of the time against GPT-5.5’s 52%, but at roughly a fifth of the price per token. That makes it about 3.5x cheaper per finding. The CTI-REALM study points the same way: the best open-weight model matched the strongest closed models on a security task at under a tenth of the cost. For a fixed token budget, that’s a lot more search.
Which models you combine matters too. Runs from the same model share the same blind spots, and some of Ramp’s bugs were only ever found by one model. There’s no single right answer here, and much depends on the harness a defender has built: some of our customers get great results from a single model family (GLM is the most popular), while others combine several.
Finally, the model has to actually do the work. Closed models’ safeguards can refuse legitimate defensive work, and every refusal is wasted budget. Open models are less likely to refuse, and models with safeguards removed, offered only to vetted defenders, avoid the problem entirely.
Source: Unconstrained CTI-REALM with Open-Weights.
The defender’s advantage
Everything that has happened over the last few months might sound bleak for defenders: attackers need one bug, and defenders need to find them all. But, as Dan Robinson from Detail.dev commented earlier this week, defenders have something attackers don’t. They have the code, the configuration, the architecture, and knowledge of how the system is actually deployed. And they can proactively search and remedy before anything ships. Shopify described exactly this approach, searching with inside knowledge, in their write-up on proactive defence just this week (Chaos Monkey).
Go back to the formula. Defenders can win on both terms. Their knowledge of the system raises : each token spent by someone who can read the source is far more likely to find a vulnerability than one spent by an attacker probing from outside. And attempts per dollar comes down to what each token costs. Attackers need one bug, defenders need all of them, so defenders need the bigger product. That makes the cost of each token the deciding factor.
That’s the bet we made when we built Doubleword. We believed the biggest volume of tokens would come from long-running background agents, where depth and width of search matter more than speed, so we optimised our inference stack for tokens delivered per dollar. Over the last three months, defensive cyber has become one of the first genuinely high-value use cases for those agents. Around 60% of the tokens we serve now go to proactive cyber defence, and that share is still climbing.
About Doubleword
Doubleword is an open model inference provider for long-running background agents. Doubleword delivers higher throughput and concurrency for a lower token cost than other providers. A leading pen-testing org reported identifying 5x more vulnerabilities per day after moving to Doubleword thanks to the significantly higher concurrencies they were able to utilise.
What Doubleword is doing to empower defenders
- Significantly lower cost per search. We optimise our stack for high-throughput use cases, which allows us to offer a significantly lower cost per search than other providers.
- Scale to 1000s of parallel agents. Our customers have scaled from 0 to 2500 parallel agents without hitting rate limits, which allows them to find more vulnerabilities faster.
- Access the models your adversaries are using. Doubleword offers frontier abliterated models to select, vetted customers, giving them access to the same technology their adversaries are using. To apply for access to these endpoints please email [email protected].
@misc{doubleword-security-is-a-token-budget-problem,
title = {Security is a token budget problem: a guide to out-searching your attackers},
author = {Meryem Arik},
year = {2026},
howpublished = {Doubleword Blog},
url = {https://blog.doubleword.ai/security-is-a-token-budget-problem},
}