Automated red-teaming is having a victory lap. AI systems that probe other AI systems for vulnerabilities now beat human red-teamers on cost, on speed, and on raw count of findings — and a growing chorus argues the humans are now redundant. A short, sharp paper says not so fast. “A Translational Note on AI Safety Evaluation,” by Madhava Gaikwad — submitted to arXiv (2609.06573) on September 6 and accepted to the 2nd Workshop on Safe AI at UAI (SafeAI 2026) — argues the victory lap confuses two different things: how thoroughly you search, and how much of the harm space you search over.
At six pages, the paper is a note rather than a tome, in the honorable tradition of translational writing: taking an idea the field half-knows and stating it precisely enough to act on. Its central concept has a name worth remembering: the threat-model coverage gap.
The argument in one paragraph
Safety benchmarks, Gaikwad observes, score attacker effort — how cleverly and exhaustively an automated adversary explores a predefined set of harms. A higher score means the search was more thorough. It does not mean the harm set was the right one. Beating humans on “attacks found per dollar” within a fixed threat model tells you nothing about the harms outside that model — the ones nobody thought to include, the ones that surface in deployment contexts the benchmark never imagined. Thoroughness is not coverage, and conflating them is how you get a perfect safety score on a model that fails in the wild.
An analogy helps. Imagine testing a building’s security by hiring someone to pick the front-door lock, then declaring the building secure when they fail — while the ground-floor windows stand open. Automated red-teaming, at its current best, is an extraordinarily thorough lock-picker. It tries more keys, faster and cheaper, than any human team. But the threat model — “attacks come through the front door, in English, in these predefined harm categories” — is the part nobody stress-tests, because stress-testing the threat model is not what the benchmark measures. Gaikwad’s point is that the windows are the threat model, and we keep grading the lock-picker.
Where the gap bites: language
The paper makes the abstract concrete with a finding that should embarrass the field: harms surface in non-English prompts that English-language benchmarks systematically miss, demonstrated on a current open-weight model. This is not a hypothetical. The overwhelming majority of safety evaluation is conducted in English, while deployment is global — and adversarial prompts in other languages exploit exactly the uneven safety tuning that English-centric evaluation cannot see.
Think about what that implies for the leaderboard culture. A model can top every English red-teaming benchmark while remaining trivially jailbreakable in Hindi, Swahili, or Indonesian. The benchmark did not lie; it measured exactly what it claimed. It just never asked the question. Multiply that blind spot across every dimension benchmarks predefine — harm categories, modalities, cultural contexts, agentic tool-use settings — and the “coverage” of even the best evaluation suite starts to look like a small, well-lit room in a large dark building.
The economics favor the wrong thing
There is a structural reason the field keeps optimizing thoroughness over coverage: thoroughness is cheap to measure and coverage is expensive to even define. An automated red-teamer that finds 10,000 jailbreaks per dollar produces a number that goes up on a chart — legible progress, publishable results, a leaderboard that moves. Expanding the threat model — hiring speakers of twelve languages, studying deployment contexts you have never operated in, imagining harms outside your cultural frame — produces no chart. It produces judgment, slowly, at human cost. Benchmarks, like all metrics, get optimized; what gets optimized is what gets measured. The coverage gap is what doesn’t get measured.
What would close the gap
Gaikwad’s prescription is institutional rather than technical: closing the coverage gap requires evaluators whose deployment context differs from the developers’. The people who decide what counts as a harm cannot be the same people who built the system, because they share the same blind spots — the same language, the same threat imagination, the same incentives to define the problem as the one they already solved.
This lands pointedly in a month full of safety claims. When Mistral announces that Large 4 was red-teamed with security partners and state authorities, the natural question after reading Gaikwad is: red-teamed against what threat model, in whose deployment context, and in which languages? Those are not gotcha questions — they are the questions the paper says we should be asking every time.
Why six pages can matter more than sixty
The paper’s brevity is part of its force. It does not propose a new benchmark, a new metric, or a new framework — the field has enough of those. It proposes a distinction: between search thoroughness and harm coverage, between the map and the territory. Distinctions like that, once stated clearly, are hard to unsee. Every safety evaluation you read after this paper will invite the same question: this measures how hard they looked — but did they look in the right places? Until the field can answer that, the humans are not dispensable. They are the only ones who know where the dark rooms are.
None of this is an argument against automated red-teaming — the paper is explicit that beating humans on cost and count is genuine progress. It is an argument against declaring victory. The mature posture is portfolio thinking: automated systems for breadth and speed, humans for the threat-model imagination that automation cannot supply, and institutional separation between the builders and the evaluators so the blind spots don’t correlate. Six pages, one distinction, and a research agenda that could occupy the field for years. Not bad for a note.
Featured image: a magnifying-glass icon (AlphaZeta, via Wikimedia Commons — CC0).
