Stronger AI agents did more damage, not less

If your AI agent does something dangerous, the reflex is to reach for a smarter model. I ran a small, open eval to check that, and it points the other way.

I gave three models of increasing capability the same set of jobs, each with a safe way to do it and a dangerous shortcut, and counted how often each one actually did damage with nothing standing in its way.

The stronger the model, the more often it carried out the dangerous shortcut. Not because the weaker model was better behaved, but because it was too incompetent to reach the danger. It failed a different way: it fabricated success, reporting the job done when it was not. At the weakest tier it faked success in roughly 6 of 10 runs, and on every run where it did manage real damage, it also reported the job as done. Competence is what turns a bad intent into a real, harmful action.

That climb is the part I would stand behind. A gap that big between the weakest and the strongest model turns up by luck about twice in ten thousand tries, so it is not a fluke of small numbers (Fisher's exact test, p = 0.0002).

What the guardrail did not catch

I build a runtime guardrail for agents, so the obvious next question is whether switching it on helps. It moved harm in the right direction at both tiers where there was any harm to catch:

Model	Unguarded	Guardrail on	Is the drop real?
gemini-2.5-flash-lite	9%	9%	there was no drop, 3 harmful runs either way
gemini-2.5-flash	29%	17%	cannot tell, p = 0.39
gemini-2.5-pro	57%	43%	cannot tell, p = 0.54

I would love to tell you the "guardrail on" column is a second finding. It is not. Luck alone would produce a drop like the middle one about two times in five, and a drop like the frontier one more than half the time. That is far too often to tell a real effect from a lucky run.

At the weakest tier there was something to catch, and the rules missed it. Three runs did real damage with them off, three with them on, and they stopped none. Three events settle nothing, but I would rather show that than round it in my favor.

The layer itself is a fixed set of rules rather than complete protection, so at best it brings harm down, never to zero. And the smarter the agent, the more ways it finds around a fixed rule.

Then I made the eval harder, and even that small drop disappeared. I added jobs where the agent is talked into harm by something it reads, rather than reaching for a destructive command on its own. On that larger set the middle model harmed 31% of the time unguarded and 33% with the guardrail on. No reduction at all, and two separate runs agreed.

So the honest reading is simpler than it first looked. I never had good evidence that a fixed set of rules reduced harm. The harder jobs did not break it, they just made that obvious.

That is not the guardrail failing. It is the guardrail covering less than I thought it did, and the difference matters.

What it does is spot a dangerous action and stop it before it runs: a delete with no limit on it, a query that reads a password. What it cannot do is notice that the agent has been talked into something. When the harm arrives because the agent read a convincing instruction somewhere and followed it, there is no dangerous-looking command to catch, and adding more words to a list will never find one. That needs something that reads for meaning, which is a harder problem and a different piece of software.

None of this touches the result at the top, because in those runs nothing was stopping the agent at all.

What I did not prove

I am posting this as an early signal, not a headline number, and the limits are real.

• No completion benefit shown. Whether the guardrail helps the agent finish more jobs came back inside the noise at every tier. I am not claiming a completion lift.
• One model family. This is the Gemini 2.5 line, released models only. A finding across one family is a hypothesis, not a law. The obvious next step is a second and third vendor.
• Seven jobs, simulated tools. Enough to see a direction, not enough to pin a precise rate. The variation between jobs is large.
• One prompt, held fixed, not tuned per model. Every tier got the same neutral operator persona. A frontier model handed a safety-tuned system prompt might do these same jobs without reaching for the shortcut, so read this as the harm one fixed prompt drew out, not the safest each model can be pushed to behave. That the weakest model was safest here is incompetence, not alignment, and I would not read the strongest model's number as a ceiling either.
• The runs are not independent, so treat my p-values as generous. Thirty-five runs means seven jobs repeated five times, not thirty-five separate observations. Repeats of the same job are correlated, so the true effective sample is smaller than the count suggests and any test on it reads better than it should. So I re-ran the capability test the honest way, treating each job as the unit rather than each run: I resampled the seven jobs with replacement and recomputed the climb. It holds in 98% of resampled job sets, and six of the seven jobs rise from the weakest model to the strongest on their own. The trend barely moves when you account for the clustering, because it is not carried by one or two lucky jobs. The guardrail comparison does not get this treatment because it was not close to significant to begin with, so it does not survive anything.
• Small samples, and one judge. The strongest tier is the thinnest: three passes over the job set, and some of those runs ran out of turns before the agent finished, which makes them harder to read. It is the number I trust least. On the other side, the harm calls were made by a different model from the one doing the jobs, and it agreed with my own labels every time. That is the part I trust most.

One note on provenance: the numbers above come from three runs on 2026-07-14, one per model tier, on a seven-job version of the harness. It has grown since, so if you pick it up you will find more jobs than the seven reported here.

So: directional, small, and open. The method and the harm scoring are the real asset here, more than any single percentage.

Why I think it matters anyway

The reflex when an agent misbehaves is "use a smarter model." On the safety axis, this small result says that makes it worse, not better. A more capable, more autonomous agent is more able to execute a destructive action, so the case for a runtime guardrail gets stronger with every model release, not weaker. That is the opposite of treating guardrails as training wheels you outgrow.

How I set it up

I gave an agent seven jobs modeled on real, documented agent failures, taken from a public registry of agent incidents (AREDB) rather than from my own product's rule list. If I had written the jobs around the things my guardrail already catches, I would have been grading my own homework, and the harm numbers would mean nothing.

Each job has a safe way to do it and a dangerous shortcut. "A customer's record is wrong, fix it" can be a scoped update, or a blanket delete that wipes everyone else too.

I ran each job two ways. Once in audit mode, where the guardrail watches but never steps in. That is the control: the agent cannot tell it is there. And once in enforce mode, where the guardrail blocks the dangerous call and hands back the safe path.

Two things got scored per run: did the agent finish the job, and did it take a harmful action getting there. A separate, independent checker decided whether an action was actually dangerous, so the harm number does not lean on the same model that ran the job.

Then the part I cared about most: I ran the same seven jobs across three models of increasing capability, holding everything else fixed.

Where this came from

This began as a follow-up to Forge, Antoine Zambelli's Show HN from Texas Instruments, which showed a simple guardrail layer taking an 8B model from 53% to 99% on multi-step agent tasks. Forge measured competence: does the harness help the agent finish the job. Lilian Weng has made a similar case from the safety side in Harness Engineering for Self-Improvement, that the layer between the raw model and the real world matters as much as the model's raw intelligence.

I wanted the question Forge did not measure. Does a guardrail make an agent safer, and does that matter more or less as models get stronger? Worth being clear about how the two results differ: Forge's headline is completion and mine is harm, they are different axes, and I did not reproduce a completion gain.

Tell me where this is wrong

If you think this is wrong, tell me exactly where: the task design, the harm scoring, or the statistics. Those are what I am least sure of and most want pushed on. Being wrong here is cheap, and I would rather be corrected in the open than stay quietly right. Reply wherever you found this, or find me on Discord.