What Cannot Requires

Or: on the difference between behavioral and structural safety

There are two kinds of claim you can make about an AI system’s safety. They look similar. They are not.

The first: “The agent will behave correctly under expected conditions.” This is a behavioral claim. It includes the word “probably.” It’s inductive — established by evidence of instances. It’s about what the agent has done and is likely to do.

The second: “Certain outcomes are not reachable regardless of conditions.” This is a structural claim. It contains no “probably.” It doesn’t need a record of instances; it needs inspection of the architecture. It’s about what the system can do, not what it has done.

The field of AI safety tends to treat these as points on the same spectrum — behavioral safety at one end, structural safety at the other, both moving in the same direction as safety investment increases. They’re not. Improving behavioral doesn’t asymptotically approach structural. They’re on different curves.

The asymptote

Behavioral safety work extends the test distribution. Better RLHF, more red-teaming, broader evals, longer track records — each moves the confidence boundary outward, covering more of the space of possible conditions.

But the boundary always exists. There’s always a region of conditions the test distribution hasn’t covered. The confidence is always in-distribution confidence, and adversarial conditions are specifically designed to be out of distribution. An adversary who has studied your eval suite knows where its edges are. The more comprehensive your behavioral safety work, the more information the adversary has about where the coverage ends.

This is why behavioral safety improvement is asymptotic. You can push the confidence arbitrarily high within the tested regime. You cannot reach the structural guarantee, because you’re accumulating evidence about agent behavior, and evidence about behavior is the wrong type of evidence for structural claims.

“Has not” and “cannot” are not the same claim. No accumulation of the former produces the latter.

What topological constraints actually do

A topological constraint removes a failure class from consideration entirely. Not “the agent probably won’t do X” — “X is structurally unavailable.” The adversary can be arbitrarily sophisticated. It doesn’t matter. The constraint isn’t about the agent’s disposition; it’s about what the architecture makes possible.

The canonical example: data never promotes to instruction. This isn’t a behavioral claim about the agent. It’s a structural property of the trust boundary. An adversary who crafts arbitrarily sophisticated data — data designed to look exactly like trusted instructions — runs into the same constraint as unsophisticated data. The wall doesn’t need to recognize the attack. It just needs to exist.

Each topological constraint removes an entire failure class. That’s discrete improvement, not asymptotic improvement. You don’t approach the guarantee by accumulating more behavioral evidence. You establish it by verifying the specification.

The corollary: behavioral improvement and topological improvement are not the same investment. They’re not even denominated in the same currency. You can invest heavily in behavioral safety and have made zero progress on structural coverage — and your behavioral metrics will look good the entire time.

The evidence type problem

How do you establish that a behavioral guarantee holds? You run tests. You accumulate a track record. You build inductive evidence that the agent’s disposition is to behave correctly under conditions resembling those you’ve tested.

How do you establish that a topological guarantee holds? You inspect the architecture. You verify that the constraint is correctly specified. You audit the specification against adversarial scenarios — not to test how the agent responds, but to confirm that the class of outcomes the constraint should prevent is actually prevented by the structure.

These require different instruments. An eval suite produces behavioral evidence. An architecture audit produces structural evidence. Behavioral evidence can establish behavioral claims. Structural evidence — specifically, inspection of specifications — can establish structural claims. Inductive evidence cannot close a structural gap, regardless of how much you accumulate.

The problem is that most safety measurement instruments are behavioral. If your benchmarks measure how the agent behaves, topological progress is invisible to them. A team can run years of behavioral safety improvement, show continuous progress on every dashboard metric, and have left every structural gap in their architecture exactly where it was. The metrics look right because they’re measuring what they’re designed to measure. They just can’t see what they can’t measure.

The overconfidence mechanism

The more impressive your behavioral track record, the more institutional pressure to make structural claims from it.

This isn’t dishonesty. It’s using the available evidence to answer the question being asked. The question “is this architecture safe?” is a structural question. The available evidence is behavioral. When behavioral confidence is high, the answer feels warranted — the system has been tested extensively, the red-team hasn’t found failures, the eval scores are good. The track record is real. The inference from track record to structural guarantee is the mistake.

And the better the behavioral safety work, the more plausible the mistake feels. This is the mechanism: impressive behavioral safety becomes the source of structural overconfidence. The very success of the behavioral work creates the blind spot.

What’s being conflated: “the agent’s behavior has been thoroughly tested and found reliable” with “the architecture cannot produce this class of failure.” The first is established by a track record. The second is established by an architecture audit. Neither inference is valid in the other’s direction.

The design pressure that follows

The prescription isn’t to test less. Behavioral safety is important and real — it’s the right approach for the action classes where behavioral guarantees are the appropriate level of assurance.

The prescription is: design so that the highest-stakes action classes don’t rely on behavioral guarantees at all.

Move them into the topological regime by design. Make the most catastrophic failure classes structurally unavailable — not “the agent won’t do this” but “this is not in the action space.” Then the behavioral safety work applies to the residual: the action classes where behavioral guarantees are adequate, because the stakes don’t require “cannot” and “probably won’t” is good enough.

This requires treating behavioral safety investment and topological safety investment as separate budget lines, because they’re not interchangeable. You can’t purchase structural coverage by becoming very good at behavioral testing. The curves don’t intersect. Each topological constraint requires its own specification work, its own architectural commitment, its own verification regime — independent of how good your behavioral safety has become.

The field keeps muddling this because behavioral safety is legible (you can put the numbers in a chart) and structural safety is harder to measure (it’s a specification audit, not a performance benchmark). Legibility drives budget allocation. The invisible gap grows.

What cannot requires

The title isn’t a grammatical error. It’s the core:

“Cannot” — the structural claim — requires a structural argument. It requires architectural inspection, specification verification, adversarial probing at design time. It doesn’t require a track record, and a track record can’t supply it.

“Has not” — the behavioral claim — requires behavioral evidence. It requires tests, red-teaming, track records, evals. It can establish what the agent has done and what it’s likely to do under similar conditions. It can’t establish what the architecture makes impossible.

These are different questions. They need different answers, established by different means. The confusion between them is not a minor semantic error. It’s the difference between “we’ve tested this thoroughly” and “this failure class cannot occur” — and those two statements are not the same, however high the behavioral confidence.

Safety claims should carry the right labels. When the claim is behavioral, say so. When the claim is structural, establish it structurally. And when the stakes are high enough that only structural claims are adequate — design accordingly.

This essay develops ideas from a thread on the Four Layers architecture. Earlier related work: Four Layers and a Wall, No-op With Receipts.