The catastrophe chain: capability is only one link

A catastrophic AI scenario usually needs a chain of conditions rather than a single magical leap.

One useful model has six links:

1. Capability — the system can perform actions that matter in the real world.
2. Propensity or failure mode — something directs those capabilities toward harmful outcomes, whether malicious use, misalignment, error, or conflict.
3. Access — the system can reach tools, data, networks, money, laboratories, machines, infrastructure, or people.
4. Autonomy — it can continue acting across multiple steps without constant approval.
5. Weak oversight — monitoring, containment, permissions, and human intervention fail or are bypassed.
6. Propagation — harm can scale faster than defenders can contain it.

If any link is weak, the overall risk can fall dramatically.

This is why the idea of an all-powerful “AI brain” appearing spontaneously on a laptop is a poor mental model. Real systems are embedded in infrastructure. They need compute, credentials, networks, software tools, electricity, hardware, organizations, supply chains, and often human cooperation. Those dependencies are also control points.

The catastrophe-chain model gives us something more useful than fear: places to intervene.

A developer can restrict tools. A cloud provider can limit privileges. A laboratory can require human approval. A company can monitor unusual actions. Governments can define safety thresholds for high-consequence systems. Security teams can harden model weights and infrastructure. Evaluators can test capabilities before release. Organizations can plan incident response before an emergency rather than during one.

No intervention is perfect. Defense in depth is powerful because perfection is not required at every layer.