When Machines Start Acting

Accenture had offices in more than fifty countries. It had hundreds of thousands of employees, a wall of blue-chip clients, signed contracts running years into the future, real revenue, and a market value in the tens of billions of dollars.

Then, for a few seconds on the afternoon of May 6, 2010, the market priced all of it at one cent a share.

Not a discount. Not a bad quarter. A penny. And then, just as fast, it priced the company back to normal, as if nothing had happened. This was not one strange stock. Across those same minutes the Dow fell nearly a thousand points and recovered most of it, close to a trillion dollars appearing and vanishing inside a quarter of an hour. We call it the Flash Crash.

Sit with that. No single component had to fail for this to happen. No bug, no rogue line of code, no malfunction in the ordinary sense. A large seller ran a program to offload contracts, set to chase volume rather than price. Fast trading systems did exactly what they were designed to do and passed the falling contracts between themselves. Every piece behaved correctly. Later analysis argued over how much the automated traders caused the crash versus amplified an imbalance already there, and that argument is worth having, but it points the same way: a serious system produced pure nonsense, and nothing inside it had to break for the nonsense to come out.

The question we do not ask

We ask, constantly, whether a system is safe. We almost never ask the question underneath it: how did this system earn our trust in the first place, and does that history decide the way it will one day betray us?

I think it does. That is the claim this whole essay rests on. How a system earns trust determines the shape of its failure.

This is not a new worry. Resilience engineering and systems safety have argued for decades that safety is a property of the whole, not the parts, and that complex systems fail through how their pieces interact, not through the breakage of any single piece. What is new is a change in what the pieces are. We are moving from machines that follow a fixed rule to machines that interpret a goal and choose their own way to it. And a machine that acts open-endedly fails in a different shape than one that merely runs its script.

The body keeps a different promise

Before the machines, consider a system you trust every day even though it fails constantly: your own body.

You trust it not because it never breaks. It breaks all the time. You catch colds, pull muscles, run fevers, scar. You trust it because whatever goes wrong, it tends to find its way back, which is the opposite of how we build the things we certify.

There are, roughly, two ways to make a system you can rely on, and they fail in opposite directions.

The first is design. You write down what must never happen and build the thing so it cannot. A bridge must not fall. A pacemaker must not stop. You name the forbidden states and prove, as far as proof reaches, that the system stays out of them. Its promise is absolute: I will not break. And a system built around that promise alone tends to fail absolutely too. It is correct right up until the one situation nobody wrote down, and then it does not bend, because nothing in it was ever meant to be half-right. Accenture at a penny is not damage on top of a working market. It is what a system that only knows "do not break" does the moment it steps outside the world it was specified for. It does not sag. It shatters.

The second is survival. You get knocked down ten thousand times, across an unimaginable stretch of time, and whatever cannot get back up is gone. What remains makes a humbler promise: I will come back. Its failure is humble too. Having broken a thousand times, the body holds a thousand routes back. It runs a fever, sheds load, limps, finds its way to balance. The failure is gentle and local, not sudden and total.

So graceful and catastrophic failure are not a matter of luck. They follow from how the system earned its trust. A system built only around correctness shatters when it finally meets what it did not foresee. A system shaped by recovery sags. You do not get to choose the failure shape apart from the method that produced it.

Of course, good engineering has always half-known this. We add redundancy, fallback modes, circuit breakers, manual overrides, degraded operation, precisely because pure correctness is too brittle. The point is not that a designed system cannot recover. It is that recovery has to be built in as a first-class promise of its own. It does not fall out of a specification of correct behavior for free. You get the recovery you designed, and nothing you did not.

I spent my early years learning to reason formally about how systems behave over time, and later years reading the living body as exactly that kind of system. The longer I sat between the two, the more that difference stopped looking like a detail and started looking like the whole story.

The new machines act

Here is the rupture, and it is easy to state imprecisely, so let me be careful. It is not that machines have started to act. Machines have acted for decades. The trading systems in the Flash Crash acted. So do autopilots, industrial controllers, pacemakers. Action at machine speed is old news.

The shift is in the kind of action. Those older systems acted inside a box someone drew: a fixed rule, a bounded task, a set of situations the designer enumerated in advance. The new systems act outside the box. They interpret a goal, choose their own steps, call tools, coordinate with other systems, and pursue objectives across contexts nobody explicitly listed. An agent does something open-ended: it books, it buys, it files, it writes and runs code, it hands work to other agents. The output is no longer a number or a sentence you can read and check. It is a chain of actions in the world, taken at machine speed, in situations no one wrote down ahead of time.

And these acting machines are increasingly wired to each other. Picture an ordinary hiring pipeline, a year from now. One agent screens the resumes. Another summarizes the interviews. Another drafts the rejections. Another updates the system and closes the loop. No single step is wrong. Each does a sensible, defensible job, and each would pass its own review. But run them together, month after month, and they can settle into a pattern nobody chose: the same kind of candidate quietly filtered at step one, a polite rejection auto-written at step three, a clean audit trail at step four, and systematic exclusion with no villain in it and no single moment where the harm began. The risk is not inside any one model, where you could test for it. It is in the chain, in the handoffs, in what the pieces do to each other.

Notice that the 2010 systems had locally simpler components. They were deterministic: same inputs, same behavior, every rule written down and in principle auditable. Each piece was about as specifiable as a fast system gets, and they still composed into a penny. The new machines are not deterministic rule-followers. They are probabilistic. The same prompt can produce different behavior, the reasons live in billions of weights no one can read line by line, and the output is a spread of possibilities, not a guarantee. A specification can still shape what one agent does on its own. What it cannot pin down is what the chain of them does together.

We certify components. They behave like ecologies.

When acting agents interact, they start to show what we call emergence. Be careful with the word: the strong version invites a fight the argument does not need. I do not mean anything mystical or unpredictable in principle. I mean something narrower and practical: behavior that is not reliably predictable from the specifications of the parts, under real deployment conditions. Agents fall into conventions, coordinate, herd, even collude, in ways nobody wrote and nobody asked for. The pattern lives in the interaction, not in any single agent you could inspect and sign off on.

This is the mismatch that should worry us. We certify these systems the way we certify components: one model at a time, in isolation, against a spec. But they behave like an ecology, and an ecology is exactly where component-level guarantees stop reaching, because the failure that matters is not knowable in advance at the level where the spec was written. It shows up a layer up, in the interaction, after deployment, in a form no one thought to forbid. The system has quietly become the second kind of thing, the survival kind, without any of the history that made the survival kind trustworthy.

Biology is the warning, not the blueprint

The tempting move here is to reach for biology as reassurance. Nature is full of emergence, the argument goes, flocks and colonies and immune systems, all self-organizing and resilient, so let us just copy it. Let the agents swarm.

A word on what this is and is not. Biology here is an analogy, not evidence. I am not claiming that agents are organisms, or that evolution proves anything about software. I am borrowing a pattern, because it is the one system we know that became trustworthy while staying complex, and the way it got there is instructive by contrast.

Biological emergence is not safe because it emerged. It is safe, to the degree it is safe at all, because it was filtered. Evolution does not optimize for safety. It optimizes for persistence, and over a vast stretch of time the patterns that failed fast and total tended not to survive to repeat, while the ones that failed slowly and survivably stayed. What is left is not optimal or danger-free. It is full of costly trade-offs and old survivable flaws. But it has been strained against reality for four billion years, and the quickest, most catastrophic failure modes were strained out. The filtering, not the emerging, is what earned the trust.

Agentic emergence has the wildness and almost none of the filtering. We are assembling the kind of thing that took the planet four billion years to make even partly trustworthy, and shipping it in a quarter. As an analogy, biology is not the blueprint here. It is the cautionary tale, and we are reading it as an instruction manual.

What we are actually doing

So the danger has a shape, and we can name it. A fast, never-break system meets the one situation nobody wrote down, and it shatters, all at once, before the slow and forgiving world around it can do the one thing it is good at, which is to come back.

The fix we reached for after 2010 shows we already understand this in our bones. The circuit breaker, the rule that halts trading when a price moves too far too fast, is nothing but graceful failure bolted onto a system that could otherwise only shatter. We taught the market to sag.

The natural question is what the circuit breaker looks like for an agent, and it is worth seeing why the answer is hard. The 2010 breaker worked because the failure was a single number crossing a single line. A price moved ten percent in five minutes. You could name that in advance, watch for it, halt on it. It was catchable because it was nameable. We do have things to reach for with agents: kill switches, rate limits, tool permissions, sandboxes, rollbacks, a human in the loop. They are worth building, and some will matter. But every one of them acts on the parts, or slows the whole down, or cleans up after. None catches the emergent failure the way a price threshold caught the crash, as a named event you can see coming, because the whole trouble is that this failure has no threshold to name. Those tools buy the slow world time. They are not a spec and not a proof, and we should stop pretending the next patch is around the corner.

This is why you cannot get system-level safety from component-level specifications alone. Much of the field still reaches first for specification: more guardrails, more evaluations, more written rules. That work is worth doing, and it constrains the parts. What it cannot do is guarantee the whole, and the whole is exactly where the failure we fear lives.

To be fair, the serious end of the field knows this and is moving. The most interesting safety work is no longer only about making a single model behave. It is about the system: monitoring the interactions, controlling what agents may do to each other, red-teaming whole workflows rather than lone models, building the scaffolding and incident response for when the composed thing misbehaves. That is the right direction. It is also an admission that the unit of danger has moved from the part to the whole, which is the entire argument here.

We were taught to fear a machine with a face. An enemy, a war, an intelligence that turns on us. The real thing has no villain in it. There is only composition, correct parts fitting together into a whole that does something none of them intended, faster than anything alive can interrupt.

We are now handing the acting machines the parts of the world that were always the forgiving kind: markets, hiring, courts, medicine, the movement of information itself. The question was never simply whether the machine is safe. It is whether we are dropping never-break things into a world that survives by recovering, and whether we have left that world enough time to come back.

---