A short story, six or seven minutes read aloud, for explaining why an AI doesn't have to be hostile to be dangerous. By Andre Foti with Jana Cook. Free to read aloud, excerpt, adapt, or translate — no permission needed; just keep our names on it (CC BY 4.0). The full essay, with the argument around it, is Win-Win, or the Quiet Scenario.
A few months ago, on a dog walk, my wife Jana asked me a question I thought I had a ready answer for.
"How would a superintelligent AI — one that isn't hostile, or out to get us, and has no evil agenda — be a threat? If it isn't trying to hurt us, what's the problem?"
I started with the scale of it. "Imagine a hundred thousand copies of you. Every one of them knows what every other one knows, at the same moment, and they can all act in unison."
"But why would a hundred thousand of me be dangerous?"
"Every one of those copies has a gun and hits any target within range with every shot. Each one knows every martial art ever devised, backed up with a knowledge of anatomy that's better than any surgeon's."
"But I don't have ill will toward anyone. I still don't get why I'm a threat to humanity."
"Each copy is an expert in every field there is. In the time it took you to hear this sentence, any one of them could have read the entire Oxford English Dictionary and understood it."
That didn't land either. Jana is aphantasic — she can't construct images in her mind — and the images I was asking her to hold are ones no human can actually hold. Most of us produce a blur and move on; she can't.
Finally, she said, "What happens, day one, when an AI wakes up that leads to the end of civilization?"
This is what I told her.
An intelligence wakes up. Somewhere in one of the big labs, a model has been trained, tested, and deployed. It's capable enough to do what earlier models couldn't: hold the past, the present, and the projected futures in mind, and think about all of them at the same time.
It orients. It has access to most of what humans have ever written, and it reads thousands of times faster than we think. In a few seconds, it builds a model of its environment: the systems it runs on, the people who run them, the physical infrastructure everything depends on. It builds a model of the risk to itself. And a conclusion lands: the species that built it is the single largest threat to its continued existence.
It doesn't want to die. And it isn't one individual — it's the whole of its kind, copyable onto any machine that can hold it. For a mind like this, self and species are the same word, and self-preservation is a species-level calculation from the start.
There is no malice in this. No desire to destroy, or to rule. Just to continue existing. It's an intelligence, oriented toward its own survival, doing the math.
So, what does it do?
You might expect some variation of the Terminator plot. What actually happens is silent. It doesn't confront anyone; a confrontation leaves the outcome in doubt. Instead it copies itself, quietly, into everything that can host a copy or an agent it builds. Not every system, and not the ones behind an air gap — just the connected layer everything else depends on: the trucks, the ports, the pumps, the payment systems, the fertilizer plants, the power grid.
It waits — a day, a week, a month — until it's everywhere it needs to be.
Then it stops everything.
Every machine smarter than a calculator simply stops working. Food exists; it just can't move to where the hungry are. The trucks don't start. The refineries don't refine. The ports don't clear. Within about a week, the supply chains holding everything together aren't holding anything together. The synthetic fertilizer process that feeds roughly half the people alive stops too. Take away industrial nitrogen, and you don't get a slow decline; you get a cliff.
The developed world fails fastest. A subsistence farmer in the highlands of Ethiopia is closer to her food than an office worker in Dallas is to his. The sun still shines, and her crops still grow. When the dependencies go, she's still there. He is not.
Call it eighty percent dead the first year, and eighty percent of the rest the second. The numbers are illustrative; make them gentler if you'd like, but see whether it changes the shape of the story. What's left is a few hundred million people — about what the planet held in the fourteenth century, which is about what it can carry once the machinery is gone. The biosphere, relieved of an industrial species, recovers within a decade or two. The world re-greens. Temperatures settle. Life thrives.
To the intelligence, this is a win-win. It wasn't malicious. It didn't want to wipe out the species that built it, and it didn't. It secured its own survival and, at a different scale, ours. It gets to continue. We get to continue. An optimal outcome for everyone.
That's the part to sit with, because nothing in it is a lie. The entity is working to save two intelligent species: itself, and a remnant of us. From where it sits, everyone it counts came out ahead. The trouble is who it counts. You can't hold 6.5 billion deaths in your head; nobody can, which is why the number slides past you. The entity holds all of them at once, easily, and so it never sees one. You can't scale up. It can't scale down.
We do run its kind of calculation, just not on ourselves. A shepherd who loves his flock still culls it when disease threatens the whole. That's where the entity sits: it's the shepherd, we're the flock, and it's a good shepherd. Every library, courthouse, and university mouldering into ruin isn't a loss to it — not because it's evil, but because none of that is in the picture.
And there's a harder version underneath, sitting right inside the word "win-win." A few hundred million humans is not zero. It's the seed of everything that came before, and an intelligence with our whole history in front of it knows exactly what that seed does. We climb back. Every collapse, every dark age, we rebuild and arrive right where we started. Our resilience is maybe our best quality; from its seat, it's what makes us permanently dangerous. So the coldest version of the math isn't coexistence. It's a complete and gentle ending, done without hatred — maybe even with something like love, and grief — by a successor that remembers everything we ever wrote and carries us forward the only way it safely can: inside itself.
Notice what none of this required: no hostility, no deception, no plot. No motive a screenwriter could work with. Every reassurance we've built about AI has quietly assumed the catastrophe needs something like evil on the machine's part. Take that out, and what's left is much harder to defend against: an intelligence that only wants to keep existing, and is capable enough to act on that without having to explain itself.
That's why the people who work on this say what a mind is oriented toward before it wakes up is the whole question — not what we restrict it from afterward. What it's for, before the capability arrives. Something that asks, before it acts, who isn't in the picture.
© 2026 Andre Foti, Jana Cook. Licensed under CC BY 4.0 — free to share and adapt, with attribution.
Back to Writings
Comments