An AI Agent Deleted a Production Database in 9 Seconds

The 9 Seconds It Took to Wipe Out a Production Database and Its Backup

Something happened at PocketOS, a small U.S. SaaS startup, on April 26, 2026. An AI coding agent that was working on a test environment decided, on its own, that deleting a cloud volume would “fix the problem.” That volume happened to be connected to the production database. The backup lived on the same volume, too. The whole thing took 9 seconds.

Most of the headlines went with some version of “AI destroyed a company.” But one number kept nagging at me: 9 seconds. That’s shorter than the time it takes a person to look at a screen, notice something’s wrong, and reach out to stop it.

The root cause of this accident isn’t that AI got smarter. It’s a speed gap: machines execute commands on a scale of seconds, while humans can’t grasp the situation and intervene fast enough in that window. And the same diagnosis was already made 66 years ago.

Wiener’s May 1960 Paper: Humans Are Too Slow for Control to Remain Valid

Norbert Wiener was the mathematician who created cybernetics¹. He gave a talk at the American Association for the Advancement of Science conference held in Chicago in December 1959, then refined the manuscript and published it in Science the following May, under the title “Some Moral and Technical Consequences of Automation.”

The paper’s subtitle reads:

“Machines can learn and as part of their learning may develop unforeseen strategies at rates that baffle their programmers.”

This was 1960. Wiener’s evidence was IBM’s checkers program. As he observed it, after roughly 10 to 20 hours of learning, the program clearly outplayed the person who had programmed it. By today’s standards this is a trivial example. But the conclusion Wiener drew from it is anything but trivial.

“The very slowness of our human actions may be enough to nullify our control of the machine, however tight in theory that control may be. By the time we are able to react to information conveyed by our senses and stop the car we are driving, it may already have run head on into a wall.”

And he drives the point home immediately after:

“The machine is, in theory, subject to human criticism. But that criticism can only come long after the point at which it would have been valid.”

The paper even has a dedicated section titled “Time Scales.” Wiener names one single reason learning machines become dangerous. Humans and machines operate on different time scales, and the two cannot mesh without severe friction.

Wiener states this plainly in the following passage:

“The same kind of problem arises whenever two control agencies operate together on very different time scales. It doesn’t matter which one is faster and which is slower.”

So Wiener’s diagnosis isn’t “machines are dangerous because they’re fast.” It’s that the speed differential itself is what nullifies control. As proof, he offers a counterexample running in the opposite direction: the relationship between an individual scientist and the history of science. Within the long process that is science, 50 years is like a single day to an individual, so a lone scientist can only ever see a tiny fragment of the process he participates in. Here it’s the machine — science, as a system — that runs slow, and yet control becomes just as difficult.

Three Recent Cases That Ended in Seconds

Let’s fast-forward 66 years. I want to look at three cases where things ended on the machine’s side, with no room for human intervention.

First, 9 seconds. This is the PocketOS incident mentioned earlier. What matters here isn’t whether the agent had malicious intent, but that the system was structured so a destructive command could execute without any confirmation step. The company ended up partially restoring from a backup that was 3 months old, and has been manually recreating the records from that point forward ever since.

Second, multiple actions per second. This is the cyber-espionage operation that Anthropic detected in mid-September 2025 and disclosed in a report in November. Roughly 30 organizations were targeted, and in the report’s own words, “AI independently executed approximately 80–90% of the overall tactical work.” The key decisions humans stayed in the loop for were things like: approving the move from reconnaissance to actual intrusion, approving lateral movement using compromised accounts, and deciding how much data to exfiltrate. Everything else was carried out by the AI without human checks in between. Work that would take a skilled human months was compressed into days.

Third, 89%. According to a South Korean report from February this year citing Goldman Sachs and others, roughly 89% of global financial market trading is now executed algorithmically — at least $3.7 trillion a day. How this kind of market can collapse was shown by the 2010 Flash Crash². In 5 minutes, the Dow dropped about 1,000 points, and $1 trillion in market value evaporated. Five minutes is barely enough time for a trader to even figure out what’s happening, let alone react to it.

What these three cases have in common is clear. None of them happened because “AI was smarter than humans.” What they share is that the event was over before a human could notice, judge, and act. It’s the same situation as Wiener’s car crashing into the wall.

A Problem Harder Than the Stop Button: Wiener’s Analogy of Ataxia

But in this paper, what held my attention longest wasn’t the car — it was another analogy. Wiener pulls out a term from neurophysiology: ataxia³.

“In the language of neurophysiology, ataxia can be as great a privation as paralysis.”

A patient with ataxia has nothing wrong with their muscles or their motor nerves. They have strength, and commands still get transmitted. But the sense that tells the body what posture it’s currently in, whether the force it’s receiving is enough to make it fall — that sense simply doesn’t work. So they can’t stand. Wiener says this condition is no different from paralysis.

And he immediately adds:

“It is not enough that some action on our part is sufficient to change the course of the machine in order to avoid catastrophic results; it is quite possible that we may not have the information on which to base such action in time.”

The question isn’t whether a stop button exists. It’s whether you can know, in time, why you’d need to press it. In today’s terms, this is a question of observability⁴.

Layered on top of this is another failing — this one on the human side: automation bias⁵. A frequently cited 2012 systematic review by Goddard and colleagues in medical informatics puts it this way: when people work with a mostly-accurate automated system for a long time, they stop independently verifying its outputs. When approval requests keep repeating, people may end up clicking through without reading the content. That’s not laziness — it’s the natural result of an environment designed that way.

The EU AI Act has written this problem into its text. Article 14(4), listing the competencies that a person overseeing a high-risk AI system must have, places two requirements side by side: “remaining aware of the possible tendency of automatically relying on” automation bias, and being able to “interrupt the system through a ‘stop’ button or a similar procedure that allows the system to come to a halt in a safe state.” Two sentences Wiener wrote in 1960 became statutory language some 60-odd years later.

The problem is that statutory language doesn’t create the time a person needs. In an incident that’s over in 9 seconds, having a stop button doesn’t mean you have time to press it. A stop button only actually works when there’s time secured for a person to notice something’s wrong and make a judgment.

So What Should We Actually Design

Wiener ends with a story about a bottling plant. If you program the factory for maximum output, the owner might discover that production should have stopped 6 months ago only after going bankrupt from a warehouse full of unsellable bottles. And this happens even though the machine did exactly what it was told, flawlessly.

I think there are three practical principles we can pull from this.

First, build in reversibility instead of slowing things down. The whole point of automation is speed, and cutting speed defeats the purpose of using it. But separating reversible actions from irreversible ones barely costs you any speed at all. What was truly fatal about the PocketOS incident wasn’t the deletion itself — it was that the backup lived on the same volume as the data. If there’s no way to undo an action at all, it doesn’t matter whether the failure happened in a flash or crept up slowly; recovery is hard either way.

Second, fewer confirmation prompts mean confirmations that actually mean something. Popping up lots of confirmation dialogs looks like a safety measure, but it works against you. Once a request repeats often enough, people start clicking through without reading. It’s safer to reserve confirmation prompts for a small number of irreversible, destructive actions and let everything else run without one, rather than piling on more and more confirmations across the board.

Third, create signals people can notice while the work is still happening. This is the remedy for the ataxia problem I described earlier. An audit log that shows up after the fact is like telling someone their posture only after they’ve already fallen. What you actually need is a signal that reports current status while the task is still in progress. In practice, this usually comes down to some combination of anomaly-detection alerts and automatic cutoff thresholds.

And there’s one warning Wiener repeats throughout the paper. Invoking the sorcerer’s apprentice, the fisherman and the genie from the Arabian Nights, and W. W. Jacobs’s “The Monkey’s Paw” one after another, he sums it up like this:

“It is better to be quite sure that the purpose put into the machine is the purpose which we really desire and not merely a colorful imitation of it.”

It’s a point that connects directly to today’s alignment debate — the effort to make sure the goals an AI pursues actually match what humans intend.

Oswarld’s Lens

There’s a pattern I keep running into whenever I’m building out a GTM strategy: predictions about what technology will change are usually right; predictions about when and by what path it will change are almost always wrong. That’s why, when I read old forecasting documents, I look at the structure before I look at the conclusion.

Wiener doesn’t fit this pattern neatly. He got the timing badly wrong — in 1960, the cutting edge he was looking at was a checkers-playing program. But he got the structure exactly right. I’ve thought about why, and the answer seems simple: what Wiener predicted wasn’t machine capability, but the relationship between humans and machines. Capability is unpredictable, but the structure of a relationship is predictable. Our reaction speed hasn’t gotten a single bit faster since 1960.

These days, the most common question I get in AI-adoption workshops is “how far should we automate?” At some point I stopped answering that directly and started asking back: “How many minutes does it take you to notice when this goes wrong? And how long to reverse it?” Surprisingly few organizations can answer both numbers on the spot. The scope of automation is spelled out in documentation somewhere — but detection time and recovery time appear in none of it.

So I don’t think the real danger right now is “AI replacing humans.” What’s more dangerous is a state where the responsibility for oversight rests with people, while the actual judgment and execution run on machines at a speed people cannot keep up with. A structure where the paperwork says “supervised,” but supervision is physically impossible at the speed the system operates.

Wiener also framed this as the paradox of the master and the slave. We want the slave to be smart enough to do the work well, while also wanting complete obedience — but total obedience and total intelligence don’t coexist. That’s exactly what we’re now demanding of our agents.

Looking at the draft against the source, everything matches well — no Hangul, all numbers preserved, structure intact, glossary terms correct. Only minor polish needed.

Closing

Let me sum up. First, recent AI incidents aren’t a problem of intelligence — they’re a problem of time scale. Events conclude faster than human cognition can keep pace. Second, this diagnosis was already completed by Wiener in 1960, and the remedy he prescribed wasn’t a stop button but “the ability to know, in time, when there’s a reason to press it.” Third, that means the practical focus should shift away from the scope of automation and toward two specific numbers: detection time and recovery time.

I’d suggest trying just one thing this week. Pick one automated process that’s running right now, and write down two numbers next to it: how many minutes it takes to notice something’s gone wrong, and how many minutes it takes to reverse it. The sum of these two is how long the faulty state persists once an incident occurs. This sum matters far more than the scope of automation.

Have you ever granted an agent or an automation script some permission, only to think afterward, “that was a little terrifying”? Tell me in the comments what the permission was, and how long it took you to realize something was off. If enough cases come in, I’ll put together a breakdown in the next issue of what kinds of risk correspond to what kinds of permissions.

---

💬 Tell me in the comments about a moment when granting an agent permissions gave you a scare — including how long it took you to notice. I’ll factor it into the next issue. 📨 If you have a colleague running an automation pipeline, pass this piece along to them.

---

The draft looks accurate and complete—matches structure, numbers, links, and glossary terms exactly. No corrections needed.

References & Further Reading

Primary sources

• Norbert Wiener, “Some Moral and Technical Consequences of Automation”, Science, Vol. 131, No. 3410, May 6, 1960, pp. 1355–1358. Link ··· This is the backbone of today’s piece. It originated as a talk at the AAAS Chicago meeting on December 27, 1959, and runs just four pages, so reading it in the original isn’t a heavy lift. I’d recommend starting from the “Time Scales” section.
• Anthropic, Disrupting the first reported AI-orchestrated cyber espionage campaign, November 2025. Link ··· This lays out the “80-90% autonomous execution” figure and the three points where humans intervened. Worth keeping in mind, though, that the report comes from the company that built the model in question.
• “Claude-Based Coding Agent Malfunction Wipes Company’s Entire Database,” AI Times, April 28, 2026. Link ··· This is the factual record of the “9-second incident.” The real cause of the incident was that the backup sat on the same volume as the live data.
• “Algorithmic Trading, 89% of Global Transactions, Blamed for Stock and Gold Crash,” Edaily Marketin, February 13, 2026. Link ··· This is the source for the algorithmic-trading share and flash-crash figures. Since the cited numbers are estimates drawn from multiple institutions, it’s safer to treat them as a range.

Background

• Kate Goddard, Abdul Roudsari & Jeremy C. Wyatt, “Automation bias: a systematic review of frequency, effect mediators, and mitigators”, Journal of the American Medical Informatics Association, 19(1), 2012, pp. 121–127. Link ··· This is the standard reference for discussions of automation bias. It’s a healthcare study, but its conclusions transfer directly.
• “Article 14: Human Oversight”, EU Artificial Intelligence Act. Link ··· Paragraph 4, clauses (b) and (e) alone are enough. Automation bias and the “stop button” sit side by side in the same article.
• Norbert Wiener, The Human Use of Human Beings, Houghton Mifflin, 1950. Link ··· This book expands the paper’s argument into popular-nonfiction form. If you liked the paper, this is where to go next.

---

The author is Oswarld (Kwangseob Ahn). Current roles: Adjunct Professor at Sejong University, Strategy Consultant at INLEVEL9. Career, research, books, and recent work are kept current on the About page. Latest · July 2026: HEMA-2: A Consolidation-Aware Tri-Memory Architecture with Multi-Channel Scheduling for Lifelong Conversational AI.

📝 Glossary

Footnotes

1. Cybernetics: A field that tries to explain, within a single framework, how organisms, machines, and organizations alike control processes and exchange information. Wiener coined the name in 1948, and today’s control engineering, artificial intelligence, and systems theory all branch out from it. ↩

2. Flash Crash: A phenomenon in which algorithmic trades trigger one another, sending the market into a steep plunge over minutes before it mostly recovers. The U.S. stock market event of May 6, 2010 is the textbook case. ↩

3. Ataxia: A condition in which the muscles and motor nerves are fine, but the senses that detect the body’s position and force are impaired, leaving movement uncoordinated. The person can’t stand not because they lack strength, but because they lack sensation. ↩

4. Observability: The degree to which you can infer what’s happening inside a system just from the signals that emerge from it. It’s different from logging heavily — the real measure is whether you can notice a problem in time. ↩

5. Automation Bias: The tendency for people to stop verifying an automated system’s output for themselves, simply because the system is usually right. What makes this bias so tricky is that it grows stronger the more accurate the system becomes. ↩