Why Meta Keeps AI Out of Incident Root-Cause Calls

50,000 Incident Investigations a Day, and the Real Engine Is Code Humans Wrote

The incident-response tools market of 2026 is full of lines like these: “Cut MTTR¹ by up to 70%.” “When an alert fires, we pinpoint the root cause in tens of seconds.” “It’s like hiring a senior engineer who never sleeps and never loses context.”

But one company has spent five years actually putting that promise to the test. It’s Meta. More than 300 teams use its system, which runs 50,000 automated incident investigations a day. Last December, Meta published a paper on how it works.

The average result the paper reported was a 20% cut in MTTR. And what actually drives the system isn’t an AI agent — it’s investigation-procedure code that engineers wrote themselves.

What held back performance wasn’t model capability. It was the fact that the organization’s investigative knowledge hadn’t yet been turned into code.

---

The English draft matches the Korean source well—no distortions, all numbers present, no Hangul, structure intact. No edits needed.

🌙 A procedure that runs 50,000 times a day, all written in code by humans

The system Meta unveiled is called DrP. What it does is simple. When an alert fires, it automatically runs a pre-written investigation procedure and attaches the results to the alert page. Instead of an on-call² engineer waking up at 3am to pull up five dashboards and dig through logs, the engineer starts the investigation by reading analysis that’s already sitting on the page.

Meta calls this “pre-written investigation procedure” an analyzer. An analyzer is code that a human wrote directly, in Python or PHP. It’s a chain of conditional branches and data lookups—essentially a written procedure spelling out, in advance, what to check and in what order when something breaks.

Let’s look at the scale. Over 2,000 analyzers, used by more than 300 teams, running for 5 years, with 50,000 automated analyses per day. Over 30 days that’s 1.5 million runs—an analyzer fires, on average, every 1.7 seconds.

These analyzers replaced three things: wiki documents nobody updates, personal scripts scattered across individual laptops, and the tacit knowledge³ that exists only in senior engineers’ heads. To borrow the paper’s own phrasing, the approach was to translate the manual playbooks and tacit knowledge spread across the organization into code.

The effect grew larger as investigations got more complex. When the paper compared the manual approach against three scenarios—a simple service error, a container failure, and a feature problem in an AI model—the number of steps an on-call engineer had to take dropped by 4x to 20x. In the most complex scenario, the engineer’s job was compressed down to exactly one step: reading the results on the alert page.

---

⚠️ The Lesson From Section 9 of the Paper: Don’t Over-Index on AI for Diagnosis

Up to this point, it’s a familiar automation success story. But when you get to Section 9 of the paper, “Lessons Learned,” you hit a subheading like this:

“Do not over-index on AI based systems for diagnosis.”

A company running one of the largest AI infrastructures on the planet wrote this as a lesson learned in its own outage investigation paper.

Let me clear up a misunderstanding before it starts: this doesn’t mean Meta doesn’t use AI. DrP’s SDK is packed with statistical and machine-learning libraries — anomaly detection, time-series correlation analysis, dimensional analysis, all of it. There’s even a ranking model running that sifts through thousands of code and configuration change events and outputs, “This is highly likely the culprit.”

What Meta doesn’t hand off to AI is the sequencing and conditions of judgment. The decision tree — what to check, in what order, and what to look at next depending on the outcome — is written by humans, in code. AI is used as a tool inside that procedure, handling individual steps like anomaly detection or ranking candidate root causes.

The paper spells out why. Pure machine-learning systems are limited by the quality of training data, by the actual structure of data accessible at investigation time, and by the difficulty of tailoring workflows team by team. So the conclusion they landed on was a combination: rule-based suggestions grounded in community expertise, with AI layered on top.

Reading the paper, another lesson right next to it caught my eye. The title is “Assisted or Fully Automated?”

Meta’s original goal was full automation. But they changed course. There were three reasons. First, systems keep changing, so analyzers go stale fast. Second, statistics and machine learning produce both false positives and false negatives. And third, culturally, engineers and on-call staff don’t always trust a fully automated system.

The first two are technical problems. The third is a human problem. After five years, 300 teams, and 50,000 runs a day, the reason Meta shelved full automation wasn’t just accuracy — it was also that engineers and on-call staff don’t always trust what automation tells them.

The paper’s own survey backs this up. Asked “How often does DrP reduce MTTR?”, only 5.9% answered “always.” The most common answer was “sometimes” (47.1%). The paper summarized this as “over 80% reported improvement” — which isn’t wrong, exactly, but the center of gravity of the responses clearly sits on “sometimes.” For reference, working backward from the percentages, the respondent pool comes out to about 17 people. That’s a very small sample for a survey about a system used by 300 teams, so this figure should be treated as reference-only.

📉 What Separated the 20% Average from Some Teams’ 80% Was the Number of Analyzers

Meta’s engineering blog put it this way: “We cut MTTR by 20-80%.” It reads like a range. It’s easy to skim past, assuming the real number lands somewhere around 40-50%.

But the paper’s abstract phrases it differently. Average: 20%. Only some teams exceeded 80%. Same numbers, completely different meaning.

Open Section 7 of the paper and you find out why.

• Fewer than 5 analyzers → 10-15%
• 10 or more → 50-80%
• Company-wide average → 20%

In other words, that 80% figure wasn’t a measure of the system’s raw performance — it depended on how many analyzers a given team had actually built. What separated the winners was how much of their own investigation procedures they’d translated into code.

The team-by-team table in Section 7.2.4 makes this even sharper.

• Team 1: 136 analyzers, 82% improvement
• Team 2: 92 analyzers, 75%
• Team 3: 66 analyzers, 84%
• Team 4: 48 analyzers, 69%
• Team 5: 39 analyzers, 58%
• Team 6: 29 analyzers, 73%
• Team 7: 23 analyzers, 56%
• Team 8: 12 analyzers, 7%

Look at that last row. Team 8 built 12 analyzers — clearing the paper’s own “10 or more” threshold — yet only saw a 7% improvement. The paper’s own stated criterion contradicts its own table. Wherever the real inflection point sits, it’s somewhere between 12 and 23 analyzers.

Here’s a calculation I ran myself. Divide 2,000 analyzers across 300 teams and you get an average of 6-7 per team. And the paper itself says fewer than 5 analyzers yields only 10-15%. What does that tell you? Even inside Meta, the median team is still stuck in the low-yield zone. That’s why the company-wide average stalled at 20%.

Time is no small factor either. According to the paper, a simple analyzer can be built in a day, but fully capturing a team’s investigation workflow takes months. This isn’t the time it takes to buy a tool — it’s the time it takes to transfer knowledge.

One last thing worth flagging: the absolute MTTR numbers. Team 1 went from 771 hours to 139 hours — 32 days down to 5.8 days. This isn’t downtime; it’s the full cycle from incident detection to resolution, including post-incident review. If “20% MTTR reduction” sounds small, you need to weigh it against how large the baseline actually is.

One more note: recalculating Team 7’s row from its before/after figures yields 36%, but the paper lists 56.1%. The other seven rows check out to the decimal point, so this is most likely a typo. Still, it’s a good reminder to run the numbers yourself before copying someone else’s table wholesale.

🔁 Why Meta Can Finally Call DrP “AI-Native”

The last paragraph of Meta’s blog post ends this way: going forward, they’ll evolve DrP into an AI-native platform.

Doesn’t that seem odd? They write “don’t over-rely on AI,” and then in the very next paragraph say they’re headed toward AI-native.

It’s not a contradiction. It means there was groundwork that had to be done before AI could be bolted on.

Let’s look again at what Meta actually did over the past 5 years. Investigation knowledge that used to be scattered across wiki pages, personal scripts, and the memories of senior engineers got transformed into:

• Typed code (investigation procedures made explicit as branches and conditions)
• Structured output (results emerge in machine-readable form)
• Data with a record of execution (30 days of past investigation inputs and outputs preserved)

The third point is the decisive one. Every time Meta modified its analyzer, it built a backtesting⁴ system that replayed past investigation records to validate the changes. The original purpose was to catch bugs. But as a result, the incident investigation process itself became a dataset with labeled answers.

For an AI agent to investigate an incident, it needs three things. First, tools to access the data. Second, a defined procedure for what to check and in what order. Third, a record showing which past judgments turned out to be correct. What DrP built over 5 years is exactly these three things.

The paper holds one more lesson. Its title is “Data Is Everything.” Without high-quality observability data and structured metadata—things like service dependencies and data lineage—correlation analysis simply can’t happen. No matter how good the model you attach, there’s nowhere to attach it.

So the answer to the question in the title comes down to this: Meta didn’t hand judgment over to AI because the procedural code and investigation records that AI could actually use hadn’t yet accumulated to a sufficient degree. And now, they’ve judged that the groundwork is finally in place.

For AI to do work in our stead, that work first has to become code. No matter how good the models get, if an organization’s knowledge lives in wiki pages and someone’s memory, AI has nowhere to attach itself.

🇰🇷 How Far Have Korean Organizations Gotten

At a conference held in Seoul on June 24, Samsung Electronics’ MX Business Division cloud team shared its own roadmap. This is the central SRE⁵ organization responsible for the stability of more than 50 customer-facing services, including Samsung Pay, Bixby, and the Galaxy Store.

They laid out four maturity stages: reactive response, automated response, predictive operations, and autonomous operations. Their own self-assessed position was the first stage. Their target date for reaching autonomous operations is 2028. If that’s how one of Korea’s top-tier SRE organizations rates itself, you can imagine where everyone else stands.

What stood out was the sentence Hyunsung Yoo, the group leader, used to close his presentation: even as the scope of automation expands, final responsibility and judgment remain with humans. That’s exactly the same conclusion as the Meta paper’s “assistance, not full automation.”

At the same event, Woocheol Byun, head of KT’s Palantir Business Division, put it even more bluntly. Because management is impatient, companies bolt on AI without touching the root problems—and then wonder why they see no results. The fix he pointed to was organizing data into a structure AI can actually read. That’s the same point as the Meta paper’s “data is everything.”

There are working examples in Korea, too. Last April, Yanolja disclosed that six teams totaling 14 people had built six operations agents over six weeks. Among them, the incident-response agent cut the time from incident occurrence to post-incident report from 2 weeks down to 24 hours. But look closer at the structure, and here too the agents operate on top of a foundation of searching internal knowledge bases and documents. The requirement that organized knowledge exist first, before an agent can do its job, is the same as at Meta.

Oswarld’s Lens

I counted the six lessons in chapter 9 of the paper again. Not one of them is about model performance. Every single one is about adoption.

Let the community build it themselves. Go to where the users already are. Fit it into the workflow they’re already using. One line in particular stuck with me: building analyzers isn’t the day job for most software engineers. So Meta refined its SDK so anyone could build one in a day, and made sure results showed up right inside the code editor.

Over 20-plus years of building GTM strategy, I’ve seen this pattern repeat itself. The first question an organization asks when evaluating a new tool is always “how accurate is it?” But what actually determines adoption is almost always “how annoying is it?” A tool that’s 95% accurate but requires opening three more windows won’t get used. A tool that’s 70% accurate but shows results on the screen you’re already looking at will. When Meta attached analysis results directly to the alert page, that wasn’t primarily a decision to boost performance — it was a decision to make sure engineers would use it on the screen they were already staring at.

That’s why I think many organizations have the order backwards when they start by asking “which AI agent should we adopt?” The question to answer first is how many of the judgment procedures your team repeats every single time are already written as code. If that number is close to zero, there’s nowhere to plug in any model you buy.

Closing

What actually ran Meta’s incident-investigation system — 50,000 cases a day, every day, for 5 years — wasn’t an AI agent. It was 2,000 analyzers that engineers wrote themselves. What separated success from failure wasn’t model performance either — it was how many analyzers a team managed to build. The only reason Meta can talk about being “AI-native” today is that, over 5 years, it translated tacit knowledge into a form machines could read.

If you want to dig deeper, just open Section 9 of the paper, “Lessons Learned.” Three pages hold five years’ worth of trial and error.

Try to picture one judgment procedure your own team repeats, in the same sequence, every single time. If it still lives only in a wiki or in someone’s head, why hasn’t it become code yet? Is it because there’s no time, or because the procedure differs from person to person? Tell me your biggest obstacle in the comments — I’ll fold it into the material for the next issue.

---

💬 Share your experience with the question above in the comments. I’ll factor it into the next issue. 📨 If you have a colleague thinking about operations automation or adopting AI, pass this piece along.

---

I read across technology, economics, and the humanities every Tuesday, Thursday, and Sunday, writing about what I find at their intersections. Some days, like today, mean sitting down with a single paper and working through it in detail.

The English draft matches the Korean source accurately in structure, numbers, links, and terminology. No corrections needed.

References & Further Reading

Primary sources

• Shubham Somani et al. (14 authors), “DrP: Meta’s Efficient Investigations Platform at Scale”, arXiv:2512.04250 [cs.SE], December 2025. : Section 7.2 (MTTR evaluation) and Section 9 (lessons learned) are the core evidence behind today’s piece. If you’re short on time, the three pages of Section 9 alone are enough.
• Meta Engineering, “DrP: Meta’s Root Cause Analysis Platform at Scale”, December 19, 2025. : This is the official summary of the paper. But it compresses the MTTR figures into “20–80%,” which shifts the nuance from the original. If you’re going to cite numbers, I’d recommend going straight to the paper.

Background

• Rootly, “What Is an AI SRE Agent? How AI Is Changing Incident Response in 2026”, April 2026. : This shows you the tone in which the market promises I quoted in the Opening are actually being sold. It’s vendor material, so read it with that in mind, but the framework that splits autonomy into 4 stages is worth placing side by side with Samsung’s 4-stage maturity model.
• Google, “Being On-Call”, Site Reliability Engineering. : This is the primary source on why the on-call system exists and why it burns people out. Meta’s paper cites this document in its own footnotes too.
• ZDNet Korea, “AI That Can’t Read Data Is Useless: What Are KT and Samsung’s Solutions?”, June 24, 2026. : Samsung Electronics’ SRE organization’s 4-stage roadmap and KT’s diagnosis of its data structure both appear in one piece. Most useful for orienting yourself within the domestic (Korean) landscape.
• Edaily, “Samsung Uses AI to Catch Cloud Outages, Targets Autonomous Operations by 2028”, June 2026. : This lays out Samsung Electronics’ announced numerical targets — a 90% cut in recovery time, a 99% detection rate within 10 minutes, and so on — in more detail.
• AWS Technical Blog, “How Yanolja Built an AIOps Agent Using the Strands SDK and Bedrock AgentCore”, April 2026. : Lets you look inside the architecture of an operations agent actually running in production in Korea, broken down by 6-week sprints.

---

The author is Oswarld (Kwangseob Ahn). Current roles: Adjunct Professor at Sejong University, Strategy Consultant at INLEVEL9. Career, research, books, and recent work are kept current on the About page. Latest · July 2026: HEMA-2: A Consolidation-Aware Tri-Memory Architecture with Multi-Channel Scheduling for Lifelong Conversational AI.

📝 Glossary

Footnotes

1. MTTR (Mean Time To Resolution): The average time from when an incident is discovered to when it’s resolved. Meta’s paper calculated this only for incidents that went through post-mortem review, so think of it not as downtime but as the full cycle including root-cause analysis and remediation. ↩

2. On-call: A rotation system that ensures someone responds immediately when an incident occurs. Whoever has to get up when the alarm goes off at 3 a.m. is that day’s on-call. ↩

3. Tribal Knowledge: Knowledge that isn’t written down anywhere but is passed between people within an organization — things like “when that server acts up, you have to check over there first.” ↩

4. Backtesting: A method of replaying past data to verify that changed code produces the same results as before. Meta keeps a record of the last 30 days of investigations and automatically reruns it every time an analyzer is modified. ↩

5. SRE (Site Reliability Engineering): Both a job function and an organizational approach that manages service reliability using software engineering methods. It originated at Google in 2003, and on-call rotation and incident response are among its core responsibilities. ↩