Six AIs Claimed Perfect IMO Scores, Only Two Were Graded

Six AIs Claimed Perfect Scores, Only Two Were Officially Graded

The International Mathematical Olympiad (IMO) wrapped up in Shanghai on July 16. 666 students sat the exam, and 7 of them scored a perfect 42.

But the following week made more noise than the exam itself. Within a single week, six different AI systems announced that they, too, had scored a perfect 42. Huawei, Xiaohongshu, and four more models from independent tests all made the claim.

Of these six, only two actually submitted their answers to the IMO organizers for grading. The rest fall outside official scoring. That includes a test graded by a Claude-based agent, and a Lean 4 formal proof from AxiomProver — both verified through methods entirely different from the official process.

What AI evaluation is missing right now isn’t harder problems. It’s a documented record of who graded the answers, under what conditions, and how. What we need isn’t a benchmark that just publishes a score — it’s an exam with defined testing conditions and a verification process.

Six perfect-score claims, six different sets of conditions

The human results are unambiguous: 7 out of 666 students scored perfectly. They worked on 6 problems over 9 hours across two days — same day, same time slot, same room. The AI results are a different story.

Official grading track: Huawei’s Celia and Xiaohongshu’s dots-note-3.0 received the problems after the human competition ended, submitted answers within a set time limit, and had those answers passed on to the IMO organizers. Xiaohongshu stated it had “strictly prohibited any form of human intervention.”

Independent testing track: The table below combines results from test environments researchers built themselves with the formal-proof results Axiom Math published for AxiomProver.

Model	First attempt	Final	Cost	Time taken
Claude Fable 5	42/42	42/42	~$51	2.5 hours
GPT-5.6 Sol (xhigh)	39/42	42/42	~$21	3.8 hours
Kimi K3	36/42	42/42	~$31	17.4 hours
AxiomProver	Formal proof	42/42	Undisclosed	~25 hours

The numbers alone are striking. Fable 5 solved everything on the first try, in under a third of the 9 hours students were given. AxiomProver is a different animal entirely — it turned the 6 problems into machine-verifiable proofs written in Lean 4¹, a formal proof language, totaling 7,722 lines. Problem 3 alone took 4,229 lines and 869 minutes.

But looking at this table, what kept nagging at me wasn’t the scores — it was the conditions.

You need to keep results from AI agents grading free-form written solutions separate from AxiomProver, whose proofs were verified in Lean 4. The same model’s score swung by 11 points depending on how its effort tier was set. And here’s the most uncomfortable part: on the same test, Meta’s Muse Spark and DeepSeek V4 Pro produced the exact same wrong answer on problem 3.

In a human exam hall, this would immediately trigger an investigation — you’d need to check why the same wrong answer appeared. It could be overlapping training data, but identical wrong answers alone don’t prove the models failed to solve the problem independently. The trouble is, benchmarks have no procedure at all for handling this situation. Both simply get logged as “1 wrong answer,” and that’s the end of it.

1062: Goryeo Erased Names From the Answer Sheet

When we talk about exams as an institution, we usually think about the questions — what was asked, how hard it was, who got it right. But if you look at the history of testing systems, the part that was actually refined with the most care wasn’t the questions. It was the procedure for preventing fraud and verifying results.

A prime example is bongmi (封彌), a sealing method. The candidate’s status, name, age, and clan seat written on the right side of the answer sheet were folded over and pasted shut, so the grader couldn’t tell whose answer it was. The practice was established during China’s Song dynasty, and in Goryeo it began in 1062, the 16th year of King Munjong’s reign, on the recommendation of an official named Jeong Yusan. It carried straight through into the Joseon dynasty.

And that wasn’t the end of it. Even with the name hidden, handwriting could give someone away. So an additional device called yeokseo (易書) — having a third party copy the entire answer out by hand — was layered on top.

Why did people a thousand years ago bother building all this? Not because they distrusted the graders. It was because for an exam result to carry social weight, it had to be possible, later on, to scrutinize how that result had been produced.

Passing the gwageo, Korea’s traditional state examination, wasn’t just a score — it was a qualification that led directly to government office. Once exam results start functioning as credentials like this, the exam stops being merely a tool for measuring ability and becomes a procedure with social consequences. And what that requires isn’t harder questions — it’s an appeals process, proctors, record-keeping, and a party accountable for the outcome.

Interestingly, the same logic shows up in how we evaluate academic papers. Blind peer review conceals identifying details — the author’s institution, their advisor — so that these don’t pressure or lend undue authority to the reviewer, before the paper is assessed on its merits. And AI evaluation is now arriving at the same juncture: its scores are starting to be used as grounds for adoption and purchasing decisions.

IOAI Gets an AI Model Track This Year, First Session on August 4

The third International Olympiad in Artificial Intelligence (IOAI) runs from August 2 to 8 in Astana, Kazakhstan. And this year, for the first time, there’s a separate track called IOAI²: AI Model Track. It’s a track where AI systems register as participants and compete on a leaderboard separate from human contestants. Registration closes tomorrow, July 27.

Here are the main provisions of the rules.

• Fixed schedule: Two 6-hour sessions on August 4 and 6, with 3 problems per session.
• No human intervention: After the initial run prompt, execution must be fully autonomous.
• Limited attempts: Up to 50 submissions per problem.
• Standardized grading environment: Execution happens on the participating organization’s own hardware, but grading runs on standardized GPUs.
• Optional disclosure of results: Participants receive their percentile score privately within 24 hours of a session, then decide within 48 hours whether to reveal their organization’s name or remain anonymous.
• Verification requirement: To receive a medal, participants must submit an execution trace² containing all model inputs/outputs, tool calls, and prompts. This record is disclosed only to the panel of judges, not to the public.
• Entry fee: One model per organization is free (sponsors get two), and each additional model costs €25,000.

Compare this list to a benchmark, and the differences become clear.

When public benchmarks are run independently and only the scores get published, the timing, number of attempts, and execution conditions aren’t standardized — and the process record can be missing entirely. This track, by contrast, fixes the exact time of the attempt, caps the number of tries, and requires a process record to be submitted before a result can be certified. And there’s a panel of judges.

If bongmi — the sealing device once used in Korean state exams to hide a test-taker’s name from the grader — worked by concealing identity, submitting an execution trace is the device that opens up the test-taking process to the judges. The direction is reversed, but the purpose is the same: making the result verifiable after the fact.

The €25,000 entry fee shouldn’t be read merely as a revenue model either. Having an entry fee means the act of taking the test carries cost and accountability. An evaluation you can run for free, infinitely, and an evaluation you pay to register for and sit on a fixed date — these produce different behavior from participants.

Evidence That Benchmarks Are Breaking Down

You might ask, “Isn’t a benchmark good enough, though?” Recent years of data say no.

First, there’s saturation³. GPQA Diamond climbed from 39% in late 2023 to 94% by early 2026. In barely two years, there’s almost no room left to push the score higher. AIME 2025 shows the same pattern—top models are bunched together in the high-80s to mid-90s. When that happens, differences in score stop meaning differences in ability. GPQA Diamond has 198 questions, and at that sample size the 95% confidence interval is roughly ±3 points. That means most marketing copy that ranks models by a 3-point gap is statistically meaningless.

Second, there’s contamination⁴. Scale AI built GSM1k, a new problem set matched in difficulty to GSM8K, and retested the same models on it. Some models dropped by as much as 13 points. And the size of that drop correlated with how often the model in question could regurgitate GSM8K questions verbatim. This suggests some models’ original scores may have been inflated by simple memorization.

Third, there’s the judge problem. When you use an LLM as the grader, a well-documented bias emerges: longer answers score 15 to 30 points higher. There’s also a bias tied to the order in which answer choices are presented. Neither is random noise—both are consistent, one-directional distortions. So if you fine-tune a model to chase these grading scores, you can end up training it to simply write longer answers.

Put these three together, and a leaderboard score stops looking like settled performance and starts looking like a claim that needs to be checked against the conditions that produced it.

Oswarld’s Lens

I see this less as a benchmarking-methodology problem and more as a corporate purchasing decision problem.

I’ve watched this scene play out countless times while building go-to-market strategies. A vendor shows up with a benchmark score. The person in charge copies that number into an internal approval document. Two months later, when performance falls short in production, nobody can be held accountable using that number — because nobody actually knows what conditions produced it in the first place. I’ve seen this happen over and over, and I keep arriving at the same conclusion: a benchmark score isn’t procurement evidence. It’s a marketing asset.

When a company adopts AI, a vendor simply saying “our model scores 94” isn’t enough to make a decision on. What’s actually needed is “under these conditions, at this time, through this process, this result was produced — and the record of it still exists.” Think of it like a financial audit. We don’t just take a company’s earnings report at face value. We trust it because there’s an auditor, there are audit working papers, and there’s a record that can be examined later.

AI performance announcements often don’t provide that kind of verifiable trail. That’s why IOAI²’s attempt catches my attention. It’s not about setting harder questions — it’s an attempt to make it possible to later verify exactly what conditions and process produced a given score. I think that direction matters far more right now.

To be clear, I don’t expect it to go smoothly the first time around. Running the exam on the vendor’s own hardware while scoring only happens in a standardized environment leaves gaps. Allowing 50 attempts leaves room for cherry-picking the single best result. A €25,000 entry fee tilts things in favor of well-resourced organizations. Even so, I think the direction is right. The shortcomings of a newly introduced system can be fixed as it’s put into practice. Bongmido — the anonymization system used in Korea’s historical state examinations — started out just covering candidates’ names, and only later evolved to conceal their handwriting as well.

Closing

1. At IMO 2026, 7 out of 666 humans scored a perfect mark, and AI systems claiming perfect scores numbered six. Of those, only two went through an official grading channel.
2. Benchmarks are already losing their discriminating power for three reasons: saturation, contamination, and judge bias.
3. The next step isn’t harder problems — it’s an exam system with proper testing conditions and verification procedures. The first experiment along those lines begins in Astana this August.

Next time someone shows you an AI performance number, don’t ask for the score. Ask this instead: “Who graded that score, when, under what conditions, and is there a record?” More often than you’d expect, the answer won’t come easily.

If you’ve ever run into a gap between benchmark scores and real-world performance while adopting an AI tool at work, tell us in the comments where the gap was widest. Once we collect enough examples, I’ll put together a “checklist of performance-verification questions for practitioners” in a future issue.

---

📨 If you know a colleague who’s evaluating AI adoption, please pass this piece along.

---

References & Further Reading

Primary sources

• IOAI, “AI Model Track: Rules and Competition Format,” International Olympiad in Artificial Intelligence 2026. Link ··· The provisions on execution-trace submission and verification scoring are the core evidence behind today’s piece. Reading the rules yourself makes the design gap between an “exam” and a “benchmark” much clearer.
• Axiom Math, “AxiomProver at IMO 2026,” GitHub, 2026. Link ··· The full Lean 4 formal proofs for all 6 problems are public. If you’re curious what a machine-verifiable proof actually looks like, open the file for Problem 3.
• Digital Applied, “Four AIs Scored a Perfect 42/42 on IMO 2026. So What?”, 2026. Link ··· A piece that tables out cost, time, and retry counts per model, plus who did the grading. It lets you separate out details that news coverage tended to blur together.
• Acing AI, “The LLM Evaluation Crisis: Contamination, Saturation, and the Judge Problem,” 2026. Link ··· This is where the GPQA Diamond confidence-interval figures and the GSM1k reproduction-experiment numbers come from. If you work with benchmarks, I’d recommend reading at least the statistics section.
• AFP, “AI catches up with humans to score 100% at top maths contest,” 23 July 2026. Link ··· A wire-service report on the Huawei and Xiaohongshu announcements. It’s also a good case study in how the press translates corporate announcements.

Background

• “Bongmi (封彌)”, Annals of the Joseon Dynasty Encyclopedia, The Academy of Korean Studies. ··· This lays out how the practice was introduced in 1062 under King Munjong of Goryeo. It’s a good way to confirm that the core of an exam system was procedure, not questions.
• “China’s Imperial Examination System (Gwageo),” Wikipedia. Link ··· A quick overview of the full context behind the Song-dynasty reforms to the imperial exam.

---

The author is Oswarld (Kwangseob Ahn). Current roles: Adjunct Professor at Sejong University, Strategy Consultant at INLEVEL9. Career, research, books, and recent work are kept current on the About page. Latest · July 2026: HEMA-2: A Consolidation-Aware Tri-Memory Architecture with Multi-Channel Scheduling for Lifelong Conversational AI.

📝 Glossary

Footnotes

1. Lean 4: A programming language for writing mathematical proofs in a form a computer can check line by line. Instead of a human reading it and deciding “this looks right,” the machine automatically catches any logical gaps. ↩

2. Execution trace: A time-ordered record of every input, output, tool call, and prompt exchanged while an AI works through a problem. Think of it less as an answer sheet and more as CCTV footage of the exam. ↩

3. Saturation: A phenomenon where, as model performance rises, evaluation problems become so easy that every top-tier model clusters near a perfect score. Once this happens, score differences can no longer distinguish ability. ↩

4. Contamination: A situation where the problems used for evaluation are already present in a model’s training data. It’s the equivalent of seeing the exam questions beforehand, so the score reflects memorization rather than ability. ↩