ICML Rejects 497 Papers Over Peer Review Rule Break

497 Papers Rejected for Breaking Review Rules

On March 18, the machine learning conference ICML¹ 2026 announced that it had rejected 497 submitted papers. The reason: the designated peer reviewers assigned to these papers had violated a no-LLM-use rule they’d agreed to, while reviewing other submissions. The organizing committee was careful to note that this wasn’t a judgment on the quality of the papers or the review reports themselves — it was a response to a broken agreement.

What matters in this case isn’t just whether AI was used. It’s whether reviewers kept the commitments they’d signed up for.

The detection method relied on hidden instructions embedded in the paper PDFs. These instructions are invisible to a normal human reader, but designed to be read by an LLM processing the document — prompting it to insert specific phrases into its review report. According to the organizing committee, they randomly selected two phrases per paper out of a pool of 170,000, and any review report that contained one of those phrases was flagged for human re-examination.

I actually built a protective tool called PDF AI SHIELD a while back, so this approach felt familiar — the idea of preserving what a human sees while intervening in how an AI reads the same document. It was gratifying to see a similar concept applied to conference peer review, but also a little bitter that it had become necessary as a way to catch rule-breaking. ICML cited research by Rao et al. as the basis for the detection method it used.

Watching this unfold, I found myself wanting to look at the broader research environment too. Now that AI makes it so easy to read documents and draft text, what exactly should researchers still be checking themselves? Holding people accountable for breaking rules and teaching people how to use these tools properly both seem necessary.

Let’s look at how this same problem shows up in PhD training and in journal peer review.

The Scope of Violations ICML Confirmed

ICML 2026 had two policies: Policy A, which banned LLM use during review entirely, and Policy B, which allowed LLMs for understanding papers, checking related work, and polishing reviews. Reviewers stated their preference, and assignment followed author requests and application status. Only reviewers who chose Policy A, or who said either policy was fine, were assigned to Policy A.

The organizing committee confirmed LLM use in 795 reviews from 506 Policy A reviewers. These reviews made up roughly 1% of all reviews and were deleted. 51 reviewers, for whom more than half of their submitted reviews were found to be in violation, had all their reviews deleted and were removed from the reviewer pool entirely.

Paper rejections applied to 497 submissions that had listed 398 of these reviewers as designated mutual reviewers². Not all 506 flagged reviewers were direct targets of paper rejection. Also, this action was a determination that a banned LLM was used somewhere in the process of writing the review — not a ruling that the entire review was AI-written.

This method cannot catch every instance of LLM use. Cases can be missed if the instructions were removed or the model didn’t follow them, and editing the output afterward makes detection harder. The committee itself explained that this approach catches only certain violations, like pasting a PDF in and copying the output verbatim. So the 1% detection rate can’t be read as the overall AI usage rate.

Even the PDF protection tool I built ran into the same limit — technical restrictions alone couldn’t fully block use. Strip the protection or read the document through another method, and the restriction falls apart. That’s why I think rules, education, and user responsibility all need to work together.

A 2025 survey by the academic publisher Frontiers also found a lot of reviewers using AI. Out of 1,645 total researcher respondents, 909 answered the question on frequency of AI use during review, and 53% of them said they used it. Among respondents with 5 years of experience or less, the review-time usage rate was 61%. The 87% figure that appears in the report is a separate number referring to early-career researchers’ overall AI use, not specifically during review. Since this survey simply asked about usage patterns, answering “yes, I used it” can’t be treated as an admission of rule violation.

The PhD Student’s Dilemma

For PhD students too, AI is a convenient yet troubling tool. A March 2026 Nature article introduces both students who use AI for literature searches, coding, and writing, and students who worry that in doing so they won’t fully develop the skills research demands.

This rapid rise in usage is also confirmed by a survey of UK undergraduates. HEPI’s 2025 report found that the share of students using AI tools rose from 66% in the 2024 survey to 92%, while the share who used AI on assessed coursework climbed from 53% to 88%. The 2025 report is based on responses from 1,041 full-time undergraduates in the UK. Note that “used it on coursework” here includes things like getting concepts explained or having material summarized, so it doesn’t mean all of them had AI write their assignments for them.

In the Nature article, Yinghui He of Tsinghua University uses ChatGPT and Gemini daily but stresses that generated code still needs to be verified. Richard Ang of the University of Western Australia recounts an experience where ChatGPT misread a question, produced an error in a fertilizer dosage calculation, and caused an experiment to fail. Getting an output is one thing; getting a calculation that actually fits the experimental design is another.

Leona Diala, a PhD student researching infectious disease modeling, describes a case where AI read a downward trend on a graph as an upward one. The point is that you need to know the field to be able to recognize a wrong answer when you see one.

Even when AI proposes an analysis or a judgment, it can be wrong. Researchers need the ability to check whether the result actually fits the research question and the data. A PhD program is, in part, the time set aside to build that ability.

What a Writing Experiment Revealed

There’s also an experiment that looked at how the writing process changes depending on how you use AI.

A team led by Nataliya Kosmyna at the MIT Media Lab published a 2025 preprint³ titled “Your Brain on ChatGPT.” 54 participants wrote essays under three conditions—without any tools, using a search engine, or using ChatGPT—and the researchers examined EEG⁴ data, the content of the essays, and participants’ ability to recall what they’d written.

Across three writing sessions, the EEG-estimated patterns of functional connectivity between brain regions differed by condition. The group that wrote without tools showed the broadest and strongest connectivity, while the LLM group showed the weakest. This is a difference in measured activity during the specific task at hand. Weaker connectivity alone doesn’t mean intelligence declined or the brain was somehow damaged.

A fourth session involved 18 participants who switched conditions. Those who had used an LLM and then switched to writing without tools showed lower alpha- and beta-band connectivity. Conversely, those who had written without tools and then switched to using an LLM showed stronger recall and activation in certain regions. The earlier LLM group had struggled to accurately quote their own writing, and reported feeling less ownership over what they’d produced.

The researchers described these findings using the term “cognitive debt” and voiced concern about the implications for learning. But this was a small essay-writing experiment, and the sample for the condition-switching session in particular was even smaller. Rather than generalizing this to research work as a whole or to long-term skill decline, it’s more accurate to read it as a study worth pursuing further—one that suggests the learning experience may shift depending on when and how AI is used.

This experiment can’t explain why the ICML reviewers broke the rules. It’s a different study, with different participants and a different task. The ICML case is a matter of broken promises and accountability; the writing experiment is a matter of tool use and the learning process. The two need to be examined separately.

So What Should Be Protected?

Universities are working out their own standards, too. Among the 217 institutions that took part in the European University Association’s (EUA) 2025 survey, 38% said they were drafting AI policies and guidelines for the first time, while 19% said they were revising existing ones. 13% reported having no guidelines at all, and 5% said their current guidelines were already sufficient. Since respondents could select up to three options, these percentages shouldn’t be added together as if they represented mutually exclusive groups.

For doctoral programs, I think institutions need to define not just how far AI use should extend, but also which parts of the process students must still do themselves.

Amina Yonis of The Page Doctor, an academic support firm cited in a Nature article, recommends using AI for literature search and organization while still doing the analysis with genuine personal understanding. Her suggestion on writing is similar: rather than handing the first draft to AI, write it yourself first and then use AI to polish it. It reads as advice meant to preserve the process of actually constructing a draft on your own.

Computational neuroscientist Natalia Bielczyk emphasizes training in forming good questions, deciding on research methods under uncertainty, and actually testing ideas. Her point is that time saved through AI should be redirected into exactly that kind of work.

Some students, by contrast, are cutting back on AI use. Samuele Lamellini, a doctoral student at the Australian National University, describes generating a graph with AI and then, not long after, being unable to recall the code or the prompts he’d used to make it. He contrasts this with a graph from his undergraduate thesis, which he’d agonized over directly for a long time. One person’s experience can’t be generalized to every student, but it’s a case that illustrates how getting a result and learning the process behind it can be two different things.

Climate scientist Alex Sen Gupta of UNSW, in a 2025 Nature essay, proposed redesigning doctoral training for the AI era. Since the tools are changing so fast, he suggests we should also question whether it’s even enough to simply identify what AI currently can’t do and teach around that.

Manikandan Palanichamy of Norway’s Østfold University College similarly stresses the need to teach responsible AI use from the earliest stages of a student’s training. His point is that students need to learn not just how to operate the tools, but also the foundational knowledge required to verify what those tools produce.

Oswarld’s Lens

Watching the ICML case unfold, two things came to mind.

The first is a pattern I’ve seen many times while building GTM strategies: how tools get adopted. When a CRM or marketing automation tool comes in, the early conversation is dominated by “this will boost efficiency.” But you also have to watch how people actually end up working once the tool is in place. AI, similarly, lets you draft a review faster—but whether it’s also fine to cut down the time spent reading the paper and checking the evidence is a separate judgment call.

The second is that ICML 2026 will be held in Seoul this July. This is a conference where Korean researchers participate as both authors and reviewers. Our own graduate schools can’t treat AI use and verification as someone else’s problem happening abroad.

I think that the more comfortable we get having AI read and summarize for us, the more deliberately we need to preserve time for direct review. That said, there isn’t enough evidence to conclude that this habit alone leads to rule violations or a decline in ability. What we need to examine first is which specific steps researchers are actually skipping.

The abilities I care about are the ones researchers need regardless of how the AI changes: framing a good question, finding evidence that actually supports a claim, and examining the ways an analysis might be wrong. Only with these abilities can you properly evaluate what AI’s help is actually worth.

If these steps keep getting skipped during training, the chance to build those abilities shrinks too. That’s why I think education needs to look not just at the finished output, but at why a particular method was chosen and what was actually checked firsthand.

Closing

What the ICML incident confirmed was a violation of an agreed-upon review rule. We can’t extend that into a verdict on researchers’ cognitive abilities. Meanwhile, the writing experiment and students’ experiences leave us with a question: even as tools grow more convenient, is the process of learning and checking things ourselves still necessary?

The more AI helps with drafts and analysis, the more researchers need to be able to explain which judgments they made themselves. Advisors and academic conferences also need to spell out, concretely, what’s permissible and what procedures should be used to verify it.

Here’s how I took the question Lamelini posed.

If AI saved you an hour, where would you spend that hour? It’s worth asking whether you’d pour it all into producing more output, or also into understanding and verifying what you’ve produced.

References & Further Reading

• Nature, AI and the PhD student: friend or foe?, 2026.03.16. An interview about students’ experiences with AI and what it means for their education.
• ICML 2026 Program Chairs, On Violations of LLM Review Policies, 2026.03.18, with a correction issued 03.26. It distinguishes between the 506 reviewers caught violating the policy and the 398 designated reviewers whose assigned submissions were rejected as a result.
• Kosmyna, N. et al., Your Brain on ChatGPT, 2025. A preprint of a small-scale essay-writing experiment.
• Sen Gupta, A., PhD training needs a reboot in an AI world, 2025.
• Frontiers, Unlocking AI’s untapped potential, 2025. It reports on 909 respondents to a question about AI use during peer review, and shows the extent of adoption among early-career researchers.
• EUA, Policies in doctoral education, 2026.01. These are the 2025 survey results; the AI-policy questions are on pages 32-33.
• Freeman, J., Student Generative AI Survey 2025, HEPI, 2025.02.26.
• Kwangseob Ahn, Vibe Coding Trial-and-Error EP.01, PDF AI SHIELD, 2025.05.23. This is the story of a PDF-protection tool I built myself — a solution I later sold outright.

The author is Oswarld (Kwangseob Ahn). Current roles: Adjunct Professor at Sejong University, Strategy Consultant at INLEVEL9. Career, research, books, and recent work are kept current on the About page. Latest · July 2026: HEMA-2: A Consolidation-Aware Tri-Memory Architecture with Multi-Channel Scheduling for Lifelong Conversational AI.

Footnotes

1. ICML (International Conference on Machine Learning): one of the leading international machine-learning conferences. The 2026 event is scheduled to take place in Seoul, July 6-11. ↩

2. Reciprocal Review: a system in which authors submitting a paper are required to designate reviewers who will, in turn, review other submissions. Under ICML’s latest action, submissions were rejected if a designated reviewer had violated the agreed-upon rules on LLM use. ↩

3. Preprint: a version of a paper released before it has completed a journal’s peer review. It allows research findings to be shared quickly, but readers should also weigh its scope and limitations. ↩

4. EEG (electroencephalography): a method of measuring the brain’s electrical activity via electrodes placed on the scalp. The functional connectivity patterns analyzed in this study reflect signal relationships during task performance — they do not mean that physical connections in the brain were severed. ↩