Ask 70 AI Models the Same Question, Get the Same Answer

If asking a different AI gets you the same answer anyway

“Write me a metaphor about time.”

One study sent this request to 25 language models and collected 50 responses from each. Among the 1,250 answers, time was often compared to a river, and a separate cluster compared time to a weaver. When the researchers mapped the meaning of these responses, the two clusters stood out clearly.

A joint research team from the University of Washington and several other institutions examined more than 70 language models and reported both repetition within the same model and similarity across different models. They called this phenomenon Artificial Hivemind. It means that even when AI models are different products, their answers can converge on the same content. The paper won a Best Paper award at NeurIPS 2025.

Seeing this result made me think about the practice of asking multiple AIs for ideas. Is switching tools alone enough to get a genuinely different perspective? Let’s look at what the research confirms—and what questions remain open.

Looking at the fragment, I compared each section carefully against the Korean source.

What Was Compared, and How

The Experimental Design: Questions Without a Single Right Answer

The research team focused on questions that don’t have one fixed correct answer.

For instance, for “what is 2+2,” you’d want the same answer every time. But requests to suggest a topic for an essay or invent a new metaphor can have many equally valid answers. The team wanted to see whether responses converged even on these kinds of questions.

They pulled 26,070 open-ended questions from WildChat, a collection of real chatbot conversations, and built a dataset called Infinity-Chat. The questions were sorted into 6 broad categories and 17 subcategories, including creative writing and idea generation. For the main experiment comparing response similarity across models, they used 100 questions drawn from this set.

Asking the Same Model the Same Question, Repeatedly

Each model was given the same question 50 times, and the researchers measured how semantically similar the resulting answers were. They also varied the token-selection method and the settings that control randomness¹.

Under top-p 0.9 and temperature 1.0, 79% of cases showed an average similarity of over 0.8 across the batch of responses. Even with min-p sampling and higher temperatures, the similarity stayed high. The finding: simply regenerating responses or tweaking settings isn’t enough to produce meaningfully diverse content.

Comparing Answers Across Different Models

Similarity remained high even when comparing different models to one another.

The paper reports average semantic similarity across major models of roughly 0.71 to 0.82. The comparison between DeepSeek-V3 and GPT-4o-2024-11-20 came out at 0.81. This score comes from converting sentences into numerical vectors and comparing them — it doesn’t mean 81% of the actual characters matched.

There were even cases of complete sentence-for-sentence matches. When asked to write an SNS slogan about success, wealth, and self-improvement, the January 25, 2025 versions of qwen-max and qwen-plus produced the exact same sentence: “Empower Your Journey: Unlock Success, Build Wealth, Transform Yourself.”

The time metaphor example we saw earlier showed how wording can vary slightly while converging on the same underlying idea. In this case, generating many answers didn’t spread ideas widely — instead, they clustered around a specific theme.

Several Possible Causes Remain

Training to Match Human Preferences

One possibility lies in how models are trained to align with human preferences. RLHF² is the best-known method here, though the paper stops short of pinning the homogenization on any single cause.

In RLHF, a reward model is built from data on which answers people prefer, and the language model is then trained to maximize that score—the goal being to generate answers that better serve users.

But if this process concentrates on the phrasing or answer styles that score highest, other perfectly valid answers may become less likely to surface. Whether preference alignment can coexist with a healthy diversity of good answers is very much an open research question.

A 2024 ICLR study by Robert Kirk and colleagues compared summarization and instruction-following tasks across two base models. RLHF handled novel inputs better than supervised fine-tuning (SFT)³, but output diversity dropped. This result, tied to one specific training method, is an important clue—but it can’t single-handedly explain homogenization across all models.

The Problem of Reflecting Disagreement Among Evaluators

The Artificial Hivemind team also examined reward models⁴ and AI judges used to score answers. When human raters disagreed, or when comparing answers of similar quality, the model’s scores lined up less well with human judgment.

For open-ended questions, multiple different answers can all be good. But if the evaluation process consistently rewards just one of them, the range of answers the model learns from ends up narrowing. The researchers argue evaluation needs to better capture this kind of divergence.

Think of it like readers disagreeing over which of two essays they prefer—what’s needed is an evaluation method that captures differences in taste. Finding the answer most people agree on and understanding how people’s preferences differ are two entirely different tasks.

The Possibility of Overlapping Training Data

Overlap in training data, and the use of synthetic data, is another plausible explanation. If multiple models train on similar internet sources, or use material generated by other AI systems, their outputs could end up resembling one another.

That said, companies don’t disclose enough about their training data and processes to say how much each factor contributed to this particular result. The research team itself noted that pretraining data, alignment methods, and memorization and data contamination all warrant further investigation.

What Does This Mean for Writing

Looking at individual pieces versus many pieces together

There’s actual research on how AI suggestions affect human writing. Anil Doshi and Oliver Hauser ran an experiment where participants wrote short stories, and some of them were given AI-generated ideas.

The stories written by people who received AI ideas were rated as better written or more entertaining. At the same time, these stories were more similar to each other than stories written without AI help. This means that even as any single piece of writing improves, diversity can shrink when you look at many pieces together. Whether the same effect shows up in actual work or long-term writing is something that needs separate study.

When you think you’ve gotten multiple opinions

A file photo of political philosopher Hannah Arendt.

What concerns me more is treating answers from multiple AIs as if they were independent opinions.

When reviewing a research idea or a business strategy, you might feel reassured by asking several models and getting similar answers. But if those answers all emerged from the same source material or similar training tendencies, agreement across multiple models doesn’t amount to independent corroboration.

The researchers themselves worry that this kind of homogeneity could lead to shared blind spots and shared errors. This particular experiment didn’t directly measure errors in medical, educational, or strategic judgment, but it does prompt a reconsideration: is simply increasing the number of models enough when reviewing an important decision?

Oswarld’s Lens

Reading this paper made me think we also need to reconsider how we evaluate AI and how we get it to compete.

In evaluations where correctness is easy to verify—like math or coding—accuracy is what matters. But for ideas and writing, we also need to check whether multiple valid answers can be generated. What worries me is a mindset so fixated on picking one “good” answer that it overlooks answers that are unfamiliar but meaningful.

In my book People Who Outsource Their Thinking: Homo Brainless, I also addressed the problem of outsourcing cognitive work. As we increasingly ask AI for ideas and writing, I think we need to be clearer about what we choose to judge for ourselves.

If we treat the first suggestion we get as merely a starting point, we risk missing other directions entirely. If I write down what I’ve already thought and observed before AI gives me an answer, I can then compare what the AI’s suggestion added—and what it left out.

Even while using AI, we can still seek out other perspectives and examine counterarguments. For important claims, we should check the original sources and scrutinize any parts that don’t match our own experience. And when writing, I believe the author should be able to explain why they chose this topic and what evidence informed their judgment.

Alongside research focused on raising performance scores, I hope more weight will be given to research on generating answers that are both appropriate and diverse—especially in creative work and idea exploration.

Closing

What stayed with me from this study is that diversity in model names doesn’t necessarily mean diversity in the answers themselves.

Regenerating a response or asking a different model can help, but it doesn’t by itself guarantee a new perspective. You still need to check what assumptions and source material the answer is actually built on.

The causes of homogenization still need more research. Only once we know how much each factor contributes — training that aligns with human preferences, evaluation methods, and training data — will we be able to pin down more precise ways to fix it.

As a writer, one thing I can do is start by organizing my own thoughts and observations first. And if a phrase the AI suggests sounds familiar, I can go back and check what that sentence is actually describing.

When I look at an idea I got from AI, I try to ask myself this: “Do I have grounds to choose this idea? Is there another idea I should weigh alongside it?”

References & Further Reading

• Liwei Jiang et al., Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond), NeurIPS 2025: this paper covers datasets, answer-similarity measures, and comparisons between human and model evaluations. It explicitly states that causal analysis of what drives this similarity falls outside its scope.
• Robert Kirk et al., Understanding the Effects of RLHF on LLM Generalisation and Diversity, ICLR 2024: compares generalization and diversity across fine-tuning methods on two base models.
• Anil R. Doshi and Oliver P. Hauser, Generative AI enhances individual creativity but reduces the collective diversity of novel content, Science Advances, 2024: a story-writing experiment involving AI-generated ideas.
• University of Washington Allen School, announcement of the NeurIPS award for the Artificial Hivemind research, January 22, 2026.
• Kwangseob Ahn, People Who Outsource Their Thinking: Homo Brainless.

The author is Oswarld (Kwangseob Ahn). Current roles: Adjunct Professor at Sejong University, Strategy Consultant at INLEVEL9. Career, research, books, and recent work are kept current on the About page. Latest · July 2026: HEMA-2: A Consolidation-Aware Tri-Memory Architecture with Multi-Channel Scheduling for Lifelong Conversational AI.

Footnotes

1. Sampling settings: the way a model chooses its next token. Temperature affects the randomness of the choice, while top-p and min-p are methods for defining the range of candidates to choose from. Min-p adjusts its threshold in proportion to the probability of the most likely token. ↩

2. RLHF: a method for training models via reinforcement learning, using reward signals derived from human preference evaluations. ↩

3. SFT: a method that further trains a model using examples of inputs and desired outputs. These examples can include tasks with a single correct answer as well as open-ended tasks like writing. ↩

4. Reward model: a model that assigns scores reflecting the preference or quality of an answer. These scores can be used to train a language model or to select among candidate answers. ↩