Solar's 42 and Mistral's 38 Used Different Rulers

The Day a 1-Trillion-Parameter Model Dropped, an Odd Comparison Followed

Reader, France’s Mistral AI recently launched the public preview of Mistral Large 4. It packs 1 trillion total parameters, with 49 billion active parameters¹ participating in any single computation. Built natively to handle text and images together, its API opened today, and the weights are scheduled for release later this month. On its official blog, Mistral introduced the model under the nickname “le Chonk”—roughly “the hefty one”—and looking at its size, that is hardly an exaggeration.

Alongside this news, grumbling began to surface in Korea: Upstage’s Solar Pro 4 is supposedly the better model, yet fails to receive a fair evaluation in the market. Solar is the proprietary Korean model lineage developed by Upstage.

Lining up the numbers alone makes the claim seem plausible. On the Intelligence Index² of independent evaluator Artificial Analysis, Solar Pro 4 scored 42 in August, while Mistral Large 4 scored 38 today. Yet these two numbers were not measured with the same ruler. Today, I want to trace when that ruler changed, and examine what actually remains of the “fair evaluation” claim once the metrics are aligned.

On September 7, the Yardstick Changed

On September 7, Artificial Analysis updated its Intelligence Index to v4.3. The approach of combining 10 tests into a single aggregate score remained the same, but the mix changed. τ³-Banking, which evaluated multi-step tool use, was dropped, and AutomationBench-AA—a 657-task workflow automation benchmark developed with Zapier—was brought in. The terminal operations test shifted from Terminal-Bench v2.1 to v4.0. The organization explained that this revision made agentic coding tasks harder while recalibrating time limits, compute caps, and scoring rubrics. It also raised the share of private, undisclosed test sets from 40% to 45% to prevent models from gaming the benchmarks through data contamination.

When tests get harder, the same model’s score drops. We were not spared from this shift either. In Issue 178 on August 5, we reported GLM-5.2 at 51 points; today, Artificial Analysis lists GLM-5.2 (max) at 34 points. The model didn’t get worse over 2 months—the markings on the yardstick changed.

The same applies to Solar Pro 4. Its 42 points was a score under the pre-revision index recorded on August 12. On the same organization’s page today, that figure is 28 points. Meanwhile, Mistral Large 4’s 38 points was measured under the post-revision v4.3.2. Put them on the same yardstick today, and their rankings actually reverse. For perspective, 38 points sits right alongside DeepSeek V4.1 Flash (39 points) in the open-weight³ camp.

But that score of 28 comes with an asterisk

Ending the story here would hardly be fair to Solar. Right now, the score of 28 on the page carries an asterisk, noting that it is an estimate and that an independent evaluation is coming soon. That means Solar Pro 4 has not yet been directly run against the revised benchmark suite. In the same table, Solar Mini 4’s score of 24 appears without an asterisk.

When Upstage introduced Solar Pro 4, it highlighted three primary improvements: terminal operations (Terminal-Bench v2.1) surged from 12 to 57, multi-step tool use (τ³-Banking) climbed from 9 to 23, and long-context reasoning (AA-LCR) jumped from 31 to 71. Yet the first two of these tests were either dropped or replaced with different versions in the September revision. In essence, the very subjects where the model posted its biggest gains have been swapped out on the exam sheet. Only re-running the evaluation will reveal whether it performs just as well on the new tests.

There is one more noteworthy detail in the design of Solar Pro 4. Upstage explains that the model was built to state that it cannot verify an answer rather than hallucinate when it lacks evidence. Artificial Analysis also confirmed a sharp rise in Solar Pro 4’s knowledge reliability score in its August evaluation, pointing out that this gain stemmed not from knowing more, but from refraining from answering when uncertain. These two explanations do not contradict each other. In tasks like contract review, where a single incorrect figure can cascade into the next step, that exact trait proves genuinely useful. It just cannot be used as proof that the model actually “knows more.”

So the Solar camp has every right to ask for one clear thing: measure us again with the same yardstick. That is an entirely different demand from complaining, “Why won’t you recognize our 42?”

Outside the Leaderboards, They Handed Over the Exam Paper

Beyond the leaderboards, an entirely different kind of verdict had already arrived. On October 5, the Seoul Economic Daily broke the news that Upstage had been selected as the top-ranked bidder for a project commissioned by the Korea Institute for Curriculum and Evaluation (KICE): “Development of AI-Based Automated Question Generation for the Suneung English Exam” (Suneung, Korea’s national college entrance exam). 13 companies competed in the tender, and KICE allocated 90% of the composite score to technical evaluation. It was a race decided by technology rather than price. While the formal contract is still pending, the target development completion date is May next year. If everything proceeds as planned, the system could be deployed in drafting actual Suneung exam questions as early as next year.

[Exclusive] Upstage to Lead AI Question Generation for Suneung English Exam[Exclusive] Upstage to Lead AI Question Generation for Suneung English Examsedaily.com

In this project, the AI is not taking the exam; it is writing it. Once operators set achievement standards, question types, vocabulary tiers, and subject materials, the system generates reading passages, multiple-choice options, correct answers, and detailed explanations, calculating predicted correct-answer rates and score-tier distributions for each new question using historical test-taker data. Dedicated agents handling question generation, verification, similarity and plagiarism screening, and difficulty prediction pass results back and forth to refine the questions. Right from the proposal stage, KICE also required the system to be designed so it could expand across every subject area of the Suneung.

A model vendor once graded on benchmark exams is now engineering the question-writing pipeline for a national entrance examination. While an intelligence index asks how high a model scores, KICE asked whether it could draft and verify Suneung English questions under these exact operational conditions. Industry observers note that Upstage’s engineering track record in developing its proprietary Solar models weighed heavily in the selection. That said, the reports do not specify which Solar model variant will actually power the system.

What Mistral Is Selling Lies Beyond the Scoreboard

Conversely, reducing Mistral Large 4 to a single intelligence index score misses much of the point. What Mistral took the trouble to emphasize in its announcement was not the number, but where and by whom this model is run.

The model was trained from scratch on 3,800 Nvidia Grace Blackwell GPUs in Mistral’s own European data centers, and the preview is served from the very same infrastructure. Mistral explicitly stated that European deployment is operated end-to-end under European law without relying on third-party digital service providers. Its training data covers over 160 languages, including every official EU tongue. In Mistral’s phrasing, it stands far ahead of open-weight models from the US and Europe, and competes directly with the world’s top open models. Factor in the promise to release the weights so customers can host it on their own servers, and the value proposition becomes clear: what Mistral is selling is an “all-European AI.”

Its cybersecurity results should be read in the same light. In an evaluation requiring models to reproduce and patch real-world open-source software vulnerabilities, it ranked 1st across all models at 82%, and solved 93% on Cybench, an evaluation of 40 Capture the Flag challenges. Yet Mistral offered an important caveat of its own: leading closed models like Claude Opus 5.5 and GPT-6 Astra scored near 0 in the reproduction and patching benchmark simply because they refused to perform the tasks. This 1st-place finish reflects a difference in refusal policies as much as a gap in capability.

Mistral turns precisely this nuance into its sales pitch. Defending an organization starts with verifying whether a flaw is real; if an engineer is blocked mid-incident response by a vendor’s safety filter, that refusal itself becomes a security risk. At the same time, Mistral noted that its refusal rate for explicitly malicious cyber requests remains higher than that of other open models. The proposition is straightforward: let the deploying organization decide what to permit and what to deny based on its own policies, rather than leaving the choice to the vendor.

The cost figures are less flattering. Pricing sits at $1.36 per 1 million input tokens and $4.18 per 1 million output tokens, working out to an average of $1.13 per index task on Artificial Analysis. According to one report, the model generated roughly 200 million output tokens across the full evaluation suite, compared to a median of 81 million for comparable models. In plain terms, it is rather verbose when doing the same work.

Mistral also noted that the model is still in training. Reinforcement learning⁴ powered by 3,000 GPUs generates 33 billion tokens daily, of which 16 billion are filtered and fed back into training—with no signs of performance saturation in sight. The company is also scaling up compute inside Europe backed by a €3 billion Series D round. Today’s score of 38 is merely a snapshot taken during the preview run.

Oswarld’s Lens

When I design enterprise deployments or AI adoption consulting, I work from one core premise: models will keep improving. I do the same when setting up air-gapped networks⁵ for public institutions. I don’t design projects around the models available right now; I build around where open-weight models are likely to be 6 to 12 months down the road.

I also have a personal rule I repeat constantly: not everyone and not every task needs artificial superintelligence. The right model is the one matched to the work and its environment.

Viewed through this lens, asking “Solar is better, so why wasn’t it picked?” compares two entirely different categories. What you check first in an air-gapped network deployment isn’t a 4-point lead on a leaderboard. You check whether you can bring the weights inside your perimeter, what the license permits, and whether the model can complete the work using your proprietary documents in your language. Solar Pro 4 is a commercial model served via API, and on-premise deployment requires a separate enterprise inquiry. Upstage offers a separate open-weight line, Solar Open 2. If you want a model in the same category as Mistral Large 4, the counterpart right now is Solar Open 2, not Solar Pro 4.

That is why I believe that rather than complaining Korean models are undervalued, demonstrating what they can achieve, on which tasks, and under what constraints is what actually shifts perception. The project with KICE (Korea Institute for Curriculum and Evaluation) is a case in point. Upstage earned its technical evaluation score not from 4 points on a generic leaderboard, but by tackling the concrete task of generating and validating English questions for Suneung (Korea’s national college entrance exam).

The Korean-language legal, medical, and administrative benchmark scores Upstage released can serve as that raw proof. However, those scores were measured inside Upstage’s internal testing environment. If they hold up when independent third parties replicate the benchmarks, the numbers will speak for themselves—without any need for grievance.

Closing

Every number on a leaderboard comes with a date and a version tag. Whenever you see a claim comparing two models, simply checking whether those two variables match—and whether the figure was directly measured or merely estimated—will filter out most of these misunderstandings.

---

💬 Have you ever used a model that felt completely different from its leaderboard ranking? Drop a quick one-line reply to let us know which model it was.

📨 If a colleague at work is putting together an AI model comparison sheet, share this post with them.

---

References & Further Reading

Primary sources

• Mistral AI, “Introducing Mistral Large 4”, October 6, 2026. ··· The source for parameter composition, training infrastructure, cybersecurity scores alongside refusal benchmarks for closed models, pricing, and reinforcement learning compute scale.
• Artificial Analysis, “Announcing the Artificial Analysis Intelligence Index v4.3”, September 2026. ··· Details what was added or removed in the September revision, and how the weighting of private benchmarks changed.
• Artificial Analysis, “Solar Pro 4 Model Page”, accessed October 6, 2026. ··· Notes that the 28 score is an estimate and explains the v4.3.2 baseline criteria.
• Artificial Analysis, “Upstage Provider Page”, accessed October 6, 2026. ··· Compares Solar Pro 4, Solar Mini 4, and Solar Pro 3 side by side.
• Artificial Analysis, “Upstage Solar Pro 4: Benchmarks and analysis”, August 2026. ··· The source for the pre-revision index score of 42, and the finding that gains in knowledge reliability stemmed from withholding answers when uncertain.
• Upstage, “Solar Pro 4: An Agentic Model Built for End-to-End Execution”, August 11, 2026. ··· Details the 3 core performance improvements, hallucination-resistant design, division of labor with Solar Open 2, and internal benchmark tables.
• Ryu Seok, “[Exclusive] Upstage Selected to Draft Suneung (Korea’s National College Entrance Exam) English Questions with AI”, Seoul Economic Daily, October 5, 2026. ··· The source for Upstage ranking 1st in the Korea Institute for Curriculum and Evaluation (KICE) procurement project, the 13 competing bidders, the 90% technical evaluation weight, and the multi-agent question-generation setup. (As an exclusive report, formal confirmation from KICE has not yet been published.)

Background

• Artificial Analysis, “DeepSeek V4.1 Flash Release Page”, September 2026. ··· Source for the 39 score compared against 38.
• Trending Topics, “Mistral Large 4 on Artificial Analysis”, October 6, 2026. ··· Reporting that compares roughly 200,000,000 output tokens against the median of 81,000,000.

Related Past Issues

• Issue 170, “Are AI Benchmarks Working as Intended?” ··· Argued for scrutinizing the evaluation process rather than the headline score. Today’s essay explores what happens when that evaluation process changes.
• Issue 178, “You Can Download It; You Just Can’t Run It” ··· Examined the divide between open weights and operational deployment capability. The GLM-5.2 score of 51 mentioned today was covered here.
• Issue 146, “Fable Can’t Go to China, but GPT Can” ··· Examined who genuinely holds the reins of sovereign AI.

---

The author is Oswarld (Kwangseob Ahn). Current roles: Adjunct Professor at Sejong University, Strategy Consultant at INLEVEL9. Career, research, books, and recent work are kept current on the About page. Latest · July 2026: HEMA-2: A Consolidation-Aware Tri-Memory Architecture with Multi-Channel Scheduling for Lifelong Conversational AI.

📝 Glossary

---

Footnotes

1. Active Parameters: In a Mixture of Experts (MoE) architecture, the model routes each token through only a selected subset of “experts” rather than activating all parameters at once. The subset used per inference pass represents active parameters. A model with 1,000,000,000,000 total parameters might activate only 49,000,000,000 per token, significantly cutting compute costs relative to its overall size. ↩

2. Artificial Analysis Intelligence Index: An aggregate score scaled from 0 to 100, calculated as a weighted average across multiple benchmark tests by independent benchmarking firm Artificial Analysis. Because a model’s score shifts whenever the underlying benchmark suite changes, scores must always be evaluated alongside their index version. ↩

3. Open Weights: Models whose trained weight files are publicly released, allowing anyone to download and run them on their own hardware. This differs from open source, which additionally discloses training data and codebases. ↩

4. Reinforcement Learning (RL): A training method where a model attempts tasks autonomously and receives rewards for successful outputs. Because tasks can scale in difficulty as the model improves, RL is widely used to continuously push the frontier of already-large models. ↩

5. Air Gap: A network physically isolated from the external internet. Used by government agencies and security-sensitive organizations, air-gapped environments cannot call external cloud APIs and therefore require models deployed entirely on local infrastructure. ↩