Why Alibaba Put 51 Billion Parameters in CPU RAM

Why Alibaba Put 51 Billion Parameters in CPU Memory, Not the GPU

At midnight on Wednesday (Korea time), Alibaba’s Qwen team released the weights for Qwen3.8-Flash-Next. This isn’t a finished flagship — it’s a preview meant to validate a design destined for the next-generation Qwen4. The model has 125 billion parameters in total, but only 6 billion of them actually activate when processing a single token. Bolted onto that is a 51-billion-parameter N-gram¹ memory that pre-stores frequently occurring token combinations.

What caught my eye in the spec sheet was where that 51-billion-parameter block lives. According to the documentation, it sits in the server’s ordinary system memory instead of GPU memory. I’ve written twice that “AI’s real bottleneck isn’t the compute chip — it’s HBM”². Here was an attempt to route around that bottleneck through design, not silicon.

What’s worth watching in this release isn’t the benchmark scores but the cost structure. Part of the parameters sit in ordinary server memory rather than GPU HBM; each published figure needs to be checked against what it was actually measured on; and the two Flash models released that same night were priced almost identically yet carried different licenses.

Why 51 Billion Parameters Live Outside the GPU

A language model starts by pulling one vector out of a dictionary for every single token. Qwen bolts on a second dictionary on top of that. It’s a table with 20 million slots, keyed by the combination of the previous two or three tokens. The model hashes the last few tokens into an address, pulls the corresponding vector out of the table, and mixes it into the second layer’s representation. Because this is a lookup rather than a multiplication, the compute cost per token barely increases.

Where you put this table matters a lot. Because the address is deterministic, knowing the token alone fixes exactly which slot to read — which means the CPU can fetch it in advance while the GPU is still computing the first layer. This prefetch³ overlaps with the first-layer computation, hiding the transfer latency entirely. You can see the result in vLLM’s deployment recipe: the GPU-side FP8 checkpoint stays at 172.78GiB, with one extra line tacked on — “51GB+ of host memory for N-gram offload.”

The point sharpens when you compare it to Mixture of Experts (MoE)⁴. MoE also cuts per-token compute, but all 512 experts still have to sit in GPU memory at once. The lookup table, by contrast, works fine sitting outside the GPU. The model card’s line — “efficient for memory-constrained accelerators, more offload-friendly than MoE” — captures exactly this difference.

Google got here first. Last February, a Google research paper on SCONE opened with the line “main memory and SSDs are far cheaper than accelerator memory” and proposed keeping N-gram embeddings off the accelerator altogether. By the paper’s own price table, system memory runs about $2 per GB versus $0.1 for SSD. That same year, Gemma 3n used a similar trick, parking per-layer embeddings on the CPU. As far as I know, this is the first time that lineage has been scaled up onto a 125 billion-parameter model, carrying 51 billion parameters’ worth of this kind of table.

The report is honest about its own limitation too: as you grow the N-gram vocabulary, training loss keeps falling, but downstream accuracy saturates at some point.

Attention that only looks at 2,048 tokens out of a million

Qwen3.8-Flash-Next’s layer structure. A set of GDN layers plus one QSA layer form a single block; the four circles represent the widened residual stream. (Source: Qwen)

The second component targets the cost of long context. Of the 48 layers, 36 are linear-attention layers called Gated DeltaNet (GDN). Because they compress the whole context into a single fixed-size state no matter how long it grows, cost scales only linearly with length. The remaining 12 layers — one in every four — carry real attention, and that’s exactly where Qwen Sparse Attention (QSA) sits.

In QSA, a lightweight indexer slices the context into small blocks, scores their importance, and then attends closely to only up to 512 blocks — or 2,048 tokens — per token. Even inside a 1 million-token context, what each attention layer actually looks at is just 2,048 tokens. This belongs to the same family as DeepSeek’s DSA from last year, but it tackles the indexer’s quadratic cost problem by compressing the keys. The report’s own numbers — 7.6x for prefill⁵ and 4.9x for decode at the 1 million-token mark — describe the speed of a single attention kernel, not the model as a whole.

The third component is the residual stream⁶, drawn in the diagram as four circles. The conveyor belt connecting the layers was widened from one lane to four, and each component pulls out only what it needs through a read gate and puts it back through a write gate. The report’s emphasis here is stability: in a stress test that pushed the learning rate to 4x the optimum, the old architecture’s loss spiked repeatedly while the new one held steady — and during actual pretraining, there wasn’t a single loss spike.

The fourth piece is the training recipe. Matrix weights use Muon, everything else uses AdamW, and by refitting the scaling laws to push up batch size and learning rate, they did away with batch warmup entirely — on the grounds that warmup burns through 18.8% more optimizer steps.

What Do “One-Ninth” and “8.6x” Actually Measure

The official announcement said “one-ninth the training cost compared to Qwen3.7-Plus,” but the report’s own wording is different. Training FLOPs⁷ are roughly one-ninth, and broken down, that’s one-third the active parameters multiplied by one-third the training tokens. On the base model, it leads in 8 of 14 benchmarks and trails in 6, by up to 2.6 points. Translated accurately, the claim is “comparable performance at one-ninth the FLOPs,” not “beats it across the board.” FLOPs and GPU-hours are different units, so whether actual training cost dropped to one-ninth isn’t something the report actually states.

The 7.6x figure is kernel speed, and the announcement’s 8.6x comes with a condition attached. It’s the prefill throughput advantage over Qwen3.7-Plus when the prefix cache⁸ hit rate is 90%. A 90% hit rate describes agentic workloads that read the same system prompt and tool definitions dozens of times over. This isn’t so much a performance boast as a number that reveals what kind of task the model was built to target.

The table most likely to get cited is the comparison against Claude Opus 4.6: SWE-bench Pro 62.5 vs. 53.4, CoWorkBench 73.9 vs. 68.2, JobBench 55.7 vs. 36.6. But three caveats attach to this comparison. First, CoWorkBench and RecreationBench are benchmarks Qwen built itself. Second, for SWE-bench Pro, only the Qwen line was remeasured on “a revised edition with problematic tasks fixed,” while the Opus number is simply carried over from its official release figures. It’s not the same exam. Third, Opus 4.6 is two generations old. Anthropic has since released 4.7 and then, on May 28, 4.8 — and GLM-5.3-Flash, released the same night, benchmarked itself against 4.8. On Humanity’s Last Exam (HLE), Opus leads, 40.0 to 35.9.

Judging purely by the scores in the press materials, the performance of a model with 6 billion active parameters is impressive. But the sentence “it beat Opus” doesn’t hold up once you factor in these three caveats. Even accounting for all of them, though, one difference remains stark: price. The production API version, Qwen3.8-Flash, was announced at $0.16 per million input tokens and $0.47 per million output tokens. Opus 4.8 runs $5 and $25. That’s a 31x gap on input and a 53x gap on output.

GLM-5.3-Flash Launched the Same Night: Similar Price, Different License

Another similarly priced model appeared the same night. Z.ai released GLM-5.3-Flash that night too, priced at $0.15 for input and $0.50 for output. It has 18 billion active parameters out of 320 billion total, and its parts list overlaps with Qwen’s—sparse-linear hybrid attention, plus mHC, which widens the residual stream. Z.ai added that “it runs entirely on Chinese AI chips.” The two Chinese Flash models ended up priced almost identically, around 15 cents input and 50 cents output per million tokens. In Artificial Analysis’s independent benchmarks, GLM-5.3-Flash scored an intelligence index of 57, ranking first among 173 models in the same price bracket, at a cost of $0.09 per task. But during the evaluation it generated 150 million output tokens—more than double the median. Even at the same per-token price, a model that uses more output tokens ends up costing more in total.

With prices this close, what’s left to differentiate is the license text. Starting with this model, Qwen switched its license from Apache 2.0 to the Qwen Community License 1.0. As recently as a month ago, Qwen3.8-27B still shipped under Apache. The new terms make two demands. Products with over 100 million monthly active users or over $20 million in monthly revenue must display the model’s name on-screen, and businesses that resell the model via API (Model as a Service) or offer coding/office AI-assistant products must sign a separate agreement with Qwen before commercial use. Internal use is exempt. Coding tools like Cursor, discussed earlier, and hosting companies like Fireworks that serve up Harvey’s model would, going by the letter of the terms, fall under this. This is my own reading, not legal advice.

The weights and architecture stay open, but the two use cases that overlap with Qwen’s own businesses—its API service (QwenCloud) and its assistant products (Qoder, QwenWork)—are now walled off behind separate agreements. This is another instance of the trend I covered in July’s “China Starts Locking the Door,” though the shape is different. It’s not a national export control—it’s a single company binding only the use cases that overlap with its own business to separate contracts. Z.ai, meanwhile, opened GLM-5.3-Flash more widely that same night, under the MIT license. Locking down and opening up split right down the middle within a single country.

A Workaround for the HBM Bottleneck

Last month, in “The China That Containment Built,” and in mid-August, in “Intelligence at Fire-Sale Prices,” I wrote that AI’s bottleneck is HBM — and both pieces ended with the observation that Korea sits astride that chokepoint. Today’s announcement adds one thing to that argument: when a bottleneck is big enough, designs emerge that route around it.

Qwen described its design rationale only as being for “memory-constrained accelerators” — it never mentioned HBM or export controls. The connection I’m drawing here is my own interpretation, but it’s not baseless. The recommended serving framework includes KTransformers, which offloads experts to CPU memory, and that same night, Z.ai announced it runs its model directly on Chinese chips. The harder HBM is to obtain, the bigger the payoff from this kind of design becomes.

51GB is just a fraction of a single server’s RAM. The place where HBM actually gets consumed heavily is output-token generation, which can’t be cached away — and that demand keeps growing regardless of this design. What’s changing isn’t the total volume of demand but which kind of memory that demand is directed toward. Until now, a bigger model meant needing more HBM; in this design, part of the growing capacity gets placed in server DRAM instead of HBM. For Korea’s memory industry, that sends two distinct signals: the bargaining power that comes from being a bottleneck won’t hold forever, and the workaround still ultimately leads back to DRAM. Made by the same companies — just products with different margins.

Oswarld’s Lens

What I spent the most time studying in this announcement wasn’t the architecture — it was the release sequence. According to TechNode, the stated reason for the disclosure was “to give the developer community a chance to prepare for the Qwen4 lineup.” Within a single day of the announcement, Unsloth and llama.cpp had added day-zero support, and recipes for vLLM and SGLang went up the same day. The effect: by the time the actual Qwen4 ships, the inference frameworks already support its architecture. Z.ai did the same thing in reverse. GLM-5.3-Flash spent six days starting August 20th circulating on OpenRouter as an unnamed free model called “Ox Alpha,” collecting real-world usage. Qwen disclosed the architecture first; Z.ai released the model anonymously first and gathered real-usage feedback.

I learned something similar running Notion’s Korea community. A platform doesn’t spread because its features are good — it spreads because other people are already moving on top of it before you arrive. If people have already built templates before launch day, there’s something ready to use the moment it ships. A lab that doesn’t disclose its model and architecture can’t play this game, because the architecture itself is the trade secret.

Compare this with a case from Korea, and the contrast becomes clear. LG’s EXAONE 4.0 report explained its choice of sliding window attention by saying “open-source frameworks widely support it,” while the K-EXAONE model card from this past January instructed users to “install our custom version until the library adds official support.” One side picks the architecture the frameworks already support; the other tries to make the frameworks support its own architecture. I keep thinking that when evaluating a domestic foundation model, maybe we should place “does vLLM run on launch day” right next to the benchmark scores.

Closing

To sum up, three points. First, the core of Qwen3.8-Flash-Next isn’t performance but design, and the heart of that design is the decision to push 51 billion parameters out of GPU memory and into CPU memory. Second, the 1/9th figure is a training-FLOPs comparison, the 7.6x figure is the speed of a single attention kernel, and the comparison chart against Opus carries three caveats: it’s a self-made benchmark, only the Qwen family was re-measured on a revised version, and the comparison is against a model two generations old. Even accounting for these caveats, the difference that clearly remains is price — and that price was nearly identical to GLM-5.3-Flash, released the very same night, at roughly 15 cents per million input tokens. What differed was the license. Third, this design doesn’t overturn the HBM-bottleneck thesis; it adds one condition to it. When a bottleneck is large enough, a design emerges to route around it — and the memory that detour ultimately uses is still DRAM.

One thing to try this week: open the logs from the agent workloads running in your organization and check the prefix-cache hit rate. If it’s close to 90%, this is exactly the kind of workload this model was built for. If it’s below 50%, then the 8.6x figure from the announcement has nothing to do with Reader.

If you’re running long-context agents in production, tell me in the comments how the cache hit rate is actually turning out, and whether prefill or decode is the heavier cost. I’ll gather the responses and follow up in the next issue with a piece on “the denominator of agent costs.”

---

💬 Tell me in the comments about the cache hit rate and cost structure of your long-context agents. I’ll factor it into the next issue. 📨 If you know a colleague wrestling with inference infrastructure, share this piece with them.

---

References & Further Reading

Primary sources

• Qwen Team, “On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability”, Alibaba Group, 2026.8.26. ··· This is today’s primary source. The “51B parked outside the accelerator,” “1/9th the FLOPs,” “8 of 14,” and the 4x learning-rate stress test are all here. Reading just the abstract and Section 1 gives you the design philosophy. Section 2.3 covers the N-gram embedding.
• Qwen, “Qwen3.8-Flash-Next model card”, Hugging Face, 2026.8. ··· This has the parameter breakdown, layer arrangement (GDN 3 : QSA 1), benchmark tables, and footnotes. Footnotes 2 and 5 under the table are the basis for what the piece calls the “revised exam paper” and “in-house benchmark.”
• vLLM, “Qwen3.8-Flash-Next Recipe”, 2026.8. ··· This is where “51GB+ of host memory” and the checkpoint size are documented. Worth checking too: the kernel-acceleration figures (10.2x, 6.6x) here differ from the ones in the report.
• Qwen, “Official Qwen3.8-Flash-Next announcement”, X, 2026.8.26. ··· This is the source for the previewed pricing of the full Qwen3.8-Flash release ($0.16 input, $0.47 output) and the “8.6x at a 90% cache hit rate” claim.
• TechNode, “Alibaba’s Qwen to open-source Qwen3.8-Flash-Next, previewing Qwen4 architecture”, 2026.8.26. ··· This is where the stated reason for the release — “giving the developer community a chance to prepare for Qwen4” — comes from.
• Anthropic, “Introducing Claude Opus 4.8”, 2026.5.28. ··· The source confirming the current generation and pricing ($5 input, $25 output) of the comparison model.
• Qwen, “Qwen Community License 1.0”, 2026.8. ··· The license is just two clauses. Clause 1 covers the model-naming obligation (100 million monthly active users or $20 million in monthly revenue), and Clause 2 covers the separate contract obligation for Model as a Service and AI Work Assistant use. Worth reading directly. Up through Qwen3.8-27B, the license was Apache 2.0.
• Z.ai, “Introducing GLM-5.3-Flash”, X, 2026.8.26. ··· The source for the 320 billion / 18 billion active parameters, the MIT license, the “pre-released as Ox Alpha” detail, the “runs entirely on Chinese AI chips” claim, and the pricing ($0.15 input, $0.50 output).
• Z.ai, “GLM-5.3-Flash model card”, Hugging Face, 2026.8. ··· This is where I confirmed the sparse-linear hybrid attention and mHC adoption, as well as the fact that the comparison model is Opus 4.8. The four recommended serving frameworks match Qwen’s.
• Bloomberg, “China’s Z.ai Made Ox Alpha, Stealth Model That Rivals DeepSeek”, 2026.8.26. ··· The report where Z.ai first acknowledged Ox Alpha as its own model.
• Artificial Analysis, “GLM-5.3-Flash Intelligence, Performance & Price Analysis”, 2026.8. ··· The source for the intelligence index of 57 (#1 among 173 models in the same price bracket), the cost-per-task of $0.09, and the evaluation output of 150 million tokens (median 64 million). This shows why per-token pricing and per-task cost diverge.
• Decrypt, “AI Model Ox Alpha Is Free, Beats Claude Fable, and Nobody Knows Who Built It”, 2026.8. ··· Traces the story from the anonymous release on August 20 to the community pinning it on GLM via its tokenizer. Also covers how the benchmark claims based on 10 tasks shifted once all 113 runs were completed.

Background

• Da Yu et al. (Google), “Scaling Embedding Layers in Language Models”, NeurIPS, 2025. ··· The original paper on SCONE, the design that keeps N-gram embeddings off the accelerator. The unit-cost table in Section 4.3 (roughly $2 per GB of system memory, about $0.1 for SSD) is the economics behind this design.
• InfoQ, “Gemma 3n Introduces Novel Techniques for Enhanced Mobile AI Inference”, 2025.7. ··· Explains Gemma 3n, which places per-layer embeddings on the CPU to run an 8 billion-parameter model on 4 billion worth of memory. A mobile-scale version of the same idea.
• LG AI Research, “EXAONE 4.0 Technical Report”, arXiv, 2025. ··· Section 2.1 states that sliding window attention was chosen because of “broad support across open-source frameworks.” This is where the contrast drawn in the Lens section comes from.
• LG AI Research, “K-EXAONE-236B-A23B model card”, Hugging Face, 2026.1. ··· Under requirements, it advises “install our version until the library officially supports it.”
• AlternativeTo, “Alibaba releases open-source Qwen3-Next model”, 2025.9. ··· A record from last September’s Qwen3-Next release. You can see it followed the exact same script as this one.

---

The author is Oswarld (Kwangseob Ahn). Current roles: Adjunct Professor at Sejong University, Strategy Consultant at INLEVEL9. Career, research, books, and recent work are kept current on the About page. Latest · July 2026: HEMA-2: A Consolidation-Aware Tri-Memory Architecture with Multi-Channel Scheduling for Lifelong Conversational AI.

📝 Glossary

Footnotes

1. N-gram: A sequence of N consecutive tokens. Two together is a bigram (like “machine learning”), three is a trigram. Memorizing combinations that frequently appear together lets you parse context faster. ↩

2. HBM (High Bandwidth Memory): Memory stacked right next to the GPU, exchanging several terabytes per second. It’s far faster and far more expensive than standard server DRAM, and right now it’s a component that’s hard to get enough of even with money in hand. ↩

3. Prefetch: A technique for fetching data before it’s needed. Like a chef pulling out ingredients for the next dish in advance, it starts reading from memory before the computation finishes, eliminating wait time. ↩

4. Mixture of Experts (MoE): An architecture that places several small expert networks inside a model and picks only a few per token. Total parameters are large, but computation per token is small. Even the unselected experts, however, still have to sit in GPU memory. ↩

5. Prefill and decode: Prefill is the stage where the entire input context is read in at once; decode is the stage where the answer is generated one token at a time. Longer input documents make prefill heavier; longer generated answers make decode heavier. ↩

6. Residual stream: The channel that carries information through from layer to layer. Each layer pulls information out of this channel, computes on it, and adds the result back in. Think of it like a conveyor belt. ↩

7. FLOPs: Floating-point operation count. It’s the unit used to measure the compute spent on training, but it’s distinct from actual GPU time or cost. Even at the same FLOPs, the bill changes depending on how continuously the GPUs are kept running. ↩

8. Prefix cache: A mechanism that reuses already-computed results when input sharing the same beginning is received again. Hit rates run high in agentic work where the same system prompt and tool descriptions are read over and over. ↩