Nvidia Raises Server Prices 15% on Memory Costs

A 15% Server Price Hike, and What Nvidia Just Admitted

Nvidia has notified major customers of a price increase on its servers.

Bloomberg reported on August 22, citing sources, that Nvidia told some of its major customers it would raise prices on servers equipped with AI chips by more than 15%. The increase applies to systems shipping from early next year, and it covers both the flagship Grace Blackwell line and the next-generation Vera Rubin. The reason: memory costs.

Bloomberg called the situation unusual. Nvidia is widely considered the most profitable company in the semiconductor industry, and yet here it is, passing on a cost increase to customers instead of absorbing it.

There’s a number that captures just how much memory now weighs on costs. According to one analysis, for the Vera Rubin system, which began shipping this month, memory accounts for 62% of total system cost.

The center of gravity in AI chip competition is shifting from compute to memory. And that shift is opening a window for NPU makers who, until now, have been overshadowed by Nvidia.

AI Memory Is Splitting Into Three Layers: HBM, HBF, and SOCAMM2

If you’ve looked at brokerage research lately, you’ll notice something new: AI memory is now being drawn as three distinct layers.

HBM is the high-bandwidth memory we already know. It sits right next to the accelerator as ultra-fast working memory — with HBM4, that’s 36GB per stack and bandwidth north of 2.8TB/s. It’s the fastest tier, but also the most expensive, and it’s constrained on both capacity and packaging.

HBF¹ is slower than HBM but holds far more. Built on NAND flash, its published specs target up to 512GB per package while aiming for 0.4–3.0TB/s of bandwidth. The idea is to pair HBM-class speed with much larger capacity. The tradeoff is NAND’s usual baggage — latency and endurance issues — and the tier is still in its early days.

SOCAMM2 is the third layer. It takes the low-power DRAM originally built for smartphones and repackages it as a server module. Compared with RDIMM, the standard server memory module, it delivers more than double the bandwidth and over 75% better energy efficiency. Because it plugs in via a connector rather than being soldered to the board, the module can be pulled out and swapped.

The capacity figures need to be read carefully, separating announcement timing from mass-production timing. Materials commonly cite “up to 256GB,” but that’s the spec Micron announced last March when it began sampling to customers. The actual mass-production workhorse is 192GB — Samsung Electronics started production first in March on its 1b process, with SK Hynix following in April on 1c. What’s more interesting, though, is news moving in the opposite direction: reports in June suggested Nvidia is considering cutting SOCAMM2 capacity from 192GB down to 96GB as low-power DRAM supply tightens. It’s a single-source report, so it shouldn’t be treated as confirmed, but if true, it’s telling — a case of a finished-product spec getting downgraded simply because the parts can’t be sourced.

Relying on HBM alone creates problems. Running a large model for inference means loading the entire model weights and conversation history into memory, and 36GB per stack hits its limit fast. You might think stacking more layers would fix it, but the higher you stack, the more heat, power draw, and yield loss you get — and the price climbs right along with it. In the end, “more of the fastest memory” is a dead end, both physically and economically. So it becomes rational to split the load: frequently accessed data stays on HBM, while larger, less time-sensitive chunks move down a layer.

Why is the number of layers growing at all? Because AI’s center of gravity has shifted from training to inference. Training concentrates computation into a short burst, which favors the fastest memory available. Inference is different — it means running large models continuously, cheaply, for a long time. Here, raw speed alone isn’t the point; what matters is how you balance speed, capacity, and the power bill. That’s why a structure that once tried to handle everything with one type of memory has split into multiple layers.

The keynote SK Hynix delivered at FMS 2026, the memory industry conference held in the US on August 4th, captures this shift perfectly. Its title was about orchestrating AI infrastructure with tiered memory for the agentic AI era.

Nvidia scrapped its own inference chip and brought in an outside LPU instead

The clearest illustration of this shift didn’t come from a memory company — it came from Nvidia. Let’s place two events from the past year side by side.

First, the chip that got shelved. In September 2025, Nvidia unveiled Rubin CPX, an inference-only accelerator. Its defining feature was using GDDR7 instead of HBM. Nvidia claimed it delivered 7.5x higher AI performance than the previous generation despite dropping the pricier HBM, and positioned it as a product specialized for million-token processing and video generation. It was effectively a declaration that “inference can run on cheaper memory.”

But at GTC 2026 this past March, Nvidia pulled the chip from its roadmap — just 6 months after unveiling it.

Second, the chip that got bought in. The Vera Rubin platform, also unveiled at GTC 2026, consists of 7 new chips: the Vera CPU, Rubin GPU, NVLink 6 switch, ConnectX-9 SuperNIC, BlueField-4 DPU, Spectrum-6 Ethernet switch, and the Groq 3 LPU².

The last item on that list — the Groq 3 LPU — isn’t a chip Nvidia designed in-house; it’s one Nvidia brought in from outside. In this architecture, Nvidia split the workload between GPU and LPU: large-scale data computation goes to the GPU, while ultra-low-latency response goes to the LPU. Nvidia said this combination boosts inference throughput for trillion-parameter models by up to 35x. Compared to the previous-generation Blackwell, the figures cited include cutting the number of GPUs needed to train mixture-of-experts models³ to a quarter, a 10x gain in both inference throughput and performance-per-watt, and a drop in cost per token to a tenth. CEO Jensen Huang declared on stage that inference had reached its inflection point.

Put the two events together and here’s how they read: Nvidia tried to build in-house the answer to “GPU-plus-HBM isn’t the best combination for inference” — and shelved it. So it bought that answer from outside instead.

Why this matters: up to now, the NPU camp has been the one making this argument. Their logic was that GPUs were born as general-purpose chips for graphics, so using them for inference creates waste — and a purpose-built inference design would have the edge in power efficiency and latency. Here, the market’s dominant player just conceded that logic through its own product lineup.

Accelerator Makers Join the Memory Standards Table

On August 4th, SanDisk and SK Hynix unveiled the first HBF standard through OCP⁴, and Google and Tenstorrent joined the standardization process as consortium members, participating in technical validation and spec-setting.

Google makes TPUs; Tenstorrent makes NPUs. That means it wasn’t just memory companies at the table where memory standards get decided — accelerator companies were there too. Samsung Electronics and Micron are still on the sidelines, watching.

The performance of inference chips is no longer determined by chip design alone — it’s heavily shaped by which memory tier the design assumes. That’s why accelerator companies are getting involved early in discussions that set the tier standards that will favor their own chips.

Let’s look at Korean companies too. Rebellions has launched its REBEL Quad, built on HBM3E, and is running field trials in Japan, Saudi Arabia, and the US. FuriosaAI has secured a deployment case with LG’s EXAONE and set a target of $100 million in revenue this year. Both companies have pitched power efficiency and low latency as their strengths in targeting the inference market.

But what I’m watching closely is what memory these companies actually use. REBEL Quad runs on HBM3E. That’s a reasonable choice for securing performance, but it also means a substantial share of the cost is locked into components shared with Nvidia. As we saw earlier, the same structure — memory eating a growing share of system cost — applies just as much to domestic NPUs. When you’re using the same expensive memory, it’s hard to win on price. If HBF or low-power DRAM tiers actually open up, domestic NPUs would gain a chance to build a fundamentally different cost structure — even before the performance race is settled.

That said, I should be honest about the other side of this too. If Nvidia folds inference into its own platform, it’s rational for customers to just stick with the ecosystem they already know. The cost of learning a new software stack can outweigh the savings on power costs. Korean NPU companies have been targeting inference — a relatively less contested space — but now the dominant player has moved directly into that same territory. Opportunity and risk are coming from the same event.

But last week’s price-hike news tilts this balance a little. If server prices rise more than 15%, the cost of sticking with the status quo keeps climbing. Meanwhile, the cost of switching ecosystems is mostly a fixed cost you pay once. So the point at which switching costs pay for themselves gets pulled forward. In fact, commentary on this report has suggested that surging server prices could accelerate Big Tech’s shift toward in-house chips.

But the same commentary also flags the other side: whether companies expand in-house chips or keep using Nvidia, they still need to secure enough memory from Samsung Electronics, SK Hynix, and Micron. Either way, memory supply is what matters — which again shows that the center of gravity has shifted from chips to memory.

So here’s how I see the future of NPUs. Going forward, what will decide outcomes won’t be benchmark scores, but which memory tier — among the now-separated tiers — a design targets. In fact, Korean companies are already branching out beyond an all-out data center war, into edge, on-device, and physical AI. Just as computing once split from mainframe-centric architecture into PCs and mobile, the outlook is that AI semiconductors will similarly fragment into inference, edge, and on-device segments.

Oswarld’s Lens

There’s something I kept confirming while working through GTM strategy. The point where a late-mover product beats the incumbent is rarely the spec sheet. It’s usually the standard, or the distribution.

You’d think that pushing performance higher under identical conditions would win, but in practice, whoever sets the rules of competition holds the advantage. That’s why I read Google and Tenstorrent joining the HBF standardization consortium not as simple technical cooperation, but as a move to stake out ground in the market. They took a seat at the table where the memory specifications suited to their own chips get decided in advance.

I think Korean NPU companies need the same thing right now. Rather than proving they’re a few percentage points better than Nvidia on performance-per-watt, securing a seat at the discussions that will set the next-generation memory hierarchy standard has far more long-term payoff. And if they’re going to ask for government support, I think that’s a better use of funding than R&D subsidies.

One caveat: this isn’t a conviction, it’s an observation with conditions attached. Joining a standard is only the starting line. Whether developers actually build and use software running on top of that standard is a separate question entirely.

Looking at the draft against the source, everything matches well: numbers (15%, 62%) are preserved, structure is intact, no Hangul remains, and the meaning is accurately conveyed. No surgical edits needed.

Closing

If I had to compress today’s story into three lines, it would look like this.

First, the reason Nvidia raised server prices by more than 15%, and the reason memory makes up 62% of Vera Rubin’s cost, are the same story. The bottleneck has shifted from compute to memory. Second, that’s why memory is splitting from a single HBM standard into a tiered structure that includes HBF and SOCAMM2 — and why Nvidia tried developing its own inference chip without HBM, shelved it, and brought in an LPU from outside instead. Third, amid all this change, the deciding factor in NPU competition is shifting from chip performance to who locks down the memory hierarchy specification first.

So the next time you read AI chip news, I’d suggest not just looking at the performance multiples — look at what kind of memory the chip was designed around. These days, that’s where more of the real information is hiding.

If you’ve ever tried to cut inference costs in your own work, I’d love to know what held you back the most. Was it chip prices, electricity bills, or the cost of migrating software? Let me know in the comments — it might become material for the next issue.

This piece analyzes industry structure and is not investment advice. The companies and figures mentioned are based on publicly available material and media reports as of their respective dates; some are drawn from exclusive reporting or industry estimates and are not confirmed fact.

---

💬 Tell us in the comments what ate up the biggest share of your inference costs · 📨 Share this piece with a colleague who follows chips or infrastructure

---

References & Further Reading

Primary sources

• Ian King et al., “Nvidia Customers Notified About AI-Related Price Hikes Above 15%”, Bloomberg, 2026. 8. 22. Link ··· This is where today’s piece starts. If the paywall blocks you, you can catch the gist through domestic outlets that quoted it. Pay less attention to the size of the hike than to the word “unusual.”
• SanDisk & SK Hynix, “Sandisk and SK hynix Advance Global Standardization of High Bandwidth Flash with Release of First OCP Technical Specification”, 2026. 8. 3. Link ··· The original document explicitly names Google and Tenstorrent as consortium members. The single most important line in today’s issue is buried in here.
• Youngho Kim, “[GTC 2026] Nvidia Unveils ‘Vera Rubin’ Specs: 3x Compute, 10x Inference Efficiency”, Electronic Times, 2026. 3. 17. Link ··· You can see the list of 7 new chips as-is here. Why the Groq 3 LPU is on that list is today’s central question.
• Kijong Lee, “Nvidia’s ‘Rubin CPX’ Launch Uncertain — No Memory or Substrate Orders Placed”, TheElec, 2026. 5. 27. Link ··· This is the record of a chip that got shelved. Coverage of a publicly announced product quietly vanishing is rare, which makes this a valuable find.
• SK Hynix Newsroom, “SK Hynix Begins Mass Production of SOCAMM2 192GB”, 2026. 4. 20. Link ··· The primary source for the bandwidth and energy-efficiency figures compared against RDIMM.

Background

• Eugene Investment & Securities, AI memory comparison data (graphic via Maeil Business Newspaper’s Maekyung Plus) ··· The table in section 1 of today’s issue comes from here. Note that the capacity figures are spec ceilings, so distinguish them from actual mass-production timing.
• Jeongho Han, “[On the Ground] Domestic NPUs Take On Nvidia’s Dominance with Inference and Physical AI”, ZDNet Korea, 2026. 5. 28. Link ··· You can hear the strategies of Rebellions, FuriosaAI, and Mobilint straight from the companies themselves.

---

The author is Oswarld (Kwangseob Ahn). Current roles: Adjunct Professor at Sejong University, Strategy Consultant at INLEVEL9. Career, research, books, and recent work are kept current on the About page. Latest · July 2026: HEMA-2: A Consolidation-Aware Tri-Memory Architecture with Multi-Channel Scheduling for Lifelong Conversational AI.

📝 Glossary

Footnotes

1. HBF (High Bandwidth Flash): Memory that stacks NAND flash in many layers to approach HBM-level speed while packing in far greater capacity. Think of it as a layer that fills the gap between HBM and SSDs. It began with a SanDisk–SK Hynix agreement in August 2025, and the first standard specification arrived this August. ↩

2. LPU (Language Processing Unit): A processor specialized for language-model inference, particularly for cutting response latency. It’s made by the American company Groq — spelled and owned completely differently from Grok, the chatbot from Elon Musk’s xAI, so don’t mix them up. ↩

3. Mixture of Experts (MoE): Instead of one giant model answering every question, the model is split internally into multiple “expert” modules, only some of which activate for any given query. This cuts compute for a given model size, which is why many recent large models have adopted it. ↩

4. OCP (Open Compute Project): An organization where multiple companies jointly develop open standards for data-center hardware. Once a spec is set here, both component makers and chipmakers design their products around it. ↩