Google Open-Sources ML Drift, Its On-Device GPU Engine

Google Just Spun Off Its On-Device GPU Engine as a Standalone Release

Google’s AI Edge team has released ML Drift under the Apache 2.0 license. It’s a compute engine that runs AI models on the GPUs inside phones, laptops, and browsers. It’s the component that’s been handling GPU acceleration under LiteRT, Google’s on-device runtime — and this time, they’ve put it out as a standalone library you can use on its own. YouTube Shorts, Google Photos, Google Meet, and the Gemini Nano feature in Chrome are all reportedly already running on this engine.

I’m reading this announcement alongside Vons because the two projects touch the same layer. I’ve been building Vons, a small judgment model. It’s a component that sits inside a plan drawn up by a larger model, and either picks one of a handful of candidates or decides not to pick any. The design targets are a model file under 64MiB, running on CPU and in the browser. So the problem ML Drift is tackling — how do you run a model on a small device’s GPU — lands right on top of the environment Vons has to run in.

Today I’ll first explain what problem ML Drift solves, and then write up how this announcement reads from the vantage point of someone building Vons.

---

Every GPU Speaks a Different Language

In a data center, you line up rows of identical accelerators to run AI. You know in advance which chip the workload will run on, so you optimize for that one target. On devices, it’s the opposite. The moment you ship an app, you have no idea which company’s GPU, running which driver version, it will land on. Android mostly uses OpenCL and OpenGL ES, Apple devices use Metal, and browsers use WebGPU. To ask for the exact same computation, you effectively need to speak four different languages to the GPU.

Google’s earlier answer to this was the GPU delegate in TensorFlow Lite¹. The problem was that it maintained separate shaders² for each backend, and the mapping of tensors³ onto GPU textures or buffers was hard-coded into backend-specific code. With three backends, that meant maintaining three near-duplicate codebases. Tensor dimensions were also capped at four, so models that needed five-dimensional tensors — say, for handling 3D spatial data — had to resort to awkwardly folding their shapes to fit.

ML Drift: Next-Gen GPU AI/ML Inference at the Edge- Google Developers BlogGoogle AI Edge has introduced ML Drift, a high-performance, universal GPU compute framework designed to accelerate on-device AI and machine learning inference. By abstracting the complexities of hardwdevelopers.googleblog.com

ML Drift’s answer breaks down into four pieces.

Tensor virtualization. It separates the shape a model thinks a tensor has from how that tensor is actually laid out in GPU memory. Shaders are written once, as templates, and the decision about which storage scheme to use on which GPU is made at initialization time, by generating code on the fly. According to the repository’s description, this shared shader language is called UCL (Unified Compute Language), and at runtime it’s translated into GLSL, MSL, and WGSL respectively. It’s a single interpreter translating into four different national languages.

Execution tuned for generative models. A conversational language model changes how it works twice over the course of a single response. The prefill⁴ stage, which reads the input sentence all at once, is bottlenecked by raw compute; the decode stage, which emits the answer one token at a time, is bottlenecked by how fast data can be pulled from memory. ML Drift swaps in different kernels and data layouts as the model transitions between stages, and in the decode stage it says it aligns the KV cache’s⁵ layout to the convolution operation and quantizes intermediate values inside the kernel itself, cutting down on unnecessary round trips to memory.

5D tensors and custom ops. It supports five-dimensional tensors natively, and it opens up a custom-op framework that lets developers register their own shaders directly. Notably, this ships alongside a SKILL.md guide meant for coding agents. Google’s claim is that an agent can write, register, and validate a custom shader in a matter of minutes.

WebGPU outside the browser. The WebGPU backend, originally written for browsers, has been compiled through Dawn — Chrome’s WebGPU implementation — so it can run outside the browser entirely. That means on Windows and Linux, the same code runs without having to separately deal with DirectX or Vulkan, while on Mac it uses the existing Metal backend. Google calls this piece a desktop preview, and cites locally running coding agents as the use case it has in mind.

How to Read Google’s Numbers

The performance figures came along with the announcement. YouTube Shorts moved its segmentation-based effects to ML Drift, cutting average frame latency by up to 40% on both Android and iOS, and Google Photos is said to be up to 2 seconds faster than with the old GPU delegate. Features like subject selection in Adobe Lightroom and Photoshop Mobile reportedly saw on-device performance improve by up to 30%, and face/style effects in Snapchat Lenses got a 30% improvement in model latency. On Gemma model benchmarks, memory overhead was said to be up to 12% lower than other frameworks.

These numbers need to be read with two caveats in mind. Most of them are “up to” figures. The announcement doesn’t specify which device or which model produced the peak number. And the comparison baseline is, more often than not, Google’s own previous engine. The 12% memory figure is attributed only to “other frameworks,” with no indication of what exactly was being compared.

This becomes clearer when you line up the numbers for the same engine in chronological order. The ML Drift paper published in May 2025 claimed speeds an order of magnitude faster than existing open-source GPU inference engines. When Google attached ML Drift to LiteRT this past January, the figure it cited was 1.4x GPU performance over TFLite. And the per-app figures in this latest announcement are 30-40%. Every time the comparison baseline shifts, the magnitude of the number shifts with it.

What deserves more weight than the numbers is a sentence about operations. Google said it will no longer add new features to the TFLite GPU delegate. It noted that existing models will keep running fine if migrated to LiteRT’s ML Drift accelerator — which makes this, in effect, a generational handoff notice. On Android, it’s available immediately through the standalone LiteRT package, and it’s coming soon to the LiteRT bundled inside Google Play Services. On the hardware side, Google said Arm, Intel, and Qualcomm jointly optimized the kernels.

One more thing worth noting: Chrome. Chrome has attached ML Drift to the built-in AI APIs (Prompt, Summarize, Writer) that run Gemini Nano. Back in issue 102 this past May, I covered the problem of Chrome downloading the 4GB Gemini Nano model without user consent — and what got unveiled this time is, in effect, the engine that actually runs that model on a laptop’s GPU. But the engine being made public and the question of installation consent are separate issues, so this announcement doesn’t resolve that question.

Where Vons Stands

Now let me turn to my own project. The line written at the top of the Vons repository reads: “Plan globally. Decide locally. Keep the host in control.” The idea is that a large model lays out the broad plan, a decision gets made locally, and execution authority stays with the program (the host) that calls the model.

Here’s the structure. A planner — something like a frontier model — hands over the current state along with a small number of predetermined candidates. Vons returns a structured answer: it either picks one of them or abstains. It doesn’t execute tools directly, and it doesn’t treat the model’s own self-reported confidence as permission to execute. Execution, policy, and user consent all remain the host’s responsibility. The research code ships with both a Direct scorer, which scores everything in a single pass, and a conditional Diffusion scorer, which produces scores through a diffusion process.

I kept the runtime footprint small, too. English-first, with a design target of 64MiB or less for model assets, running on CPU/WASM with WebGPU as optional. The TypeScript SDK runs inference inside the browser via onnxruntime-web. It only runs on whichever backend is declared — WASM or WebGPU — and I made sure it doesn’t silently fall back to the other one if the accelerator is unavailable. I also built a Chrome extension version and submitted it to the Chrome Web Store on September 25; as of this writing, it’s still pending review.

INLEVEL9/Vons · Hugging FaceWe’re on a journey to advance and democratize artificial intelligence through open source and open science.huggingface.co

That said, Vons is a 0.1.0 research preview. The trained weights haven’t been released, and there are no publicly reviewed figures yet for browser latency or memory usage. In the repository, I’ve noted that any unmeasured value should be read as “not available,” not as zero.

Where the Two Projects Meet

Looking at the same layer doesn’t mean solving the same problem. Put them side by side, and the differences show up first.

Category	ML Drift	Vons
What it is	GPU compute engine	Judgment model and SDK
Models targeted	Everything from video effects to Gemma-class generative models	Small graders under 64MiB
Question it answers	How do I run this model fast on any GPU?	How do I make this decision near the edge instead of on a big model?
Execution path	OpenCL, Metal, WebGPU, OpenGL ES	onnxruntime-web’s WASM and WebGPU

Much of the optimization work ML Drift put in — things like the prefill-to-decode transition or KV cache batching — is built for large models that generate tokens sequentially. Vons scores a candidate once and stops, so none of that optimization applies to it directly. And in general, for very small models, the fixed cost of moving data to the GPU and launching kernels can outweigh the computation itself, which is why GPU doesn’t always beat CPU. That’s part of why Vons keeps WebGPU as an opt-in rather than a default.

On the flip side, there’s a point where the two line up exactly. The Vons SDK includes a local MCP⁶ server meant to be attached to coding agents, and there, the WebGPU option is deliberately blocked. A child process spawned from the terminal can’t get a browser’s WebGPU adapter. So to use WebGPU with Vons, you have to call the SDK directly from inside a browser or an extension.

What ML Drift did with Dawn sits right on the other side of that wall. It made browser-only WebGPU code run as-is outside the browser. Google naming local coding agents as a use case for its desktop preview, and Vons’s MCP server dropping WebGPU support at that very spot — these are two sightings of the same gap, from opposite sides. Still, Vons currently sits on the ONNX Runtime path, so using ML Drift would mean either converting the model to LiteRT format or hand-writing the computation graph with ML Drift’s C++ API. Just because the engine has opened up doesn’t mean you can switch over right away.

Oswarld’s Lens

There’s something I always say: not everyone, not every task, needs superintelligence. There’s a model suited to the job and the environment.

My own workflow is split along those lines, too. Overall planning, orchestration, the first-draft blueprint—I still hand those to Claude or GPT-6. But the long, repetitive grind I’ve pushed over to DeepSeek or GLM. Vons is an experiment in pushing that same division of labor down one more level. A large model narrows down the candidates, a small on-device model decides which one to pick, and the host decides whether to actually execute it.

Read Google’s ML Drift announcement through this lens, and you’ll see Google running the same division of labor on the hardware side. Writing shaders gets handed off to a coding agent armed with a SKILL.md file, while the layer that a small team couldn’t handle alone—the fragmentation across on-device GPUs—gets blanketed by a shared engine. Making the model small is still on the maker’s shoulders, but now there’s an additional path that doesn’t require shouldering, single-handedly, the job of running that model across dozens of different GPUs.

Still, an open engine doesn’t mean open decision-making power. As we saw with the Chrome case in Issue 102, how easily a model gets loaded onto a device and who decides what that model is allowed to do are two separate questions. The fact that Vons doesn’t treat confidence scores as permission, and instead leaves execution to the host, is itself a design answer to that second question.

Closing

ML Drift is an engine that translates the different GPU languages of every device into one common tongue, and Google has made it the default path going forward, succeeding TFLite. If you’re building small models, this announcement reads less as a set of performance numbers and more as news that the road to on-device GPUs is finally being consolidated into one.

---

💬 Is there a task among Reader‘s work that a small on-device model could handle on its own, without calling a large model? Tell me in the comments.

📨 If you know a colleague wrestling with on-device AI or agent design, pass this piece along.

References & Further Reading

Primary sources

• Google AI Edge Team, “ML Drift: Next-Gen GPU AI/ML Inference at the Edge”, 2026. ··· This is the primary source for this issue. The architecture changes, per-product performance figures, and the announcement discontinuing TFLite GPU delegate support all come from here.
• Google AI Edge, “google-ai-edge/ml-drift”, GitHub. ··· The official README laying out UCL, tensor virtualization, and supported backend configurations.
• Jiuqiang Tang et al., “Scaling On-Device GPU Inference for Large Generative Models”, arXiv:2505.00232, 2025.5. ··· The paper that first disclosed ML Drift’s design. The source of the “single-digit multiple” claim.
• Paul Krill, “Google’s LiteRT adds advanced hardware acceleration”, InfoWorld, 2026.1.28. ··· The source of the “1.4x over TFLite” figure that emerged when ML Drift was bolted onto LiteRT.

Vons

• Kwangseob Ahn, “inlevel9-com/Vons”, GitHub. ··· Lays out the scope and limitations of the research preview, along with links to the paper and technical report.
• “@vons/decision-sdk README”, GitHub. ··· Explains how WASM vs. WebGPU is chosen, and why the MCP server has WebGPU blocked off.

Past issues worth reading alongside this one

• Issue 102, “What Chrome’s Hidden 4GB AI Model Reveals About Install Consent” ··· The story of the very model that this newly revealed engine runs inside Chrome.
• Issue 178, “Kimi K3 Has 2.8 Trillion Parameters, Open Weights — So Why Can’t You Run It?” ··· Covers the diverging math behind giant models that need server racks and small models that run on-device.

The author is Oswarld (Kwangseob Ahn). Current roles: Adjunct Professor at Sejong University, Strategy Consultant at INLEVEL9. Career, research, books, and recent work are kept current on the About page. Latest · July 2026: HEMA-2: A Consolidation-Aware Tri-Memory Architecture with Multi-Channel Scheduling for Lifelong Conversational AI.

📝 Glossary

Footnotes

1. Delegate: A TensorFlow Lite connector component that hands off certain operations during model execution to an accelerator like a GPU to compute instead. ↩

2. Shader: A small program that runs on the GPU. Originally used to draw graphics on screen, it’s now also used to offload AI computation to the GPU. ↩

3. Tensor: A multi-dimensional bundle of numbers that an AI model works with. A single photo is 3-dimensional (width, height, color), and bundling multiple photos together makes it 4-dimensional. ↩

4. Prefill and decode: Prefill is the stage where a language model reads in the entire prompt at once; decode is the stage where it generates the answer one token at a time. ↩

5. KV cache: A memory space where a language model stores intermediate results so it doesn’t have to recompute what it already read earlier. It grows larger as the context gets longer. ↩

6. MCP (Model Context Protocol): A standardized protocol for how AI agents connect to external tools and data. Vons’s MCP server serves as the channel through which an agent asks Vons to make a decision. ↩