Google TPUs Hit 700 Tokens Per Second in Meta Production

Google has successfully integrated its custom tensor processing units into Meta's production AI infrastructure, marking a significant shift in the competitive landscape for artificial intelligence hardware. This move was enabled by the release of a new, PyTorch-native version of the SGLang inference engine, which allows the chips to operate without requiring extensive code rewrites. Simultaneously, Google announced an open-source version of vLLM, another popular inference engine, to support this new integration.

The strategic pivot allows both inference engines to run natively on Google's hardware using the PyTorch framework, moving away from the previous reliance on Google's own JAX system. Zongwei Zhou, Meta's lead for PyTorch on TPUs, confirmed that the new SGLang-TorchTPU stack is already live in production at Meta. He described the achievement as the result of a close three-way collaboration between Meta, Google, and the SGLang team, noting that the system provides a remarkably mature serving environment for PyTorch workloads.

Jana van Greunen, director of PyTorch engineering at Meta, emphasized the importance of hardware diversity in a statement accompanying the announcement. She noted that Meta cannot be constrained by a single hardware architecture. According to Greunen, the new TorchTPU initiative enables the company to deploy its PyTorch workloads directly to Google's chips, a capability that Google co-developed with Meta's engineering team.

This is not the first time Meta has experimented with Google's silicon. The older PyTorch/XLA bridge, which Google now terms a predecessor to the current system, was used by Meta research teams from 2020 to 2023 before contracts were canceled. SemiAnalysis reported that these earlier efforts were primarily aimed at accommodating Meta's preference to avoid migrating to JAX. While the exact location of Meta's new production units is not public, reports from last year suggest talks included Google Cloud rentals for 2026 and purchases for 2027, with a multi-year rental agreement reportedly signed.

The timing of the announcement is notable, coming just one day after Amazon revealed that it is open-sourcing TorchNeuron, a native PyTorch backend for its Trainium chips. Both developments appear designed to loosen the software lock-in that has long favored Nvidia in the AI sector. By making alternative hardware more accessible to the dominant PyTorch ecosystem, Google and Amazon are aiming to fragment the market.

Performance benchmarks suggest the new setup is highly competitive. A tuned vLLM configuration serving Moonshot AI's Kimi K3 model, which has 2.8 trillion parameters, reportedly achieved over 700 tokens per second for a single user. This test used the seventh-generation Ironwood TPU. The figures come from Inferact, the startup behind vLLM, which helped build the new stack. The test utilized 16 Ironwood chips with speculative decoding, a method that speeds up inference using a small draft model. In comparison, vLLM's published setup on 16 Nvidia GB200s reached 452 tokens per second.

Inferact also reported that its kernels for Kimi K3 and Qwen 3.8 27B deliver roughly 1.4 to 2 times the throughput of the GB200 baseline at small batch sizes, even without speculative decoding. These results indicate that Google's hardware can match or exceed Nvidia's performance in specific inference scenarios, particularly when optimized for specific models.

Looking ahead, the SGLang-TorchTPU project plans to expand its model support, starting with DeepSeek V4.1 Flash and Reflection Beam. RadixArk, the startup founded by the creators of SGLang, aims to provide Day 0 TPU support for new open models by the first quarter of 2027. This means that when new models are released, they will have immediate compatibility with the hardware. Google, Meta, and RadixArk are scheduled to share further details at the SGLang Summit 2026, taking place on November 12 and 13 in San Francisco.