Thursday, 12 November 2026, story 5 of 7
Google Cloud open-sources TPU Sync for KV cache and weight transfer
Apache 2.0 library moves KV cache and weights across TPU memory and network. Google reports large TTFT cuts; builds are development-only.
What happened
Google Cloud has open-sourced TPU Sync, a C++ library that moves KV cache blocks and model weights between TPU HBM, host DRAM, shared storage and the data center network, InfoQ reports. It is licensed Apache 2.0. JAX and PyTorch wheels exist, but only as development builds, and the repository warns that APIs are changing rapidly.
Google says median KV transfer latency for Qwen3 235B in prefill/decode disaggregation fell from several seconds to about 230 ms. It reports average TTFT cuts of 60% for that model and 90% for a 480B model at low concurrency, and up to 90% for prefix caching on 64K-context workloads.
What it means
The numbers are Google's own, and the source reports no independent test. The TTFT cuts are quoted at low concurrency, so teams running busy clusters should not assume the same gain.
The practical draw is integration. TPU Sync works as the KV connector for vLLM-TorchTPU, as a backend for SGLang-TorchTPU, and pairs with llm-d 1.0 on GKE, so teams on those stacks have a route to try it. The limits are real, though. It is validated only on TPU v7x and v6e, the PyTorch wheel needs a torchtpu build that is not on a public index, and the storage tier and RL weight sync are experimental. That makes it a project to test, not yet to depend on. Watch for a stable release, a public torchtpu and outside benchmarks.
What to do: If you serve long-context models on TPUs with vLLM or SGLang, test TPU Sync on a staging cluster and pin to the repository's last-known-good commit.
Source
Filed under Open source. Published .
"What it means" is analysis, not reporting. Found an error? Write to hello@vibitely.com.