Vibitely Daily

Friday, 20 November 2026, story 6 of 7

Google shows Streaming DiLoCo cuts cross-slice traffic 64x in MaxText

In Google's tests on two TPU v5p-32 slices, throttled links left step time flat at 5.76 s while data parallel slowed to 55.11 s.

Illustration: Two stacks of blocks joined by a thin pipe, with a steady line of small beads passing through it, beside a second pipe jammed by one oversized block.

What happened

Google says Streaming DiLoCo, implemented in MaxText, its open-source JAX training framework, cut cross-slice traffic on Qwen3-32B from 4,258.8 GB to 67.0 GB over 65 steps, a 64x to 65x reduction. The method syncs one model fragment per step instead of the whole model every H steps.

In Google's benchmark on two TPU v5p-32 slices, with inter-slice bandwidth throttled by Linux traffic control, Streaming DiLoCo held 5.76 s per step at both 30 and 10 Gbit/s. Data Parallel took 19.41 s and 55.11 s. Vanilla DiLoCo took 6.39 s and 7.55 s. Google says loss matched the Data Parallel baseline.

What it means

The gain appears where bandwidth is scarce. Unthrottled, all three methods landed at about 5.6 to 5.9 s per step, so a well-connected single datacenter gains little. The evidence is also emulated: bandwidth was capped between two slices, which is not the same as running over real metro or wide-area links. The post reports no latency tests and no runs with more than two learners.

The convergence claims are Google's own, from Qwen3-32B and 4,000-step runs on two MoE models, and the source text cuts off before the MoE results are fully described. Teams limited by cross-slice bandwidth have a reason to test this in MaxText on their own links. The post gives no pricing or support terms for Google Cloud.

What to do: If cross-slice bandwidth limits your training, benchmark Streaming DiLoCo in MaxText on your own links and compare loss against your data-parallel baseline before committing.

Source

Filed under Research. Published .

"What it means" is analysis, not reporting. Found an error? Write to daily@vibitely.com.