Vibitely Daily

Sunday, 15 November 2026, story 6 of 7

Nvidia shows GLM-5.3 at 607.8 tokens per second on two DGX Stations

Nvidia splits the 464 GB checkpoint across two GB300 GPUs; throughput falls to 385.0 tokens per second at concurrency 32.

Illustration: Two identical stone blocks joined by a short bridge, together carrying one large arch, while a row of small spheres passes through and bunches up at the narrow end.

What happened

Nvidia has published a tutorial for serving Z.ai's GLM-5.3 on two DGX Stations with vLLM. In its AIPerf run on the SPEED-Bench Coding dataset, overall output throughput peaks at 607.8 tokens per second at concurrency 16, Nvidia reports. At concurrency 32 it falls to 385.0 tokens per second, with mean request latency of 453.5 seconds.

The NVFP4 checkpoint has 753 billion parameters and needs about 464 GB, against 252 GB of HBM3e on one GB300 GPU. Two Stations linked over ConnectX-8 RDMA split the weights with tensor parallelism and serve one OpenAI-compatible endpoint. The source gives no hardware price.

What it means

The useful number may be the drop, not the peak. Overall throughput went from 607.8 to 385.0 tokens per second when concurrency doubled from 16 to 32, so a self-hosted endpoint of this size likely has a sweet spot that a team must find by measurement. Nvidia itself tells readers to rerun AIPerf with their own prompts.

The 607.8 figure was measured on coding prompts with Nvidia's settings. Multi-token prediction drafts four tokens per step, and Nvidia says the gain depends on how often drafts are accepted, which varies with the prompt. Other workloads may land elsewhere. The setup is also heavy: two GB300 Stations, 464 GB of checkpoint on each disk, a context ceiling of 98304 tokens. It is a reference for teams that already own this hardware, more than a path for everyone else.

What to do: If you plan to self-host a model this size, benchmark your own prompts across several concurrency levels before sizing hardware; Nvidia's own numbers drop at 32.

Source

Filed under Chips and compute. Published .

"What it means" is analysis, not reporting. Found an error? Write to daily@vibitely.com.