Text Diffusion Lifts H100 Throughput 10- to 30-Fold for Latency-Bound Enterprise AI
Updated
Updated · O'Reilly Media · Jul 20
Text Diffusion Lifts H100 Throughput 10- to 30-Fold for Latency-Bound Enterprise AI
1 articles · Updated · O'Reilly Media · Jul 20
Summary
Latency-bound enterprise AI jobs can leave eight-H100 clusters badly underused because autoregressive models generate one token at a time and cannot wait to build large batches.
At batch size one, autoregressive decoding has arithmetic intensity near 1, while diffusion refines token blocks in parallel at intensities in the hundreds, making single-stream requests compute-bound instead of memory-bound.
Inception Labs has reported about 1,100 tokens per second on H100s and roughly 1,000 on Blackwell with Mercury models, while Google’s Gemini Diffusion is in enterprise preview ahead of broader 2026 availability.
The gain is narrow but material: offline document queues already batch efficiently on standard models, but subsecond tasks such as code completion, onboarding checks and agentic document steps can require far fewer nodes under diffusion.
Economically, owned eight-H100 capacity still struggles against cheap APIs near $0.40 per million tokens, but diffusion can win where data must stay on-premises or frontier API output costs $5 to $15 per million.