Go Deeper, Stay Fast
Parallel Looped Transformer (PLT)
InfuseLoop
Simulated playback at measured speed ratios. Quality marks describe this example.
Source data
The charts compare measured eight-loop quality and decode-only latency, using the 0.6B models trained on 6B tokens. Latency uses one GB200 at batch size 32 with 2K-token prompts and outputs, separately from the illustrative playback.
These continuations are unchanged checkpoint outputs selected from 64 prompts. The animation uses uniform token intervals scaled from an earlier, validated 256-token benchmark on one GB200: about 7.7Ă— faster for InfuseLoop than LT, and nearly the same rate as PLT. It is a simulated illustration, not a timing measurement of this passage. One slowdown factor applies to every model.
Why InfuseLoop
- Depth is useful, but serial loops are costly. LT reuses one backbone for additional computation; PLT overlaps loop stages across predictions to reduce the sequential decoding cost.
- Parallel execution leaves useful states unrouted. Vanilla PLT passes only the preceding in-trajectory state between loops, leaving completed neighboring-trajectory outputs outside the recurrent input.
- Additional loops need better information flow. Our vanilla PLT runs exhibit training spikes and limited depth scaling, motivating richer inputs from representations the pipeline has already computed.

How it works
- Use non-stale states. Aggregate outputs from the immediately preceding execution cycle to reuse the freshest available contextual representations.
- Route through shallower future states. Select states from later prediction trajectories at shallower loops, preserving token-parallel training and identical training–inference routing without approximations.
Information flow
State availability

Depth scalability and Pareto efficiency
InfuseLoop improves depth scalability while preserving efficient parallel execution, expanding the performance–latency Pareto frontier of looped Transformers.
Depth scaling
Language modeling improves with depth
Downstream gains persist across depths
Performance–latency Pareto frontiers
Performance–latency frontier: validation loss
Performance–latency frontier: downstream accuracy
Long-context efficiency
Throughput approaches the non-looped baseline
Shared KV keeps additional cache storage bounded
Training curves, measurement details, and source data

Each quality point is a separately trained 0.6B model using 6B tokens. The latency benchmark uses one GB200 at batch size 32, 2K-token prompts and outputs, and the median of three decode-only runs after prefill. Context sensitivity uses 64-cycle windows, separately from the full-generation latency benchmark. CUDA Graphs, FA4, and custom G-SWA kernels are used. Cache overhead is computed from shared global KV, loop-local windows, and aggregation states. Larger and MoE models remain untested.