LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformers

Sanglyul Cho1,*, Langqing Cui2,*,†, Sehoon Kim2, Dongsu Han2, Insu Han2,‡
1Seoul National University, 2KAIST *Equal contribution   †Project lead   ‡Corresponding author
Paper Code

Looped Transformers are powerful yet suffer from slow decoding due to the inherently sequential execution of recurrent depths. LoopSpec decouples recurrent computation from the strictly sequential token-by-token decoding schedule, enabling pipelined self-speculative decoding and achieving up to 6.81× lossless acceleration on Raven models and 3.36× on Ouro models.

LoopSpec

LoopSpec is a training-free, self-speculative decoding framework for Looped Transformers. An early recurrent state drafts a token and starts computing its continuation while the original branch continues toward verification. Because every depth uses the same recurrent block, active branches can advance together in one batch.

A deeper residual second proposal with a gating mechanism provides a recovery path when the early draft is wrong. Verification keeps the accepted continuation and prunes incompatible branches, preserving the target model’s output distribution.

Decoding demo: Looped Transformer and LoopSpec

Loop Depth = 4
Prompt x1:5The Capital of France isTarget continuation: Paris and it…
Committed output
Recurrent Depth 1234Accepted tokenPruned branchActive branch

Ready to decode

    Speedups relative to autoregressive decoding

    Results across benchmarks, models, and decoding settings.

    Temperature
    Model

    Raven adapts pretrained models with recurrent layers. Ouro is a family of pretrained Looped Transformers. Explore their model collections on Hugging Face.

    LoopSpec w/ 1st proposalLoopSpec w/ 1st & 2nd proposal- - Autoregressive baseline

    How LoopSpec Works

    LoopSpec uses intermediate recurrent states to draft future tokens, without an additional draft model or training. Each draft starts a speculative continuation while its parent branch continues toward full-depth verification. These branches share the same recurrent block and advance together in a batched forward pass.

    A residual second proposal with gating mechanism provides a fallback for the first proposal. We sample the second proposal from the residual distribution between the distribution of first proposal depth and second proposal depth, instead of directly sample from the second proposal depth distribution. The gating mechanism creates the additional branch only when confidence in the first proposal drops, reducing the number of active branches and their computational cost.

    At full depth, LoopSpec verifies the first proposal and, if needed, the second. It accepts the draft token with standard rejection sampling. This preserves the target model’s output distribution while keeping computation already performed for future tokens.

    Citation

    @misc{cho2026loopspecpipelinedselfspeculativedecoding,
  title={LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformers},
  author={SangLyul Cho and Langqing Cui and Sehoon Kim and Dongsu Han and Insu Han},
  year={2026},
  eprint={2609.17184},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url={https://arxiv.org/abs/2609.17184}
}