F ractal LLM : Lossless Self-Speculative Decoding with Layer Embedded Self-Compression (2025)

Abstract: Autoregressive decoding in large language models (LLMs) necessitates a full forward pass for each generated token, significantly increasing inference latency. To address this limitation, we propose Fractal-LLM, a lossless self-speculative decoding method that embeds a compressed model within selected decoder layers of the original model. Specifically, our approach generates multiple draft tokens in parallel by injecting compressed layers into selected decoder layers. These draft tokens are subsequently verified through a single forward pass of the original model, ensuring the final outputs exactly match those produced by the original model. Experimental results across diverse benchmarks—including GSM8K, XSUM, CNN/DailyMail, and HumanEval—demonstrate that our method achieves substantial inference speed-ups (up to 2.47×) compared to standard autoregressive decoding, without requiring any additional training.

## introduction recently, large language models (llms) have demonstrated remarkable performance across various natural language processing tasks. however, autoregressive decoding requires a full forward pass for each generated token, causing significant latency in practical applications. speculative decoding methods @xcite mitigate this issue by proposing multiple draft tokens simultaneously and verifying them with fewer forward passes, thus reducing overall decoding steps. yet, existing approaches often require either an external draft model @xcite , additional decoding heads integrated within the original model @xcite , or employ selfspeculative strategies @xcite ., * equal contribution. † corresponding author. in this paper, we introduce fractal-llm, which embeds compressed layers within selected decoder layers, enabling parallel draft token generation in a single forward pass. generated tokens are subsequently verified by one additional forward pass of the original model, ensuring that outputs precisely match those of standard autoregressive decoding. this approach significantly reduces inference latency without auxiliary models or extra training. the contributions of this paper are summarized as follows: (1) layer embedded self-compression: we introduce a novel design that embeds a compressed model into selected decoder layers for parallel token generation without removing original layers; (2) lossless self-speculative decoding: we generate and verify tokens within a single model in parallel, preserving output quality identical to the base model; (3) efficiency analysis: we conduct a detailed study of how the number of embedded layers and the size of the draft window jointly affect inference speed-up. ## related work speculative decoding. early research on speeding up autoregressive generation introduced the idea of speculative decoding, in which a small draft model proposes multiple tokens for each step, and the main (larger) model verifies them in a single forward pass @xcite . this approach can effectively reduce the number of sequential decoding steps, but often requires an additional model that must be trained or at least carefully aligned to the target llm. some work, such as medusa @xcite , extends this paradigm by adding multiple look-ahead heads, constructing a tree of candidate continuations in parallel. recently, eagle @xcite further enhanced speculative decoding by introducing feature-level autoregression, which leverages the top-layer outputs of an llm for predicting token features, and a context-aware dynamic draft tree for improved token acceptance rates. ouroboros (zhao et al., 2024) extends speculative decoding by enabling draft models to generate entire phrases in one step, and subsequently concatenates high-quality phrases selected during verification to significantly enhance decoding efficiency. instead of relying on a second model, recent selfspeculative techniques generate drafts by skipping layers or using partial forward passes within the same network, then apply the full model to verify and correct these drafts @xcite @xcite . however, these methods commonly retain a strictly token-by-token mechanism, thus limiting potential speedups from token-level parallelism. another way to reduce latency is to generate several tokens per iteration. block-wise or multi-token algorithms @xcite split the sequence, guess a chunk of tokens in parallel, then verify or refine them before continuing. look-ahead decoding @xcite uses a modified attention mask to simultaneously predict multiple future tokens without additional training. ## evaluation setup. we benchmark three open llama checkpoints-llama-3-3b, llama-3-8b @xcite , and codellama-13b @xcite . for each llama-3-3b, llama-3-8b we randomly select 300 inputs each from gsm8k @xcite , xsum @xcite , and cnn/dailymail @xcite . codellama-13b is evaluated on the full humaneval @xcite test set. throughput (tokens / s), relative speed-up, and theoretical flops are measured under identical settings. all experiments were conducted using rtx 3090 gpus. detailed implementation and hyperparameters are described in appendix d. results. we further evaluate our method on the 13b codellama model on humaneval (