Spaces:

yuhuili

/

EAGLE

like 1

Sleeping

We introduce EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency), a new baseline for fast decoding of Large Language Models (LLMs) with provable performance maintenance. This approach involves extrapolating the second-top-layer contextual feature vectors of LLMs, enabling a significant boost in generation efficiency.

• EAGLE is:
  • 3x faster than vanilla decoding (13B).
  • 2x faster than Lookahead (13B).
  • 1.6x faster than Medusa (13B).
    - provably maintaining the consistency with vanilla decoding in the distribution of generated texts.
    - trainable (within 1-2 days) and testable on 8x RTX 3090 GPUs. So even the GPU poor can afford it.
  • combinable with other parallelled techniques such as vLLM, Mamba, FlashAttention, quantization, and hardware optimization.