MASSV : Multimodal Adaptation and Self-Data Distillation for Speculative Decoding of Vision-Language Models (2025)

Abstract: Speculative decoding significantly accelerates language model inference by enabling a lightweight draft model to propose multiple tokens that a larger target model verifies simultaneously. However, applying this technique to vision-language models (VLMs) presents two fundamental challenges: small language models that could serve as efficient drafters lack the architectural components to process visual inputs, and their token predictions fail to match those of VLM target models that consider visual context. We introduce Multimodal Adaptation and Self-Data Distillation for Speculative Decoding of Vision-Language Models (MASSV), which transforms existing small language models into effective multimodal drafters through a two-phase approach. MASSV first connects the target VLM’s vision encoder to the draft model via a lightweight trainable projector, then applies self-distilled visual instruction tuning using responses generated by the target VLM to align token predictions. Comprehensive experiments across the Qwen2.5-VL and Gemma3 model families demonstrate that MASSV increases accepted length by up to 30% and delivers end-to-end inference speedups of up to 1.46x compared to conventional text-only drafting baselines on visually-grounded tasks.

## introduction large language models (llms) have transformed artificial intelligence by delivering breakthrough capabilities in reasoning @xcite , code generation @xcite , and natural language understanding @xcite @xcite . however, these achievements come with substantial computational costs, particularly during inference. the fundamental constraint arises from * work completed while on internship at cerebras systems. while sd has been well-studied for text-only models, extending it to vision-language models (vlms) introduces unique challenges. vlms process multimodal inputs by mapping image features and text tokens into a joint embedding space, enabling sophisticated visual reasoning capabilities @xcite . this multimodal conditioning presents two fundamental challenges for sd: (1) architectural incompatibility, as small language models lack the components to process visual inputs, and (2) distribution mismatch, as unimodal draft models cannot effectively capture the visually-grounded nature of the target vlm's outputs. previous approaches have addressed these challenges either by excluding image tokens entirely or by training small multimodal models from scratch @xcite . the former approach fails to leverage visual information, while the latter requires substantial computational resources and may still suffer from distribution misalignment. @xcite explored ensemble-based methods that combine multiple drafting strategies through batch inference. however, these ensemble approaches do not fundamentally address the distribution mismatch between draft and target models, instead relying on averaging predictions from multiple unaligned drafters. neither of these approaches fully exploit the poten- ## preliminaries we establish the necessary background for our approach. first, we review sd, an inference acceleration technique that uses a smaller draft model to propose tokens that are verified by a larger target model. second, we describe vlms, which combine visual encoders with language models to process multimodal inputs. finally, we discuss how sd has been adapted for vlms, including the text-only drafting baseline we compare against. speculative decoding is a technique for accelerating llm generation without altering the distribution of the generation output @xcite . in each iteration of the algorithm, a draft model m q generates multiple draft tokens that are verified in parallel by the target model m p . the algorithm continues iterating until an end-of-sequence (eos) token is generated or the max sequence length is reached. formally, let x 1:@xmath0, x 2 , ..., x t be the input sequence for the current iteration. m q first autoregressively samples γ draft tokens x t+1:t+γ , where token x t+i is sampled with probability q(x t+i |x 1:t+i-1 ). next, m p computes the probabilities p(x t+i |x 1:t+i-1 ) for @xmath1, 2, ..., γ + 1 in parallel with one forward call. these probabilities are used to evaluate the draft tokens sequentially, with the probability of accepting token slm caption phase 1: multimodal projector pretraining multimodal projector target's vision encoder image cross entropy loss slm instruction multimodal projector target's vision encoder image cross entropy loss target vlm drafting for the target vlm slm multimodal projector target's vision encoder image text draft tokens decoded autoregressively multimodal alignment of the drafter vlm phase 2: self-distilled visual instruction tuning (1) the two-phase training methodology consisting of multimodal projector pretraining followed by self-distilled visual instruction tuning, and (2) the deployment configuration for draft token generation during speculative decoding inference. components marked with the snowflake remain frozen during training to preserve their parameters, while components with the flame are trainable. this architecture enables efficient knowledge transfer from the target vision-language model to the smaller draft model while maintaining alignment in their token distributions. a new token is sampled from the residual distribution norm(max(p(•|x 1:t+i-1 )-q(•|x 1:t+i-1 ), 0)) and the iteration ends. sampling from the residual distribution ensures the output distribution of the speculative decoding algorithm is the same as the target's output distribution. in the degenerate case where sampling is disabled (@xmath2, the algorithm simplifies to greedy decoding. the draft model generates tokens by selecting x t+@xmath3:t+i-1 ). during verification, token x t+i is accepted if and only if x t+@xmath4:t+i-1 ). if rejected, the token is set to arg max x p(x|x 1:t+i-1 ). vision-language models (vlms) process multimodal inputs, consisting of visual and text tokens, by mapping the tokens into a joint embedding space. a vlm consists of three components: a vision encoder ϕ i , multimodal projector g θ , and a language model m p . given an input consisting of tokens x 1:t and visual information i, a vlm first extracts m features h 1:@xmath5. these image features are then projected into the joint embeddings space finally, the vlm samples the next token x t+1 from p(•|x 1:t , v 1:m ), where p(•|•) denotes the conditional probability distribution of m p . note that directly using sd to accelerate a vlm on multimodal inputs requires the drafter to also be a vlm. however, @xcite show that a small language model (slm) can be used as an effective drafter by conditioning it only on the text tokens in the input. concretely, given an slm drafter m q , the draft token x t+i is sampled from q(•|x 1:t+i-1 ) for @xmath6, ..., γ. we refer to this as text-only drafting and use it as the baseline in our experiments. ## methodology we introduce a method to adapt an slm into an effective draft model for llava-style vlms, which employ a modular architecture of separate vision encoder and language model components connected via a projection layer that maps image features into the language model's embedding space. our approach integrates the target vlm's frozen vision encoder into the slm through a randomly initialized mlp-based projector, preserving architectural compatibility while enabling visual processing. we then align the adapted model with the target vlm through a two-phase training protocol: (1) the projector is pretrained on paired imagetext data to establish visual grounding; and ## ablation studies we investigate the impact of self-distilled visual instruction tuning on distribution alignment, and we examine whether multimodal capability provides meaningful benefits over text-only drafting. ## related work speculative decoding has emerged as a promising technique for accelerating llm inference without compromising output quality. this approach leverages smaller, faster draft models to autoregressively generate multiple candidate tokens, which are then verified in parallel by the larger target model in a single forward pass @xcite . the theoretical foundations of this technique were established by identifying conditions under which speculative proposals preserve the original model's output distribution @xcite . recent advancements include treestructured variants @xcite @xcite , self-drafting @xcite @xcite , n-gram-based @xcite and retrieval-based @xcite that further enhance inference efficiency. however, these approaches have primarily focused on text-only models, where the draft and target operate within the same modality space. multimodal speculative decoding. extending speculative decoding to vision-language models introduces fundamental challenges absent in unimodal settings. @xcite conducted initial explorations in this domain by evaluating several draft model variants with the llava-7b architecture @xcite . their analysis across image question-answering, captioning, and reasoning tasks revealed modest token acceptance rates, with the multimodal variant achieving only marginal improvements over text-only counterparts. detailed traces demonstrated that while drafters successfully predicted function words and repeated tokens, they struggled with visually-grounded content, highlighting two fundamental challenges: (1) architectural misalignment between drafters and visionlanguage targets, and (2) distributional divergence between text-only priors and visually-informed out-puts. @xcite introduced a batch-based approach that combines predictions from multiple drafting methods to increase the likelihood of token acceptance. while their ensemble technique improves empirical performance without parameter overhead, it operates primarily as a post-hoc aggregation mechanism rather than addressing the underlying distributional divergence between individual drafters and the target model. our massv framework directly addresses these limitations through principled vision-language alignment techniques. draft model alignment. self-distillation uses a model's own outputs as training targets, extending traditional knowledge distillation approaches. while @xcite showed self-distillation can bridge distribution gaps during language model fine-tuning and @xcite demonstrated its effectiveness in mitigating catastrophic forgetting in pruned models, we extend these insights to multimodal drafting. in particular, sd 2 @xcite apply sdd to fine-grained sparse draft models, aligning them closely with their original dense counterparts and yielding substantially higher mean accepted lengths than undistilled sparse drafters. unlike previous work, we explicitly formulate self-distillation as an optimization for token acceptance probability in the speculative decoding framework. by training our draft model on responses generated by the target vlm itself rather than fixed dataset labels, we align the draft model's distribution with that of the target. ## conclusion in this work, we present massv, a method to transform slms into highly efficient speculative drafters for vlms. massv addresses challenges like architectural incompatibility and distribution mismatch by grafting the frozen vision encoder of the target vlm onto the draft model via a trainable projector and aligning the drafter's token distribution through fine-tuning on self-generated vision-language data. across both qwen2.5-vl and gemma3 model families, massv increases mean accepted length by 16-30% with end-to-end inference speedups of up to 1.46x. ablation studies show that sdd is critical for distribution alignment, and full multimodal drafting consistently outperforms text-only approaches. given its generalizability and demonstrated performance gains, massv presents a readily deployable solution for significantly accelerating vlm inference across diverse architectures and tasks.