S teer VLM : Robust Model Control through Lightweight Activation Steering for Vision Language Models (2025)

Abstract: This work introduces SteerVLM, a lightweight steering module designed to guide Vision-Language Models (VLMs) towards outputs that better adhere to desired instructions. Our approach learns from the latent embeddings of paired prompts encoding target and converse behaviors to dynamically adjust activations connecting the language modality with image context. This allows for fine-grained, inference-time control over complex output semantics without modifying model weights while preserving performance on off-target tasks. Our steering module requires learning parameters equal to 0.14% of the original VLM’s size. Our steering module gains model control through dimension-wise activation modulation and adaptive steering across layers without requiring pre-extracted static vectors or manual tuning of intervention points. Furthermore, we introduce VNIA (Visual Narrative Intent Alignment), a multimodal dataset specifically created to facilitate the development and evaluation of VLM steering techniques. Our method outperforms existing intervention techniques on steering and hallucination mitigation benchmarks for VLMs and proposes a robust solution for multimodal model control through activation engineering.

## introduction large vision language models (vlms) demonstrate remarkable capabilities across a wide range of tasks such as image captioning @xcite , visual question-answering @xcite , and more. yet, effectively eliciting their full potential remains a challenge. naive prompting does not guarantee optimal performance on tasks that are highly dependent on following instructions @xcite . techniques such as chain-of-thought prompts @xcite and few-shot examples @xcite have been shown to significantly improve large language model (llm) performance without modifying the underlying model. this suggests the presence of an "elicitation overhang": a gap between a model's potential and our ability to fully access it @xcite . we often lack a complete understanding of how to best extract the desired behavior or knowledge from the model. prompt engineering is a primary method for guiding model behavior, but its effectiveness diminishes with complex inputs or nuanced desired outputs @xcite . in vlms, where models must integrate both visual and textual inputs, prompt engineering becomes less effective and makes it harder to elicit good vlm responses. to address the limitations of prompt-based approaches, we present steervlm, an inference time intervention based steering approach that aligns model outputs with desired instructions (figure 1 ). rather than relying solely on input prompts, we utilize latent space vector arithmetic to modify a model's internal activations, specifically its hidden states, during inference. according to recent literature @xcite , directly modifying the model's activation can more effectively steer a vlm's response than prompt engineering alone. recent lightweight approaches adjust llm outputs without resorting to full-scale fine-tuning or prompt-tuning. however, these methods often exhibit several limitations. existing methods often require extracting static steering vectors from a dataset containing steered and unsteered responses @xcite @xcite . furthermore, many methods require interventions at a predetermined, hyperparameter-tuned layer and apply the same uniform steering vector for each generated token @xcite . finally, a significant challenge is that they often do not adapt well to the complexities of a multimodal setting @xcite . all these factors can limit their overall adaptability and effectiveness in steering. in contrast, we introduce a novel, parameterized steering mechanism that enhances control over model behavior without the inflexibility and potential loss of generalizability associated with fine-tuning. we achieve this by training a lightweight steering module that learns to predict adjustments by analyzing pairs of prompts that encode both the desired target behavior and its converse. at inference time, this trained module dynamically computes and applies these adjustments to the vlm's intermediate hidden states. our contributions extend these ideas into the multimodal vlm setting, where language must be grounded in vision. here, the steering is applied to the language module of the vlm -post projection of visual features in the language space. this approach allows us to explore controlled generation in multimodal models, following prior methods that intervene solely on the llm backbone of vlms @xcite . unlike prior methods that are often limited to direct subtraction of averaged activations @xcite , our module learns a more complex, non-linear mapping from these target-converse activations. this allows it to selectively amplify or suppress relevant activation patterns, effectively learning to ignore irrelevant signals for robust, token-specific steering signals that can be adaptively applied across multiple layers. in comparison to simple prompting-based techniques, and previous steering techniques @xcite @xcite , steervlm is capable of modeling the relationship encoded in semantically rich prompt pairs within the vlm's activation space. this leads to better control over the model's outputs. we also introduce vnia (visual narrative intent alignment), a multimodal dataset specifically designed to support the development and evaluation of steering mechanisms for vision language models. to our knowledge, vnia is the first such dataset to provide steered responses directly conditioned on images, addressing a key resource gap for vlm steering research. in summary, our contributions are as follows foot_0 : ## dataset we also contribute vnia (visual narrative intent alignment), a multimodal steering dataset designed to train and evaluate our steering module. we randomly sampled 61,391 images from the cc3m dataset @xcite and generated steered responses to prompts using qwen2.5-vl-72b ( bai prompt sampling first, we used gpt-4o (ope-nai et al., 2024) to generate pairs of mutually exclusive prompts for various topics (see appendix the goal is to create a list with contrasting pairs that span these areas, offering a rich mix of relationships between positive and negative perspectives. the pairs of target and converse prompts were manually inspected by the authors. since the dataset contains only 463 prompts, applying human-in-the-loop techniques with a set of filtering criteria was manageable. the criteria included ensuring diversity in both the topics of the prompts (e.g., cooking, running, astronomy, etc.) and their semantics (e.g., love-hate, easy-difficult, intriguing-confusing), removing duplicates, and ensuring mutually exclusive semantics. for example, consider the pair "volunteering is fulfilling" vs. "volunteering is time-consuming." while both can be true simultaneously, similar to the approach in (turner et al., 2023), we aimed to ensure a degree of mutual exclusivity between the target and converse prompts. next, the set of prompts was split into distinct training and evaluation sets. we then matched these prompts with the sampled images using clip embedding scores. image-prompt pairing a key requirement was to ensure the steering prompts were relevant to the image content but not so trivially descriptive that the steering task offered little challenge. to achieve this balance between relevance and difficulty when selecting the final image-prompt pairs, we employ an adaptive, entropy-based nucleus sampling threshold where the value of top-p is selected based on the sharpness or flatness of the probability distribution obtained from softmax-normalized clip outputs. this method ensures that the chosen image and steering prompt pair share some correlation without being obviously or directly linked. as seen in figure 5 , we conduct an ablation study for entropy thresholds τ ∈ [0.1, 0.9] (over a batch size of 1024) to quantify the trade-off between clip scores and diversity measured by the number of unique prompts. in qualitative terms, overly low thresholds select a small subset of the target versus converse prompts lowering the diversity of the prompts in the dataset whereas overly high thresholds increase sample rejection and random prompt assignments, undermining dataset consistency and size. by adopting τ = 0.6 as our default (with an optional τ = 0.7 setting for diversity-critical applications), we ensure an optimal balance between steerable outputs and steering signals. ## training the supervised fine-tuning stage is essential to stabilize the steering module due to the random initialization of model weights and the shared nature of the architecture across layers. the sft alignment stage provides quick adaptation to the steering task. given that our dataset (section 4) size is approximately 20-25% smaller than typical supervised fine-tuning datasets and considering stabilization requirements of the shared module across all layers, we trained steervlm on 8 a100 gpus for 5 epochs. additionally, we keep steering strength λ as 1, and learning rate as 3e -4 with cosine learning rate scheduler during training. we used standard cross-entropy loss to optimize log probabilities of tokens aligned towards the target behavior. ## evaluation setup we compare our approach against prior works using an evaluation set derived from the vnia benchmark. we extract five topic steering vectors from vnia and generate 150 image-prompt pairs per vector as training data for methods requiring examples to construct steering vectors. an additional set of 20 samples per vector is held out for evaluation. we extract steering vectors for methods that require it from the training set. for all experiments, steervlm is applied in a zero-shot setting, i.e. on unseen p + /p -. gpt-o4-mini (openai, 2025) is used as the judge model to evaluate the performance on this task (appendix a.2.3). we also benchmark our work on hallucination mitigation. we use 1,000 training samples from the ohd dataset @xcite , which consists of coco @xcite images paired with both faithful and hallucinated captions. for baselines requiring negative examples to compute steering vectors, we randomly select one hallucinated caption from the "adversarial" category for each training instance. evaluation is conducted using the standard ohd benchmark, which reports pope f 1 score and accuracy across the popular, adversarial, and random categories @xcite . as before, our method operates zero-shot, using handcrafted positive (target behavior) and negative (converse behavior) prompts related to hallucination (listed in appendix in contrast, our method demonstrated improved quality of steered responses. the experiment was evaluated using the qwen2.5-vl-72b model @xcite as an automated judge, based on criteria detailed in the appendix ## ablation study to justify the architectural choices of our steering module, we conducted ablation studies comparing performance against several variations and baselines. qwen2.5-vl-72b @xcite is used as the judge model with the judging criteria detailed in the appendix a.2.3. steervlm surpassed prompt engineered zeroshot and one-shot methods at steering on the vnia evaluation dataset. we compared against standard zero-shot prompting and one-shot prompting with manually engineered prompts designed to guide the model towards a steered response. the steeringgate module and the unsteered context vector u are essential to steervlm's training and inference stability, and qualitative performance. we assessed variations of our module: one lacking the steeringgate mechanism to evaluate its importance (especially in stable training), and another without the unsteered context vector to highlight its contribution to layer agnosticity. we also applied steering only to layers showing strong effects (figure 6 ), highlighting the module's role in adaptive selection and dynamic steering. steeringgate's dimension-specific steering enhances the model's qualitative performance. lastly, we compare against a variant of the steer-inggate module that applies a uniform sigmoid flow-of-control value across all dimensions to demonstrate the benefit of our dimension-specific steering approach. steervlm is robust to semantic shifts. we evaluated steervlm's robustness to semantic shifts in prompting by testing varied prompt phrasings for 3 randomly selected steering vector prompts. as detailed in appendix a.4.4, performance remained highly stable despite these semantic shifts. we noted minor performance changes only when introducing more abstract concepts (e.g., 'energizing' to 'chaotic'), confirming the model's effective generalization against common linguistic variations. steervlm steers embeddings towards target semantics and away from converse semantics. figure 7 , analyzes steering specificity by projecting embeddings onto the target-converse semantic task: describe the image concisely. ## qualitative analysis we present two qualitative analyses in however, steervlm understands the context of the target prompt and invokes the desired behavior in a naturally compelling way. ## conclusion in this paper, we introduce steervlm, a lightweight steering module that operates at inference time for fine-grained vlm control. our method utilizes target and converse prompts to isolate and amplify behaviors aligned with the target and divergent from the converse. steervlm is a lightweight module comprising only 0.14% of the main model's parameters, and is orthogonal to existing prompting techniques and fine-tuning. its additive steering capabilities and layered approach to model control, along with token-wise and dimension-wise control over latent variables within the model's activation space, allow for precise steering control. steervlm consistently outperforms existing steering methods in both quantitative benchmarks and qualitative assessments, proving its performance in eliciting steered responses.