B c QLM : Efficient Vision-Language Understanding with Distilled Q -Gated Cross-Modal Fusion (2025)

Abstract: As multimodal large language models (MLLMs) advance, their large-scale architectures pose challenges for deployment in resource-constrained environments. In the age of large models, where energy efficiency, computational scalability and environmental sustainability are paramount, the development of lightweight and high-performance models is critical for real-world applications. As such, we propose a lightweight MLLM framework for end-to-end visual question answering. Our proposed approach centres on BreezeCLIP, a compact yet powerful vision-language encoder optimised for efficient multimodal understanding. With only 1.2 billion parameters overall, our model significantly reduces computational cost while achieving performance comparable to standard-size MLLMs. Experiments conducted on multiple datasets further validate its effectiveness in balancing accuracy and efficiency. The modular and extensible design enables generalisation to broader multimodal tasks. The proposed lightweight vision-language framework is denoted as BcQLM (BreezeCLIP-enhanced Q-Gated Multimodal Language Model). It offers a promising path toward deployable MLLMs under practical hardware constraints. The source code is available at https://github.com/thico0224/BcQLM .

## introduction multimodal learning integrates visual and linguistic modalities, which has become a key direction in building more general and human-like ai systems. by combining information from different modalities, such systems can better understand complex real-world scenarios and perform more advanced reasoning and decision-making tasks. among various multimodal tasks, visual question answering (vqa) has emerged as a widely * corresponding author. to effectively address the inherent challenges of vqa and other complex multimodal tasks, mainstream approaches have predominantly adopted a two-stage architecture. pretrained encoders such as clip @xcite , vit @xcite , and bert @xcite are first used to extract semantic features from images and texts, providing strong unimodal representations and initial cross-modal alignment through large-scale pretraining. these features are then passed to large language models (llm), which serve as decoders to perform reasoning and generate answers. this paradigm has been adopted by several representative systems, including blip-2, qwen and llava @xcite @xcite . these models integrate visual understanding with language modelling, which enables them to generate contextually relevant and accurate answers across a wide range of question types. however, existing multimodal large models often suffer from large parameter sizes and high computational costs, limiting their deployment on edge devices. this hinders practical use in scenarios requiring efficient on-device inference, such as healthcare @xcite , remote education @xcite , disaster response (kumbam and vejre, 2024) and assistive technologies @xcite . in these settings, cloud-based solutions may be infeasible due to connectivity or latency issues @xcite , underscoring the need for lightweight yet capable multimodal systems. to address this issue, we propose bcqlm (breezeclip-enhanced q-gated multimodal language model). breezeclip is a lightweight vision-language encoder designed to significantly reduce computational costs while enabling efficient multimodal representation. it replaces the original bert and vit backbones in clip with compact transformer modules inspired by the inverted bottleneck design @xcite . to train breeze-clip, we construct image-text pairs from the gqa dataset @xcite , leveraging its rich scene graph annotations that provide detailed object relationships and spatial context. we adopt a dual training strategy: contrastive learning ensures visual and textual features are well aligned in a shared embedding space, while knowledge distillation from a clip teacher @xcite transfers high-level semantic alignment to breeze-clip. this enables breezeclip to achieve strong performance despite its lightweight architecture. a lightweight decoder based on llama-3.2-1b @xcite is first employed to generate answers conditioned on fused multimodal features, achieving strong performance with significantly reduced computational overhead. building on this, we introduce a q-gated cross-modal fusion module that dynamically adjusts the contribution of visual and textual features according to the input question, which enables fine-grained, question-aware interaction between modalities. these components together constitute an efficient end-to-end multimodal question answering system designed for real-world, resourceconstrained deployment scenarios. our primary contributions are thus as follows: ## related work we consider prior work leading to recent advances in mllms, with connections to multimodal fusion techniques and vision-language tasks, such as vqa, highly relevant to our work. ## method the proposed bcqlm (breezeclip-enhanced q-gated multimodal language model) is a lightweight multimodal large language model designed for visual question answering under resource constraints, to support effective multimodal reasoning in resource-limited environments. the architecture comprises three main components. first, a compact vision-language encoder, breeze-clip, is proposed to extract semantic features from both images and text. breezeclip is trained using contrastive learning and knowledge distillation from a clip teacher to ensure efficient representation learning. next, we propose a question-guided cross-modal attention module (q-gcam) to dynamically fuse visual and textual features based on the semantics of the input question. finally, a lightweight language decoder based on llama-3.2-1b generates the final answer. an overview of the framework is shown in figure 1 . it follows a two-stage design: (1) breezeclip is first pretrained via distillation and contrastive learning; and (2) the pretrained breezeclip is then used as a frozen feature extractor, working with q-gcam and an llama-3.2-1b decoder to perform visionlanguage generation. this balances performance and efficiency, and enables bcqlm to support effective multimodal reasoning in resource-limited environments.