L ong LL a VA : Scaling Multi-modal LLM s to 1000 Images Efficiently via a Hybrid Architecture (2025)

Abstract: Expanding the long-context capabilities of Multi-modal Large Language Models (MLLMs) is critical for advancing video understanding and high-resolution image analysis. Achieving this requires systematic improvements in model architecture, data construction, and training strategies, particularly to address challenges such as performance degradation with increasing image counts and high computational costs. In this paper, we propose a hybrid architecture that integrates Mamba and Transformer blocks, introduce data construction methods that capture both temporal and spatial dependencies, and employ a progressive training strategy. Our released model, LongLLaVA (Long-Context Large Language and Vision Assistant), demonstrates an effective balance between efficiency and performance. LongLLaVA achieves competitive results across various benchmarks while maintaining high throughput and low memory consumption. Notably, it can process nearly one thousand images on a single A100 80GB GPU, underscoring its potential for a wide range of multi-modal applications.

## introduction the rapid advancement of mllms @xcite @xcite has demonstrated their remarkable capabilities across various applications @xcite @xcite . however, multi-image scenario remain an important yet to-be-explored aspect. in particular, expanding the context of mllms to understand longer videos @xcite , higher-resolution images @xcite , and make decisions based on more historical messages @xcite is crucial for enhancing user experience @xcite and further broadening mllms' application scope @xcite . however, extending the context length of mllms to improve their usability poses challenges related to degraded performance and high computational costs when processing more images. to maintain the performance in longer context, some studies @xcite have concentrated on curating long-context training data involving multiple images to enhance performance. additionally, other research efforts have explored new training strategies @xcite @xcite to mitigate performance declines. regarding the issue of high computational costs, @xcite have made strides in improving multi-node efficiency by reducing communication costs. however, a significant gap persists in accelerating core on-node computation for long visual contexts without sacrificing performance. an integrated architectural solution addressing both performance and efficiency is thus needed. to tackle these challenges, we propose longllava, featuring a hybrid architecture for efficient acceleration. our solution focuses on three aspects: multi-modal architecture, data construction, and training strategy. ## longllava to address the aforementioned challenges and enhance the model's adaptability to long-context, multi-image scenarios, we introduce improvements from three perspectives: multi-modal model architecture (sec. 3.1), data processing protocol (sec. 3.2), and training strategy (sec. 3.3). ## conclusion in this study, we introduce longllava, an innovative hybrid architecture model that excels in longcontext multi-modal understanding. the model integrates mamba and transformer blocks, leveraging temporal and spatial dependencies between multiple images to construct data, and employs a progressive training strategy. longllava demonstrates competitive performance across various benchmarks while ensuring efficiency, setting a new standard for long-context mllms.