AVG - LL a VA : An Efficient Large Multimodal Model with Adaptive Visual Granularity (2025)

Abstract: Recently, large multimodal models (LMMs) have achieved significant advancements. When dealing with high-resolution images, dominant LMMs typically divide them into multiple local images and a global image, leading to a large number of visual tokens. In this work, we introduce AVG-LLaVA, an LMM that can adaptively select the appropriate visual granularity based on the input image and instruction. Specifically, we first apply the multiple pooling layers to obtain visual tokens at different granularities. Then we propose a visual granularity router, which includes a Transformer layer, an MLP layer, and a voter layer, used to select the appropriate visual granularity based on the image and instruction. Furthermore, we put forward RGLF, a novel training paradigm that aims at aligning the granularity predicted by the router with the preferences of the LMM, without the need for additional manually annotated data. Extensive experiments and analysis show that AVG-LLaVA achieves superior performance across 11 benchmarks, as well as significantly reduces the number of visual tokens and speeds up inference (e.g., an 85.3% reduction in visual tokens and a 2.53 \times increase in inference speed on the AI2D benchmark).

## introduction recently, the field of artificial intelligence (ai) has witnessed a significant advancement in large multimodal models (lmms) @xcite @xcite @xcite , marking a further step toward artificial general intelligence (agi). most existing lmms follow the structure of llava @xcite , which includes a vision encoder to embed figure 1 : an example of vqa from mscoco @xcite . notably, responding to question 1 necessitates fine-grained visual information, whereas responding to question 2 requires only coarse-grained visual information. due to the limitations imposed by the fixed aspect ratio (e.g., 1:1) and low resolution (e.g., 336×336) used by visual encoders (e.g., clip-vit @xcite ), early lmms face challenges in processing high-resolution images with different aspect ratios. to deal with this limitation, dominant models, such as llava-next @xcite , dynamically divide each input highresolution image into multiple local images. these local images are encoded separately, and their tokens are then concatenated with the tokens of the original global image. this approach will lead to longer visual token sequences, such as 2880 visual tokens for a 672×672 image. however, in practice, such fine-grained visual information is not always necessary, and in some cases, coarse-grained visual information can even be more beneficial for model predictions. for instance, as shown in figure 1 , when the model is asked to recognize the number on the jersey, it requires relatively fine-grained visual information. in contrast, determining the color of the jersey only necessitates coarse-grained visual information. in this paper, we propose adaptive visual granularity llava (avg-llava), an lmm that can adaptively select the appropriate visual granularity based on the input image and instruction. the basic intuition behind our model is that humans only scrutinize images carefully when answering difficult questions; otherwise, a brief glance is sufficient. as displayed in figure 2 , avg-llava extends llava-next with a visual granularity scaler and a visual granularity router. the visual granularity scaler performs multiple rounds of pooling on visual tokens, each time halving the number of visual tokens, thus obtaining a series of visual features with different granularities. the visual granularity router adaptively selects the appropriate visual granularity features based on the input multi-granularity visual features and text features. by doing so, for images and instructions that do not require fine-grained details, the number of visual tokens can be reduced, which not only speeds up inference but also may improves performance. this performance enhancement likely stems from the reduction of redundant information, as selecting appropriate visual granularity makes it easier for the model to answer questions based on images effectively. besides, we observe that it is challenging to train the visual granularity router directly through visual instruction tuning @xcite . this may be because the router cannot learn the distinctions between different visual granularities from visual instruction tuning, making it difficult to learn how to select the most appropriate visual granularity based on the image and instruction. to deal with this issue, we propose a novel training paradigm called ranking granularity based on lmm feedback (rglf). this paradigm aligns router probabilities of multiple granularities with lmm preferences by a ranking loss @xcite , effectively aiding the router in distinguishing between different visual granularities and selecting the appropriate one. we further evaluate avg-llava on 11 benchmarks including tasks from various types (e.g., general vqa and text-oriented vqa, etc.). extensive experimental results show that avg-llava can effectively reduce the number of visual tokens and improve inference speed (e.g., an 85.3% reduction in visual tokens and a 2.53× increase in inference speed on the ai2d @xcite benchmark) while achieving better performance under the same base llm. to perceive images with higher resolutions, qwen-vl @xcite increases the input resolution of the visual encoder to 448×448 and introduces an additional training stage. along this line, both vary @xcite and mini-gemini @xcite include two vision encoders: one is an additional introduced high-resolution vision encoder, and the other is the original low-resolution vision encoder. unlike the methods mentioned above, sphinx @xcite and monkey @xcite enlarge the input image to a high resolution, and then divide it into a fixed number of local images, which are individually encoded using an image encoder to obtain local image tokens. subsequently, the original global image tokens are concatenated with all local image tokens to feed into the llm. furthermore, llava-next @xcite enumerates various resolutions and adaptively selects the one that most closely matches the input image resolution. although these methods can achieve better performance, they significantly increase the number of visual tokens, as the computational complexity scales quadratically with the number of input tokens, resulting in higher inference costs. vision token reduction for lmms. recently, several methods are proposed to reduce the visual tokens for lmms, including visual token compression and pruning. for example, llava-uhd @xcite adopts a qformer-like @xcite structure to compress visual tokens, while mg-llava @xcite employs a convolution layer to compress high-resolution visual features. in contrast, crossget @xcite introduces a cross-modal token for leveraging crossmodal information to make decisions on token selection and merging. llava-prumerge @xcite employs the similarity between the class token and other tokens as a key criterion for pruning and merging vision tokens. furthermore, fastv @xcite ) finds that most image tokens receive inefficient attention after the second decoder layer, and thus prunes half of the image tokens. similarly, vtw @xcite adopts a more aggressive strategy to prune all visual tokens at a certain layer. unfortunately, despite the above methods effectively reducing the number of visual tokens, they often lead to a certain degree of decline in model performance. more recently, llava-m 3 @xcite obtains multi-granularity visual features by merging visual tokens through pooling, enabling manual control of the tradeoff between inference cost and performance. significantly different from aforementioned methods, our model is a dynamic neural network @xcite that can adaptively select the appropriate visual granularity based on the input image and instruction, improving model performance while reducing the number of visual tokens. ## conclusion in this work, we propose avg-llava, an lmm that can adaptively select appropriate visual granularity based on input image and instruction. besides, we introduce rglf, which aligns routerpredicted probabilities of multiple granularities with lmm preferences by a ranking loss, effectively helping the model learn to distinguish between different granularities. experimental results show that avg-llava not only exhibits superior performance across 11 benchmarks, but also significantly reduce the number of visual tokens and speed up inference in tasks that do not require finegrained information. in future work, we aim to develop different visual granularity scaling networks to obtain richer visual granularity and integrate the two-stage training into a single stage to improve efficiency.