Advancing General Multimodal Capability of Vision-language Models with Pyramid-descent Visual Position Encoding (2025)

Abstract: Vision-language Models (VLMs) have shown remarkable capabilities in advancing general artificial intelligence, yet the irrational encoding of visual positions persists in inhibiting the models’ comprehensive perception performance across different levels of granularity. In this work, we propose Pyramid-descent Visual Position Encoding (PyPE), a novel approach designed to enhance the perception of visual tokens within VLMs. By assigning visual position indexes from the periphery to the center and expanding the central receptive field incrementally, PyPE addresses the limitations of traditional raster-scan methods and mitigates the long-term decay effects induced by Rotary Position Embedding (RoPE). Our method reduces the relative distance between interrelated visual elements and instruction tokens, promoting a more rational allocation of attention weights and allowing for a multi-granularity perception of visual elements and countering the over-reliance on anchor tokens. Extensive experimental evaluations demonstrate that PyPE consistently improves the general capabilities of VLMs across various sizes. Code is available at https://anonymous.4open.science/r/PyPE-34EE.

## introduction large language models (llms) @xcite demonstrate significant universal capabilities that contribute to the pursuit of general artificial intelligence. however, language constitutes only one aspect of communication. visual information plays a crucial role in augmenting and enhancing our understanding of the world. consequently, there is a growing interest in the development of vision-language models (vlms) @xcite @xcite that can process and integrate visual modality. to effectively leverage the powerful contextual understanding capabilities of llms, vlms project visual information to the same dimensionality as textual embeddings through specific projection layers @xcite @xcite , which are then directly embedded into the text sequence to form the input for the foundation llms, enabling cross-modal alignment and instruction-following learning using next-token prediction. despite their commendable progress, the typical processing of visual information does not align with the distribution patterns of visual elements. since visual information is composed of fixedsized patches obtained through raster scanning, patches located closer to the bottom right corner of the image are positioned nearer to the instruction tokens within the sequence. due to the long-term decay from rotary position embedding (rope) @xcite , visual tokens closer to the instruction tokens will be more likely to receive higher attention weights, and vice versa. this is counterintuitive, as the importance of visual information is not defined by the order of raster-scanning. @xcite observe a similar phenomenon by visualizing the attention information flow from instruction tokens to visual tokens in the first layer of the decoder. consequently, they propose concentric causal attention (cca), which starts assigning the position indexes of images from the peripheral and ends in the center, to alleviate the long-term decay in rope and improve causal attention following 2d spatial locality of images. although cca is both intuitive and effective, its applicability is constrained by the assumption that all significant elements related to the instructions are situated at the center of the image. this assumption inherently results in a loss of detail, limiting its effectiveness in capturing comprehensive information. to further investigate the impact of raster-scan and concentric pe on the fine-grained modeling of visual information, we extend the visualization to all layers of the decoder. as illustrated in figure 1 , cca demonstrates exceptional performance in the first layer, alleviating the long-term decay caused by rope in the raster-scan approach, thereby directing the model's attention to more significant areas. however, in the subsequent layers, both methods largely maintain the same attention patterns as observed in their respective third layers, with changes only occurring in the final layer. a similar phenomenon, namely "aggregation pattern", is observed in opera @xcite , where both llms and vlms tend to generate new tokens by concentrating on a limited number of summary tokens (also referred to as anchor tokens @xcite ) rather than considering all preceding tokens. this tendency towards partial overtrust leads to the neglect of fine-grained image tokens, resulting in the generation that may be hallucinatory and do not accurately reflect the image content. moreover, it has been demonstrated in opera that more hallucinations are generated when more summary tokens appear in the context. to this end, we present pyramid-descent visual position encoding (pype), a novel position assignment approach for visual tokens, to alleviate the long-term decay induced by rope, avoid the "aggregation pattern" in the llm, and ensure a comprehensive understanding of visual contents. pype reorganizes the flattened visual tokens into the 2d shape and assigns visual position indexes from the periphery to the center. this reduces the relative distance between interrelated visual elements, as well as the distance between significant visual elements and instruction tokens, thereby ensuring a more rational allocation of attention weights. furthermore, to mitigate the impact of anchor tokens on the model's fine-grained percep-tion of visual elements, we draw inspiration from pyramid vision transformer (pvt) @xcite : consistently combining global and local receptive fields. pype gradually expands the central receptive field, i.e., the central region of the position index matrix, at predetermined intervals of layers. specifically, we expand the central region of the position index matrix by a circle every certain number of layers. such expansion weakens the anchor tokens and enhances the model's ability to perceive visual elements at varying levels of granularity (more cases can be found in section 5.4). with extensive experiments on visual question answering and general multimodal benchmarks, pype consistently improves general perception capabilities across vlms of different sizes. in a nutshell, the main contributions of this work are as follows: (i) we make an in-depth analysis of how position encoding affects visual perception in vlms. (ii) our proposed pype effectively mitigates longterm decay and the "aggregation pattern", which helps better perceive visual elements at different granularities. (iii) extensive evaluations demonstrate the superior performance of pype, a simple yet effective method that applies to any vlms. ## empirical results and analysis we evaluate the visual capabilities of the models trained with the pype through various visual question answering and general multimodal benchmarks. this novel position encoding demonstrates highly competitive performance at different scales. our proposed method consistently delivers top-tier performance across most evaluation metrics, frequently surpassing other baselines. ## conclusion in this work, we conduct an in-depth analysis of how visual position encoding affects visual perception in vlms (vision-lnaguage models), particularly from the aspect of long-term decay and the "aggregation pattern" (also known as "anchor tokens").