DAPE - BR : Distance-Aware Positional Encoding for Mitigating Object Hallucination in LVLM s (2025)

Abstract: Large Vision–Language Models (LVLMs) have garnered substantial interest owing to their impressive ability to interpret visual inputs and converse with users.Nevertheless, LVLMs still suffer from object hallucination – generating descriptions for objects that are absent from the image, which undermines reliability and hinders real-world deployment. We propose DAPE-BR, a positional-alignment scheme that (i) preserves the pretrained weight order while globally—- visual–text distances, (ii) embeds an isotropic fused patch-distance metric, and (iii) applies a patch-distance causal mask to enforce spatial causality. Extensive experiments on POPE, MMStar and SQA show that DAPE-BR consistently reduces hallucinations and boosts.

## introduction lvlms @xcite @xcite @xcite excel at parsing images and conducting natural-language conversations, but they still hallucinate-describing objects that are missing or mis-characterised in the picture @xcite @xcite . these factual slips erode user trust and impede reliable, real-world deployment, so reducing object hallucination has become a central research priority. to tackle this challenge,various strategies have been developed to mitigate object hallucination in lvlms. a widely adopted approach involves employing post-hoc correction through revisor models @xcite @xcite , which refine generated outputs to suppress hallucinated descriptions. another promising avenue enhances supervised fine-tuning by enriching the diversity of instruction-tuning datasets(liu et @xcite . although these methods effectively reduce object hallucination, they rely on high-quality annotations, which entail substantial manual labor and make them costly to deploy in practice. recently, several studies have explored training-free techniques that mitigate object hallucination by directly correcting inaccuracies during the autoregressive decoding phase of lvlms @xcite @xcite . while these training-free heuristics operate at the output layer, another line of work revisits the input sidenamely, the positional encoding itself. a recent study @xcite traces object hallucination in lvlms to the problem of "long-term decay" in rotary position embeddings (rope) @xcite : as token indices increase, rope's sinusoidal phases shrink exponentially and thus cause tokens at higher positions to overshadow earlier visual context. the authors counter this by adding a lightweight positional-alignment module (figure 1a ) that rescales and recenters the phases before decoding and then fine-tunes the model. this modification reduces hallucinations but still suffers from three issues. first, its concentricring encoding merges all patches on the same ring into a single index, slashing the positional vocabulary from h × w to o(min{h, w }) and causing under-utilization and aliasing (border patches dominate, and objects that cross rings split encodings). second, because the algorithm scans patches from the periphery inward, the last visual token fed to the language model is an arbitrary border patch rather than the bottom-right patch used during pretraining. as a result, the learned visualtext offset is disrupted, and additional fine-tuning becomes necessaryhence the method is not truly trainingfree. third, since the indices depend only on the radius, angular information is lost: patches that are opposite each other on a ring (approximately 2r apart) appear co-located, whereas adjacent patches in neighboring rings seem distant. this angular insensitivity introduces anisotropic distortions that hinder consistent 2-d spatial reasoning. building on this analysis, we propose dape-br (see figure 1b ), a two-stage positional-alignment module that first scans visual tokens from the bottom-right corner in a special raster order and rectifies the causal mask, then encodes each token with the mean of its euclidean, manhattan, and chebyshev distances, thereby preserving twodimensional continuity and markedly reducing hallucinations during lvlms training; this scheme shortens global tokeninstruction paths without disturbing pretrained attention, introduces an isotropic fused-distance metric that removes directional bias, and applies a three-step pipelinedistance fusion, shell quantization, and reverse causal maskingto confine attention to truly causal regions. our key contributions include: • refines dape-br positional encoding, enriching spatial indices and strengthening visual grounding. position encoding in transformers. because self-attention is order-agnostic @xcite , researchers have proposed several positional-encoding schemes, including sinusoidal @xcite , learnable (dosovitskiy et al., 2020), and relative approaches @xcite @xcite , the last of which excels on variablelength inputs @xcite . rotary position encoding (rope) encodes positions by rotating embedding pairs @xcite ; this design boosts linear attention and enables large-scale pre-training in llama @xcite , and it is now being explored for vision tasks @xcite . in this work, we examine whether ropes long-term decay contributes to object hallucination and how to curb it without costly retraining. ## dape-br method core contributions of the proposed dape-br (figure 2 ). our approach can be summarized in three interconnected aspects: order-consistent re-indexing. without any additional supervised fine-tuning, dape-br globally shrinks the relative distances from most visual tokens to the instruction token, effectively counteracting the hallucination-prone long-range decay inherent to rope. the first patch-distance metric and anisotropy removal. we are the first to explicitly measure pairwise distances between image patches and to encode them in the positional indices. this fused metric removes the directional anisotropy inherent in prior ring-based or raster layouts, enabling the model to perceive patch separations isotropically across the grid. three-stage pipeline. the method proceeds through (i) fused distance computation, (ii) shell quantization that converts the distance into discrete indices, and (iii) a causal mask that allows a query to attend only to keys in the same or inner shells. in the following subsections,we elaborate on each stage in the following subsections, starting with the definition of the fused distance. ## experiments all experiments were implemented in pytorch and run on nvidia a100 80gb gpu hardware. we evaluate dape-br on the unified llava-v1.5-7b @xcite foundation model across three complementary benchmarks, each chosen to stress a different aspect of multimodal grounding. specifically, we use the pope @xcite dataset to test object hallucination suppression, mmstar @xcite dataset to assess image-grounded reasoning accuracy, and sqa @xcite dataset to evaluate multi-turn dialog consistency. this evaluation design ensures that dape-br is validated across these critical angles, demonstrating its ability to improve visual grounding and reduce spurious object generation. ## conclusion we propose dape-br, a positional-alignment scheme that mitigates object hallucination in lvlms. it adds three light componentsorderconsistent re-indexing, fused distance-aware patch encoding, and reverse causal maskingwithout changing pretrained weights. across pope, mm-star, and sqa it surpasses prior methods; each module contributes, and their combination yields the largest gains. thus, precise positional realign-ment,rather than extra data or fine-tuning, markedly improves grounding and is expected to extend to deeper layers, longer sequences, and other multimodal models.