Visual Self-Refinement for Autoregressive Models (2025)

Abstract: Autoregressive models excel in sequential modeling and have proven to be effective for vision-language data. However, the spatial nature of visual signals conflicts with the sequential dependencies of next-token prediction, leading to suboptimal results. This work proposes a plug-and-play refinement module to enhance the complex spatial correspondence modeling within the generated visual sequence. This module operates as a post-pretraining step tojointly refine all generated tokens of autoregressive model, enhancing vision-language modeling under a shared sequential prediction framework. By leveraging global context and relationship across the tokens, our method mitigates the error accumulation issue within the sequential generation. Experiments demonstrate that the proposed method improves the generation quality, enhancing the model’s ability to produce semantically consistent results.

## introduction autoregressive (ar) models have achieved remarkable success in recent years across multimodal tasks @xcite @xcite @xcite @xcite @xcite . large language models (llms) based on autoregressive modeling @xcite @xcite ) encode text as sequences of tokens and predict each token sequentially based on the preceding tokens. this next-token prediction paradigm effectively captures sequential dependencies and complex semantic relationships in the text, and shown to excel in tasks like question answering @xcite @xcite ) and text generation @xcite . building on this success, recent works have extended autoregressive modeling to visual data @xcite @xcite @xcite . among them, large vision model (lvm) @xcite stands out for its focus on in-context tasks. lvm encodes images (or video frames) as a sequence of tokens and uses next-token prediction to solve various image translation and generation tasks. the focus on in-context learning makes lvm well-suited to handle vision tasks that require contextual information for generating coherent and consistent outputs. unlike text, visual tokens exhibit strong spatial correlations rather than purely sequential dependencies. this spatial structure necessitates attention to both prefix and future tokens for coherent generation. consequently, generating tokens solely based on preceding tokens may compromise global contextual awareness, and the sequential nature of generation further exacerbates this by allowing early errors to accumulate, degrading both visual fidelity and semantic consistency. to address these challenges, we propose a plugand-play self-refinement module that operates on the generated tokens while keeping the pretrained backbone frozen for the post-pretraining finetuning (see figure 1 ). specifically, the proposed module jointly optimizes the generated tokens under the supervision of the target image. the discrete tokens are mapped to the embedding space to facilitate the learning. to this end, the broader view enables consistent corrections across the generated token sequence, mitigating errors stemming from the prefix context in the autoregressive generation. we summarize the contributions as follows: 1) we introduce a post-pretraining finetuning operation to improve the next-token prediction for the visual data, without harming the generalizeability of the pretrained ar model. 2) we propose to learn a plug-and-play lightweight module to facilitate the refinement, with small-scale data and negligi- ## related work autoregressive modeling serves as main streamline for image generation. llamagen @xcite ) develops text-conditioned visual generation, aim @xcite studies scalability in model capacity, data quantity and objective function, rar @xcite rearranges token orders for bidirectional correspondence modeling, elm @xcite studies the design space of llms for vision tasks, and var @xcite incorporates visual inductive bias via a next-scale prediction mechanism. while these methods primarily focus on unconditional, and text/class-conditional generation, they fail to utilize images as context. lvm @xcite address this gap by leveraging context image pairs for inferring tasks and generation, allowing a series of perception tasks @xcite or reconstruction tasks @xcite @xcite . however, its reliance on next-token prediction leads to error accumulation and struggles to model spatial correspondences @xcite .