L a D i C : Are Diffusion Models Really Inferior to Autoregressive Counterparts for Image-to-Text Generation? (2024)

Abstract: Diffusion models have exhibited remarkable capabilities in text-to-image generation. However, their performance in image-to-text generation, specifically image captioning, has lagged behind Auto-Regressive (AR) models, casting doubt on their applicability for such tasks. In this work, we revisit diffusion models, highlighting their capacity for holistic context modeling and parallel decoding. With these benefits, diffusion models can alleviate the inherent limitations of AR methods, including their slow inference speed, error propagation, and unidirectional constraints. Furthermore, we identify the prior underperformance of diffusion models stemming from the absence of an effective latent space for image-text alignment, and the discrepancy between continuous diffusion processes and discrete textual data. In response, we introduce a novel architecture, LaDiC, which utilizes a split BERT to create a dedicated latent space for captions and integrates a regularization module to manage varying text lengths. Our framework also includes a diffuser for semantic image-to-text conversion and a Back&Refine technique to enhance token interactivity during inference. LaDiC achieves state-of-the-art performance for diffusion-based methods on the MS COCO dataset with 38.2 BLEU@4 and 126.2 CIDEr, demonstrating exceptional performance without pre-training or ancillary modules. This indicates strong competitiveness with AR models, revealing the previously untapped potential of diffusion models in image-to-text generation.

## introduction in recent years, there has been a surge of impressive applications of diffusion models in text-to-image generation tasks @xcite ; @xcite . however, the inverse process of image-to-text generation remains less explored. some pioneering efforts @xcite have attempted to integrate diffusion models into text generation tasks. they mainly followed the traditional encoder-decoder framework, utilizing the diffusion model as a text decoder. subsequent research @xcite introduces visual capability into this paradigm by treating visual inputs as special tokens or encoded hidden states, thereby extending the research scope to the realm of multi-modal tasks, such as imageto-text generation. however, their performance has consistently lagged behind that of auto-regressive (ar) models @xcite @xcite . only through intricate architecture @xcite or external data @xcite can they barely achieve comparable results, raising doubts about whether diffusion models have inherent limitations, potentially making them less suitable for the image-to-text task. in this study, we aim to dispel this doubt by deeply reexamining the diffusion-based image-totext paradigm and unveiling its distinct benefits. unlike conventional ar approaches that sequentially generate captions token by token (fig. 1a ), diffusion-based models take gaussian noise as input and iteratively denoise it under image guidance to simultaneously produce the entire caption (fig. 1b ). this non-autoregressive (nar) diffusion-based model exhibits three key advantages: (1) parallel decoding: diffusion-based models emit all tokens in parallel, significantly reducing inference time for lengthy target captions. as illustrated in fig. 2a , the inference time of ar models like blip @xcite proliferates as text length increases, while our model can emit all tokens concurrently, ensuring stable inference time regardless of the length increase. for instance, when the caption length reaches 16, our model is approximately 3× faster than blip. despite the above benefits, the underperfor-mance of diffusion models on image-to-text tasks hinders their popularity. upon examining prior diffusion-based models, we deduce that their unsatisfactory performance primarily stems from two factors: (i) two significant gaps in translating between images and text, i.e., 1) the gap between visual information and textual representation, and 2) the gap between high-level text semantics and specific words. simultaneously addressing both gaps within the previous paradigm as shown in fig. 3 , proves to be a challenging task for diffusion models. nar) text decoder retains its ability to generate discrete tokens from latent space. during training, a text encoder is employed to generate ground-truth text latent codes, and during inference, it can be discarded. technically, we leverage bert (devlin et al., image space discrete text space image latent space text latent space sample ours data flow previous works' data flow image encoder text encoder text decoder diffuser diffuser (decoder) figure 3 : comparison of the pipeline between our ladic and that of previous diffusion-based models. we introduce text latent space to alleviate the burden on the diffuser. 2019) to generate the text latent space and propose a post-processing submodule including normalization and reassignment procedures for addressing problems like variable length of text. furthermore, the diffuser serves as a bridge between image and text, aiming to fit the distribution of the text latent space conditioned on the image, wherein we utilize a cross-attention mechanism for better modality fusion. lastly, during inference, we propose the back&refine technique to provide more interaction between tokens. at a high level, we intuitively summarize our model in fig. 1(b ). the generation process commences with gaussian noise, and at each step, we subtract a certain amount of noise conditioned on the image. as the number of steps increases, the denoised text latent, or sentence feature of bert in our model, converges toward the ground-truth sentence feature. ultimately, the welldenoised sentence feature is passed through the nar frozen text decoder to concurrently generate discrete words. we conducted experiments mainly on the coco dataset @xcite to validate our model's capabilities. remarkably, without pretraining or external modules, our model achieves 38.2 bleu@4 and 126.2 cider, significantly surpassing both diffusion-based methods and traditional nar models. in addition to the unique advantages discussed earlier, our model also matches the performance of well-established pretrained ar models and outperforms blip in image paragraph captioning. these results underscore the potent generative ability and immense potential of diffusion models in image-totext generation. we aspire that our work offers a fresh perspective, fostering future research on diffusion models for image-to-text generation or even other text-centered multimodal generation tasks. ## methodology in this section, we introduce our diffusion-based image captioning model, ladic. in § 3.1, we present the overall architecture of ladic, including its training and inference pipeline. subsequently, from § 3.2 to § 3.4, we offer a detailed illustration. ## conclusion in this paper, we reexamine the diffusion-based image-to-text paradigm and introduce a novel architecture, denoted as ladic. our model attains state-of-the-art performance among diffusionbased methods and demonstrates comparable capabilities with some pre-trained ar models. moreover, our extensive experiments reveal the exciting advantages of diffusion models over ar models in considering more holistic contexts and emitting all tokens in parallel. consequently, we posit that diffusion models hold substantial potential for imageto-text generation and we anticipate that our work will open new possibilities in this field.