Masked Diffusion Captioning for Visual Feature Learning (2025)

Abstract: We learn visual features by captioning images with an image-conditioned masked diffusion language model, a formulation we call masked diffusion captioning (MDC). During training, text tokens in each image–caption pair are masked at a randomly chosen ratio, and a decoder conditioned on visual features is trained to reconstruct the original text. After training, the learned visual features can be applied to downstream vision tasks. Unlike autoregressive captioning, the strength of the visual learning signal in MDC does not depend on each token’s position in the sequence, reducing the need for auxiliary objectives. Linear probing experiments across a variety of academic-scale models and datasets show that the learned visual features are competitive with those produced by autoregressive and contrastive approaches.

## introduction multimodal models that learn the cross-modal associations between images and language have driven many recent advances in visual representation learning @xcite @xcite ). an intuitively appealing approach is to pose this problem as visual captioning: first, train an image-conditioned language model to generate text captions from images, and then use its learned visual features for downstream tasks. however, the popular formulation of captioning as autoregressive language modeling often yields visual features that perform worse than those from alternative vision-language learning approaches. one major reason for this is the asymmetry in the learning signal @xcite : later text tokens can be predicted so well from the earlier ones that the image becomes decreasingly important as the sequence progresses from left to right. a variety of approaches have addressed this issue by augmenting the objective with right-to-left generation @xcite , multimodal decoder an emerging line of work in the natural language processing community has applied masked diffusion language models (mdlms) to text generation @xcite @xcite . instead of producing text in a fixed order, these methods randomly mask tokens at each iteration and train a model to reconstruct the original text. during training, the fraction of masked tokens is chosen randomly, enabling the model to reconstruct text given arbitrary numbers of masked tokens. previous work has shown that such models can generate high-quality text via ancestral sampling, optimize variational bounds, and learn language features that transfer well to downstream tasks @xcite . in this paper, we learn visual features through masked diffusion captioning (mdc): using an image-conditioned masked diffusion language model to generate text captions from images (fig. 1 ). unlike autoregressive models, the amount of text conditioning each token receives is not determined by its position in the sequence; instead, each token provides a position-independent amount of visual supervision. since we primarily use captioning as a means of learning features rather than as an end in itself, our approach is closely related to methods that learn visual features with imageconditioned bert @xcite . however, instead of using a fixed masking ratio, we sample ratios randomly during training and weight the loss as a function of the ratio. we evaluate our approach on academic-scale models and datasets, establishing an effective training recipe for masked diffusion captioning. our experiments suggest that the resulting model learns useful visual features across multiple datasets and encoder architectures (e.g., cc12m @xcite with @xcite ). these features achieve performance that is competitive with autoregressive and contrastive methods on a variety of linear probing experiments for visual recognition tasks. we also find that the model's ability to approximately estimate the likelihood of a given caption can be used to match images to their captions successfully, resulting in competitive performance on compositionality benchmarks @xcite . additionally, we find that imageconditioned bert, a special case of our model, can achieve features competitive with those of other learning approaches when properly tuned, typically by choosing a large masking ratio that requires the model to rely heavily on the visual signal. ## related work image captioning for visual representation learning. contrastive vision-language pretraining @xcite @xcite @xcite learns strong visual features through the discriminative task of contrastive learning. there is a line of work that seeks to obtain good visual representations by captioning, where the model is supervised at the token level. this paradigm of feature learning through generative pretraining can produce both visual features and captioning models capable of generating text for specific tasks. virtex @xcite utilizes forward (left-to-right) and backward (right-to-left) captioning to learn visual features. simvlm @xcite treats visual patches as the prefix and employs a single prefix language modeling objective for supervision. blip @xcite uses synthetic captions to improve the quality of image-text pairs. similarly, coca @xcite leverages both contrastive learning and image captioning objectives. recently, cappa @xcite has shown that captioning can produce strong visual encoders as competitive as those from contrastive learning on large datasets. it augments the autoregressive captioning objective with parallel decoding (i.e., where all tokens are masked, and the model must reconstruct the text). following this direction, locca @xcite and siglip 2 @xcite employ captioning as a pretraining task. additionally, there is prior work @xcite @xcite @xcite ) that aims to improve text quality for image-text pairs. like these approaches, we learn visual features via image captioning, but we do so using a single masked diffusion language modeling objective, instead of an autoregressive or hybrid approach. vision language models. contrastive learning methods, such as clip @xcite , have provided scalable and effective approaches for image-language learning. largescale datasets @xcite @xcite @xcite have contributed significantly to this success. these models @xcite @xcite @xcite can perform visual recognition @xcite @xcite in a zero-shot manner by computing similarities between image and text embeddings. recently, with the advancement of large language models (llms) @xcite @xcite @xcite , multimodal models @xcite @xcite @xcite @xcite @xcite have been developed that perform vision tasks through language, given visual input processed by vision encoders @xcite @xcite . these models demonstrate a variety of multimodal capabilities @xcite @xcite @xcite @xcite . despite their success, they often fail to capture complex relationships between images and language, such as compositionality @xcite . autoregressive language models. autoregressive language models factorize the joint probability of a sequence into a product of conditional nexttoken probabilities and are trained with maximumlikelihood estimation (teacher forcing) to predict each token given its left context. the paradigms of next-token prediction and gpt-style models @xcite @xcite ) laid the foundation for the success of large language models @xcite @xcite @xcite @xcite @xcite . using autoregressive models for image captioning is also common practice @xcite . diffusion language models. diffusion models were first proposed by @xcite and later popularized for continuous data by ddpm @xcite and score matching @xcite . more recently, diffusion-based language models have gained significant attention. these methods can be broadly divided into two categories: (1) embedding-space diffusion @xcite and (2) discrete-state diffusion @xcite @xcite @xcite @xcite @xcite . @xcite first introduced diffusion models with discrete state spaces over binary random variables, which were extended by @xcite to categorical data using uniform categorical noise. d3pm @xcite introduced various transition matrices (uniform, absorbing, discretized gaussian, and token embedding distance) for discrete-time markov chains, while @xcite extended this to continuous-time markov chains (ctmc). concrete score matching @xcite generalized score matching @xcite to discrete domains, and sedd @xcite further proposed score entropy for optimization. both mdlm @xcite and md4 @xcite derived simplified expressions of the elbo for masked diffusion language models. other work @xcite has suggested that input time embeddings are unnecessary for discrete diffusion language models. more recently, smdm @xcite demonstrated the scalability of masked diffusion language models, and llada @xcite scaled them to rel-atively large sizes. building on these advances, our paper focuses on applying masked diffusion language models to visual representation learning through image captioning. vision-language masked modeling. a variety of recent methods have learned visual feature learning used masked language modeling @xcite @xcite @xcite @xcite @xcite . however, these methods have typically focused on learning joint visual-linguistic representations through early fusion. sariyildiz et al. (2020) first identifies candidate tokens in the caption that correspond to visual concepts, typically nouns, adjectives, or verbs, then randomly masks one of them and trains the model to predict it using both the image and the surrounding text. similarly, @xcite and @xcite extend masked modeling to both vision and language. in contrast, our work mainly focuses on learning visual representations from the captioning objective only by using masked diffusion language modeling. it avoids the need to choose a single (possibly dataset-dependent) masking ratio, and can directly generate text. ## method we propose to learn visual features by generating text captions from images using an imageconditioned masked diffusion language model, an approach we call masked diffusion captioning (mdc). ## experiments we pretrain our models and benchmark them against other approaches. ## conclusion in this work, we introduce masked diffusion captioning (mdc), an image-conditioned masked diffusion language model designed to learn visual representations. our results demonstrate that masked diffusion captioning effectively learns visual features, outperforming previous masked language modeling variants by using a unified noise schedule. in addition, masked diffusion captioning can generate reasonable captions and exhibits strong vision-language compositionality. we conduct evaluations to establish an effective training recipe for masked diffusion captioning. overall, our study suggests that masked diffusion language models offer a compelling alternative to autoregressive approaches for learning visual representations from image caption pairs.