Denoising Large-Scale Image Captioning from Alt-text Data Using Content Selection Models (2022)

Abstract: Training large-scale image captioning (IC) models demands access to a rich and diverse set of training examples that are expensive to curate both in terms of time and man-power. Instead, alt-text based captions gathered from the web is a far cheaper alternative to scale with the downside of being noisy. Recent modeling approaches to IC often fall short in terms of performance in leveraging these noisy datasets in favor of clean annotations. We address this problem with a simple yet effective technique of breaking down the task into two smaller, more controllable tasks – skeleton prediction and skeleton-based caption generation. Specifically, we show that sub-selecting content words as skeletons helps in generating improved and denoised captions when leveraging rich yet noisy alt-text–based uncurated datasets. We also show that the predicted English skeletons can further cross-lingually be leveraged to generate non-English captions, and present experimental results covering caption generation in French, Italian, German, Spanish and Hindi. We also show that skeleton-based prediction allows for better control of certain caption properties, such as length, content, and gender expression, providing a handle to perform human-in-the-loop interpretable semi-automatic corrections.

## introduction in the last demi-decade, nlp fields have ventured into reaping the benefits of utilizing large scale raw (uncurated) data from web-crawls. this trend aligned with new uncurated image-captioning datasets like conceptual captions @xcite . while these uncurated datasets are superior in terms of size and diversity, they are inferior to well curated datasets @xcite in terms of noise in the captions. the content in the alt-text for the image is often distorted by the intent or context in which the image is presented. for example, the ground truth alt-text caption for a house is 'house for sale' instead of 'front view of a house'. this hinders the use of these large noisy datasets to the fullest extent. we present a simple two-staged approach by separating the content selection from caption generation as illustrated in figure 1 . in contrast to most ic approaches @xcite , which hallucinate incorrect content from noisy training data (i.e 'custom posters' and 'wedding'), our approach first focuses on denoising the content words (i.e 'collection' and 'book') that are further used to generate a relevant caption. we refer to this sequence of concept words that are key pieces of information consistent with the image as a skeleton. sub-selecting skeleton words that curb noisiness are automatically extracted from the alttext captions. we focus on language-based skeletons that are derived from captions @xcite @xcite , rather than expensive visual-based skeletons derived from image, e.g., scene graphs, @xcite , which are hard to scale. more concretely, we introduce an intermediate task of distantly supervised skeleton prediction in the end to end ic pipeline: the end-to-end task of ic (f θ : @xmath0: skeleton prediction (f θ : @xmath1: i, @xmath2, where i is the image, s is the skeleton, and c is the caption @xcite @xcite . we present a comparison between encoding, decoding and autoencoding these skeletons. as such, our skeleton prediction solution addresses the semantic gap problem @xcite . we illustrate the effectiveness of this approach on uncurated noisy datasets in the following ways. ## related work content selection from vision: there is a rich body of work in improving content selection for ic @xcite , mainly focused on scene graph based skeletons @xcite @xcite . however, these annotations with objects and relations are expensive, thereby constraining the scaling up to multiple languages and diverse concepts. our work delegates this responsibility of identifying content to the language modality by using inexpensive off the shelf tools for weak supervision. content selection from language: an orthogonal body of work relies on skeletons derived from language using hierarchical phrase modeling @xcite , semantic attention @xcite , attribute lstm @xcite , skeleton based attribute filling @xcite , adaptively merging topic and visual information @xcite , multimodal flow @xcite and concept guided attention @xcite . note that all these prior works uti-lize human curated gold datasets such as coco @xcite and @xcite with clean coupling between captions and images. however, scaling them to large and diverse concepts is expensive. we utilize uncurated silver standard datasets with the advantages of richness and diversity at the cost of noisy text. hence we show the effectiveness of a dual staged approach that denoises the captions by skeleton prediction. cross-lingual and controllable captions: past work on cross-lingual captioning focused on translation @xcite , fluency guidance @xcite , using large datasets @xcite and more recently by pivoting on source language captions @xcite . we go a step further and pivot on the predicted english skeleton to improve multilingual captions due to the dearth of similar off the shelf tools in other languages. we qualitatively explore controlling length via skeletons which was explored before via adding length to decoder @xcite . other controllable aspects include stylistic captions @xcite language @xcite which are potential extensions to our work. interpretable natural language skeletons: despite remarkable advancements of large scale endto-end models, recent work identifies spurious correlations in datasets that potentially lead to high performance @xcite . to mitigate this, researchers began to dissect intermediate components of models with the goal of interpretability to humans @xcite @xcite as opposed to implicit explanation @xcite . our work is an instance of explaining captions through skeleton predictions similar to recent work on rationalizing answer predictions for question answering @xcite . we view the intermediate skeleton layer as an interpretable model prediction that helps us study key subtle dataset attributes, such as gender bias. ## our approach ic requires paired examples of images and captions (i, c), where @xmath3, c 2 , ..., c m ), which are often expensive to gather. under this paradigm, end-to-end model training attempts to perform a match between the semantic concepts present in i and c, starting from image, region, and object level features and mapping them to various c i s. in contrast, our approach uses intermediate skeletons as an effective way to leverage noisy, uncurated alt-text based captions to train a model to generate more visually informative captions. an overview of both the stages is presented in fig. 1 . ## experiments and results hyperparameters: our transformer model uses 6 encoder and 6 decoder layers (unless specified otherwise), with 8 heads for multiheaded attention. captions are subword-tokenized with a vocab size of 8,300. the models are optimized with adam and an initial learning rate of 3.2e -5 . we use minibatches of size 128, and train for 1m steps. the token embedding and filter sizes are both 512. we experimented with several values for both frequency thresholding for skeleton words at 20, 50, 100 and k at 2, 4, 8, 16 for top-k selection for multilabel classification model. we manually selected the values that optimize the model performance based on manual examination as conducting human evaluations with more hyperparameters is very expensive especially with unreliable automatic metrics. ## controllability: qualitative discussion the dual-stage modeling using skeelton decomposition can be a double-edged sword: it can be an information bottleneck, limiting the ability to train the model in an end-to-end manner; but, it brings forth the advantage of increased interpretability and thereby the ability to use the intermediate stage results to control the final caption. we present aspects of caption controllability by altering the skeleton to explore effects on caption length, informativeness, and gender specificity. this section discusses the utility of this dual staged model for controllability qualitatively with skeae architecture. automatic intervention at the skeleton level involves non-trivially selecting related concepts for each image, and we leave this for follow-up work. instead, we present an empirical study only to semiautomatically control gender specificity in two of the languages. we plan to conduct experiments to compare with other models @xcite focused on controllability for follow-up work. effect of length of skeletons on captions: for applications that limit the caption lengths due to ui restrictions, the ability to control the length is important. the length of the skeleton correlates with the number of caption words, as shown in figure 6 . for 2 or 3 skeleton words, the percentage of captions monotonically decreases with the number of caption words, with the mode at 4-word captions. thus, for skeletons of size 2, captions of length 4 are much more frequent than captions of length 6 or 8. for longer skeletons, we see that the mode shifts to the right: with skeletons of size 5, the caption length peaks between 8 and 10 words. fig 7 illustrates this qualitatively. effect on gender specificity: current models often make embarrassing mistakes when generating captions that mention gender. the availability of a skeleton provides a direct handle for human-inthe-loop correction of such biases, at a pre-captiongeneration stage. this is more robust compared to caption post-processing, especially for highly inflected languages. to illustrate this, we compare the number of times 'man' appears in the captions generated by our baseline versus our dual-stage model after automatically modifying the skeleton (replacing 'man' to the gender-neutral word 'person' in the skeleton). over the t2 dataset, the baseline caption generates 'man' 13 times, and the automatic control mechanism via our model reduces this by 46% (to 7 occurrences) in english. in hindi, the equivalent of 'man' (a ú ) is generated 10 times, and it is reduced to a gender neutral word ( kt) by 70% (to 3 occurrences). ## conclusions scaling image captioning models practically mandates training on noisy and uncurated data available on web. our work presents an approach that denoises learning from such large yet diverse web-scaled data with alt-text annotations by subselecting content as intermediate skeletons. we experimentally demonstrate that this approach improves the captions significantly in human evaluations on out-of-domain test data by converting meta and story like captions to more visually informative captions. we also demonstrate the transferability of english skeleton words to improving captions in five other languages. additionally, the naturallanguage interpretable skeleton layer gives us a way to better control and perform human-in-theloop corrections of model predictions. we believe that this is a promising direction towards unpaired ic and also has potential for semi-automatic interventions to correct or interact with the skeletons to guide the final captions. in this work, our main focus is denoising alt-text captions using skeletons and using them for crosslingual captioning. in future, we plan to explore the effect of denoising in pretraining large multimodal models (blip @xcite , uniter @xcite , vilbert @xcite ) as base architectures by automatically cleaning captions, similar to how blip has an additional classifier to subselect captions that are not noisy. appendix h presents a broader impact of our work.