Altogether: Image Captioning via Re-aligning Alt-text (2024)

Abstract: This paper focuses on creating synthetic data to improve the quality of image captions. Existing works typically have two shortcomings. First, they caption images from scratch, ignoring existing alt-text metadata, and second, lack transparency if the captioners’ training data (e.g. GPT) is unknown. In this paper, we study a principled approach Altogether based on the key idea to edit and re-align existing alt-texts associated with the images. To generate training data, we perform human annotation where annotators start with the existing alt-text and re-align it to the image content in multiple rounds, consequently constructing captions with rich visual concepts. This differs from prior work that carries out human annotation as a one-time description task solely based on images and annotator knowledge. We train a captioner on this data that generalizes the process of re-aligning alt-texts at scale. Our results show our Altogether approach leads to richer image captions that also improve text-to-image generation and zero-shot image classification tasks.

## introduction human social interactions often gravitate towards engaging with individuals who exhibit a higher level of intelligence. this inherent social behavior underscores the aspiration to develop ai agents that surpass the average human intelligence. the pursuit of creating such advanced ai agents hinges significantly on the quality of the training data, which ideally encapsulates superhuman intelligence. however, in the context of image captioning, most existing training data is designed for naive and well-known visual concepts that provide little value to an average user, e.g., a caption "a dog is walking in the park" offer minimal utility to most users unless specific accessibility needs are present, e.g., for individuals with visual impairments. the primary issue with these captions lies in their lack of detail; they fail to convey nuanced information about the images, such as the breed of the dog or the specific name or location of the park. moreover, while alternative text (alt-text) in webcrawled data often contains detailed and concrete visual descriptions, current captioning models generally ignore this information. instead, these models tend to generate captions solely based on the image content, which misses the opportunity to enhance the relevance and accuracy of the captions. additionally, advancements in caption quality often lack transparency and are not easily reproducible. for instance, recent developments such as llava @xcite and sharegpt4v @xcite utilize high-quality captions derived from proprietary models like gpt-4v. while these models benefit from high-quality annotations, they are built on processes that are not openly shared. this lack of disclosure presents significant challenges in terms of scalability, intellectual property rights, data integrity and privacy. the use of such proprietary models in industry applications is fraught with risks, particularly when the implementation details remain undisclosed. this paper presents a principled approach to enhance caption quality and develops a parameterefficient captioner capable of scaling re-captioning efforts. we assume each image contains information that the caption needs to align with using natural language. although obtaining the real-world information from an image or generating a perfect ground-truth caption might be challenging, we demonstrate that caption quality can be improved relatively by iteratively refining captions to better describe the visual content (e.g., adding information on specific objects, colors, spatial relations or more fine-grained named entities). our key insight is that the creator who posts an image along with its associated alt-text is likely the most knowledgeable expert regarding the concrete visual concepts within that image (e.g., knowing that the animal is an "iguana" instead of just an "object," "animal," or "lizard"). it would be difficult for an average annotator to provide similar level of detail within a short annotation timeframe. instead, these annotators could offer weak yet complementary supervision by either removing non-existent information from the alt-text or describing missing objects using more general concepts ("lizard" instead of "iguana"). building on this insight, we introduce altogether, an approach to improve image captions through the process of re-aligning existing alt-texts with the image content. we instantiate this idea in two forms (i) through human annotation to create a fine-tuning dataset and (ii) through a parameter-efficient captioner that can re-caption billions of images when fine-tuned for this task. for annotation (i), we perform multiple rounds of alt-text realignment to preserve concrete visual concepts while adding or removing relevant information, as depicted in fig. 1 . starting with the initial alt-text, which may partially overlap with the image, subsequent annotation rounds iteratively refine the captions to achieve better alignment with the image's information. using this data, we can train a captioner (ii) that is capable of generalizing this process by reading, grounding, and transforming alt-texts into dense captions at scale. we evaluate our re-aligned captions across captioning, generative and discriminative tasks. with a lightweight text decoder, our captioner surpasses alt-texts by 4% in clip @xcite score and outperforms state-of-the-art captioners on a challenging test set, which we annotate based on a subset of the wit (wikipedia image-text) dataset @xcite . we further evaluate our approach on text-to-image (t2i) generation, where we observe significant improvements in similarity between generated images and text prompts when training latent diffusion models with synthetic captions. for discriminative tasks, we obtain 1.1% absolute accuracy improvement over 26 zero-shot classification datasets and a 3% gain on retrieval tasks, when using synthetic captions to supplement clip training. an interesting observation we make is that generative and discriminative tasks require widely different ratios (100% vs. 15%) of synthetic data. dense captioning. while image captioning is well-studied, generating dense captions precisely aligned with the original images has gained more attention recently. mscoco-style captions @xcite are brief and describe main objects, limiting their value for aligned image-text pairs due to their brevity, general concepts, and constrained image distribution. the dci dataset @xcite overcomes the brevity issue but still suffers from the other limitations. docci @xcite and imageinwords (iiw) @xcite address these challenges for specific datasets using clustering or iterative refinement with object detection tools. our work proposes a general process to improve caption quality for web images, paving the way for further advancements in this area. ## altogether: re-aligning alt-texts this section presents our method for re-aligning alt-texts to produce dense captions with concrete visual concepts, which we later ( §4) instantiate in a parameter-efficient captioner scalable to billions of images. we structure this section into three main parts: ( §3.1) revisiting the image captioning task, ( §3.2) incorporating re-alignment into existing captioning frameworks, as well as designing annotation tasks ( §3.2.1) and learning mechanisms ( §3.2.2) for re-aligning alt-texts. ## altogether: implementation details in this section, we first discuss the annotation and training data for our captioning model in §4.1. then we describe captioner architecture in §4.2. ## evaluation our evaluation spans three areas: (i) human annotations, (ii) captions generated from our captioner, and (iii) downstream tasks using our synthetic captions (i.e., text-to-image generation and zero-shot image classification).