Precision or Recall? An Analysis of Image Captions for Training Text-to-Image Generation Model (2024)

Abstract: Despite advancements in text-to-image models, generating images that precisely align with textual descriptions remains challenging due to misalignment in training data. In this paper, we analyze the critical role of caption precision and recall in text-to-image model training. Our analysis of human-annotated captions shows that both precision and recall are important for text-image alignment, but precision has a more significant impact. Leveraging these insights, we utilize Large Vision Language Models to generate synthetic captions for training. Models trained with these synthetic captions show similar behavior to those trained on human-annotated captions, underscores the potential for synthetic data in text-to-image training.

## introduction recent advancements in diffusion models such as stable diffusion @xcite , dalle 3 @xcite , emu @xcite , and imagen @xcite , have demonstrated remarkable capabilities in image synthesis. despite these achievements, challenges persist in generating images that accurately align with the given text inputs @xcite . one issue is the misalignment between training captions and images, where captions either describe only a portion of the image or fail to describe the image content accurately. recent research efforts have focused on enhancing caption quality using large language models (llm) @xcite or large vision language models (lvlm) @xcite . nevertheless, there is a scarcity of in-depth analysis on how specific factors influence the efficacy of text-to-image model training. in this paper, we evaluate captions based on two metrics: precision and recall. we train the textto-image (t2i) model using human-annotated cap1 the data and code is available at https://github.com/ shengcheng/captions4t2i . tions that vary in levels of precision and recall. we then assess the model's capability for text-image alignment, specifically focusing on compositional ability. as illustrated in figure 1 , our findings indicate that while combinations of high precision and high recall yield the best results, generating captions with high precision is generally more beneficial. however, if the model is prone to hallucination, adding more diverse details can also enhance performance. building on these insights as shown in fgiure 1, we explore whether the same observation hold when using lvlms to generate synthetic captions for t2i training. given the variability across different lvlms, some models may emphasize diversity at the expense of precision, leading to increased hallucination, while others may prioritize precision but sacrifice diversity. we evaluate the precision and recall of captions generated by various lvlms and then use them to train t2i models. our findings confirm that the compositional capabilities of the t2i models are consistent with our previous con-clusions, underscoring the critical role of precision in caption generation. the major contributions of this paper are: improving captions for text-to-image model training models such as pixart-α @xcite , dalle 3 @xcite , and stable diffusion 3 @xcite emphasize the critical role of high-quality captions in their training. these systems incorporate captions synthesized by lvlms @xcite @xcite into their training processes to enhance t2i generation capabilities. in particular, @xcite investigates how synthetic captions contribute to improved t2i generation. however, these models do not specifically examine how caption quality impact the effectiveness of their training processes. ## analysis of image captions for t2i training dataset construction our study utilizes the dense caption dataset @xcite foot_0 , which comprises 8,012 images and 99,445 submasks derived from the sam dataset @xcite . each image in the dataset is segmented into multiple submasks, and both the main image and its corresponding submask images, known as subimages, are each paired with at least one detailed, human-annotated caption. we refer to the captions associated with the main image as the main captions and those linked to the subimages as subcaptions. these captions have been condensed to summarized versions of no more than 77 tokens using the llama2-70b model @xcite . to create negative captions, the llama2-70b model modifies these sentences by altering their structure, editing content, or reshuffling words to form new sentences with the original vocabulary. our dataset is constructed based on these elements. in our study, we focus solely on how captions, including those for specific subregions within images, align with the overall image. therefore, we do not use the subimages derived from submasks; instead, our dataset is built around the subcaptions associated with these submasks. to control the recall of the caption, each image is described not just by a basic caption that outlines the overall scene, but also by multiple subcaptions that provide detailed descriptions of specific regions, as shown in to validate the model's ability to generate images accurately aligned with the provided text, we use the t2i-compbench @xcite , which includes five tasks. the first three tasks evaluate the model's capacity to accurately generate multiple objects in terms of correct color, shape, and texture. these tasks convert the text prompts into questions, which are tested using the blip2 model @xcite . the spatial task examines the model's understanding of spatial directives like 'left' and 'right' using an object detection algorithm @xcite . the non-spatial task focuses on object interactions and employs the clip model @xcite to evaluate alignment. ## insight for synthetic captions for t2i training building on the findings from our previous analysis, this section investigates whether the observed impacts of precision and recall on performance also apply to synthetic captions. we conducted experiments using synthetic captions generated by three different lvlms: llava @xcite , blip2 @xcite , and uform 3 . each model was given the instructions: "describe the image concisely," with a limit of fewer than 77 tokens per caption. the images for training are sourced from the mscoco dataset @xcite , which includes 118k images. we fine-tuned the stable diffusion v2.1 and stable diffusion xl base 1.0 @xcite 4 model using these captions for 100,000 iterations with a batch size of 8 and a learning rate of 1e-4. due to the high cost of human verification for the precision and recall of synthetic captions, we use a modified version of the faithscore @xcite for evaluation. initially, we filter out the descriptive content of the captions. then, adapting from the original method in @xcite , which decomposes captions into entity, count, color, relation, and other, we refine our decomposition to better fit the compositional evaluation metric by using entity, shape, color, texture, spatial, non-spatial, and other. each sentence within these categories is then assessed for correctness. for the first two steps, we utilize the gpt-3.5 api @xcite ., 3 https://huggingface.co/unum-cloud/uform-gen2-qwen500m 4 https://huggingface.co/stabilityai/stable-diffusion-xlbase-1.0 method llava blip uform # of entites 4.90 2.12 6.52 faithscore 0.911 0.931 0.831