Exploring the Potential of Dense Information in Multimodal Alignment (2024)

Abstract: Despite the success of data augmentation in improving CLIP model, existing methods that utilize LLM or SAM to enrich the information in captions still suffer from several limitations, including insufficient detail and excessive hallucinations, ultimately resulting in compromised alignment and masking the true potential of dense information. This can lead to erroneous conclusions about CLIP’s ability to handle rich data, impeding the development of more effective models. To address the limitations of existing methods, we introduce a novel pipeline that generates highly detailed, factually accurate captions for images, which facilitates in-depth analysis of the potential for dense information in multimodal alignment. Contrary to previous findings, our investigation revealed that lengthening captions boosts performance across diverse benchmarks, even surpassing the effectiveness of meticulously crafted hard negative samples. Building on these insights, DELIP is introduced, demonstrably enhancing both foundational multimodal alignment and compositional reasoning abilities. Finally, we explore strategies to expand the context window of the text encoder, unlocking the potential of richer data for CLIP and paving the way for advancements in leveraging dense information for multimodal alignment.

## introduction clip @xcite learns alignment between text and visual modalities through contrastive learning on a massive dataset of image-text pairs. its vision encoder has been widely applied to various downstream tasks, such as image captioning @xcite @xcite , visual grounding @xcite ), * work done during visiting student period. dci @xcite proposed a new benchmark to test the fine-grained understanding figure 1 : data generation pipeline. after obtaining a coarse-grained global description, we crop the image into four parts to improve the relative resolution. we then request gpt-4v to generate descriptions for each part and fill the detailed information into the coarse-grained global description. finally, we rewrite the resulting description, which contains rich details, to generate descriptions of different lengths and styles. in this paper, we propose delip, density enhanced language image continual pretraining, to analyze the impact of dense information on multimodal alignment. specifically, we first propose a crop-then-merge data generation pipeline to generate detailed descriptions with fewer hallucinations, which is the basis for subsequent experiments. then, we enhance the density of text information and image information respectively to study the impact on performance. we find that without using any tricks, we can outperform those methods that use hard negative examples on both the subcrop-caption matching task and the negatives discrimination task, which demonstrates the great potential of dense information in multimodal alignment. we then explore the strategy of expanding context windows and find that it can further improve the representation ability of the model. we also obtain some conclusions that are opposite to previous studies. through analysis, we find that this is because the dense captions used in previous studies is either shorter or more hallucinatory. we hope that our research can help to recognize the potential of dense information and inspire future work to scale it up to a larger scale, which we call information density scaling law. summarizing, our key contributions are: ## expand the context length previous works were hindered by a context window size of 77, preventing the model from effectively utilizing the information contained in longer captions. to address this limitation and harness the potential of denser text information, we explore increasing the context window to 512 tokens and analyze its impact on model performance. ## future work firstly, in this paper, we use gpt-4v to generate descriptions for each image or subcrop and then use gpt-4 to merge these generated descriptions into a global description with detailed details while minimizing hallucinations. however, calling the gpt-4 api is expensive. therefore, it is necessary to explore how to use open-source mllms to generate high-quality descriptions and how to use open-source llms to merge different descriptions together. secondly, we have only divided a single image into four parts, which still may overlook some details if the original image's input resolution is high and contains many details. a potential direction worth exploring is to divide the image into more parts to generate descriptions separately, which would improve the overall richness of details. however, it is essential to note how to effectively merge these individual parts without introducing addi-tional hallucinations (e.g., if an object is incorrectly divided into multiple parts), which is a problem worth studying. thirdly, when merging the descriptions of each part, we explicitly marked the relative position of each part in the original image (e.g., top left, bottom left, top right, bottom right) in the prompt. however, if the image is divided into more parts, marking the position information of each part becomes difficult. a straightforward idea is to use a coordinate system for annotation. we leave this part of the exploration work for the future. ## conclusion we conducted a systematic analysis of the impact of dense information on multimodal alignment. analyzing the impact of increasing the density of text information and vision information on model performance, we found that data rich in detail and with less hallucination improved the model's fine-grained alignment and enhanced its ability to distinguish negatives. interestingly, simply using density-enhanced training data already outperformed using carefully designed hard negative samples. we also tried various strategies to further increase context windows, finding that it further improved the model's representation ability, showcasing the great potential of dense information for enhancing multimodal alignment.