World to Code: Multi-modal Data Generation via Self-Instructed Compositional Captioning and Filtering (2024)

Abstract: Recent advances in Vision-Language Models (VLMs) and the scarcity of high-quality multi-modal alignment data have inspired numerous researches on synthetic VLM data generation. The conventional norm in VLM data construction uses a mixture of specialists in caption and OCR, or stronger VLM APIs and expensive human annotation.In this paper, we present World to Code ( W2C ), a meticulously curated multi-modal data construction pipeline that organizes the final generation output into a Python code format. The pipeline leverages the VLM itself to extract cross-modal information via different prompts and filter the generated outputs again via a consistency filtering strategy. Experiments have demonstrated the high quality of W2C by improving various existing visual question answering and visual grounding benchmarks across different VLMs. Further analysis also demonstrates that the new code parsing ability of VLMs presents better cross-modal equivalence than the commonly used detail caption ability. Our code is available at https://github.com/foundation-multimodal-models/World2Code.

## introduction fueled by the rapid development of vision-language models (vlms) @xcite @xcite @xcite and diffusion models (dms) @xcite , collecting detailed and concrete high-quality captions for each image becomes more and more urging. however, expensive and tedious human labeling for high-quality imagetext pairs further incurs the necessity of a cheap and reliable data construction pipeline without human intervention. related works on image-text data curation can be divided into two main streams. distillation-based methods leverage closed-source commercial products (e.g., gpt-4v @xcite ) with the state-of-the-art performance for image caption @xcite @xcite . another line of work curates an image caption pipeline with existing vlms to filter high-quality image-text for the training of better vlms. these methods usually combine opensource llms @xcite and different visual specialists @xcite @xcite @xcite @xcite to endow existing vlms with new abilities, e.g., pixel grounding in glamm @xcite . however, the dependency on a mixture of specialists and human feedback in filtering noisy generations @xcite makes it difficult to scale the generated data and automate the process. recent progress shows that generated results of llms @xcite and vlms @xcite for prompts with similar meanings should be alike and we can help filter out noisy generated texts and captions by consistency checking among multiple prompt instructed results. in light of the above evidence, we present a self-instructed data construction pipeline, coined w2c . w2c autonomously extracts and articulates specific content from images, and enhances the reliability of the generated image captions by employing consistency filtering by assessing the outputs through multiple instructed prompt consistencies. the overall pipeline reduces requested specialists and frees off expensive human feedback as shown in figure 1 . in addition, we leverage the idea from human-machine interaction and organize the model-generated responses into a python code format, following @xcite and text2reward @xcite . experiments have shown that our proposed w2c can improve vlms on various visual question-answering benchmarks. to be specific, w2c performs the best in 7 out of 9 vqa benchmarks on llava-next-7b, and 6 out of 9 vqa benchmarks on llava-next-13b. furthermore, w2c also improves few-shot evaluations on two widely used vqa benchmarks including gqa and mme. especially, on the 2-shot evaluation of gqa, the method achieves over 5 accuracy gains across different vlms. our contribution is summarized in threefold: ## related work vision language models with the emergence of llms @xcite @xcite @xcite , vlms @xcite @xcite have demonstrated exceptional capabilities in visual recognition and understanding, achieving remarkable results on various vlm benchmarks @xcite @xcite each image. however, the expensive human annotation required in as-1b and the complicated construction pipeline in glamm have greatly limited the potential of data scaling. in this work, we try to answer whether synthetic data can improve vlms on classical vqa benchmarks @xcite @xcite to avoid tedious data collection. recent progress in synthetic data generation for llms @xcite @xcite shed light on the possibility of multi-modal data construction by leveraging consistency in generation to filter invalid data. @xcite presents the consistent reasoning path generation demonstrating better performance in cot. @xcite uses the generator-validator consistent data for training and can effectively improve llms on various tasks. @xcite further shows that the generator-validator consistency in most vlms is prone to be correct. code representation for visual tasks code representations can formally encode various structure information in a scene. eureka @xcite and text2reward @xcite parse a scene into python codes and encourage llms to generate programmable dense rewards. vistruct @xcite takes the first step in visual code intelligence by decomposing the codevisual representation into multiple components including object recognition, object grounding, attribute detection, relation detection, and event detection. @xcite further introduces a curriculum learning approach to endow vlms with the aforementioned four abilities. however, the heavy dependency on supervised human-labeled datasets and the complicated curriculum learning pipeline limits its potential. this work investigates an effective data-constructing pipeline based on code-vision representation. ## method our data construction pipeline shares some similarities with glamm @xcite , where both methods focus on the region-level caption of the whole image. w2c further extend glamm to support generation-validation consistency filtering by exploring different organization formations of the labeled elements and present how vlms boost themselves on basic multi-modal understanding tasks. to make a comprehensive and systematic exposi-tion of our w2c entire pipeline, the following will be divided into three parts for discussion: ## conclusion this paper presents w2c , an enhanced data construction pipeline that only leverages existing vlms themselves for detail and compositional captions for an image, which is further organized in python code format. we present that existing vlms can improve themselves on the understanding benchmarks in various scenarios, significantly reducing the need for a mix of visual specialists and heavy human annotations. moreover, additional experiments show that the new code parsing ability of vlms presents better capability in fully describing the image, with notable improvement in the few-shot evaluation on downstream tasks when the raw images are not provided. our proposed w2c not only enhances the original capabilities on the widely used multi-modal understanding benchmarks but also endows existing vlms with detailed and executable multi-modal parsing ability.