Bridging the Gap between Authentic and Answer-Guided Images for C hinese Vision-Language Understanding Enhancement (2024)

Abstract: “The objective of the Chinese Vision-Language Understanding Evaluation (CVLUE) is to comprehensively assess the performance of Chinese vision-language multimodal pre-trained models in multimodal modeling and understanding across four tasks: Image-Text Retrieval, Visual Question Answering, Visual Grounding, and Visual Dialog. To enhance the models’ performance across various multimodal tasks, this paper propose a multimodal information understanding enhancement method based on answer-guided images. Firstly, we propose task-specific methods for answer-guided image generation. Secondly, the authentic and answer-guided images are fed into the model for multimodal fine-tuning, respectively. Finally, training objectives are set for different tasks to minimize the gap between the answer-guided images and authentic images, thereby supervising the results produced by the authentic images utlizing answer-guided images. The experimental results demonstrate the effectiveness of the proposed method.”

## introduction the chinese vision-language understanding evaluation (cvlue) aims to assess chinese visionlanguage multimodal pre-trained models from multiple perspectives, including image-text retrieval (itr), visual question answering (vqa), visual grounding (vg), and visual dialog (vd). this comprehensive evaluation is designed to measure the multimodal modeling and understanding capabilities of these models. exploring the diverse dimensions of chinese multimodal pre-trained models not only refines modeling strategies and optimization algorithms, but also significantly enhances the models' ability in multimodal information comprehension and interaction. researchers can also gain a deeper insight into their practical applications within real-world chinese contexts. to enhance the capabilities of chinese vision-language multimodal pre-trained models across various multimodal tasks, we propose a method to strengthen chinese multimodal comprehension by bridging the gap between authentic and answer-guided images. firstly, we propose a method to generate answer-guided images for each task. specifically, for the itr task, we generate images according to image captions utilize chinese text-to-image generation models. the synthetic images effectively highlight key textual information through attributes such as color, quantity, and orientation. for the vqa and vd tasks, we adopt an image generation approach based on questions and answers, integrating answer information into the image generation process. for the vg task, images with grounding information are used as answer-guided images. secondly, both the answer-guided images and authentic images are fed into the model, respectively. thirdly, we bridge the gap between authentic and answer-guided images during the training process, thereby enabling the answer-guided images to supervise and elevate the results yielded by authentic images. experimental results across various tasks demonstrate the effectiveness of our proposed method in enhancing chinese multimodal understanding. @xcite represents a modular architecture multimodal vision-language model, which is trained through a unified framework to learn multi-grained visual-language alignments. this model enhances its comprehension of weak-correlated image-text pairs by associating visual concepts such as objects, regions, and images with text descriptions, facilitating the execution of various imagetext tasks without the need for additional image annotations. moreover, x 2 -vlm exhibits commendable cross-lingual and cross-domain adaptability. by replacing text encoders tailored for specific languages or domains, it effectively adapts to image-text tasks across different languages and domains without the necessity for related pre-training, demonstrating its potential and flexibility in multimodal applications. thus, we have selected x 2 -vlm as the pre-trained model for this chinese image-text multimodal evaluation. ## participating system this section provides a detailed description of the methods and strategies employed in our evaluation. our method can be devided into two stages: the generation of answer-guided images and the multimodal fine-tuning for each task. to obtain answer-guided images, we design different prompts for text-to-image generation according to different tasks. we also set specific training objectives to accommodate the characteristics of each task during the multimodal fine-tuning stage. ## conclusion in the chinese vision-language understanding evaluation task, we propose a system that enhances chinese text-image multimodal understanding by bridging the gap between authentic images and answerguided images. firstly, we propose an image generation module, utilizing the text-to-image generation model or answer information from the train set to generate answer-guided images for different tasks. secondly, we send the authentic and answer-guided images into the model respectively during the fine-tuning stage. thirdly, we design specific training objectives for different tasks to encourage the representation or prediction consistency between the two images. our proposed method improves the performance of our model over the baseline model across all the tasks on the validation set.