C hinese Vision-Language Understanding Evaluation (2024)

Abstract: “This paper introduces our systems submitted for the Chinese Vision-Language Understanding Evaluation task at the 23rd Chinese Computational Linguistics Conference.In this competition, we utilized X2-VLM and CCLM models to participate in various subtasks such as image-text retrieval, visual grounding, visual dialogue, and visual question answering. Additionally, we employed other models to assess performance on certain subtasks. We optimized our models and successfully applied them to these different tasks”

## introduction in today's era of information explosion, multimodal understanding and interaction between vision and language have become increasingly important. with the rapid development of deep learning technology, vision-language pre-training models can establish associations between images and text, demonstrating powerful performance. this competition encompasses various subtasks, spanning image-text retrieval, visual grounding, visual dialogue, and visual question answering. these tasks not only assess the model's comprehension and processing abilities with multimodal data but also evaluate its adaptability and robustness across diverse application scenarios. through participation in these tasks, our goal is to validate the effectiveness of different vision-language pre-training models and delve into their potential in practical deployments. the image-text retrieval task requires the model to retrieve relevant images based on textual descriptions or find matching text based on images. this demands the model to efficiently encode and align image and text features. the visual grounding task requires the model to accurately locate target objects in images, testing its fine-grained feature extraction capabilities. the visual dialogue task involves understanding the content of images and generating natural dialogues based on context. the visual question answering task requires the model to answer questions based on image content, assessing its visual understanding and language generation capabilities. in this competition, we successfully completed the above tasks by optimizing and adjusting multiple models, including the x 2 -vlm model. this paper provides a detailed introduction to our research methods, experimental process, and results, and summarizes the performance and achievements of the models in each task. additionally, we will share the challenges and solutions encountered during the competition, aiming to offer references for future research and applications. ## methodology the main models discussed in this paper include x 2 -vlm @xcite , cclm @xcite , chinese clip (redford et al.), and ofa @xcite . x 2 -vlm is a general pre-training model capable of handling tasks that combine vision and language. clip, proposed figure 2: the overall structure of the ofa model by openai, is a multimodal model that can embed images and text into the same semantic space. ## conclusion in the chinese vision-language understanding evaluation task of the 23rd chinese computational linguistics conference, we designed and submitted a multi-model system to participate in several subtasks, including image-text retrieval, text retrieval, visual question answering, visual localization, and visual dialogue. we primarily utilized the x 2 -vlm, cclm, chinese clip, and ofa models, leveraging their strengths to enhance performance in each subtask. the experimental results demonstrate that the x 2 -vlm model excelled in image-text retrieval and visual grounding tasks, particularly showcasing strong capabilities in aligning image and text features. the cclm and ofa models also exhibited good performance in specific tasks. through model optimization and data preprocessing, we successfully enhanced the system's performance in each subtask. in future work, we plan to further optimize these model architectures and explore additional data augmentation and preprocessing methods to further improve model performance. additionally, we aim to apply these models to a broader range of multimodal tasks to verify their applicability in diverse scenarios. overall, the outcomes of this competition underscore the potential and advantages of these multi-model systems in tackling complex multimodal tasks, offering valuable insights and guidance for future research and applications. c c l 2 0 2 4