Cultural Concept Adaptation on Multimodal Reasoning (2023)

Abstract: Developing cultural adaptation methods is important, which can improve the model performance on the low-resource ones and provide more equitable opportunities for everyone to benefit from advanced technology. Past methods primarily focused on multilingual and multimodal capabilities, and the improvement of multicultural competence is still an unexplored problem. This is largely due to the difficulty of data scarcity and expensive annotation. In this paper, we navigate this uncharted territory by leveraging high-resource cultures to facilitate comprehension of low-resource ones. We first introduce an annotation-free method for cultural-concept adaptation and construct a concept mapping set. To facilitate the model’s comprehension of cultural-concept mappings, we propose a new multimodal data augmentation called CultureMixup. This approach employs a three-tier code-switching strategy on textual sentences. Additionally, it uses a cultural concept-based mixup method for the images. This combination effectively generates new data instances across culture, phrase, word, and image levels. For visually grounded reasoning across languages and cultures, experimental results on five languages show that our method consistently improves performance for four existing multilingual and multimodal models on both zero-shot and few-shot settings.

## introduction the computer vision (cv) and natural language processing (nlp) communities have witnessed significant strides in multilingual and multimodal research in recent years. for instance, xlms (conneau and lample, 2019) have substantially surpassed the previous benchmarks in cross-lingual tasks. the uc 2 model @xcite learns to represent input tokens in a context-aware manner by leveraging both linguistic and visual content. ## method to overcome data scarcity and costly labeling, we initially propose an annotation-free method for cultural-concept adaptation, which constructs a concept mapping set. to support the model's understanding of cultural-concept mappings, we subsequently introduce a novel cultural concept-based multimodal data augmentation technique. by training the model on both original and augmented data, we significantly boost the model's performance. ## conclusion to attack the difficulties of data annotation and scarcity, we propose an annotation-free cultural adaptation method and design a novel cultural concept-based multi-modal data augmentation to generate the new data example. by training the model on the augmented dataset, key results indicate that our methods consistently and statistically outperform the baselines. in the future, we plan to apply our method to more downstream tasks related to culture. employing curriculum learning and designing more refined training strategies according to the difficulty of different languages and cultural concepts is also worth exploring. at the same time, how to further extend our method to make it more applicable to multi-modal models based on autore-gressive generation, such as gpt-4-v foot_5 , is also highly worthwhile to explore.