SEA - VQA : S outheast A sian Cultural Context Dataset For Visual Question Answering (2024)

Abstract: Visual Question Answering (VQA) is a critical task that requires the simultaneous understanding of visual and textual information. While significant advancements have been made with multilingual datasets, these often lack cultural specificity, especially in the context of Southeast Asia (SEA). In this paper, we introduce SEA-VQA aiming to highlight the challenges and gaps in existing VQA models when confronted with culturally specific content. Our dataset includes images from eight SEA countries, curated from the UNESCO Cultural Heritage collection. Our evaluation, comparing GPT-4 and GEMINI models, demonstrates substantial performance drops on culture-centric questions compared to the A-OKVQA dataset, a commonsense and world-knowledge VQA benchmark comprising approximately 25,000 questions. Our findings underscore the importance of cultural diversity in VQA datasets and reveal substantial gaps in the ability of current VQA models to handle culturally rich contexts. SEA-VQA serves as a crucial benchmark for identifying these gaps and guiding future improvements in VQA systems.

## introduction visual question answering (vqa) is the task of answering questions based on an image. as exemplified in figure 1 , one may ask a question involving an object or an action in an image. the vqa system accepts the question and picture as input and answers the questions based on the image's contents. therefore, the performance of vqa depends on the ability of the model to understand textual and visual information simultaneously. given its applications in various domains, such as healthcare, autonomous driving, and assistive technologies, vqa is pivotal in advancing human-computer interaction by enabling machines to comprehend and respond to complex visual content and textual queries. past efforts in vqa evaluation datasets have generally focused on measuring the reasoning, common knowledge, and image understanding of models. initially, these datasets @xcite used real-world images paired with straightforward questions, requiring direct answers based on visible elements. over time, the emphasis shifted towards complex reasoning with datasets like clevr @xcite and gqa @xcite , which present questions that demand comprehension of relationships, quantities, and spatial awareness. more recent datasets focus on improving generalization across various visual data types and question formats, testing the capabilities of vqa models from multiple perspectives and reasoning tasks @xcite @xcite @xcite @xcite . although these datasets have been instrumental in advancing vqa, they often lack multicultural aspects. multilingualism is typically achieved by translating existing queries into multiple languages, which does not fully capture cultural specificities @xcite @xcite . this approach overlooks the nuances and contextual knowledge unique to different cultures, limiting the robustness of vqa systems in diverse settings. for instance, xgqa @xcite introduces cross-lingual vqa but focuses more on language translation rather than cultural context. vi-clevr (tran et al., 2023) explores visual reasoning in vietnamese, but it is limited to a single culture and language. in particular, we develop a culturally specific dataset tailored to the region depicted in the image, incorporating a wider range of languages, including low-resource languages, particularly from southeast asia. this approach aims to improve the generalizability of vqa systems and address the current limitations in evaluating vqas on sea languages, which remains an open question in the field. our approach involves designing a datagathering pipeline based on the utilization of large language models, such as gpt-4, to formulate questions and answers based on culturally specific images. to ensure quality in question generation, we leverage metadata for cultural questions, including cultural names, countries, and im-age descriptions, to assist the multi-modal large language model (mllm) system in generating accurate questions. additionally, human oversight in the quality-checking process ensures the integrity of the data. our dataset comprises 515 images, 1,999 questions, and 53 cultures from 8 countries, focusing on the traditions of each culture and the reasoning behind each answer. this culturally specific approach aims to improve the generalizability of vqa systems and address the current limitations in evaluating vqas on sea languages, thereby ensuring robustness across diverse cultural and linguistic contexts. ## methodology to formulate our dataset, the data creation pipeline consists of four steps: (i) image curation, (ii) attribute extraction, (iii) qa generation, and (iv) data quality assurance. ## conclusion and future work in conclusion, we propose a vqa dataset for the southeast asian cultural context called sea-vqa. our dataset is generated from mllm while using humans in the data quality assurance process. using this approach, we are able to generate 1,999 questions from 8 countries and 53 cultures with limited human efforts. results from assessments using our sea-vqa dataset reveal that, although mllms demonstrate reasonable performances in standard vqa benchmarks, there is a gap in understanding local cultural knowledge. in future work, we aim to apply this process to other underrepresented languages and dialects from the region. we plan to explore more languages and images from open-source projects in sea, i.e., @xcite , to extend from monolingual to multilingual vqas. additionally, we will explore generating vqas using multiple models to improve accuracy and robustness. we also plan to add more attribute extraction methods to create more variation in vqas. in addition, we also plan to explore the integration of virtual reality technology to enhance the richness of the dataset.