Broaden the Vision: Geo-Diverse Visual Commonsense Reasoning (2021)

Abstract: Commonsense is defined as the knowledge on which everyone agrees. However, certain types of commonsense knowledge are correlated with culture and geographic locations and they are only shared locally. For example, the scenes of wedding ceremonies vary across regions due to different customs influenced by historical and religious factors. Such regional characteristics, however, are generally omitted in prior work. In this paper, we construct a Geo-Diverse Visual Commonsense Reasoning dataset (GD-VCR) to test vision-and-language models’ ability to understand cultural and geo-location-specific commonsense. In particular, we study two state-of-the-art Vision-and-Language models, VisualBERT and ViLBERT trained on VCR, a standard benchmark with images primarily from Western regions. We then evaluate how well the trained models can generalize to answering the questions in GD-VCR. We find that the performance of both models for non-Western regions including East Asia, South Asia, and Africa is significantly lower than that for Western region. We analyze the reasons behind the performance disparity and find that the performance gap is larger on QA pairs that: 1) are concerned with culture-related scenarios, e.g., weddings, religious activities, and festivals; 2) require high-level geo-diverse commonsense reasoning rather than low-order perception and recognition. Dataset and code are released at https://github.com/WadeYin9712/GD-VCR .

## introduction commonsense reasoning endows machines with high-level reasoning ability to understand situations with implicit commonsense knowledge. suppose that there is a scene where a woman is wearing a bridal gown at a party. an ideal ai system with commonsense knowledge should be able to infer that this woman is attending a wedding and likely to be the bride. recently, the field of commonsense reasoning is progressing with the development of large-scale benchmark datasets @xcite , intended to cover a wide range of commonsense knowledge, such as physical interactions @xcite , social conventions @xcite , and commonsense grounded in vision @xcite . however, existing benchmarks are often composed by data from sources in certain regions (e.g., western movies) and overlook the differences across groups in different regions foot_0 due to factors including cultural differences. in the aforementioned wedding example, while brides are usually in white in western weddings, they often wear red and their faces are covered with a red cloth in traditional chinese weddings. if a model is unaware of regional characteristics or incapable of capturing the nuanced regional characteristics, it leads to a disparity in performance across different regions @xcite . this motivates us to study how well a model trained on existing commonsense annotations can generalize to tasks requiring commonsense beyond western regions. in this paper, we mainly focus on regional commonsense with visual scenes. as shown in figure 1 , the three images all describe a wedding but the dresses of the grooms and brides are different, reflecting the regional characteristics of the wedding scenario. in this paper, we introduce a new evaluation benchmark, geo-diverse visual commonsense reasoning (gd-vcr), following the settings of the visual commonsense reasoning (vcr) task @xcite . vcr consists of multiple-choice questions paired with images extracted from movies or tv series primarily in western regions. gd-vcr includes 328 images, which are mainly sourced from movies and tv series in east asian, south asian, african, and western countries. the images are paired with 886 qa pairs, which need to be answered with geo-diverse commonsense and thorough understanding of the images. an example is given in figure 1 . gd-vcr benchmark addresses geo-diverse commonsense, such as "what are [person1] and [person2] participating?". with the help of these questions, it can manifest how models behave differently and reveal potential issues with geographical bias in commonsense reasoning. gd-vcr is one of the first benchmarks to evaluate model's reasoning ability on the task which requires geo-specific commonsense knowledge. we first study how well a model trained on vcr can generalize to questions involving geospecific commonsense. experimenting with two pre-trained vision-and-language (v&l) models, vi-sualbert @xcite and vilbert @xcite , we observe that two models achieve 64%-68% accuracy over the qa pairs on images from western regions, while their accuracy on images from east asian region ranges around 45%-51%. the significant performance disparity suggests that the commonsense learned in these models cannot be generalized well across different regions. we further investigate the reasons why the model exhibits such disparity based on the results of vi-sualbert. we first find that the performance gap on the images from western and non-western regions is large on the scenarios involving regional characteristics, such as weddings, religion and festivals. we also discover that the disparity is related to the reasoning difficulty of qa pairs. on the qa pairs only requiring basic visual recognition, e.g., "what's [person3] wearing? [person3] is wearing a suit.", the model achieves relatively similar performance over the four regions; however, the gap enlarges when the questions involve higherlevel reasoning with commonsense and rich visual contexts. by presenting the gd-vcr benchmark, we call upon the researchers to empower ai systems with geo-diverse commonsense such that they are capable of conducting high-level reasoning on data from different regions. ## related work commonsense reasoning benchmarks. recently, there has been an emergence of commonsense reasoning benchmarks @xcite @xcite @xcite @xcite , which cover a great variety of commonsense knowledge including visual, social, physical, and temporal commonsense. however, these commonsense benchmarks are mostly constructed by annotators from certain regions (e.g., the us and uk) using specific languages (e.g., english). xcopa @xcite and x-csr @xcite are two multilingual benchmarks, but most samples in both benchmarks are simply translated from english and cannot reflect the regional characteristics. different from previous benchmarks, gd-vcr focuses on geo-diverse commonsense instead of viewing commonsense as a universal monolith. vision-and-language tasks. a long line of research seeks to build vision-and-language datasets that test a model's ability to understand the visual world and how it is grounded in natural language. the tasks take on various forms, such as phrase grounding @xcite , visual question answering @xcite , and visual reasoning @xcite . to solve these tasks, a wide range of visual grounding skills are required. however, in existing tasks, little consideration is taken into reasoning on the images with regional characteristics. geographic bias. geographic bias is a serious issue that may cause harmful effects on certain groups of people. in computer vision, researchers @xcite find that most images from two large-scale image datasets, imagenet @xcite and openimages @xcite , are amerocentric and eurocentric. when a model trained on these datasets is applied to images from other regions, the performance will drop drastically. there also exists geographic bias in language technology. for example, it underlies natural language processing @xcite @xcite and automatic speech recognition @xcite ) models. our work seeks to reveal and test the geographic bias in the visual commonsense reasoning task and models. ## benchmark construction to build a geo-diverse visual commonsense reasoning benchmark, we design a three-stage annotation pipeline, following the original vcr dataset. 1) we first ask annotators to collect images from movies and tv series in western, east asian, south asian, and african countries. 2) we request annotators to design questions and write down the right answers according to the collected images. 3) we generate answer candidates automatically and formulate multiple-choice questions. the overview of our pipeline is illustrated in figure 2 . we elaborate on the three stages in the following. ## model performance and human evaluation over gd-vcr we are interested in the following questions: 1) can a model trained on the original vcr dataset (mostly western scenarios) generalize well to solve reasoning questions require commonsense specific to other regions? 2) do humans show similar trend when dealing with questions require regional commonsense that they are not familiar with? our experiments are conducted with two visionand-language models @xcite and vilbert @xcite . we fine-tuned the two pre-trained models on the original vcr training set and evaluate them on gd-vcr. all the experimental results are the average of 3 runs with different seeds. implementation details are listed in appendix c. ## analyses of performance disparity as we observe large performance gaps between western and non-western data in section 4.1, in this section, we inspect the pre-trained v&l model to analyze the reasons behind such performance disparity on two aspects, 1) regional differences of scenarios and 2) reasoning level of qa pairs. we analyze the visualbert model, since its performance gap is more evident. ## conclusion we propose a new benchmark, gd-vcr, for evaluating v&l models' reasoning ability on the qa pairs involving geo-diverse commonsense knowledge. experimental results show that the v&l models cannot generalize well to the images regarding the regional characteristics of non-western regions. based on visualbert's results, we find that 1) the scenarios such as wedding, religion and festival, which require geo-diverse commonsense knowledge to be understood, and 2) the reasoning difficulty of qa pairs are highly associated with the performance disparity. for broader impact, we hope that the gd-vcr benchmark could broaden researchers' vision on the scope of commonsense reasoning field and motivate researchers to build better commonsense reasoning systems with more inclusive consideration.