GD - COMET : A Geo-Diverse Commonsense Inference Model (2023)

Abstract: With the increasing integration of AI into everyday life, it’s becoming crucial to design AI systems to serve users from diverse backgrounds by making them culturally aware. In this paper, we present GD-COMET, a geo-diverse version of the COMET commonsense inference model. GD-COMET goes beyond Western commonsense knowledge and is capable of generating inferences pertaining to a broad range of cultures. We demonstrate the effectiveness of GD-COMET through a comprehensive human evaluation across 5 diverse cultures, as well as extrinsic evaluation on a geo-diverse task. The evaluation shows that GD-COMET captures and generates culturally nuanced commonsense knowledge, demonstrating its potential to benefit NLP applications across the board and contribute to making NLP more inclusive.

## introduction culture plays a significant role in shaping an individual's worldviews, beliefs, behaviours, and communication styles @xcite . a considerable portion of what is commonly referred to as commonsense knowledge is not universal but rather culture-specific, including social norms, values, traditions, and more. an example of cultural differences is greetings, which may involve a handshake in western cultures, bowing in some asian cultures, a 'namaste' gesture in india, or 'wai' in thailand. with ai systems becoming increasingly ubiquitous in society, it is imperative to go beyond the western cultural perspective @xcite . lack of cultural awareness may lead to models perpetuating stereotypes and reinforcing societal inequalities @xcite @xcite , impeding their effectiveness for users from non-western countries. in this paper, we focus on a popular model for commonsense reasoning, comet @xcite , which is based on an english language model (lm) and further trained on commonsense inferences collected from north american crowdsource workers @xcite . consequently, the model exhibits a certain bias towards the north american cultural perspective. as evidenced by fig. 1 , comet displays limited familiarity with the concept of a german pancake, erroneously interpreting the term "dutch baby" in a literal sense. we identify a need for more inclusive commonsense reasoning models and propose gd-comet: geo-diverse comet. as demonstrated in fig 1 , gd-comet gained the culturally relevant knowledge to interpret "dutch baby" as a legitimate dish. gd-comet is similarly based on an english lm but is trained on a knowledge base of cultural knowledge @xcite prior to training on comet's original training data. this simple approach is effective, as judged by both human evaluations as well as extrinsic evaluation on a geodiverse task @xcite . gd-comet can potentially benefit many downstream nlp applications where the user population is diverse. foot_0 2 background ## gd-comet we present gd-comet, a geo-diverse version of comet. the goal of gd-comet is to generate high-quality commonsense inferences for concepts and events pertaining to both western and non-western cultures. rather than collecting a large-scale geo-diverse dataset in the style of atomic, we split the training into two phases: (1) training the underlying lm on geo-diverse data; ## intrinsic evaluation to evaluate the quality of gd-comet, we construct a set of input sentences pertaining to 5 diverse cultures ( the annotations yielded a substantial interannotator agreement with κ = 0.656 for comet and 0.702 for gd-comet, measured with average cohen's kappa @xcite across cultures. results. ## extrinsic evaluation traditional benchmarks often fall short in testing models' knowledge and comprehension of diverse cultural contexts. to show gd-comet's utility for downstream tasks, we evaluate on a multimodal task, gd-vcr (sec 5.1). we develop a model inspired by vlc-bert @xcite ) that generates inferences and incorporates them into a vision and language (v&l) model (sec 5.2). we show that gd-comet improves the performance on gd-vcr upon an array of baselines (sec 5.3) and demonstrate the inferences contributing to the performance gains (sec 5.4). ## conclusion this work challenges the current notion of universally applicable commonsense knowledge by introducing gd-comet, a geo-diverse variant of comet. gd-comet can generate culturallynuanced commonsense inferences for a broad range of cultures. our comprehensive evaluation confirms the effectiveness of gd-comet in incorporating and leveraging cultural cues. we view our work as a step towards developing more inclusive and culturally-aware ai systems.