E n CBP : A New Benchmark Dataset for Finer-Grained Cultural Background Prediction in E nglish (2022)

Abstract: While cultural backgrounds have been shown to affect linguistic expressions, existing natural language processing (NLP) research on culture modeling is overly coarse-grained and does not examine cultural differences among speakers of the same language. To address this problem and augment NLP models with cultural background features, we collect, annotate, manually validate, and benchmark EnCBP, a finer-grained news-based cultural background prediction dataset in English. Through language modeling (LM) evaluations and manual analyses, we confirm that there are noticeable differences in linguistic expressions among five English-speaking countries and across four states in the US. Additionally, our evaluations on nine syntactic (CoNLL-2003), semantic (PAWS-Wiki, QNLI, STS-B, and RTE), and psycholinguistic tasks (SST-5, SST-2, Emotion, and Go-Emotions) show that, while introducing cultural background information does not benefit the Go-Emotions task due to text domain conflicts, it noticeably improves deep learning (DL) model performance on other tasks. Our findings strongly support the importance of cultural background modeling to a wide variety of NLP tasks and demonstrate the applicability of EnCBP in culture-related research.

## introduction psychological research has revealed that people from different cultural background behave differently in the ways they think @xcite , talk @xcite , write @xcite @xcite , and express emotions @xcite @xcite . nlp researchers have applied cultural background information to model differences in linguistic expressions across culture groups especially for psycholinguistic tasks foot_0 , e.g., distributional perspective identification @xcite and sentiment analysis @xcite . in prior research, culture groups are usually defined by official language @xcite (e.g., us, uk, and india are considered part of the same culture group) or, even more coarse-grained, by ideology @xcite ) (e.g., "western" countries and "eastern" countries). these settings typically overlook the nuanced cultural differences across or within countries, and they do not provide useful information for modeling different language use behavior in mono-lingual contexts. to study culture-specific linguistic expressions in the same language and to apply culture-related knowledge to other nlp tasks, we build encbp, a cultural background prediction dataset in english. following @xcite , we assume that language use patterns are more consistent inside each country or each district in a large country, e.g., states in the us. as such, we first construct news corpora by sampling news articles covering five frequently discussed and controversial topics from major news outlets in five english-speaking countries and four geographically dispersed states in the us. we then break the articles down to paragraphs and annotate them with the country and state codes of the news outlets to construct the country-and district-level subsets of encbp. we refer to the two subsets as encbp-country and encbp-district. to ensure annotation quality, we randomly sample 20 instances from each culture group and have them validated manually by local residents using amazon mechanical turk (mturk). the annotation accuracies and inter-validator agreement rates are both high for all the validation sets, supporting the correctness of the labels and demonstrating the differences in writing style across culture groups. in addition, we benchmark encbp for cultural background prediction with three widely-used nlp model architectures, namely bilstm, bert @xcite , and roberta @xcite . among the three models, the roberta-base model achieves the best overall performance, scoring 82.96 in f1-macro on the encbp-country and 73.96 on encbp-district. the better performance of bert and roberta over bilstm implies the importance of deep neural network architectures and large-scale pre-training for the challenging textbased cultural background prediction task. we conduct both quantitative and qualitative analyses on encbp to show the differences in linguistic expressions across culture groups. for the quantitative analysis, we fine-tune a bert model on the corpus with each cultural background label and evaluate it on the corpora of all the culture groups. results show that all the fine-tuned models are more compatible with the cultural domains of their training corpora and less compatible with the those of other corpora, with perplexity differences ranging from 0.43 to 14.90. for the qualitative analysis, we manually analyze sentence structures and the choices of words or phrases in instances randomly sampled from encbp to illustrate culturespecific english expressions. furthermore, we evaluate a bert model on nine psycholinguistic (sentiment analysis and emotion recognition), syntactic (named entity recognition), and semantic (paraphrase identification, natural language inference, semantic textual similarity, and text entailment) tasks to examine how modeling culture-specific english writing styles benefits the performance of nlp models. the models that incorporate cultural background information perform noticeably better on the named entity recognition (ner) task, most semantic tasks, and the sentiment analysis (sa) tasks. in our emotion recognition (er) evaluation on the go-emotions dataset, however, the performance is slightly harmed by incorporating cultural background information. this is likely due to the imbalanced cultural background distribution in the dataset, as the evaluation performance of bert clearly improves on emotion, another er dataset. on the paraphrase identification (pi) task, while the model performs better with cultural information incorporated, the improvement is lower than those on sa and ner tasks. this result suggests that differentiating linguistic expressions with the same semantic meaning may introduce additional noise to semantic tasks. our analyses and evaluations support the importance of cultural background modeling for a wide range of nlp tasks and show that encbp can con-tribute to future culture-related nlp research. the contributions of this paper are three-fold: • we construct, manually validate, and benchmark encbp, a mono-lingual news-based cultural background prediction dataset; • we qualitatively and quantitatively examine the distinctions in writing style from different culture groups; and • we show the effect of introducing cultural background information to nine downstream nlp tasks to showcase the importance of cultural information in natural language understanding. ## dataset construction this section introduces the construction, validation, and benchmarking of the encbp dataset. the encbp dataset adopts a multi-class classification objective. the labels are country codes of news outlets for the coarse-grained subset (encbp-country) and us state codes for the finer-grained subset (encbp-district). ## cultural domain compatibility this section examines whether linguistic expressions are clearly separable across culture groups in encbp through lm evaluations. we also manually examine representative linguistic expressions associated with each label to illustrate the differences in linguistic expression across cultures. ## experiments and analyses since cultural background labels are expensive to annotate, most nlp models forego the use of this information to opt for larger training data amount. for example, bert is trained on wikipedia text written in styles from mixed cultural backgrounds without access to cultural background information of the writers. using the encbp dataset we constructed, this section examines the relatedness between the cultural background prediction task and multiple other nlp tasks via model probing. we also examine the effectiveness of cultural feature augmentation, i.e., augmenting dl models on downstream nlp tasks with culture-specific writing style information. specifically, we evaluate a bert-base-cased model with two common information injection methods, namely two-stage training and mtl, on nine syntactic, semantic, and psycholinguistic tasks. ## conclusion and future work this paper presents encbp, a mono-lingual newsbased cultural background prediction dataset containing country-level (coarse-grained) and districtlevel (finer-grained) cultural background labels. through manual validation on mturk and cultural domain compatibility evaluations, we find that writing style clearly differs across countries and districts, confirming that cultural background has a substantial effect on writing style even in the same language. we also benchmark the dataset with state-of-the-art nlp models to show that, though challenging, different english expressions across cultural backgrounds can be identified and classified into culture categories by dl models. additionally, our evaluations on downstream nlp tasks of various types show that cultural feature augmentation is able to improve the performance of dl models on various semantic, syntactic, and psycholinguistic tasks. while the performance of the bert model is negatively affected by introducing cultural background information on an er dataset, the imbalanced distribution of cultural backgrounds in its corpus may account for the performance drop. our results demonstrate that cultural feature augmentation with encbp is a practical way of improving the performance of dl models on various nlp tasks, as long as the text domains of encbp and the downstream tasks are not too divergent. future work can extend our research to examine cultural differences in social media writings, which reflect even finer-grained cultural distinctions and are much noisier and difficult to annotate or validate than news articles.