T y D i P : A Dataset for Politeness Classification in Nine Typologically Diverse Languages (2022)

Abstract: We study politeness phenomena in nine typologically diverse languages. Politeness is an important facet of communication and is sometimes argued to be cultural-specific, yet existing computational linguistic study is limited to English. We create TyDiP, a dataset containing three-way politeness annotations for 500 examples in each language, totaling 4.5K examples. We evaluate how well multilingual models can identify politeness levels – they show a fairly robust zero-shot transfer ability, yet fall short of estimated human accuracy significantly. We further study mapping the English politeness strategy lexicon into nine languages via automatic translation and lexicon induction, analyzing whether each strategy’s impact stays consistent across languages. Lastly, we empirically study the complicated relationship between formality and politeness through transfer experiments. We hope our dataset will support various research questions and applications, from evaluating multilingual models to constructing polite multilingual agents.

## introduction whether politeness phenomena and strategies are universal across languages or not have been controversial among sociologists and linguists. while @xcite claimed their universality, other followup work @xcite claimed how communication patterns can differ based on cultures and other social constructs such as gender @xcite and domains. to contribute to the linguistic study of crosscultural politeness, we collect politeness labels on nine typologically and culturally diverse languages, hindi, korean, spanish, tamil, french, vietnamese, russian, afrikaans, and hungarian. this language set covers five scripts and eight language families. we follow the seminal work @xcite closely, focusing on politeness exhibited in requests as they involve the speaker imposing on the listener, requiring them to employ various politeness techniques. to capture rich linguistic strategies that can be lost in translation @xcite , we collect sentences written in each target language. to minimize the domain shift among languages, we collect examples in each language from their respective wikipedia user talk pages, where editors make requests about administrative and editorial decisions. crowdsourcing labels in low-resource languages is challenging. thus, we carefully design an annotation process that includes a translation task to evaluate annotator's language proficiency and a model-in-the-loop qualification task which filters workers whose labels diverges from highly confident predictions from multilingual models. after this process, we observe high agreements among the annotators in our dataset despite the subjectivity of the task. interestingly, the annotators agree with each other more when assigning politeness score on requests in their native languages compared to assigning politeness score on requests in english, which is their second language. equipped with our new multilingual politeness dataset, we evaluate zero-shot transfer ability of existing multilingual models in predicting politeness -subjective and pragmatic language interpretation task. pretrained language models @xcite fine-tuned on annotated english politeness data @xcite show competitive performances on all languages, weighing in the universality of politeness phenomena across languages. we also witness impressive zeroshot performance of a high-capacity pretrained language model @xcite . we observe a degradation in classification performances when we translate the target language (via google translate api) to english, suggesting politeness might not be preserved in the current machine translation model. despite the simplicity of classification task, we report a substantial difference between the estimated human accuracy and the best model accuracy (over 10% difference in accuracy in six out of nine languages). lastly, we provide two studies delving into politeness phenomena. we map english politeness strategy lexicon to create politeness strategy lexicon in nine languages by using tools like automatic translation, lexicon alignment @xcite and large-scale corpora in the same domain. despite the limitations of automatic lexicon mapping, we largely observe consistent correlation with politeness score for each politeness strategy in nine languages we study, with some interesting exceptions. we then compare the notion of politeness and formality which has been studied in multilingual setting @xcite . our empirical results supports that notions of politeness and formality cannot be used interchangeably. however, when we control for semantics, politeness classifier judges the formal version of the same sentence as more polite than its informal variant. we release our annotated data and aligned politeness lexicon to support future work. our dataset can support various end applications, such as building multilingual agents optimized for politeness @xcite , developing a translation model that preserves politeness level @xcite , evaluating the impact of different pretraining corpus and modeling architecture for modeling subjective tasks in a wide range of languages @xcite , understanding cultural-specific politeness strategies, and many more. ## tydip: multilingual politeness dataset motivation our goal is to construct high-quality multilingual evaluation data with native content, covering a wide range of languages on the task of politeness prediction. following prior work @xcite , we focus on identifying politeness in requests, where requests involve speaker imposing on the listener. this scenario elicit speakers to employ diverse strategies to minimize the imposition of requests, or apologizing for the imposition @xcite . for each request text, we aim to collect a graded politeness score (between -3 and 3, with 0.5 increment). language selection we chose hindi, korean, spanish, tamil, french, vietnamese, russian, afrikaans, and hungarian. our criteria for selecting languages were (1) covering low resource language when possible, (2) languages with rich discussion on wikipedia editor forum and (3) languages where we can recruit native speaker annotators on a crowdsourcing platform, prolific. 2 source sentence collection we source requests from wikipedia user talk pages from target language wikipedia dumps. 3 each request is a part of a conversation between editors on wikipedia. we follow the pre-processing step from prior work @xcite , extracting each request as a sequence of two successive sentences where the second sentence ends with a question mark (?). we present one example here: "i'm somewhat puzzled by your recent edits on the harper page, which have left two different sets of footnotes. could you please explain your rationale for the change?" ## predicting politeness equipped with politeness data for nine languages, we evaluate cross-lingual transfer performance of multilingual language models @xcite . we are interested in following research questions: models we study two fine-tuned pretrained language models, one english model (roberta @xcite ) and one multilingual model (xlm-roberta @xcite ) which supports all nine languages we study. 2013 ), while gpt3 model is prompted in a zero-shot fashion. when input lang. column is "en", we use google translate api to translate the target language into english. we randomly split data from @xcite to yield 1,926 training and 251 evaluation examples in english. with this training dataset, we fine-tuned each model for five epochs with a batch size of 32 and an learning rate of 5e-6 on a quadro rtx 6000 machine. we use the large variants for both models. at inference time, we translate the target language requests into english using google translate api (optional for xlmr model, necessary for roberta model). we use one large-scale language model, gpt3 @xcite davinci-002, in a zeroshot prompting setup with the following prompt: is this request polite? <input @xmath0, we compute the probabilities for two options for next token -"yes" and "no" respectively, which map to "polite" and "impolite" labels respectively. designing prompts for each language is non-trivial, so in this initial study we use this exact same english template for all languages. results in terms of xlmr model, the results were fairly split on whether it is better to use automatically translated english input, matching the training data, or using the target language input as is. using the text in english showed better performances in and four (hindi, spanish, french, russian) and using the target language input was better in five languages (korean, tamil, vietnamese, afrikaans, and hungarian). using the target language yields a slightly better performance, questioning whether automatic translation maintain the politeness level. large-scale language model, gpt3, even used in a zero-shot fashion without much prompt engineering @xcite shows competitive performances, significantly outperforming the majority baseline. similar to xlmr, using target language as is showed better performance than using translated text (70.6 vs. 66.8) on average, and in seven out of nine languages. comparing performances across languages is tricky as the annotation was done by different sets of annotators on different items for each language. to put these numbers in context, we provide a comparison between estimated human performance and model performance in the next section. would human agreement be lower on languages with weaker model performance? comparison with human agreement to compute a comparable number between the annotators and models, we use our original 3-way annotated data before aggregating politeness score. we treat one annotator's label as the human prediction and consider the other two as references, taking their mean to get the gold politeness score. we repeat this random sampling process for each example in test set 1,000 times and plot the distribution of accuracy scores in figure 3 . annotators shows varying degree of agreements -we notice a particularly stronger agreement in korean and hungarian, but overall we observe strong agreement, hovering around 90%. interestingly, models significantly underperform in these languages with high human agreement, making the gap between human and model performance large. six out of the nine languages have a gap of at least 10%, and two of them being greater than 15%. the original english study presents a list of politeness strategies along with each strategies relation to assigned politeness score. they found many statistically significant correlations between politeness strategies and human perception, such as words belonging to gratitude lexicon (appreciate), counterfactual modal (could/would) correlates with being polite, and starting the sentence with first person pronoun correlating with being impolite. developing such a politeness lexicon for each language requires expert annotation, which can be infeasible for low-resource languages with a fewer language-proficient researchers @xcite . thus, we aim to automatically generate politeness strategies for other languages from the english ones. for this initial study, we focus on lexicon-based strategies (15 out of 20 strategies), excluding strategies involving dependency parsing. ## transfer between formality and politeness while we are not aware of computational linguistic studies in politeness covering multiple languages, prior work @xcite has explored formality in four languages (english, french, italian and portuguese). in this section, we study the connections between the formality and politeness. would formally written sentences perceived as more polite by our classifier? we use gyafc @xcite ) and x-formal @xcite , two datasets containing informal sentences from the l6 yahoo answers corpus foot_9 and four formal rewrites for each sentence (dataset statistics can be found in the appendix g). in does this mean formality and politeness are not linked? upon inspection (see in their original work, rao and tetreault (2018) report that commonly used techniques to make sentences formal include phrasal paraphrases, punctuation changes, expansions, contractions, capitalization and normalization which are fairly stylistic. would such rewriting make sentences to be perceived more polite? we investigate this by further looking into (informal, formal) sentence pairs -for each version of the sentence in the pair, we compute their politeness probability (as assigned by the classifier) and report percentage of pairs where formal version of the same sentence were viewed as more polite than its impolite counterpart. the right column in multilingual models recent progresses in pretrained language models have brought better representation for multitude of languages. multilin-gual language models like mbert @xcite , xlmr @xcite , based on the transformer architecture, are pretrained with the masked language modeling objective on a large amount of corpora @xcite spanning over 100 languages. while the community also recognizes the varying quality of unlabeled data in a range languages @xcite , such multilingual models provide improved representations for modeling low resource languages. when finetuned on downstream task data in a single language, these models make reasonable predictions in multiple languages @xcite . multilingual models have also been evaluated in a prompting setup for different tasks like machine translation @xcite and different multilingual nlu tasks @xcite @xcite . multilingual benchmarks despite recent progresses in nlp resources and benchmarks, partially powered by affordable crowdsourcing @xcite , linguistic resources in low resource languages are still severely limited to compared to resources in english @xcite . many existing datasets are translated from english data @xcite . while translating approach for dataset construction have advantage of ensuring similar data distribution across languages, data collected in such fashion will not reflect the language usages of diverse population, introducing translationese which can be different from purely native text @xcite . we provide resources for nine typologically diverse languages, capturing a subtle phenomena of politeness. ## conclusion we present tydip, a corpus of requests paired with its perceived politeness score spanning nine languages. we evaluate multiple multilingual models in zero-shot politeness prediction and find that they are able to perform well without being trained on data from the same language, while not reaching human-level performances yet.