Comparing Styles across Languages (2023)

Abstract: Understanding how styles differ across languages is advantageous for training both humans and computers to generate culturally appropriate text. We introduce an explanation framework to extract stylistic differences from multilingual LMs and compare styles across languages. Our framework (1) generates comprehensive style lexica in any language and (2) consolidates feature importances from LMs into comparable lexical categories. We apply this framework to compare politeness, creating the first holistic multilingual politeness dataset and exploring how politeness varies across four languages. Our approach enables an effective evaluation of how distinct linguistic categories contribute to stylistic variations and provides interpretable insights into how people communicate differently around the world.

## introduction communication practices vary across cultures. inherent differences in how people think and behave @xcite influence cultural norms, which in turn, have a significant impact on communication @xcite . one key way that cultural variation influences communication is through linguistic style. in this work, we introduce an explanation framework to extract stylistic differences from multilingual language models (lms), enabling cross-cultural style comparison. style is a complex and nuanced construction. in communication, style is heavily used to convey certain personal or social goals depending on the speakers' culture @xcite . for example, cultures that are high-context 1 tend to use more indirect language than those that are lowcontext @xcite , and cultures that have a high power-distance 2 tend to have more formal 1 conversations in high-context cultures have a lot of subtlety and require more collective understanding. @xcite 2 power distance refers to the strength of a society's social hierarchy. @xcite thanks, but do you really think spiders are insects? please stop editing as soon as possible, i'd really appreciate it. interactions in a workplace setting. @xcite . for example, consider the two conversation snippets in figure 1 . though these two utterances are direct translations of each other, a multilingual lm fine-tuned for politeness classification outputs different labels, with the chinese utterance labeled as impolite. we ask a bilingual english/chinese speaker to provide further insights, and she observes that the same request to "stop editing as soon as possible" sounds more aggressive @xcite in chinese than in english, as cultural norms in china do not typically condone giving such harsh, direct instructions to a stranger. stylistically appropriate language is crucial to successful communication within a certain culture. however, multilingual lms sometimes struggle to en please es por favor zh 请 ja お願い します token-level importances (1) multilingual lexica creation (2) feature set aggregation en ja es zh languages lexical categories en style lexica shap multilingual style lms en hi .14 yes .25 ... ja やあ .46 はい .19 ... zh 你好.23 是 -.18 ... es hola .21 sí -.06 ... ja/es/zh style lexica ## a framework for multilingual style comparison in this section, we detail our two-part framework for multilingual style comparison. (1) multilingual lexica creation takes a curated style lexica and uses embedding-based methods to refine lexica translation into another language, creating a set of parallel lexical categories. (2) feature set aggregation maps extracted feature importances from trained lms to these parallel lexical categories across languages. this framework helps us to interpret what multilingual lms learn about style during training, and allows us to meaningfully compare how style is expressed across languages. ## a holistic politeness dataset politeness, like all linguistic styles, is a complex, nuanced construction. in order to compare how politeness differs across languages, it is necessary to analyze the full distribution of conversational data, without any oversimplifying assumptions. we follow the process of the stanford politeness corpus @xcite and tydip @xcite when creating and evaluating our politeness dataset, but with three key differences: ## comparing politeness via politelex politelex @xcite consists of curated lexica to measure politeness in english and chinese, based on the twenty politeness strategies introduced by @xcite . we use mlc to expand politelex to cover all four of our languages. chinese politelex has some additional lexical categories, such as taboo words and honorifics, to account for cultural and linguistic norms in chinese that do not have an english equivalent. as politeness is expressed more similarly within eastern and western cultures than between them @xcite , we use english politelex as the seed for spanish and chinese politelex as we additionally show the frequency of each category (% of total sentences that contain a word from the category.) the seed for japanese, to create a set of four lexica with parallel, comparable categories. when purifying our expanded lexica in spanish and japanese, we use the full set of 41,000 scraped talk pages to calculate internal correlation and remove uncorrelated words from each category. next, we fine-tune xlm-roberta models @xcite on our holistic politeness dataset (see appendix a.4 for training details) and use the shap library (lundberg and lee, 2017) to extract shapley values for each utterance. finally, we apply feature set aggregation to calculate importance scores for each politelex category. ## comparing politeness via dialogue acts given our dataset is the first politeness dataset to include multiple types of sentences (i.e. dialogue acts), we additionally apply the second part of our framework to dialogue act groupings as categories. in the previous section, we compared how linguistic expressions of politeness differ across languages. in this section, we seek to compare how the linguistic form of politeness differs as well. to classify the dialogue acts of each utterance, we machine translate our dataset to english. we then run a trained english dialogue act classifier provided by @xcite on the translated dataset and label each sentence of an utterance with one of 42 dialogue acts @xcite . as dialogue acts are sentence-level, we modify equation ( 1 ) to aggregate over all tokens in a given sentence, as opposed to all tokens in a given word. finally, we treat each dialogue act as a unique category (analogous to a lexical category) and use our feature set aggregation method to map sentencelevel shap values to their corresponding dialogue acts across our four languages. results. figure 5 shows the average importance of each dialogue act, with the frequency of each dialogue act given in parentheses. once again, we observe some similarities across languages: conventional openings, conventional closings, and sentences of thanks are strong indicators of politeness across languages. however, statements appear to have differing roles across languages. declarative statements dialogue act example conventional opening "hi all, this article is in desperate need of attention." declarative statement "as it stands, we do not know the place or date of shapiro's birth." opinion statement to an english speaker, both the english sentence and the chinese translation appear to be similar levels of politeness. however, american annotators label the english question as "neutral" while chinese annotators label the chinese question as "slightly rude," highlighting the ways in which cultural norms influence perceptions of politeness. interestingly, we do not observe any major differences in the frequency of dialogue acts across languages; conversations in all four languages appear to have a similar distribution of dialogue acts, though the average politeness of each dialogue act often varies based on language. results for all dialogue acts are shown in figure a8 . ## ablation analysis the politelex category-level importances in figure 4 and dialogue act importances in figure 5 are dependent on the shapley values extracted from fine-tuned xlm-roberta models. in this section, we analyze the effect of using alternate models and training paradigms. effect of model size. to investigate the role of lm size and architecture, we fine-tune llama-2-7b @xcite to analyze politeness. comparing the results from llama-2-7b to the results from xlm-roberta, we notice stability in the direction of importance score (i.e. positive, negative, and neutral lexical categories/dialog acts are stable across both lms). interestingly, we observe differences in the magnitude of importance score (e.g. llama-2-7b sees the "greeting" category in chinese to be a larger indicator of politeness than xlm-roberta does). ## related work multilingual style. previous work on multilingual style predominantly focuses on training lms to perform cross-lingual and multilingual style classification and style transfer. key styles studied include formality @xcite @xcite and emotion @xcite @xcite , with another body of work focusing on style-aware multilingual generation with any subset of chosen styles @xcite . explaining style. one line of work builds on existing techniques @xcite to explain style within a single lm @xcite . another line of work interprets style lms by comparing learned features to those humans would consider important @xcite , mapping feature attributions to topics @xcite , and training models to output relevant features alongside their predictions @xcite . politeness. @xcite presents one of the earliest quantitative analyses of linguistic politeness, with @xcite following in a multilingual setting. other computational work focusing on politeness uses lms to generate or modify text with a specified politeness level @xcite @xcite . previous work focused on multilingual style has little emphasis on investigating how style differs amongst languages. additionally, most work on explaining style lms is english-scoped, and thus, developed methods do not easily allow for cultural comparison. we are the first to present a method to compare styles across languages and provide quantitative, human-interpretable insights into how communication differs globally. ## conclusion in this work, we present a framework to extract the knowledge implicit in trained multilingual lms. this knowledge enables comparison of styles across languages via a human-interpretable common language for explanation. we also provide the first holistic multilingual politeness dataset, which we hope will encourage future research exploring cultural differences in style. our framework provides insights into how style is expressed differently across languages. these insights can improve multilingual lms -understanding how and why an lm generation is not stylistically appropriate can inform culturallyadaptable models. these insights can also help people learning a second language become more aware of the language's culturally-informed stylistic nuances. at a higher level, our framework provides a general methodology for explaining lms; using feature attributions such as shapley values to provide explanations in terms of human-interpretable categories (e.g. lexica and dialogue acts) gives explanations that are both grounded in the model and useful to humans.