In What Languages are Generative Language Models the Most Formal? Analyzing Formality Distribution across Languages (2023)

Abstract: Multilingual generative language models (LMs) are increasingly fluent in a large variety of languages. Trained on the concatenation of corpora in multiple languages, they enable powerful transfer from high-resource languages to low-resource ones. However, it is still unknown what cultural biases are induced in the predictions of these models. In this work, we focus on one language property highly influenced by culture: formality. We analyze the formality distributions of XGLM and BLOOM’s predictions, two popular generative multilingual language models, in 5 languages. We classify 1,200 generations per language as formal, informal, or incohesive and measure the impact of the prompt formality on the predictions. Overall, we observe a diversity of behaviors across the models and languages. For instance, XGLM generates informal text in Arabic and Bengali when conditioned with informal prompts, much more than BLOOM. In addition, even though both models are highly biased toward the formal style when prompted neutrally, we find that the models generate a significant amount of informal predictions even when prompted with formal text. We release with this work 6,000 annotated samples, paving the way for future work on the formality of generative multilingual LMs.

## introduction natural language processing (nlp) systems are used worldwide across multiple cultures, audiences, contexts, communication goals, demographics, and languages. thus, it is essential that these models be able to adapt to the sociocultural context of its users. as described by @xcite , linguistic style is one of the major dimensions by which cultures vary in nlp technologies. in this work, we focus on formality. formality is a stylistic property of language that can impact how we perceive a text. it typically carries information about the culture of the speaker (or writer), is constrained by the context of the message, and can impact the communicative goal of a text @xcite . generating text with a desired level of formality can be useful for different nlp applications @xcite . for example, controlling the tone of machine translation models @xcite @xcite , designing chatbots with formality awareness to respond to user-preferred conversational style @xcite , or assisting users to change the formality level of their writings @xcite @xcite . generative language models have demonstrated capabilities in producing cohesive texts and solving nlp tasks with zero/few-shot learning @xcite @xcite , even in multilingual scenarios @xcite @xcite @xcite . multilingual language models are trained with large amounts of text from different sources. that training process could make the model biased towards a certain level of formality because of the data of each language as well as cross-lingual transfer @xcite @xcite , limiting the capabilities of the model to adapt to the different cultures of an nlp application. this work analyzes the formality level of two multilingual language models: xglm @xcite and bloom @xcite , across five languages, namely arabic, bengali, english, french, and spanish. to do so, a native/proficient speaker of each language evaluates the generation outputs of each model into three categories: formal, informal, and incohesive. this evaluation allows us to analyze the generations across three different dimensions: the cohesiveness of the generations, foot_0 the formality bias given neutral prompts, and the formality preservation given formal/informal prompts. as an example, we show in ## formality across different languages we start by defining formality in the five languages of our study. ## experiments we evaluate different dimensions of formality of the generations of two popular generative multilingual language models: xglm (lin et al., 2021b) and bloom @xcite , in five languages: arabic, bengali, english, spanish, and french. we hypothesize that the influence of high-resource languages in the corpus can involve biases in the formality of the whole models. to see their behavior in different scenarios, we employ distinct variations of prompt lengths and formality. in addition, we tweak some parameters when generating to avoid incohesive outputs. xglm (lin et al., 2021b) is a multilingual language model trained with 500 billion tokens belonging to 30 languages of common crawl. xglm has five sizes ranging from 564 million to 7.5 billion parameters. bloom (scao et al., 2022) is also a multilingual generative language model trained on around 341 billion tokens from a corpus of 59 languages (13 of them are programming ones) to democratize huge pre-trained language models. bloom was released in different sizes ranging from 560 million to 176 billion parameters. xlgm and bloom are decoder-only transformers pre-trained on a similar set of languages with a comparable amount of data. we compare checkpoints of similar size (i.e. we compare xglm 2.9b with bloom 3b and xglm 7.5b and bloom 7.1b foot_2 ). regarding the proportion and data sources on which both models were trained, bloom was trained on a more varied set of domains than xglm in spite of the xglm corpus being larger. in addition, the bloom corpus has a more balanced distribution of the amount of data of the languages evaluated in this study. more details about the quantity and sources of both models can be found in appendix c. ## results & analysis we start by analyzing the cohesiveness of each model. we then focus on the cohesive text for analyzing formality. ## discussion formality bias when present in multilingual models, which are increasingly popular nowadays, can lead to undesirable outcomes. for example, using "please" is common among north american english native speakers in requests, even among close friends, while in arabic, it could be considered awkward, if not rude, in conversations among close friends @xcite . a usage example of language models is solving downstream tasks using prompting techniques for zero-shot learning, such as @xcite 's work on questionanswering. prompting has also been used with large language models for conversational chatbots such as chatgpt @xcite . as prompting is becoming popular, we must understand that prompting a model that exhibits formality bias could be a barrier to getting the expected output. furthermore, depending on the application, formality bias could even lead to sometimes unwanted misunderstandings @xcite and conflicts if the models, for example, are not able to generate text in the formality style of the users' expectations. controlling llms generations has been taken into consideration in recent work, such as in @xcite , where they fine-tune a language model @xcite intending to align the model with the intent of the users using reinforcement learning from human feedback (rlhf) @xcite . future work could analyze the impact of rlhf on the formality distributions present in language models. furthermore, our work focused only on two pre-trained models with up to 7b parameters. the same analysis could be conducted for larger models such as gpt-3 and bloom(175b). finally, the increase in the number of multilingual language models calls for more work on their biases. ## conclusion in conclusion, we analyzed the formality level of the generations of two large-scale generative language models, xglm and bloom, ranging from 2b parameters to 7b parameters. we first observed the cohesiveness of the predictions. we found that bloom(7.1b) predicts significantly more cohesive text than xglm(7.5b) for english, french, and spanish. second, we showed that, across all five languages, both models tend to generate formal text when prompted neutrally. finally, we found that the formality of the prompt highly impacts both models. in most cases, they generate the same style as the prompt, with slight differences between the models depending on the language. our analysis is based on the annotations of 1,200 generations in arabic, bengali, english, french, and spanish. we release them with this paper opening future avenues for modeling the formality of generative multilingual language models.