What an Elegant Bridge: Multilingual LLM s are Biased Similarly in Different Languages (2024)

Abstract: This paper investigates biases of Large Language Models (LLMs) through the lens of grammatical gender. Drawing inspiration from seminal works in psycholinguistics, particularly the study of gender’s influence on language perception, we leverage multilingual LLMs to revisit and expand upon the foundational experiments of Boroditsky (2003). Employing LLMs as a novel method for examining psycholinguistic biases related to grammatical gender, we prompt a model to describe nouns with adjectives in various languages, focusing specifically on languages with grammatical gender. In particular, we look at adjective co-occurrences across gender and languages, and train a binary classifier to predict grammatical gender given adjectives an LLM uses to describe a noun. Surprisingly, we find that a simple classifier can not only predict noun gender above chance but also exhibit cross-language transferability. We show that while LLMs may describe words differently in different languages, they are biased similarly.

## introduction the way we perceive the world is not only affected by our culture @xcite , but also the language we speak @xcite . the relationship between cognition and language has been of interest for a long time @xcite , especially through the lens of gender @xcite . recent advances in large language models (llms), that match human performance on multiple tasks, provide an exciting opportunity to study the relationship between the psycholinguistic biases of humans and those of machines. while it is unclear whether the latter relationship exists, it would be a more scalable, affordable, and even ethical @xcite alternative to human studies. figure 1 : probing the bias of multilingual llms. we prompt a llm to describe gendered nouns using adjectives. this allows us to study psycholinguistic biases of llms. for example, if the generated adjectives are predictive of the nouns's gender, we can, by training a binary classifier, predict grammatical gender by only looking at the adjectives a llm uses to describe a word. in this work, we revisit the study of @xcite in the era of llms. to see how grammatical gender affects cognition, @xcite ask speakers of languages with grammatical gender (where nouns have assigned genders) to describe various objects, finding that the language a person speaks affects the attribution of masculine or feminine characteristics to objects. for example, a spanish speaker (where "bridge" is masculine) might describe a bridge with words like "strong" or "sturdy", while a german speaker (where "bridge" is feminine) might use terms like "elegant" or "beautiful". however, several subsequent studies fail to replicate such results @xcite @xcite , which is but a symptom of the replication crisis in psychology @xcite @xcite . similarly, studies in the field of nlp that examine the way gendered nouns are used in text corpora @xcite , find conflicting evidence on whether there is a relationship between grammatical gender and cognition. the existence of gender bias has been well stud-ied for word embeddings @xcite @xcite , as well as a range of nlp systems, such as ones for machine translation @xcite , image and video captioning @xcite , or sentiment analysis @xcite . more recently, the social biases of llms have been studied @xcite . while the multilingual capabilities of llms have been extensively evaluated, showing they perform well on machine translation @xcite @xcite as well as various multilingual benchmarks @xcite , the evaluation of biases in the multilingual setting is less mature. contrary to recent work showing that multilingual llms have different biases for different languages @xcite , we find that when it comes to gendered nouns, llms are biased in a similar way, as the biases are predictive of each other. in this paper, we loosely follow the protocol of @xcite and prompt llms to describe nouns using adjectives in different languages. specifically, we focus on open-sourced llms (llama-2 @xcite and @xcite ). we select 10 languages that have grammatical gender (e.g, german and spanish), and use the llms to describe gendered nouns using adjectives. this allows us to see how adjectives co-occur across languages. our most important findings are that (i) a simple classifier can predict the gender of a noun using the adjectives used to describe it, and (ii) such a classifier reliably transfers across languages, suggesting llms are biased similarly in different languages. ## method in this work, we are interested in the adjectives a multilingual llm uses to describe gendered nouns when asked in different languages. here, we describe how we generate such adjectives, and how we examine whether they are predictive of the grammatical gender of the nouns. ## conclusion in this work, we revisit the psycholinguistic experiments of @xcite , confirming the hypothesis of their work applies to llms, where different words are used to described masculine and feminine nouns. our most surprising finding is that we can reliably zero-shot transfer a classifier that predicts grammatical gender across languages. this shows that while llms might think differently on different languages, they are biased similarly when it comes to grammatical gender. we hope this work inspires others to explore psycholonguistic experiments applied to llms, and to drive a discussion of whether such results can be useful to inform or motivate human experiments.