Universal Patterns of Grammatical Gender in Multilingual Large Language Models (2025)

Abstract: Grammatical gender is a fundamental linguistic feature that varies across languages, and its cross-linguistic correspondence has been a central question in disciplines such as cognitive science and linguistic typology. This study takes an information-theoretic approach to investigate the extent to which variational usable information about grammatical gender encoded by a large language model generalizes across languages belonging to different language families. Using mBERT as a case study, we analyze how grammatical gender is encoded and transferred across languages based on the usable information of the intermediate representations. The empirical results provide evidence that gender mechanisms are driven by abstract semantic features largely shared across languages, and that the information becomes more accessible at the higher layers of the language model.

## introduction grammatical gender is a nominal category (e.g., masculine, feminine, and neuter) that continues to challenge linguists due to the complexity of gender systems across languages and the rules governing its assignment to nouns @xcite . these rules vary cross-linguistically and cannot always be inferred from a noun's surface form. for instance, the german das mädchen 'the neut girl' is grammatically neuter despite denoting a female entity, and common concepts such as sun differ in gender across languages, masculine in french (le soleil) but feminine in german (die sonne). in addition to its linguistic implications, the study of grammatical gender provides insights into cognitive science @xcite @xcite , assists second language learners in navigating the seemingly arbitrary rules of gender assignment @xcite , and helps reducing gender bias in language models @xcite . examining grammatical gender from a typological perspective can further illuminate shared linguistic principles contributing to the assignment of grammatical gender across languages. recent studies in computational linguistics provide clues, based on static multilingual embeddings, about the existence of universal patterns in the assignment of grammatical genders, transferable across several languages @xcite . however, the linguistic depth and extent of the universal patterns of grammatical gender have remained unexplored, primarily because the multilingual word embeddings do not provide a clear mechanism in distinguishing between formal and semantic features. in particular, it is still unclear whether the linguistic patterns that drive such universalities emerge at the morphological or semantic levels and how the gender system across languages might be related at these levels @xcite . on the other hand, previous studies have shown large language models (llms) normally encode linguistic information in a more transparent and structured way, allowing for an access into distinct linguistic levels @xcite @xcite @xcite . lower layers primarily encode surface-level and morphological features, middle layers capture syntactic structure, and higher layers represent semantic properties and more abstract linguistic features @xcite . additionally, later studies show that multilingual llms are capable at capturing the universal aspects of languages at their intermediate representations @xcite , including the grammatical abstractions such as gender @xcite . building upon these studies, we employ mbert @xcite to investigate the universal and language-specific aspects of grammatical gender across different linguistic levels, such as morphology and semantics. mbert, a multilingual encoder-only language model trained on a diverse set of languages, provides a structured distribution of linguistic information across its intermediate representations. this allows us to systematically examine grammatical gender at multiple linguistic levels. moreover, its shared feature space across languages facilitates universal analyses. taking an information theoretic strategy, we investigate universal aspects of grammatical gender based on the amount of information transferable (generalizable) across gender systems of languages. specifically, we extend the concept of variationalusable (v-usable) information @xcite to measure the extent to which the gender information from a source language is generalizable to a target language. a high amount of generalizable information is interpreted as evidence of structural similarities between the gender systems of the source and target languages. in addition to the cross-lingual analysis, the application of v-usable information is also motivated as it allows us to effectively measure the intra-lingual complexity of gender systems. our experiments on a typologically diverse set of languages provide empirical evidence that linguistic information about gender is largely generalizable across languages with similar gender categories, while their genealogical relationship plays a secondary role. additionally, we show that linguistically driven complexities of gender systems are reflected in the hidden representations of the language model, leading to variations in usable information in our intra-lingual analysis. furthermore, our layer-wise analysis of usable information highlights the varying contributions of intermediate representations to gender encoding, both within and across languages. finally, further examination of intermediate representations confirms the role of both morphology and semantics in gender representation, with semantic aspects proving to be more generalizable across languages. overall, this study adopts a computational approach to explore the relationships between different systems of grammatical gender based on their encoding in the intermediate representations of a large language model. specifically, the contributions of this study include: ## grammatical gender grammatical gender is an abstract system of noun classification found in many languages, often overlapping with, or considered as subset of, noun class systems @xcite . it is generally considered an inherent property of the noun itself @xcite , with determiners, adjectives, and sometimes verbs agreeing with the noun in gender. although grammatical gender is frequently correlated with biological sex, it is distinct from it, as evidenced by instances of gender-sex mismatches-for example, in german, das mädchen ('the neut girl') is grammatically neuter despite referring to a female entity. furthermore, grammatical gender should not be conflated with nominal declension classes @xcite . common gender categories include masculine, feminine, neuter, and common, with indo-european languages typically featuring masculine/feminine/neuter (e.g., german, russian), neuter/common (e.g., danish, dutch), and masculine/feminine (e.g. french, italian) gender systems. in contrast, two-gender systems are common in afro-asiatic languages @xcite . the function of grammatical gender is still debated: some researchers suggest it aids in referent identification or categorization for cognitive processes such as storage and retrieval @xcite @xcite , while others dismiss it as 'historical junk' @xcite . ## method our investigation of gender transfer spans both layers of a language model and languages. specifically, we assess gender transferability across languages by measuring the information each intermediate layer provides for gender prediction. for this purpose, we adopt v-usable information @xcite , an extension of shannon mutual information @xcite that accounts for computational constraints. the v-usable information in a random variable x for predicting a category y is defined as the difference in conditional entropy between predictions based on x and a baseline prediction with no input features (denoted as φ): a higher value of i v (y ; x) indicates that x significantly reduces uncertainty in predicting y . in our setting, x is a random vector in an embedding space formed by a hidden layer of a language model while processing nouns in a given language, and y represents a probability vector of grammatical genders. to quantify the amount of information encoded in the embedding space of a source language i, denoted as x i , for predicting grammatical genders in a target language j, denoted as y j , we extend equation 1 as: intuitively, i v (y j ; x i ) measures the usable information that the embeddings from the source language provide for predicting gender in the target language. a high value of i v (y j ; x i ) suggests a strong similarity between the gender systems of the source and target languages. in some cases that the gender systems are highly different from each other, for example when a gender category is seen in a language but not in the other (e.g., common is in danish but not in arabic), i v (y j ; x i ) can be negative or an invalid number. we set the negative values and invalid numbers to zero to satisfy the non-negativity constraint of usable information and manage the nan exceptions. for a given language pair and hidden layer, we calculate the marginal entropy h v (y j | φ) in equation 2 based on the gender distribution of the target language and approximate the conditional entropy h v (y j | x i ) using a light classifier trained on embedding-gender pairs from the source language i. the cross-entropy loss on a test sample from the target language j is then used as an estimate of to address class imbalance, crossentropy loss is weighted by the gender distribution in the source language. ## experiment setup we investigate transfer learning of grammatical gender across a typologically diverse set of languages with different grammatical gender systems and minimal lexical similarity, as outlined in following @xcite , we extract embeddings from the formal representations of nouns provided in the form column of the universal dependencies treebanks. in cases where tokens are divided into subtokens, we average their embeddings. since the experiments are based on cross-lingual transfer, morphosyntactic gender indicators from the source language are absent in the target language input. this design helps ensure that model performance reflects genuine cross-linguistic generalization rather than reliance on surface-level lexical cues in the target language. for each sentence in a language, we construct a dictionary that maps the contextual embeddings of its nouns to their grammatical gender. the embeddings are extracted from all layers of mbert, resulting in an embedding matrix of 768 × 13 for each occurrence of a noun in a language. a known limitation of using mbert in multilingual settings is that model performance can vary across languages due to differences in their representation in the pretraining data @xcite . to mitigate this imbalance and ensure crosslinguistic comparability, we downsample the total number of nouns per language to match the smallest sample size in arabic (5,151 nouns). for beja, a total of 555 nouns are included. we estimate the usable information based on 7 × 13 logistic regression models, corresponding to the number of languages and hidden layers. each classifier is trained for 30 epochs with early stopping (patience of 5 epochs). we use adamw as the optimizer with a manually tuned learning rate of 5×10 -5 , and a learning rate scheduler that reduces the learning rate every 10 epochs by a factor of 0.1. the train-validation-test split for all languages is 80-10-10. training is performed on the training (80%) and validation (10%) splits, while crosslingual entropy is estimated using the test split (10%). for each source-target language pair, 2 is estimated as the average cross-entropy loss over five random seeds. ## results in this section, we present and analyze the results of our experiments through 1) intra-lingual analysis of the usable information, 2) their transferability across languages, and 3) their variations across layers and languages. ## conclusions this study explores the cross-linguistic transferability of grammatical gender in multilingual language models, focusing on the extent to which gender information generalizes across languages with different gender systems. using variational-usable (v-usable) information, we quantify how grammatical gender is encoded within and across languages in mbert. our findings reveal that gender information is more transferable between languages that share similar gender categories, whereas genealogical relationships play a secondary role. through intra-lingual analysis, we demonstrate that the complexity of a language's gender system is reflected in the amount of usable information available in the intermediate representation of mbert. our cross-lingual results highlight that languages with two-gender systems, such as arabic and italian, exhibit the highest transferability, particularly to languages with similar gender distinctions. in contrast, languages with more complex gender systems, such as german and russian, show reduced transfer due to the added complexity of declension systems and irregularities in gender assignment. a layer-wise analysis further reveals that intermediate representations in mbert play a critical role in encoding gender information. gender distinctions are captured more effectively in the middle-to-upper layers, supporting the idea that semantic information is more generalizable across languages than purely morphological features. overall, our findings contribute to a deeper understanding of how grammatical gender is represented in multilingual llms and offer insights into universal aspects of grammatical gender. future research could extend this analysis to other multilingual models (e.g., mgpt, bloom) and investigate additional factors influencing gender transfer, such as word frequency effects, training data composition, and finer-grained linguistic features. expanding the study to a broader range of languages beyond indo-european and afro-asiatic families would further enhance our understanding of crosslinguistic gender representation.