Predicting and Explaining F rench Grammatical Gender (2021)

Abstract: Grammatical gender may be determined by semantics, orthography, phonology, or could even be arbitrary. Identifying patterns in the factors that govern noun genders can be useful for language learners, and for understanding innate linguistic sources of gender bias. Traditional manual rule-based approaches may be substituted by more accurate and scalable but harder-to-interpret computational approaches for predicting gender from typological information. In this work, we propose interpretable gender classification models for French, which obtain the best of both worlds. We present high accuracy neural approaches which are augmented by a novel global surrogate based approach for explaining predictions. We introduce ‘auxiliary attributes’ to provide tunable explanation complexity.

## introduction grammatical gender is a categorization of nouns in certain languages which forms a basis for agreement with related words in sentences, and plays an important role in disambiguation and correct usage @xcite ). an estimated third of the current world population are native speakers of gendered languages, and over one-sixth are l2 speakers. having a gender assigned to nouns can potentially affect how the speakers think about the world @xcite . a systematic study of rules governing these assignments can point to the origin of and potentially help mitigate gender biases, and improve gender-based inclusivity @xcite . grammatical gender (hereon referred to by gender) need not coincide with "natural gender", which can make language acquisition more challenging. for example, irish cailín (meaning "girl") is assigned a masculine gender. works investigating the role of gender in acquiring a new language * equal contribution @xcite have found that the speakers of a language with grammatical gender have an advantage when acquiring a new gendered language. automated generation of simple rules for assigning gender can be helpful for l2 learners, especially when l1 is genderless. tools for understanding predictions of statistical models, for example variable importance analysis of @xcite , have been used even before the widespread use of black-box neural models. recently the interest in such tools, reformulated as explainability in the neural context @xcite , has surged, with a corresponding development of a suite of solutions @xcite @xcite . these approaches typically explain the model prediction by attributing it to relevant bits in the input encoding. while faithful to the black box model's "decision making", the explanations obtained may not be readily intuited by human users. surrogate models, which globally approximate the model predictions by a more interpretable model, or obtain prediction-specific explanations by perturbing the input in domainspecific ways, have been introduced to remedy this problem @xcite . we consider a novel surrogate approach to explainability, where we map the feature embedding learned by the black box models to an auxiliary space of explanations. we contend that the best way to arrive at a decision (prediction) may not necessarily be the best way to explain it. while prior work is largely limited to the input encodings, by designing a set of auxiliary attributes we can provide explanations at desired levels of complexity, which could (for example) be made to suit the language learner's ability in our motivating setting. our techniques overcome issues in prior art in our setting and are completely language-independent, with potential for use in broader natural language processing and other deep learning explanations. for illustration, we examine french in detail where the explanations require both meaning and form. ## related work we consider the problem of obtaining rules for assigning grammatical gender, which has been extensively studied in the linguistic context @xcite @xcite @xcite , but these studies are often limited to identifying semantic or morpho-phonological rules specific to languages and language families. in computational linguistics, prediction models have been discussed in contextual settings @xcite and the role of semantics has been discussed @xcite . @xcite use information-theoretic tools to quantify the strength of the relationships between declension class, grammatical gender, distributional semantics, and orthography for czech and german nouns. classification of gender using data mining approaches has been studied for konkani @xcite . in this work we look at explainable prediction using neural models. the noun gender can be predicted better by considering the word form @xcite . rule-based gender assignment in french has been extensively studied based on both morphonological endings @xcite and semantic patterns @xcite . these studies carefully form rules that govern the gender, argue merits and demerits that often involve factors beyond what rules concisely explain the patterns. further they are organized as tedious lists of dozens of rules, and evaluated only manually on smaller corpora (less than 8% the size of our dataset). @xcite show that it is possible to learn the gender by using a small set of annotated words, with their proposed algorithm combining both contextual and morphological models. the encoding of grammatical gender in contextual word embeddings has been explored for some languages in @xcite . they find that adding more context to the contextualized word embeddings of a word is detrimental to the gender classifier's performance. moreover these embeddings often learn gender from contextual agreement, like associated articles, which are not suitable for explanation @xcite . in contrast, here we will study the role of semantics in gender determination by learning an encoding of the lexical definition of the word, along with the role of form. in modern applications of machine learning, it is often desirable to augment the model predictions with faithful (accurately capturing the model) and interpretable (easily understood by humans) explanations of "why" an algorithm is making a certain prediction @xcite . this is typically formulated as an attribution problem, that is one of identifying properties of the input used in a given prediction, and has been studied in the context of deep neural feedforward and recurrent networks @xcite . the attributes are usually just input features (encoding) used in training. by studying how these features, or perturbations thereof, propagate through a network, one obtains faithful explanations which may not necessarily be easy to interpret. in this work, we consider explanations obtained using auxiliary attributes which are not used in training, but correspond to a simpler and more intuitive space of interpretations. we learn a mapping of feature embedding (learned by the black-box neural model) to this space, to approximate faithfulness, at the profit of better explanations. a similar local surrogate based approach is considered by @xcite , but it involves domain-specific input perturbations (e.g. deleting words in text, or pixels in image inputs) for explanation. ## dataset we extract french words, their definitions and phonetic representations from dbnary @xcite , a wiktionary-based multilingual lexical database. the words are filtered so that only nouns tagged with a unique gender are retained (for example voile which has senses with both genders is removed). for words with multiple definitions but the same gender, we retain the one that appears first as the semantic feature. we retrieve 124803 words, which are split 90-10-10 into train, validation and test sets respectively. the class distribution of the resulting dataset is not skewed, with 58% masculine and 42% feminine words. ## results and discussion the best orthographic model achieves an accuracy of 92.5%, whereas the semantic model alone achieves only 77.23%. combining the features from the two models leads to a gain in the accuracy of the classifier, to 94.01%. we can conclude that for french, the gender can be predicted robustly by the word orthography, but adding semantic information can further improve prediction. adding phonology to the mix does not seem to help much. this may be attributed to the fact that phonological forms contain less information than the orthographical forms in french, e.g. lit /li/ (bed, m.) and lie /li/ (dregs, f.). not only are the written forms phonetic here (i.e. pronunciation is typically unambiguous given spelling) but they often contain additional (e.g. etymological) information which may be missing in the spoken forms. a more detailed error analysis and comparison of model pairs is presented in appendix a. we define a 'good explanation' to be one with high model fidelity (measured by f1) and if it involves fewer rules (more easily interpretable). this can be quantified in the case of decision trees as the length of path from root to leaf node, when making a prediction. a class with higher average decision tree path length for its sample is less interpretable. we observe the trade-off between achieving interpretability and model accuracy for masculine and feminine classes (figure 1 ) and for clusters generated via embeddings (figure 2 ). the clusters are generated so that within a gender class, a distinction could be made for nouns that could have different rules, so that easier explanations per class could be generated. both figures 1 and 2 show that increasing size of the tree, always increases f1 score, but that comes at the cost of interpretability due to higher number of decision rules. some example features that distinguish the different clusters are noted in appendix b. we see in figure 1 that the explainability is higher for feminine nouns than masculine. this is consistent with the fact that there are many rules to indicate the feminine gender (such as words ending in -ine, -elle, -esse), whereas masculine gender is a default category leading to more complex, and harder to explain rules. ## conclusion orthography predicts the grammatical gender in french with high accuracy, and adding semantic features can improve this prediction. the blackbox embedding can be explained by simpler decision tree models over a given auxiliary explanation space, both locally and globally. global explanations lead to fewer rules across examples but are more complex on individual instances. explainable gender prediction can be useful to language learners and gender bias researchers. a cross-linguistic extension of our study is deferred to future work.