Disambiguating Grammatical Number and Gender With BERT (2021)

Abstract: Accurately dealing with any type of ambiguity is a major task in Natural Language Processing, with great advances recently reached due to the development of context dependent language models and the use of word or sentence embeddings. In this context, our work aimed at determining how the popular language representation model BERT handle ambiguity of nouns in grammatical number and gender in different languages. We show that models trained on one specific language achieve better results for the disambiguation process than multilingual models. Also, ambiguity is generally better dealt with in grammatical number than it is in grammatical gender, reaching greater distance values from one to another in direct comparisons of individual senses. The overall results show also that the amount of data needed for training monolingual models as well as application should not be underestimated.

## introduction a challenge in natural language processing (nlp) resides in the accurate automatic sense disambiguation of words and phrases. an often cited example of ambiguity is the one between bank ("an institution where one can place and borrow money and take care of financial affairs.") and bank ("an edge of river, lake, or other watercourse.") in english. 1 while a person is able to get the right meaning of the word from context, the same skill is now expected from contextual language models. the goal of our work, however, is to find out if and how well contextual meaning representations can handle sense ambiguities in grammatical gender and number. further, this paper aims to show differences in disambiguation in between different 1 definitions taken from wiktionary: https://en. wiktionary.org/wiki/bank languages, ambiguity types, and pre-existing models. for these tasks, german and spanish are used for gender and number ambiguities, and english for number ambiguities only (see section 3). ambiguity in grammatical number is similar to the example mentioned above, as the plural form of a word might also mean something different from the mere quantity. in this case, the language model (lm) needs to disambiguate the meaning of plurals considering only the current context. for grammatical gender ambiguity -that is, words that can occur in more than one gender, with their meaning dependent on that factor -additional cues to the current meaning can be found in words that are connected to the currently observed ambiguous word, such as accordingly gendered determina or adjectives. this specific skill of disambiguation of lms is tested on bert word embeddings (see section 4.2), based on data of three different languages: german (section 3.1) and spanish (section 3.2), which contain both types of ambiguity, and english (section 3.3), which contains only ambiguity in grammatical number, but has been included due to the large amount of available data for testing (see section 4.1). ## related work while existing work on ambiguity mostly focuses on a more general definition of the term, grammatical gender and number have mostly been left aside. however, there has been research on gender bias in language models. @xcite found a gender-bias for english in bert similar to the gender-bias occurring with names of professions. for german, this effect has been found to be less strong, due to morphological gender markings. seeing how a language without grammatical gender shows a higher gender bias than a language with such, this topic appears to be rather drifting in a direction of biological gender and social prejudice connected with gender, which, according to @xcite , can be solved by grammatical gender in form of morphological markings, as this occurs in german. these results show a possibility to search for a gender bias in the disambiguation process of bert for languages such as german or spanish. concerning ambiguity in grammatical number, @xcite analyses how plural entries included in princeton wordnet (princeton university, 2010) are encoded in word embeddings using word2vec @xcite and glove @xcite , thereby showing the convergence between distributed representations of plurals bearing a specific sense, as well as their symbolic representation proposed in wordnet. they bring up examples like people -peoples: both words occur in plural form. however, people can be treated as a singular term, meaning 'the body of citizens of a state or country' (definition given by princeton wordnet (princeton university, 2010)). peoples can be used as the plural of people, but can also refer to 'the human beings of a particular nation or community or ethnic group' (definition given by princeton wordnet (princeton university, 2010)). this shows that: considering sense already during training, instead of researching on existing models, has been shown to be a promising approach (see @xcite , @xcite al. (2019) ) for english data. these approaches add information on senses or glosses to the training process, thereby considering this additional knowledge in all applicational usage. morphological approaches have also shown promising results in english, german @xcite and hebrew @xcite . future research in that direction will show if these approaches can improve disambiguation of those ambiguities that contain relevant morphological cues (see @xcite ). observations of morphological content of sentences, including grammatical number and gender, have shown that bert achieves better results for morphologically simple languages, in comparison to morphologically more complex languages such as german or russian (see @xcite ). this work shows that bert is able to detect and disambiguate such morphological content based on contextual cues, but does not achieve human-like results. ## ambiguity in grammatical gender and number the present work takes focus on ambiguity occurring in grammatical number and gender. ambiguity in grammatical number occurs if a word w has a plural p that has a standing meaning of its own, that is not only expressing a quantity associated with the word. an example in english is given by the word glass; its plural glasses can either mean multiple items of the object named glass, or an object that is worn on one's face to increase eyesight. while both meanings of glasses appear to be etymologically related, they do mean different things, and one meaning is independent of the meaning of the singular word glass. ambiguity in grammatical gender is given when a word w is ambiguous, while w can be assigned to at two or more different grammatical genders g and g', and the standing meaning is dependent on the assigned gender. a german example is displayed in ## methodology in the following sections, the methods used to compute and compare embedding vectors are described. section 4.1 describes the data and its necessary information. section 4.2 lists the pretrained bert models that have been used to retrieve results, which then have been evaluated using the methods described in section 4.3. the actual results and evaluation outcomes are later described in section 5. ## discussion given the results found in section 5.1, bert can disambiguate ambiguity in grammatical number in english similarly well as non-ambiguous word pairs. the same goes for ambiguity in grammatical gender in german. overall, ambiguity in grammatical number in german could also be disambiguated well, but not as well as for the same type of data in english. a reason for this outcome might be more available data to compute word embeddings from or possible differences in amounts of data per language during training of the individual models. ambiguity in grammatical number could be dismbiguated better than for grammatical gender, which is possibly due to additional morphological cues within the word, or in some cases a more notable difference within the provided context. given the results found in section 5.2, the monolingual model bert-base-uncased outperformed the multilingual model bert-base-multilingual on english data. concerning ambiguity in grammatical gender in german, both models performed similarly well on both ambiguous and non-ambiguous data. in the case of ambiguity in grammatical number in german, the monolingual model outperformed the multilingual model, achieving slightly greater results for ambiguous data, and much greater results for non-ambiguous data. overall did the multilingual model perform better on german data, especially on ambiguity in grammatical gender, while it performs similarly well on ambiguity in grammatical number for both languages. the english monolingual model achieved much larger distances overall in comparison to the german monolingual model. using monolingual models is rewarded with better disambiguation, showing that it is well worth the time invested in the creation of such models, as the multilingual counterpart does well, but by far not as good. it is, however, a great way to include minority languages with not enough data to create individual language models. for languages like english and german, however, monolingual models appear to be the way to go, at least for disambiguation tasks. ## conclusion this work shows that basic bert models for english are better adapt to specific types of ambiguity than those for other languages like german, shown by the greater distances reached in the given evaluation methods. however, the multilingual model did perform better on german data than on english data, showing that a change of focus can improve results on languages with less data, but overall does not perform as well as language specific models. this work shows a need for well-trained monolingual models, which appear to provide a better possibility of focus on disambiguation tasks. further, available data such as provided by wiktionary @xcite ) needs to increase. interesting expansions of this work could come with approaches such as sensebert @xcite ) -a bert model that is trained on wordnet (princeton university, 2010) supersenses -which might be able to achieve better results in disambiguation, and goes along with the findings of @xcite . switching from word level to a morphological level could also bring up interesting results @xcite , @xcite ).