This is a BERT . Now there are several of them. Can they generalize to novel words? (2020)

Abstract: Recently, large-scale pre-trained neural network models such as BERT have achieved many state-of-the-art results in natural language processing. Recent work has explored the linguistic capacities of these models. However, no work has focused on the ability of these models to generalize these capacities to novel words. This type of generalization is exhibited by humans, and is intimately related to morphology—humans are in many cases able to identify inflections of novel words in the appropriate context. This type of morphological capacity has not been previously tested in BERT models, and is important for morphologically-rich languages, which are under-studied in the literature regarding BERT’s linguistic capacities. In this work, we investigate this by considering monolingual and multilingual BERT models’ abilities to agree in number with novel plural words in English, French, German, Spanish, and Dutch. We find that many models are not able to reliably determine plurality of novel words, suggesting potential deficiencies in the morphological capacities of BERT models.

## introduction in recent years, large-scale pre-trained neural network models have transformed the landscape of natural language processing (nlp) research. this approach to nlp became prominent after several models such as bert @xcite achieved new state of the art performance on a wide range of nlp tasks such as natural language inference. the successful performance of bert and other models like it on natural language understanding tasks suggests that they may be learning valuable general linguistic competencies. however, it is not clear whether these models are able to generalize these competencies to unseen words. with the large training sets of these models ( 3.3 billion tokens in @xcite al. (2019) ), their state-of-the-artestablishing performance may feasibly have been achieved without ever being tested on a word that was not in the training set. nevertheless, bert may need be concerned about unseen words. increasingly, there is an interest in creating bert and bert-like models trained on large corpora of languages other than english. in comparison to english, many of the world's languages exhibit a much greater amount of inflectional morphology. however, most of the results motivating this explosion of bert models are in english nlp. it is unclear, then, how well bert will generalize to languages with complex morphology. while bert models are being developed for other languages, many of these models have been less comprehensively evaluated than english bert. for instance, the publicly available turkish @xcite ) bert model (one of the most morphologically complex languages for which a bert model is available) has only been evaluated on named entity recognition and part-of-speech tagging. it is unclear, then, how well the model would fare on more complex nlp tasks. in this work, we investigate bert's ability to capture this type of information by studying its ability to identify the correct plural form of novel words in english, french, spanish, dutch, and german. we find that bert is able to distinguish plural and singular forms to perform number agreement significantly above chance in all languages. however, many bert models perform substantially worse on novel words than on words in the training set, even when prompted with an example that shows the singular form, a task which humans are known to be capable of @xcite . this indicates that even simple morphological capacities are not reliably acquired in a human-like way in the bert training paradigm, showing room for improvement in future models. ## background bert is part of a growing research direction of pretraining deep learning models, often a variant of a "transformer" architecture @xcite , on large amounts of natural language data using some variant of a language modelling objective. this line of research includes other such successful models as elmo @xcite and xlnet @xcite . all of these models are trained on very large corpora, with elmo being the smallest (trained on 1 billion tokens in @xcite -in contrast to bert's 4 billion tokens in @xcite ). all of these models are also highly computationally intensive to train, so it is desirable to avoid training new bert models. bert uses a transformer-based architecture, making it bidirectionally sensitive. it is trained on a masked language modelling objective, meaning that it takes in as input a sequence with some words replaced with a [mask] token, and is expected to output the original sequence. to enable this, a final fully connected layer and softmax is added after the transformer encoder to produce the desired output. this means bert is "out of the box" capable of answering exactly those questions that can be posed as replacing [mask] tokens. bert-like models are also generally so-called open-vocabulary language models, meaning they can assign a probability to any string. this enables them to give probabilities to novel words and novel forms of known words, giving bert the capacity to learn morphological generalizations. this is achieved through the use of subword segmentation, in which a strategy such as byte-pair encoding (bpe) @xcite or unigram lm segmentation (such as wordpiece @xcite and the related sentencepiece) is used to turn words into a sequence of multi-character tokens. these segmentation strategies use statistical methods to determine which multi-character tokens are added to their vocabularies, meaning that high-frequency sub-word strings will more likely be added as tokens. these tokens may or may not correspond to morpheme boundaries. if they do not, then models that rely on them will encounter the same morpheme expressed in many distinct tokens, requiring the model to learn agreement for all tokens which may contain, e.g., the plural affix. this may mean that uncommon segments containing inflectional affixes will be less reliable in agreement, since they have no relation in representation to frequently-occurring subwords containing the same inflection. ## methods this work focuses on bert's ability to recognize novel words as singular or plural. this construction was chosen for its testability (through number agreement on verbs) and its disparity in complexity between languages. in english, french, dutch and spanish, a large majority of plurals are derived according to rules that can be expressed simply in terms of adding a suffix corresponding to the suffix of the base noun. further, in french and spanish, the plurality of a noun is unambiguous if it is preceded by a determiner. ## results cross-linguistically, there is no consistent trend in whether the model is able to use the prime to achieve better performance. while flaubert was the only model to achieve statistically significant gains (@xmath1.01) in the non-word case with the addition of the prime, this gain was also significant (@xmath2.05) in the real word case, suggesting deeper issues with this model's agreement capabilities generally. many models were slightly hurt by the inclusion of the prime, suggesting that they may be spuriously agreeing with the prime, even across a sentence boundary. as one might expect, the german bert models had the lowest average performance on the nonword conditions, with no model surpassing 0.80 accuracy. however, only french models achieved an accuracy of greater than 0.90 in any non-word case. given that the correct form for french and spanish agreement can be determined from the noun's article alone, it is surprising that the spanish models do not fully utilize this heuristic; this heuristic may explain the high french performance. ## discussion to investigate whether the lower novel-word performance was related to the segmentations of the novel words, we measured how often each non-word was associated with an error. we found inconsistent results across models. bertje was found to perform especially poorly on 4 out of 24 non-words, incorrectly choosing a singular verb for a plural form of the word 93% of the time. on investigating the segmentations, these words were found to be segmented to "[unk]" by the tokenizer. this model uses the standard sentencepiece unigram tokenizer foot_4 , ostensibly the same as many of these other models. typically, this tokenizer is considered to be open-vocabulary, yet it fails to segment these subwords, indicating that this is not strictly true for this very popular implementation. if these 4 words are disregarded, in the no-prime case this model achieves an accuracy of 0.93, the highest of all non-french models across languages. while this error is of substantial concern, it affects only the dutch bertje results, as no other models were found to have this behavior. other models were found to frequently fail on the plural or singular forms of some words, such as beto, which 50% of the time identified "comanas" as a singular word form. some models, such as the english model and flaubert, instead seemed to be uncertain about the plurality of all forms, making a moderate amount errors at roughly equal rates across non-words. in the case of the english model, the model has a bias towards plurality, with plural accuracy 0.19 greater than singular accuracy; however, flaubert makes agreement errors at roughly equal rates regardless of whether the subject is singular or plural. with the german models, accuracy @xmath3.88 for singular non-words, many of which are disambiguated by their determiner. accordingly, most errors are plural word forms which the model identified as singular. both monolingual german models showed a pattern of having many plural forms that were identified as @xmath4. most of the remaining forms in each model were correctly identified as @xmath5, indicating that these models are relatively certain in their predictions. unfortunately, no clear relationship to how closely the segmentation pattern matches the morphology was found with whether the correct verb is selected for a given non-word. however, it is possible that the frequency of the final subword segment occurring as a plural affix in a german corpus would be more predictive of which segments are likely to result in errors. ## conclusion these results suggest that bert models have some understanding of morphology when applied to novel words (or at least the plurals in a few germanic and romance languages). performance is much significantly better than chance in simple agreement cases, even when no prime is given. this shows that the bert models have learned something about what plural and singular forms "look like." however, non-word performance is not helped especially by the inclusion of a priming sentence, indicating that the bert models in ques-tion may not have learned to recognize new words and apply rules to them, as humans might. further work should investigate what types of primes affect the performance and how. the model performance here represents a best case for bert's morphological capacity on novel words. the plural construction is extremely common in text and is connected to phenomena like agreement which additionally pressures it to be learned on a non-semantic level. further, the simple sentences studied here allow for the potential of n-gram-level agreement heuristics which are not possible in the general case. that bert struggles to capture morphology in this way is likely not due to a lack of training data. there are two potential culprits: the tokenization method and the training objective. the flaubert results especially indicate that the masked language modeling objective may not sufficiently encourage agreement. cross-linguistically, the models seem not to have picked up on how to use the information in the prime. in addition, the subword tokenization methods used by bert and bert-like models make the morphology learning problem significantly more complicated. this is because the plural morpheme is connected to some number of final characters of a word as a single token, meaning even plurals formed in the same way may be represented differently. this work points to a need for subword segmentation strategies that more closely mirror a language's morphology than current approaches like unigram segmentation or byte-pair encoding. in this aspect of language, there remains a large gap between bert's behavior and human performance.