D ago BERT : G enerating Derivational Morphology with a Pretrained Language Model (2020)

Abstract: Can pretrained language models (PLMs) generate derivationally complex words? We present the first study investigating this question, taking BERT as the example PLM. We examine BERT’s derivational capabilities in different settings, ranging from using the unmodified pretrained model to full finetuning. Our best model, DagoBERT (Derivationally and generatively optimized BERT), clearly outperforms the previous state of the art in derivation generation (DG). Furthermore, our experiments show that the input segmentation crucially impacts BERT’s derivational knowledge, suggesting that the performance of PLMs could be further improved if a morphologically informed vocabulary of units were used.

## introduction what kind of linguistic knowledge is encoded by pretrained language models (plms) such as elmo @xcite , gpt-2 @xcite , and bert @xcite ? this question has attracted a lot of attention in nlp recently, with a focus on syntax (e.g., @xcite and semantics (e.g., @xcite . it is much less clear what plms learn about other aspects of language. here, we present the first study on the knowledge of plms about derivational morphology, taking bert as the example plm. given an english cloze sentence such as this jacket is . and a base such as wear, we ask: can bert generate correct derivatives such as unwearable? the motivation for this study is twofold. on the one hand, we add to the growing body of work on the linguistic capabilities of plms. most plms segment words into subword units @xcite , e.g., unwearable is segmented into un, ##wear, ##able by bert's wordpiece tokenizer @xcite . the fact that many of bert dcl this jacket is [mask] wear [mask] . un ##able to bert, mask out derivational affixes, and recover them using a derivational classification layer (dcl). type examples prefixes anti, auto, contra, extra, hyper, mega, mini, multi, non, proto, pseudo suffixes ##able, ##an, ##ate, ##ee, ##ess, ##ful, ##ify, ##ize, ##ment, ##ness, ##ster contributions. we develop the first framework for generating derivationally complex english words with a plm, specifically bert, and analyze bert's performance in different settings. our best model, dagobert (derivationally and generatively optimized bert), clearly outperforms an lstm-based model, the previous state of the art. we find that dagobert's errors are mainly due to syntactic and semantic overlap between affixes. furthermore, we show that the input segmentation impacts how much derivational knowledge is available to bert, both during training and inference. this suggests that the performance of plms could be further improved if a morphologically informed vocabulary of units were used. we also publish the largest dataset of derivatives in context to date. 1 ## derivational morphology linguistics divides morphology into inflection and derivation. given a lexeme such as wear, while inflection produces word forms such as wears, derivation produces new lexemes such as unwearable. there are several differences between inflection and derivation @xcite , two of which are particularly important for the task of dg. 2 first, derivation covers a much larger spectrum of meanings than inflection @xcite , and it is not possible to predict in general with which of them a particular lexeme is compatible. this is different from inflectional paradigms, where it is automatically clear whether a certain form will exist @xcite . second, the relationship between form and meaning is more varied in derivation than inflection. on the one hand, derivational affixes tend to be highly polysemous, i.e., individual affixes can represent a number of related meanings @xcite . on the other hand, several affixes can represent the same meaning, e.g., ity and ness. while such competing affixes are often not completely synonymous as in the case of hyperactivity and hyperactiveness, there are examples like purity and pureness or exclusivity and exclusiveness where a semantic distinction is more difficult to gauge @xcite . these differences make learning functions from meaning to form harder for derivation than inflection. derivational affixes differ in how productive they are, i.e., how readily they can be used to create new lexemes @xcite . while the suffix ness, e.g., can attach to practically all english adjectives, the suffix th is much more limited in its scope of applicability. in this paper, we focus on productive affixes such as ness and exclude unproductive affixes such as th. morphological productivity has been the subject of much work in psycholinguistics since it reveals implicit cognitive generalizations (see @xcite for a review), making it an interesting phenomenon to explore in plms. furthermore, in the context of nlp applications such as sentiment analysis, productively formed derivatives are challenging because they tend to have very low frequencies and often only occur once (i.e., they are hapaxes) or a few times in large corpora @xcite . our focus on productive derivational morphology has crucial consequences for dataset design (section 3) and model evaluation (section 4) in the context of dg. ## dataset of derivatives we base our study on a new dataset of derivatives in context similar in form to the one released by @xcite , i.e., it is based on sentences with a derivative (e.g., this jacket is unwearable .) that are altered by masking the derivative (this jacket is .). each item in the dataset consists of (i) the altered sentence, (ii) the derivative (unwearable) and (iii) the base (wear). the task is to generate the correct derivative given the altered sentence and the base. we use sentential contexts rather than tags to represent derivational meanings because they better reflect the semantic variability inherent in derivational morphology (section 2). while @xcite use wikipedia, we extract the dataset from reddit. 3 since productively formed derivatives are not part of the language norm initially @xcite , social media is a particularly fertile ground for our study. for determining derivatives, we use the algorithm introduced by @xcite , which takes as input a set of prefixes, suffixes, and bases and checks for each word in the data whether it can be derived from a base using a combination of prefixes and suffixes. the algorithm is sensitive to morpho-orthographic rules of english @xcite , e.g., when ity is removed from applicability, the result is applicable, not applicabil. here, we use bert's prefixes, suffixes, and bases as input to the algorithm. drawing upon a comprehensive list of 52 productive pre- for this study, we extract all derivatives with a frequency @xmath0, 128) from the dataset. we divide the derivatives into 7 frequency bins with regarding the number of affixes, we confine ourselves to three cases: derivatives with one prefix (p), derivatives with one suffix (s), and derivatives with one prefix and one suffix (ps). 6 we treat these cases separately because they are known to have different linguistic properties. in particular, since suffixes in english can change the pos of a lexeme, the syntactic context is more affected by suffixation than by prefixation. ## related work plms such as elmo @xcite , gpt-2 @xcite , and bert @xcite have been the focus of much recent work in nlp. several studies have been devoted to the linguistic knowledge encoded by the parameters of plms (see @xcite for a review), particularly syntax @xcite @xcite and semantics @xcite @xcite . there is also a recent study examining morphosyntactic information in a plm, specifically bert @xcite . there has been relatively little recent work on derivational morphology in nlp. both @xcite and @xcite propose neural architectures that represent derivational meanings as tags. more closely related to our study, @xcite develop an encoder-decoder model that uses the context sentence for predicting deverbal nouns. hofmann et al. (2020b) propose a graph auto-encoder that models the morphological well-formedness of derivatives. ## conclusion we show that a plm, specifically bert, can generate derivationally complex words. our best model, dagobert, clearly beats an lstm-based model, the previous state of the art in dg. dagobert's errors are mainly due to syntactic and semantic overlap between affixes. furthermore, we demonstrate that the input segmentation impacts how much derivational knowledge is available to bert. this suggests that the performance of plms could be further improved if a morphologically informed vocabulary of units were used.