Superbizarre Is Not Superb: Derivational Morphology Improves BERT ’s Interpretation of Complex Words (2021)

Abstract: How does the input segmentation of pretrained language models (PLMs) affect their interpretations of complex words? We present the first study investigating this question, taking BERT as the example PLM and focusing on its semantic representations of English derivatives. We show that PLMs can be interpreted as serial dual-route models, i.e., the meanings of complex words are either stored or else need to be computed from the subwords, which implies that maximally meaningful input tokens should allow for the best generalization on new words. This hypothesis is confirmed by a series of semantic probing tasks on which DelBERT (Derivation leveraging BERT), a model with derivational input segmentation, substantially outperforms BERT with WordPiece segmentation. Our results suggest that the generalization capabilities of PLMs could be further improved if a morphologically-informed vocabulary of input tokens were used.

## introduction pretrained language models (plms) such as bert @xcite , gpt-2 @xcite , xlnet @xcite , electra @xcite , and t5 @xcite have yielded substantial improvements on a range of nlp tasks. what linguistic properties do they have? various studies have tried to illuminate this question, with a focus on syntax @xcite and semantics @xcite @xcite . one common characteristic of plms is their input segmentation: plms are based on fixed-size vocabularies of words and subwords that are generated by compression algorithms such as bytepair encoding @xcite and wordpiece @xcite . the segmentations produced by these delbert with derivational segmentation (s d ), on the other hand, separates prefix and stem by a hyphen (right panel). the two likelihoods are averaged across 20 models trained with different random seeds. the average likelihood of the true class is considerably higher with delbert than with bert. while superbizarre has negative sentiment, applausive is an example of a complex word with positive sentiment. here, we address a central aspect of this issue, namely how the input segmentation affects the semantic representations of plms, taking bert as the example plm. we focus on derivationally complex words such as superbizarre since they exhibit systematic patterns on the lexical level, providing an ideal testbed for linguistic generalization. at the same time, the fact that low-frequency and out-of-vocabulary words are often derivationally complex @xcite makes our work relevant in practical settings, especially when many one-word expressions are involved, e.g., in query processing @xcite . the topic of this paper is related to the more fundamental question of how plms represent the meaning of complex words in the first place. so far, most studies have focused on methods of representation extraction, using ad-hoc heuristics such as averaging the subword embeddings @xcite @xcite or taking the first subword embedding @xcite @xcite . while not resolving the issue, we lay the theoretical groundwork for more systematic analyses by showing that plms can be regarded as serial dual-route models @xcite , i.e., the meanings of complex words are either stored or else need to be computed from the subwords. contributions. we present the first study examining how the input segmentation of plms, specifically bert, affects their interpretations of derivationally complex english words. we show that plms can be interpreted as serial dualroute models, which implies that maximally meaningful input tokens should allow for the best generalization on new words. this hypothesis is confirmed by a series of semantic probing tasks on which derivational segmentation substantially outperforms bert's wordpiece segmentation. this suggests that the generalization capabilities of plms could be further improved if a morphologically-informed vocabulary of input tokens were used. we also publish three large datasets of derivationally complex words with corresponding semantic properties. foot_0 2 how are complex words processed? ## related work several recent studies have examined how the performance of plms is affected by their input segmentation. @xcite show that tokenizing inflected words into stems and inflection symbols allows bert to generalize better on non-standard inflections. @xcite pretrain roberta with different tokenization methods and find tokenizations that align more closely with morphology to perform better on a number of tasks. ma et al. (2020) show that providing bert with character-level information also leads to enhanced performance. relatedly, studies from automatic speech recognition have demonstrated that morphological decomposition improves the perplexity of language models @xcite . whereas these studies change the vocabulary of input tokens (e.g., by adding special tokens), we show that even when keeping the pretrained vocabulary fixed, employing it in a morphologically correct way leads to better performance. 14 most nlp studies on derivational morphology have been devoted to the question of how semantic representations of derivationally complex words can be enhanced by including morphological information @xcite @xcite @xcite , and how affix embeddings can be computed @xcite @xcite . @xcite , @xcite propose sequence-to-sequence models for the generation of derivationally complex words. hofmann et al. (2020a) address the same task using bert. in contrast, we analyze how different input segmentations affect the semantic representations of derivationally complex words in plms, a question that has not been addressed before. ## conclusion we have examined how the input segmentation of plms, specifically bert, affects their interpretations of derivationally complex words. drawing upon insights from psycholinguistics, we have deduced a conceptual interpretation of plms as serial dual-route models, which implies that maximally meaningful input tokens should allow for the best generalization on new words. this hypothesis was confirmed by a series of semantic probing tasks on which delbert, a model using derivational segmentation, consistently outperformed bert using wordpiece segmentation. quantitative and qualitative analyses further showed that bert's inferior performance was caused by its inability to infer the complex-word meaning as a function of the subwords when the complex-word meaning was not stored in the weights. overall, our findings suggest that the generalization capabilities of plms could be further improved if a morphologically-informed vocabulary of input tokens were used. stevens, george kurian, nishant patil, wei wang, cliff young, jason smith, jason riesa, alex rudnick, oriol vinyals, greg corrado, macduff hughes, and jeffrey dean. 2016. google's neural machine translation system: bridging the gap between human and machine translation. in arxiv 1609.08144. yi yang and jacob eisenstein. 2017. overcoming language variation in sentiment analysis with social attention. transactions of the association for computational linguistics, 5:295-307. zhilin yang, zihang dai, yiming yang, jaime carbonell, ruslan salakhutdinov, and quoc v. le. 2019. xlnet: generalized autoregressive pretraining for language understanding. in advances in neural information processing systems (neurips) 33.