To Split or Not to Split: Composing Compounds in Contextual Vector Spaces (2023)

Abstract: We investigate the effect of sub-word tokenization on representations of German noun compounds: single orthographic words which are composed of two or more constituents but often tokenized into units that are not morphologically motivated or meaningful. Using variants of BERT models and tokenization strategies on domain-specific restricted diachronic data, we introduce a suite of evaluations relying on the masked language modelling task and compositionality prediction. We obtain the most consistent improvements by pre-splitting compounds into constituents.

## introduction contextual word embedding models such as bert @xcite have opened up exciting avenues in computational lexical semantics. but since their vocabulary size is limited, these models rely on sub-word tokenization, potentially splitting words into smaller units that are not morphologically or semantically motivated. this linguistically counter-intuitive mechanism does not seem to be detrimental, at least judging by the models' success on downstream tasks. we challenge this assumption by investigating a type of linguistic structure where it is especially important for sub-word representations to be meaningful. specifically, we vary and analyze bert representations of german noun compounds. 1 these are conventionally represented as a single orthographic word (e.g. zitronensaft 'lemon juice') with two clearly identifiable constituents (zitrone 'lemon' and saft 'juice'). but bert would tokenize our example as [zit, ##ronen, ##sa, ##ft], ignoring both constituents and instead including semantically unrelated sub-word fragments which occur in a highly diverse set of also irrelevant contexts. moreover, these suboptimal representations are especially problematic in under-resourced settings, where a compound may only occur a few times. this study aims to identify the most robust bert representations of german compounds by experimenting with variants derived from the base and re-trained models using three tokenization strategies: the default tokenizer, a re-trained tokenizer, and pre-splitting compounds into their constituents. we evaluate on (i) the model's likelihood of predicting a compound in the masked language modelling task, additionally using the semantic relatedness of these predictions to the target compound as defined by germanet paths; and (ii) the correlation of model-internal measures of compound-constituent similarity with human compositionality ratings. in order to maximally enforce the need for robust representations, we run the experiments foot_1 on a diachronic corpus as a notoriously restricted type of data. the most consistent improvements are obtained when compounds are pre-split into constituents; as a simple but efficient preprocessing step, this has direct implications for contextual representations of other types of complex words. ## prior work impact of tokenizers on contextual embeddings @xcite and @xcite evaluated multilingual contextual language models on downstream tasks such as question answering and event detection. they noted that a lack of representation of some languages in the tokenizer vocabulary for models like multilingual-bert hurts the performance, but that this effect can be mitigated with a monolingual tokenizer @xcite or learning a function to aggregate sub-word tokens and to compensate for the relatively higher fragmentation of tokens in under-represented languages @xcite . these works were an important step toward addressing the negative effect that sub-word-tokenization can have on semantic representation, going beyond the approach by @xcite to use the first sub-word token as the representation of a longer sequence. @xcite found contextualized embeddings to be more successful than static embeddings as a basis for classification tasks involving detecting semantic differences from the typical meaning of several kinds of multi-word expressions. @xcite explored the geometry of contextual vector spaces, showing that randomly selected words' vector representations are more similar (via cosine similarity) than would be expected if the vectors were distributed uniformly. they also found that vector representations formed from later layers were more context-specific, and those from earlier layers functioned better when used to create static representations. these works will inform our baseline expectations for forming contextual semantic representations. ## resources here we describe our corpus of historical german texts used to train and adapt semantic representations, human compositionality ratings of german compounds, and the lexical taxonomy germanet. deutsches textarchiv -dta the dta (berlin-brandenburgischen akademie der wissenschaften, 2022) is a diachronic, curated selection of german texts of various genres. we use a portion of the corpus to train and fine-tune our models. we rely on the orthographically modernized, normalized, lemmatized versions of the texts made available in the dta. all of ≈4m sentences (≈89m tokens) were shuffled, and 10% were held out as evaluation data. human compositionality ratings compositionality ratings measure the degree to which the compound's head or modifier constituents contribute to the meaning of the compound, in our case ranging from 1 (totally unrelated to compound meaning) to 6 (fully accounts for compound meaning). we rely on the ratings for german noun-noun compounds in the ghost-nn dataset (schulte im @xcite , but exclude any compound occurring fewer than 20 times in our training data, leaving 185 noun-noun compounds as our set of test items. germanet is a taxonomy of 'synsets', sets of one or more words that are synonymous @xcite . the primary connection between synsets of nouns is established by hypernymy and hyponymy relationships. paths and distances in this network can be used as a measure of semantic distance @xcite . ## system configurations configuration pre-train dta re-train split base base-ft-dta voc-rt-dta split re-training or fine-tuning bert configurations base and base-ft-dta use the base tokenizer vocabulary, while voc-rt-dta and split re-train the tokenizer. 5 all configurations other than the base bert model are trained on dta data. the base-ft-dta configuration retains the base tokenizer's vocabulary, so its training on dta data constitutes fine-tuning of the base model. ## evaluation here we present our two evaluation perspectives: the in-context masked-language-model prediction of target compounds, and the decontextualized exploration of how the compounds and constituents are represented in each model's embedding space. ## conclusion we endeavored to investigate the effects of subword tokenization on the semantic representation of german noun compounds with contextualized embedding models, using the contrasting perspectives afforded by an in-context and a decontextualized evaluation. splitting compounds prior to training bert embeddings resulted in better in-context performance, thus meeting or surpassing the fine-tuned base model, while making more effective use of limited training data. furthermore, the split model's higher performance compared with the voc-rt-dta model suggests an advantage for this pre-processing approach in data-limited settings, as both of these models were only trained on the dta data. in the decontextualized evaluation, word2vec models surpassed all bert models, where the base model had the best performance, and the split model was competitive with the finetuned model. it remains to be seen whether the advantages of splitting compounds would remain if this technique was applied over the full pre-training of german bert, but overall, we recommend paying closer attention to tokenizer granularity in limited data contexts.