The Impact of Word Splitting on the Semantic Content of Contextualized Word Representations (2024)

Abstract: When deriving contextualized word representations from language models, a decision needs to be made on how to obtain one for out-of-vocabulary (OOV) words that are segmented into subwords. What is the best way to represent these words with a single vector, and are these representations of worse quality than those of in-vocabulary words? We carry out an intrinsic evaluation of embeddings from different models on semantic similarity tasks involving OOV words. Our analysis reveals, among other interesting findings, that the quality of representations of words that are split is often, but not always, worse than that of the embeddings of known words. Their similarity values, however, must be interpreted with caution.

## introduction with the appearance of pre-trained language models (plms) such as bert @xcite and @xcite , there has been an interest in extracting, analyzing, and using contextualized word representations derived from these models, for example, to understand how well they represent the meaning of words @xcite or to predict diachronic semantic change @xcite . most modern plms, however, operate at the subword level-they rely on a subword tokenization algorithm to represent their input, like word-piece @xcite or byte pair encoding (bpe) @xcite . this way of representing words has advantages: with a fixed, reasonably-sized vocabulary (oov), models can account for out-ofvocabulary words by splitting them into smaller units. when it comes to obtaining representations for words, a subword vocabulary implies that not all words are created equally. words that have to be split (''split-words'') need a special treatment, different from words that have a dedicated embedding (''full-words''). there are reasons to believe that the semantics of split-words is more poorly represented than that of full-words. first, it is generally assumed that longer tokens tend to contain more semantic information about a word @xcite because they are more discriminative. the subword representations making up split-words must be able to encode the semantics of all words they can be part of. it has also been noted that tokenization algorithms tend to split words in a way that disregards language morphology @xcite , and some of them favor splittings with more subword units than would be necessary @xcite . in fact, a more morphology-aware segmentation seems to correlate with better results on downstream nlp tasks @xcite . in this study, we investigate the impact that word splitting (and how we decide to deal with it) has on the quality of contextualized word representations. we rely on the task of lexical semantic similarity estimation, which has traditionally been used as a way of intrinsically evaluating different types of word representations @xcite . we set out to answer two main questions: figure 1 : example of one of our settings where we calculate the cosine similarity between the representations of an oov word and a known word. we test different ways of creating one embedding for an oov word ( §4), such as avg and lng, on two similarity tasks ( §3). as ''sub-tokens'' (as opposed to ''full-tokens'' which do not start with ''##''). subword tokenization presented itself as a good compromise between character-level and word-level models, balancing the trade-off between vocabulary size and sequence length. character-based representations are generally better than subword-based models at morphology, part-of-speech (pos) tagging, and at handling noisy input and out-of-domain words; but the latter are generally better at handling semantics and syntax @xcite @xcite . because of these advantages, most modern plms rely on subword tokenization: bert uses wordpiece; roberta, xlm (conneau and lample, 2019), gpt-2 @xcite and gpt-3 @xcite use bpe or some variant; t5 @xcite relies on sentence-piece @xcite . several studies have pointed out that splitting words may be detrimental for certain tasks, especially if segmentation is not done in a linguistically correct way. @xcite compare two subword tokenization algorithms, bpe and unigramlm @xcite , and find that the latter, which aligns better with morphology, also yields better results on question answering, textual entailment, and named entity recognition. work on machine translation has shown benefits from using linguistically informed tokenization @xcite as well as algorithms that favor segmentation into fewer tokens @xcite . in fact, @xcite note that multilingual bert's (mbert) tokenizer segments much more in some languages than others, and they demonstrate that a dedicated monolingual tokenizer plays a crucial role in mbert's performance on numerous nlp tasks. similarly, @xcite show that increased fertility (i.e., the average number of tokens generated for every word) and number of split-words correlate negatively with mbert's performance on epidemiologic watch through multilingual event extraction. however, the effect that (over)splitting words-or doing so disregarding their morphology-has on similarity remains unclear. nayak et al. (2020) explore a similar question to ours using the bert model, but compare the similarity between a word representation and their sub-token counterpart (e.g., night with ##night). we argue, however, that even if they represent the same string, sub-tokens and full-tokens have different distributions and the similarity between them is not necessarily expected to be high. 3 their experiments additionally involve a modification of the tokenizer. we instead compare representations of whole words using the models' default tokenization, and we work with representations of words extracted from sentential contexts and not in isolation. multiple approaches have been proposed to improve on the weak aspects of vanilla subword tokenization, such as the representation of rare, out-of-domain, or misspelled words @xcite @xcite , and its concurrence with morphological structure @xcite . @xcite devise flota, a simple segmentation method that can be used with pre-trained models without the need for re-training a new model or tokenizer. it consists in segmenting words prioritizing the longest substrings available, omitting part of the word in some cases. flota was shown to match the actual morphological segmentation of words more closely than the default bert, gpt-2, and xlnet tokenizers, and yielded an improved performance on a topic-based text classification task. el @xcite propose characterbert, a modified bert model with a character-level cnn intended for building representations for complex tokens. the model improves bert's performance on several tasks on the medical domain. we test the flota method and the characterbert model in our experiments to investigate their advantages when it comes to lexical semantic similarity. the split-words in our study are existing words-we do not include misspelled termswith a generally low frequency. there has been extensive work in nlp focused on improving representations of rare words, which are often involved in lower-quality predictions than those of more frequent words @xcite @xcite , also in bert @xcite . our goal is not to study the quality of rare word representations per se, but rather the effect of the splitting procedure on the quality of similarity estimates. given the strong link between splitting and frequency, we also include an analysis controlling for this factor. ## similarity tasks and data we evaluate the representations' lexical semantic content on two similarity tasks. in this section we describe the creation of an inter-word similarity dataset ( §3.1) as well as the dataset used in our within-word similarity experiments ( §3.2). ## results and analysis in this section we analyze the results obtained on the split-sim ( §5.1) and wic ( §5.2) datasets. ## discussion we have seen that when examined separately, word pairs involving split-words often obtain worse quality similarity estimations than those consisting of full-words; but this depends on the type of word: split polysemous nouns are better represented than non-split ones. this holds across the models and tokenizers tested, and also when evaluating on words in a narrower frequency range. this shows that word splitting has a negative effect on the representation of many words. we have also seen that in normal conditions, performance on 1-split is generally the worst one, due mainly to a larger disparity in frequencies of the words in a pair. our analysis has also confirmed the hypothesis that words that are split in a way that preserves their morphology obtain bet16 a partial explanation is that same pairs have a slightly stronger tendency of being t (77% of same 2-split pairs are t, vs 66% of diff 2-split pairs). we have noted that similarities for the different split-types are found in different ranges; notably, similarities between two split-words tend to be higher than similarities in 0-and 1-split pairs. naturally, this has an effect on the correlation calculated on the full dataset, which is lower than when considering each split-type separately. it would be interesting to develop a similarity measure that allows comparison across split-types, which could rely on information from the rest of the sentence, like bertscore @xcite . another simple way to make similarities comparable would be to bring 2-split similarities to the 0-split similarity range by subtracting the average similarity value obtained in 0-split. the best value to use, however, may vary depending on the application. one surprising finding relates to the impact of the number of subwords: similarity estimations are not always more reliable on words involving fewer tokens. this was especially the case for bert, where we saw that the first token is generally the least useful in building a representation. given the tendency for the first token to be the longest, this has put the other strategies tested (wavg and lng) at a disadvantage. from our within-word experiments we confirm that word form is reflected in the representations and has a strong impact on similarity, but this does not necessarily mean that comparing words with distinct morphological properties (e.g., singular vs plural) would be detrimental in the inter-word setting. in the within-word setting, same pairs compare two equal word forms, whose representation at the initial (static) embedding layer is identical. diff pairs, instead, start off with different static embeddings, which results in an overall lower similarity. in split-sim, all comparisons are made, by definition, between different words. the fact that two words have different morphological properties may thus have a smaller impact on results. most of our findings are consistent between the two kinds of task (inter-and within-word) and across models. one exception is cbert, which does not assign higher similarities to 2-split pairs when comparing instances of the same word; and the lng strategy, which is more useful withinword than inter-word. avg is, however, the best strategy overall. one direction for future work would be to find a pooling method that closes the gap in performance between split-types. our experiments only involve one language (english), spearman's correlation, and cosine similarity, although our methodology is not restricted to a single similarity or evaluation metric. extending this work to more languages is also possible, but less straightforward, due to the need for suitable datasets. ## conclusion we have compared the contextualized representations of words that are segmented into subwords to those of words that have a dedicated embedding in bert and other models. we have done so through an intrinsic evaluation relying on similarity estimation. our findings are relevant for any nlp practitioner working with contextualized word representations, and particularly for applications relying on word similarity: (i) out of the tested strategies for split-word representation, averaging subword embeddings is the best one, with few exceptions; (ii) the quality of splitword representations is often worse than that of full-words, although this depends on the kind of words considered; (iii) similarity values obtained for split-word pairs are generally higher than similarity estimations involving full-words; (iv) the best layers to use differ across split-types; (v) a higher number of tokens does not necessarily, as intuitively thought, decrease representation quality; (vi) in the within-word setting, word form has a negative impact on results when words are split. our results also point to specific aspects to which future research and efforts of improvement should be directed. we make our split-sim dataset available to facilitate research on split-word representation.