Word Boundary Information Isn’t Useful for Encoder Language Models (2024)

Abstract: All existing transformer-based approaches to NLP using subword tokenisation algorithms encode whitespace (word boundary information) through the use of special space symbols (such as ## or _) forming part of tokens. These symbols have been shown to a) lead to reduced morphological validity of tokenisations, and b) give substantial vocabulary redundancy. As such, removing these symbols has been shown to have a beneficial effect on the processing of morphologically complex words for transformer encoders in the pretrain-finetune paradigm. In this work, we explore whether word boundary information is at all useful to such models. In particular, we train transformer encoders across four different training scales, and investigate several alternative approaches to including word boundary information, evaluating on two languages (English and Finnish) with a range of tasks across different domains and problem set-ups: sentence classification datasets, NER (for token-level classification), and two classification datasets involving complex words (Superbizarre and FLOTA). Overall, through an extensive experimental setup that includes the pre-training of 35 models, we find no substantial improvements from our alternative approaches, suggesting that modifying tokenisers to remove word boundary information isn’t leading to a loss of useful information.

## introduction transformer @xcite pretrained language models for nlp, such as bert @xcite and the gpt family @xcite , typically use subword tokenisation algorithms, such as wordpiece @xcite , to process text. previous work @xcite has shown that such methods have limited alignment with word morphology, resulting in worsened downstream performance for various tasks @xcite @xcite . in fact, it has been shown that the morphological validity of tokenisation can be improved by removing all whitespace markers (and hence word boundary (wb) information) from the tokenisers @xcite . however, the full impact of this modification on downstream performance is unknown, and the question of whether wb information is at all useful to models is as yet unanswered. in this work, we first perform a morphological evaluation of wordpiece and wordpiece ′ , a version which has been modified to have no wb information. we find that wordpiece ′ significantly improves the alignment with morphological gold standard references. then, we evaluate wordpiece and wordpiece ′ as tokenisers on downstream tasks. we also introduce models which modify wordpiece ′ by including wb information in various ways -either explicitly through the input or implicitly through the pretraining objective. much interest recently has been in the scaling laws of language models @xcite , and a direction towards training larger models. on the other hand, there has been recent work investigating sample-efficient pretraining on datasets of a developmentally plausible size @xcite . in companion to such work, we train our models across four training scales, from approximately 6m params and 250m tokens at the lowest scale to approximately 370m params and 23b tokens at the highest scale. across these scales we pretrain all of our models and evaluate in english on four downstream tasks the findings of our work are as follows: (1) we show that modifying wordpiece to remove wb information (giving wordpiece ′ ) substantially improves the morphological validity of the resulting tokenisations across english and finnish; (2) across four training scales, we find that wordpiece ′ outperforms wordpiece on downstream tasks involving complex words, and gives better performance across most datasets at the lower training scales; ## tokenisers one particular design choice of subword tokenisers used by transformer models is the addition of prefixes such as "_" and "##" in order to encode space information, hence representing word boundaries in languages with spaces between words. previous work @xcite has investigated the impact of these prefixes, showing they lead to less morphologically valid tokenisations, and also to a reduced efficiency, since the dual representation of subwords (e.g. "beat" and "_beat") gives a vocabulary redundancy (of approximately 9%). as such, removing these tokens for unigram @xcite and bpe @xcite has been shown to have a beneficial effect on downstream performance for complex word tasks, whilst retaining equivalent performance in general natural language understanding tasks. we refer readers to @xcite for a full analysis, but here we focus on wordpiece ′ -wordpiece modified such that wb information is removed. we train this model and the default on 1 million sentences from wikipedia for two languages (english and finnish). we show an example of the tokenisations generated by this compared to the default for english in figure 1 . we perform a morphological evaluation of wordpiece ′ compared to wordpiece across the two languages, shown in ## models the sequences generated by wordpiece ′ have no word boundary information, which means some information is lost when using it to encode sequences. we aim to answer the question of whether such information is at all useful to transformer encodersi.e. can it be incorporated in an alternative way to improve performance? we investigate transformer encoders pretrained using the masked language modelling (mlm) task, and then finetuned on downstream tasks (pretrain-finetune paradigm). english finnish len precis. f1 len precis. f1 wordpiece 3.29 24.8 33.8 3.21 28.3 38.9 wordpiece ′ 2.75 42.6 52.7 2.86 34.7 45.0 @xmath0, where word boundary embeddings are passed in the input. mlm 2 @xmath1, either directly as input (both in pretraining and finetuning), or through a modification of the pretraining task. ## results we report our full results across all individual datasets for all models in the appendix (tables 12 and 13 ). here, we look at the overall metrics from the four tasks across the training scales, and present our main findings. we note that the plots produced (figures 5 and 6 , and figures 9 to 12 in the appendix) are approximately logarithmic in training scale, and we reproduce them using a scale factor on the x-axis in the appendix: figures 15 to 19. firstly, we compare wordpiece and wordpiece ′ in at the lower scales (+1.5 and +2.1 for the v low and low training scales, respectively). we note that at the higher scales, the differences are within two standard deviations of the baseline, so these results are consistent with those by @xcite . for ner, on the other hand, we find that wordpiece ′ performs worse than wordpiece across all training scales except v high, where they perform equivalently. looking at the individual dataset performances ( for the complex word tasks, wordpiece ′ substantially outperforms wordpiece: averaging across the training scales, we get 1.1 average increase for superbizarre, and 4.5 average increase for flota. the relative performance difference is most significant for superbizarre: at the v low scale, we would require approximately 20 times the training scale for wordpiece to match wordpiece ′ (figure 15c in the appendix). in general, we find the performance differences to decrease as the training scale increases, as expected, foot_2 however this effect seems significantly less for superbizarre, which still has a large performance difference at the v high training scale (+0.9). next, we look at the models that attempt to use wb information, with results in comparing wordpiece ′ and the implicit variant (shown also in figure 9 in the appendix), we find that adding the extra loss term gives mixed results across the four tasks and training scales. we do however see that at the v low and low training scales, the implicit model improves performance for ner (+1.7 and +1.2, respectively). since this prediction task is very similar to the finetuning task of token classification, this may explain the effect on performance. the additional mlm head increases the total loss (see figure 13 in the ap-pendix), but when we look at the evaluation accuracies for the two mlm heads (figure 14 in the appendix), we see that the default mlm head has very similar accuracies to the wordpiece ′ baseline. we also note that for the very low training scale, there is a 3.5% (relative) improvement in default mlm accuracy, which could be contributing to the performance improvement -in a low resource scenario (both compute and data), the extra prediction task may help to leverage additional information. next, we look at the explicit variants. naively including the wb information through additional tokens leads to decreased performance across all tasks except for flota, where there is no substantial performance difference (figure 10 in the appendix). overall, these differences are small: around 1 for glue, 0.5-2 for ner, 0.1-0.3 for superbizarre. this is despite a significantly lower mlm loss (approximately 60%: figure 13 in the appendix) due to the high probability of wb tokens, and the fact that this model trains for around 40% fewer epochs ( finally, we look at two approaches to including wb information during finetuning only (figure 12 in the appendix) -with wb tokens or binary index wb embeddings. we find that neither of these approaches improve over the baseline, with the wb tokens approach performing overall slightly worse: averaged across all training scales and datasets, we get 57.5 for default wordpiece ′ , 57.5 for wordpiece ′ f/t binary index, and 57.3 for wordpiece ′ f/t wb tokens. this corroborates the results by @xcite , who find that adding position embeddings after pretraining without them does not lead to improved performance. on average, including the wb embeddings during finetuning decreases training stability (increased standard deviation across seeds). ## discussion overall, we find that incorporating word boundary information in transformer encoders, either explicitly or implicitly, does not lead to substantial performance improvements. this suggests that: a) modifying tokenisers such as wordpiece to remove space information does not result in the loss of useful information, b) the default mlm task is sufficient for such models to pretrain effectively. the pre-tokenisation step of splitting on whitespace prevents tokens from ever crossing word boundaries, which is perhaps a sufficient restriction. our results indicate the importance of a morpheme compared to a word as the key feature which contributes to meaning. for english, across all models and training scales, we only see a weak correlation between performance on ner and glue -if we compare the difference compared to wordpiece ′ for the implicit and explicit models, we find a correlation with pearson's ρ = 0.332. the superbizarre task is significantly less affected by model scaling than the other tasks we evaluate on, but much more affected by the choice of tokeniser. this suggests that morphologically valid tokenisation is vital for generating good representations of complex words in the absence of context. this task is also less likely to be dependent on spurious correlations (annotation artefacts) in the data. all of our models at the high and v high training scales outperform the dev results reported by @xcite on the flota arxiv-s datasets using their tokenisation method. we hypothesise this is likely an effect of hyperparameters, e.g. we use a batch size of 32 rather than their 64, and we use a learning rate scheduler with warm-up, whereas they do not. there is previous work which has passed additional position indices to transformer models. @xcite introduce a model for neural textto-speech called png bert which uses wordposition embeddings to provide alignment between phonemes and graphemes at the word level. in nlp, @xcite introduce segatron, a model which modifies the transformer-xl @xcite with two additional position embeddings: a sentence index and a paragraph index. they also apply the same modifications to bert, giving segabert. they find that segabert gives lower validation losses during pre-training, lower language modelling perplexities, and improves upon the glue score of bert. @xcite include pos tags as additional input embeddings during bert pretraining, which they find to reduce performance on (super)glue @xcite and msgs @xcite . there has also been work which has modified the pretraining objective of transformer models. @xcite introduce various alternatives to mlm, and pre-train models using them, finding that default mlm is superior in the higherparameter setting. there have been various works using linguistically-motivated pretraining objectives @xcite , with the closest to our work being that by @xcite , who find improved performance through simply adding additional mlm heads for linguistic tasks and summing their losses. ## conclusion in this work we investigate whether word boundary information is useful for transformer encoders. in particular, we start with wordpiece ′ , a version of wordpiece modified to remove word boundary in-formation, and show that it leads to more linguistically meaningful tokenisations, as well as improved performance on tasks involving morphologically complex words, whilst having no significant effect on performance for general domain tasks across english and finnish. we also investigate modifications to the default model architecture which involve incorporating word boundary information, either explicitly (through the input), or implicitly (through the pretraining task), and through pretraining or finetuning alone. across all models and training scales, we find that these modifications give no substantial improvements in performance, which suggests transformer encoders can perform well without word boundary information, either in the form of prefixes ("##" or "_"), word boundary tokens, word boundary embeddings, or through a modification to the pretraining task. 0 10 20 30 40 50 60 70 80 90 epochs 0 2 4 6 8 10 wordpiece train wordpiece valid wordpiece' train wordpiece' valid (a) v low 0 5 10 15 20 25 30 epochs 0 2 4 6 8 10 (b) low 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 epochs 0 2 4 6 8 10 (c) high figure 8: training and valid losses for wordpiece and wordpiece ′ across three training scales for finnish. |train| (k) |dev| (k) metric domain cola (warstadt et al., 2018) 8.5 1 matthew's correlation books and journal articles sst-2 (socher et al., 2013) 67 1 accuracy film reviews mrpc @xcite 3.7 0.4 f1 / accuracy online news sts-b @xcite 5.8 1.5 pearson / spearman correlation various qqp ( https://quoradata.quora.com/first-quora-dataset-release-question-pairs ) 364 40 f1 / accuracy quora questions mnli @xcite 393 9.8 accuracy various qnli @xcite 105 5.5 accuracy wikipedia rte @xcite 2.5 0.3 accuracy wikipedia and news superbizarre-arxiv @xcite 58 19 f1 arxiv papers superbizarre-reddit @xcite 51 17 f1 reddit flota @xcite 1