Bridging the Gap between Subword and Character Segmentation in Pretrained Language Models (2023)

Abstract: Pretrained language models require the use of consistent segmentation (e.g., subword- or character-level segmentation) in pretraining and finetuning. In NLP, many tasks are modeled by subword-level segmentation better than by character-level segmentation. However, because of their format, several tasks require the use of character-level segmentation. Thus, in order to tackle both types of NLP tasks, language models must be independently pretrained for both subword and character-level segmentation. However, this is an inefficient and costly procedure. Instead, this paper proposes a method for training a language model with unified segmentation. This means that the trained model can be finetuned on both subword- and character-level segmentation. The principle of the method is to apply the subword regularization technique to generate a mixture of subword- and character-level segmentation. Through experiment on BERT models, we demonstrate that our method can halve the computational cost of pretraining.

## introduction the use of large pretrained language models (plms) has become the dominant approach for tackling nlp tasks and applications @xcite @xcite . one notable characteristic of these models is that the segmentation algorithm must be determined before pretraining the model. given a pretrained model, users are expected to employ a consistent segmentation algorithm. for example, a common convention is to use a family of subword-level segmentation algorithms @xcite @xcite with a sufficiently large vocabulary; for example, 8k @xcite , 30k @xcite , 50k @xcite , or character-level segmentation must be used to insert a missing comma in a given input sentence. ## background as explained in section 1, our method is based on a subword segmentation algorithm and a corresponding regularization technique, namely, subword regularization @xcite . in this paper, we employ byte pair encoding (bpe) @xcite and bpe-dropout @xcite for subword segmentation and subword regularization, respectively 4 . this section briefly describes the main ideas underlying both methods. ## method originally, bpe-dropout was developed for the purpose of regularization, that is, to improve a model's robustness to noise and segmentation errors. conversely, in this study, we used this technique as a means of training a language model that is compatible with both subword-and character-level segmentations. our idea originated from the characteristics of the segmentation performed by bpe-dropout (figure 3 (b) ), that is, a sequence of two subwords ne west can be segmented as a sequence of both characters and subwords ne w e st. we expect that a model trained with such a mixed segmentation can be compatible with both subword-and character-level segmentation. as a result, the need for independently pretraining language models for dedicated types of segmentations can be eliminated, and thus, the computational cost of pretraining can be halved. our method is extremely straightforward: during pretraining, we simply apply the off-the-shelf bpedropout algorithm to the input. thus, the method requires neither modification of the model architecture nor the addition of model parameters. once the model is pretrained, we set the dropout probability p according to the desirable segmentation, and then perform finetuning. for example, if a task of interest requires character-level segmentation, we set @xmath0.0 and then finetune the model. ## experiments we demonstrate the effectiveness of unified segmentation on pretrained bert @xcite models on japanese benchmark datasets. specifically, we demonstrate that unified segmentation achieves performance comparable to that of both subword-and character-level bert. it should be noted that the aim of unified segmentation is neither to achieve state-of-the-art performance on benchmark datasets, nor to outperform its counterparts (i.e., bert models pretrained on either subwordor character-level segmentation alone). instead, we aim to achieve comparable performance. this is because, given such results, the independent training of subword-and character-level bert models can be eliminated, thereby saving the computational cost of pretraining. ## conclusion in this study, we investigated the effectiveness of incorporating subword regularization as a means of training a language model with unified segmentation. our method enables the pretraining of a single model that is applicable to both subword-and character-level segmentation. this can significantly reduce the computational cost of pretraining. as a future work, we will investigate the effectiveness of this method to the pretraining of other language models, such as the encoder-decoder model @xcite and decoder-only model @xcite .