Multi-view Subword Regularization (2021)

Abstract: Multilingual pretrained representations generally rely on subword segmentation algorithms to create a shared multilingual vocabulary. However, standard heuristic algorithms often lead to sub-optimal segmentation, especially for languages with limited amounts of data. In this paper, we take two major steps towards alleviating this problem. First, we demonstrate empirically that applying existing subword regularization methods (Kudo, 2018; Provilkov et al., 2020) during fine-tuning of pre-trained multilingual representations improves the effectiveness of cross-lingual transfer. Second, to take full advantage of different possible input segmentations, we propose Multi-view Subword Regularization (MVR), a method that enforces the consistency of predictors between using inputs tokenized by the standard and probabilistic segmentations. Results on the XTREME multilingual benchmark (Hu et al., 2020) show that MVR brings consistent improvements of up to 2.5 points over using standard segmentation algorithms.

## introduction multilingual pre-trained representations @xcite @xcite are now an essential component of state-of-the-art methods for cross-lingual transfer @xcite . these methods pretrain an encoder by learning in an unsupervised way from raw textual data in up to hundreds of languages which can then be fine-tuned on annotated data of a downstream task in a high-resource language, often english, and transferred to another language. in order to encode hundreds of languages with diverse vocabulary, it is standard for such multilingual models to employ a shared subword vocabulary jointly learned on the en excitement fr excita/tion de auf/re/gung pt excita/ção el en/j/ousi/asmïc ru волн/ение despite this, there is not much work studying or improving subword segmentation methods for cross-lingual transfer. @xcite empirically compare several popular word segmentation algorithms for pretrained language models of a single language. several works propose to use different representation granularities, such as phrase-level segmentation @xcite or character-aware representations @xcite for pretrained language models of a single highresource language, such as english or chinese only. however, it is not a foregone conclusion that methods designed and tested on monolingual models will be immediately applicable to multilingual representations. furthermore, they add significant computation cost to the pretraining stage, which is especially problematic for multilingual pretraining on hundreds of languages. the problem of suboptimal subword segmentation has drawn more attention in the context of neural machine translation (nmt). specifically, subword regularization methods have been proposed to improve the nmt model of a single language pair by randomly sampling different segmentations of the sentences during training @xcite . however, these methods have not been applied to multilingual nmt or pretrained language models and it is similarly not clear if they are useful for cross-lingual transfer. in this paper, we make two contributions to close this gap. first, we perform the first (to our knowledge) empirical examination of subword regularization methods on a variety of cross-lingual transfer tasks from the xtreme benchmark @xcite . we demonstrate that despite its simplicity, this method is highly effective, providing consistent improvements across a wide variety of languages and tasks for both multilingual bert (mbert; @xcite and xlm-r @xcite models. analysis of the results shows that this method is particularly effective for languages with non-latin scripts despite only being applied during english fine-tuning. further, we posit that naively applying probabilistic segmentation only during fine-tuning may be sub-optimal as it creates a discrepancy between the segmentations during the pretraining and finetuning stages. to address this problem, we propose multi-view subword regularization (mvr; fig. 1 ), a novel method-inspired by the usage of consistency regularization in semi-supervised learning methods @xcite -which utilizes both the standard and probabilistically segmented inputs, enforcing the model's predictions to be consistent across the two views. such consistency regularization further improves accuracy, with mvr finally demonstrating consistent gains of up to 2.5 points over the standard practice across all models and tasks. we analyze the sources of the improvement from consistency regularization and find that it can be attributed to both label smoothing and self-ensembling. ## background: subword segmentation here, we first discuss two common deterministic segmentation methods based on byte pair encoding (bpe) and unigram language models (ulm), discuss their probabilistic variants, and explain how to incorporate them in training. ## subword regularization for cross-lingual transfer while sub-optimal word segmentation is a challenge in monolingual models, it is an even bigger challenge for multilingual pretrained models. these models train a shared subword segmentation model jointly on data from many languages, but the segmentation can nonetheless be different across languages, stemming from two main issues. first, the granularity of segmentation differs among languages, where the segmentation model tends to over-segment low-resource languages that do not have enough representation in the joint training data @xcite . fig. 2 shows the distribution of words from languages from different language families based on the number of subwords they are split into. 2 we can see that the majority of english words are not segmented at all, while many languages only have less than half of the words unsegmented. notably, even though burmese (my) is a language with little inflectional morphology, almost a quarter of the words are segmented into more than nine subwords. second, the segmentation might still be inconsistent between different languages even if the granularity is similar, as explained in tab. 1. for example, neither the english word "excitement" nor the same word in french "excita/tion" are overly segmented, but segmenting the english word into "excite/ment" would allow the model to learn a better cross-lingual alignment. despite these issues, few methods have tried to address this subword segmentation problem for multilingual pretrained models. @xcite propose to adapt a pretrained multilingual model to a new language by augmenting the vocabulary with a new subword vocabulary learned on the target language, but this method might not help for languages other than the target language it adapts to. @xcite propose to separately construct a subword segmentation model for each cluster of related languages for pretraining the multilingual representations. however, directly modifying the word segmentation requires retraining large pretrained models, which is computationally prohibitive in most cases. in this paper, we instead propose a more efficient approach of using probabilistic segmentation during fine-tuning on labeled data of a downstream task. as mismatch in segmentation is one of the factors harming cross-lingual transfer, we expect a model that becomes more robust to different varieties of segmentation in one language will be more accommodating to differing segmentations in other languages during inference. despite the simplicity of this method it is, as far as we are aware, unattested in the literature, and we verify in § 5.3 that it significantly improves the cross-lingual transfer performance of multilingual pretrained models. ## multi-view subword regularization previous attempts at sr have mainly applied it to models trained from scratch for tasks such as mt. however, the situation is somewhat different when fine-tuning pre-trained representations, in which case the original pre-trained models are generally not trained on sampled segmentations. this discrepancy between the segmentation of the english labeled data and the segmentation of english monolingual data during pretraining might hurt the ability of the model to take full advantage of the parameters learned during the pretraining stage. to reduce this pretraining-fine-tuning discrepancy, we propose multi-view subword regularization (mvr), a method for learning from multiple segmented versions of the same data and enforcing the consistency of predictions over different segmentations. ## analyses in this section, we perform several analyses to better understand the behavior and root causes of the accuracy gains realized by our method. ## related work several works propose to optimize subwordsensitive word encoding methods for pretrained language models. ma et al. (2020) uses convolutional neural networks @xcite on characters to calculate word representations. @xcite propose to add phrases into the vocabulary for chinese pretrained language models. however, they focus on improving the vocabulary of pretrained representations of a single language, and they require modification to the model pretraining stage. @xcite propose to cluster related languages together and run subword vocabulary construction on each language cluster when constructing vocabularies for mbert. their method is also applied at the pretraining stage and could be combined with our method for potential additional improvements. our method is also related to prior work that optimize word representations for nmt and language modeling. character level embeddings have been utilized instead of subword segmentation for nmt @xcite @xcite and language modeling @xcite . @xcite propose a multilingual word embedding method for nmt that relies on character n-gram embedding and a latent semantic embedding shared between different languages. @xcite show that character n-gram based embedding performs better than bpe for morphologically rich languages. @xcite propose to learn the optimal segmentation given a subword vocabulary for nmt. our method is inspired by semi-supervised learning methods that enforce model consistency on unlabeled data. several self-training methods utilize unlabeled examples to minimize the distance between the model predictions based on the unlabeled example and a noised version of the same input @xcite @xcite . @xcite use knowledge distillation on unlabeled data to adapt models to a new language. @xcite propose to mask out different parts of the unlabeled input and encourage the model to make consistent prediction given these different inputs. these methods all focus on semi-supervised learning, while our method regulates model consistency to mitigate the subword segmentation discrepancy between different languages. ## conclusion we believe that the results in this paper convincingly demonstrate that standard deterministic subword segmentation is sub-optimal for multilingual pretrained representations. even incorporating simple methods for subword regularization such as bpe-dropout at fine-tuning can improve the crosslingual transfer of pretrained models, and our proposed multi-view subword regularization method further shows consistent and strong improvements over a variety of tasks for models built upon different subword segmentation algorithms. going forward, we suggest that some variety of subword regularization, mvr or otherwise, should be a standard component of the fine-tuning of pre-trained representations that use subword segmentation.