Overlap-based Vocabulary Generation Improves Cross-lingual Transfer Among Related Languages (2022)

Abstract: Pre-trained multilingual language models such as mBERT and XLM-R have demonstrated great potential for zero-shot cross-lingual transfer to low web-resource languages (LRL). However, due to limited model capacity, the large difference in the sizes of available monolingual corpora between high web-resource languages (HRL) and LRLs does not provide enough scope of co-embedding the LRL with the HRL, thereby affecting the downstream task performance of LRLs. In this paper, we argue that relatedness among languages in a language family along the dimension of lexical overlap may be leveraged to overcome some of the corpora limitations of LRLs. We propose Overlap BPE (OBPE), a simple yet effective modification to the BPE vocabulary generation algorithm which enhances overlap across related languages. Through extensive experiments on multiple NLP tasks and datasets, we observe that OBPE generates a vocabulary that increases the representation of LRLs via tokens shared with HRLs. This results in improved zero-shot transfer from related HRLs to LRLs without reducing HRL representation and accuracy. Unlike previous studies that dismissed the importance of token-overlap, we show that in the low-resource related language setting, token overlap matters. Synthetically reducing the overlap to zero can cause as much as a four-fold drop in zero-shot transfer accuracy.

## introduction zero-shot cross-lingual transfer is the ability of a model to learn from labeled data in one language and transfer the learning to another language without any labeled data. transformer @xcite based multilingual models pre-trained on unlabeled data from multiple languages are the stateof-the-art means for cross-lingual transfer @xcite . while pretraining based cross-lingual transfer holds great promise for low web-resource languages (lrls), such techniques are found to be more effective for transfer within high web-resource languages (hrls) @xcite . vocabulary generation is an important step in multilingual model training, where vocabulary size directly impacts model capacity. usually, the vocabulary is generated from a union of hrl and lrl data. this often results in under-allocation of vocabulary bandwidth to lrls, as lrl data is significantly smaller in size compared to hrl. this under-allocation of model capacity results in lower lrl performance @xcite , as mentioned previously. in response, prior research has explored development of region-specific models (antoun et al.; @xcite , generating vocabulary specific to language clusters @xcite , and exploring relatedness among languages to build better lms for lrls @xcite . however, none of these methods have utilized relatedness among languages for better vocabulary generation during multilingual pre-training. in this paper, we hypothesize that exploiting language relatedness can result in an overall more effective vocabulary, which is also better representative of lrls. closely related languages (e.g., languages belonging to a single family) have common origins for words with similar meanings. we show some examples across three different families of related languages in given text from a mix of high and low webresource languages (hrl and lrl, respectively), byte pair encoding (bpe) @xcite and its variants like wordpiece @xcite and sentencepiece @xcite prefer frequent tokens, most of those from the hrls. this would cause most long hrl tokens to get included, leaving only a limited budget of short tokens for the lrl. any sub-token level overlap between hrl and lrl could get lost in this process. in a zero-shot setting, since available supervision is hrl based, this creates a bottleneck when transferring supervision from hrl to lrls. oversampling lrls is a common strategy to offset this imbalance but that hurts hrl performance as shown in @xcite . in this paper, we propose overlap bpe (obpe). obpe chooses a vocabulary by giving token overlap among hrl and lrls a primary consideration. obpe prefers vocabulary units which are shared across multiple languages, while also encoding the input corpora compactly. thus, obpe tries to balance the trade-off between cross-lingual subword sharing and the need for robust representation of individual languages in the vocabulary. this re-sults in a more balanced vocabulary, resulting in improved performance for lrls without hurting hrl accuracy. recently k et al. (2020); conneau et al. (2020b) concluded that token overlap is unimportant for cross-lingual transfer. however, they studied language pairs where either both languages had a large corpus, or where the languages were not sufficiently related. we focus on related languages within a family and observe drastic drop in zeroshot accuracy when we synthetically reduce the overlap to zero (58% f1 drops to 17% for ner, 71% drops to 30% for text classification). this paper offers the following contributions • we present obpe, a simple yet effective modification to the popular bpe algorithm to promote overlap between lrls and a related hrl during vocabulary generation. obpe uses a generalized mean based formulation to quantify token overlap among languages. ## related work transformer-based multilingual language models such as mbert @xcite and xlm-r @xcite ) are now established as the de-facto method for zero-shot cross-lingual transferability, and thus hold promise for low resource domains. however, recent studies have indicated that even the current state-of-the-art models such as xlm-r (large) do not yield reasonable transfer performance across low resource target languages with limited data @xcite . this has led to a surge of interest in enhancing cross-lingual transfer of multilingual models to the low-resource setting. we categorize existing work based on the stage of the pre-training pipeline where it is relevant: input data in the data creation stage, @xcite propose over-sampling of lrl documents to improve lrl representation in the vocabulary and pre-training steps. @xcite specifically target related languages and propose transliteration of lrl documents to the script of related hrl for greater lexical overlap. we deploy both these tricks in this paper. tokenization @xcite study that even the tokenization step could have a crucial impact on performance accrued to each language in a multilingual models. they propose the use of dedicated tokenizer for each language instead of the automatically generated multilingual mbert tokenizer. however, they continue to use the default mbert vocabulary generator. @xcite highlighted the importance of subword tokens in the vocabulary and proposed use of the bpe algorithm @xcite for efficiently growing such a vocabulary incrementally. variants like wordpiece @xcite and sentencepiece @xcite either build on top of bpe or follow a very similar process. @xcite is a variant method that chooses tokens based on unigram lm score. we obtained better results with bpe and continued with that. all these bpe variants incrementally add subwords based on overall frequency in the combined corpus, and they all ignore language boundaries. chung et al. (2020) observed that such a combined approach could under-represent several languages, and proposed instead to separately create vocabularies for clusters of related languages and take a union of each cluster-specific vocabulary. however, within each cluster they continue to use the default vocabulary generator. our approach can be used as a drop-in replacement to further enhance the quality of the cluster-specific vocabulary that they obtain. @xcite ; @xcite propose a soft-decoupled encoding approach for exploiting subword overlap between lrls and hrls. however, their focus is nmt models and does not easily integrate in existing multilingual models such as mbert. @xcite targets tok-enization compatibility based purely on vocabulary size and does not focus on choosing the tokens that go in the vocabulary. pre-training and adaptation several previous works have proposed to include additional alignment loss between parallel @xcite or pseudo-parallel @xcite sentences to co-embed hrls and lrls. another approach is to design language-specific adapter layers @xcite @xcite that can be easily fine-tuned for each new language. @xcite leverages the pre-trained embeddings of lexically overlapping tokens between the vocabulary of pre-trained model and that of unseen target language to initialize the corresponding embeddings of target language. however, they did not attempt to increase the fraction of such tokens in the vocabulary. ## overlap-based vocabulary generation we are given monolingual data d 1 , ..., d n in a set of n languages @xmath0, ..., l n } and a vocabulary budget v. our goal is to generate a vocabulary v that when used to tokenize each d i in a multilingual model would provide cross-lingual transfer to lrls from related hrls. we use l lrl to denote the subset of the n languages that are low-resource, the remaining languages l -l lrl are denoted as the set l hrl of high resource languages. existing methods of vocabulary creation start with a union d of monolingual data d 1 , ..., d n , and choose a vocabulary v that most compactly represents d. we first present an overview of bpe, a popular algorithm for vocabulary generation. ## experiments we evaluate by measuring the efficacy of zeroshot transfer from the hrl on four different tasks: named entity recognition (ner), part of speech tagging (pos), text classification(tc), and cross-lingual natural language inference (xnli). through our experiments, we evaluate the following questions: 1. is obpe more effective than bpe for zeroshot transfer? (section 4.2) 2. what is the effect of token overlap on overall accuracy? (section 4.3) 3. how does increased lrl representation in the vocabulary impact accuracy? (section 4.4) we report additional ablation and analysis experiments in section 4.5. ## conclusion in this paper, we address the problem of crosslingual transfer from hrls to lrls by exploiting relatedness among them. we focus on lexical overlap during the vocabulary generation stage of multilingual pre-training. we propose overlap bpe (obpe), a simple yet effective modification to the bpe algorithm, which chooses a vocabulary that maximizes overlap across languages. obpe encodes input corpora compactly while also balancing the trade-off between cross-lingual subword sharing and language-specific vocabularies. we focus on three sets of closely related languages from diverse language families. our experiments provide evidence that obpe is effective in leveraging overlap across related languages to improve lrl performance. in contrast to prior work, through controlled experiments on the amount of token overlap between two related hrl-lrl language pairs, we establish that token overlap is important when a lrl is paired with a related hrl.