How Effective is Byte Pair Encoding for Out-Of-Vocabulary Words in Neural Machine Translation? (2022)

Abstract: Neural Machine Translation (NMT) is an open vocabulary problem. As a result, dealing with the words not occurring during training (a.k.a. out-of-vocabulary (OOV) words) have long been a fundamental challenge for NMT systems. The predominant method to tackle this problem is Byte Pair Encoding (BPE) which splits words, including OOV words, into sub-word segments. BPE has achieved impressive results for a wide range of translation tasks in terms of automatic evaluation metrics. While it is often assumed that by using BPE, NMT systems are capable of handling OOV words, the effectiveness of BPE in translating OOV words has not been explicitly measured. In this paper, we study to what extent BPE is successful in translating OOV words at the word-level. We analyze the translation quality of OOV words based on word type, number of segments, cross-attention weights, and the frequency of segment n-grams in the training data. Our experiments show that while careful BPE settings seem to be fairly useful in translating OOV words across datasets, a considerable percentage of OOV words are translated incorrectly. Furthermore, we highlight the slightly higher effectiveness of BPE in translating OOV words for special cases, such as named-entities and when the languages involved are linguistically close to each other.

## introduction one of the key challenges of neural machine translation (nmt) @xcite is vocabulary sparsity; irrespective of the amount of data available for training. as a consequence, all nmt models suffer from out-of-vocabulary (oov) words. accordingly, a significant proportion of sentences in the test set have oov words. even with millions of sentence pairs in training data, 1 15% of test sentences contain oov words, while with limited training data, 2 oov words appear in more than 60% of the test sentences. earlier approaches to tackling the oov problem include using a very large vocabulary @xcite , backing off to a dictionary look-up @xcite , and copying oov words from source to the target sentence @xcite . however, most recent approaches are based on splitting the words into smaller units and can be divided into: language-specific approaches @xcite , language-agnostic approaches (sennrich et al., 1 russian-english from wmt 2 kazakh-english from wmt ## experimental setup datasets we use german-english, russian-english, and romanian-english as language pairs for our experiments. the main reasons to select these languages are twofold: the data sizes are large enough to eliminate the effect of the amount of data on translation quality, especially for rare words. also, since two common types of oovs are inflected and compound words in general, we choose russian, romanian, and german with varying degrees of morphology and compound words and as a representative of slavic, romance, and germanic languages, respectively. for the russian-english direction, we use the yandex corpus, common crawl, news commentary, and wiki titles from wmt2020. we preprocess the data by limiting the length of the sentences to 200 tokens and removing sentence pairs with a source/target length ratio exceeding 1.5, following previous work @xcite . we use the concatenation of newstest2017, newstest2018, and newstest2019 for evaluation. as german-english training set we use europarl, common crawl, and news commentary from wmt2017 and for the test set we use newstest2014. also for romanian-english, we use all available training data from wmt2016 and newstest2016 for evaluation purposes. the data prepossessing pipeline for german-english and russian-english is similar to russian-english. we end up with 2.64m, 3.95m, and 612k training sentences and 1078 / 1363, 1830 / 2411, and 926 / 1385 oov word 3 throughout the paper, oov refers to an actual non-segmented out-of-vocabulary word, unless otherwise stated. ## data annotation in order to analyze bpe usability for different oov words, we randomly sample 400 unique oov words from the set of all oov words for each language. first, we manually label oov types. for this annotation process, we employ a highly qualified native annotator for each language. it is worthwhile mentioning that one given oov word may belong to more than one category. then, we extract their corresponding translation from the reference sentence. below, we explain how we extract the translations of oov words from the hypothesis sentences and also how we assign quality labels to them in more details . ## conclusion in this paper, we analyze the translation quality of oov words in bpe segmented datasets. our analysis shows that while bpe has brought significant improvements to nmt in terms of automatic evaluation metrics, the translation quality still suffers from oov words. our experiments show that splitting oov words into subwords is more effective where there is higher degree of language similarity. also, there is a strong correlation between the translation quality and the amount of attention received by oov words. on the other hand, there is no evidence to support that the translation quality is dependent on the frequency of bpe segment n-grams in the training data. moreover, we find that the translation quality is better for named entity oov words compared to other word types, especially for language pairs with more lexical similarity. furthermore, we showed that automatic evaluation metrics such as bleu are not able to capture the effectiveness of a word segmentation method for translations of oovs. therefore, manual analysis on the translation quality of oov words is essential to compare different approaches, although it needs annotators in each language and it is very laborious. in future work, we compare suggested approaches for morphologically-rich languages at the word level.