Bilingual Subword Segmentation for Neural Machine Translation (2020)

Abstract: This paper proposed a new subword segmentation method for neural machine translation, “Bilingual Subword Segmentation,” which tokenizes sentences to minimize the difference between the number of subword units in a sentence and that of its translation. While existing subword segmentation methods tokenize a sentence without considering its translation, the proposed method tokenizes a sentence by using subword units induced from bilingual sentences; this method could be more favorable to machine translation. Evaluations on WAT Asian Scientific Paper Excerpt Corpus (ASPEC) English-to-Japanese and Japanese-to-English translation tasks and WMT14 English-to-German and German-to-English translation tasks show that our bilingual subword segmentation improves the performance of Transformer neural machine translation (up to +0.81 BLEU).

## introduction subword units have recently been widely used in neural machine translation (nmt) to solve open vocabulary problems. byte pair encoding (bpe) @xcite is a dominant subword segmentation method for nmt, but it is designed for segmented languages in which words are divided by spaces. @xcite has proposed a subword segmentation method based on a unigram language model, that can be applied to non-segmented languages such as chinese and japanese. both bpe and the unigram language model tokenize sentences by minimizing the number of segments under a limitation on subword vocabulary size, which relies on a data compression principle. in these existing segmentations, a sentence is segmented without considering its translation, and therefore the segmented sentence might not be optimal for nmt. this paper proposes a new subword segmentation method for nmt, "bilingual subword segmentation," which tokenizes sentences by using subword units induced from bilingual sentences. the proposed method is based on a unigram language model like @xcite because we aim to improve translation performance for non-segmented languages 1 . in particular, the proposed segmentation tokenizes bilingual sentences (i.e., training data for nmt) by selecting subword sequence pairs with similar numbers of segments, from segmentation candidates of the source and target language sentences obtained by a unigram language model. for segmentation of monolingual source language sentences (i.e., test data for nmt), an lstm-based subword segmenter for the source language is preliminarily learned from the source side of segmented bilingual sentences, and monolingual source language sentences are tokenized by the learned subword segmenter. our bilingual segmentation encourages one-to-one mappings between segments across languages because it minimizes the difference between the number of segments in a sentence and that of its translation. as a result, subword units segmented by our bilingual segmentation can be expected to be more helpful for nmt than conventional subword units. for example, consider the situation in which two japanese compound words "設計法 (design method)" and "計測装置 (measurement instrument)" occur many times in training data. when a conventional subword segmentation method is used, each must be merged into one subword unit because the conventional method minimizes the number of segments according to the data compression principle. as a result, these segments in training data are useless for translation of "計測法 (measurement method)". on the other hand, when the proposed segmentation method is used, these segments must be decomposed into "設計 (design) 法 (method)" and "計測 (measurement) 装置 (instrument)", respectively, because our method puts the number of japanese subword units near the number of english subword units (i.e., 2). thus, segmented training data would be useful for the translation of "計測 (measurement) 法 (method)" because an nmt model can learn translations of "計測 (measurement)" and "法 (method)" from constituents of "計測 (measurement) 装置 (instrument)" and " 設計 (design) 法 (method)", respectively. we evaluate the proposed subword segmentation method on wat japanese-to-english (ja-en) and english-to-japanese (en-ja) translation tasks with the aspec @xcite . we also evaluate our proposed method on wmt14 english-to-german (en-de) and german-to-english (de-en) translation tasks. these experiments show that the proposed subword segmentation method improves the performance of transformer nmt @xcite on all translation tasks (up to 0.81 point improvement in bleu). ## subword segmentation based on the unigram language model this section describes the subword segmentation method based on the unigram language model @xcite , that is the basis of our proposed segmentation method. the unigram language model assumes that each subword occurs independently and that the occurrence probability of a subword sequence p (x) is formulated as follows: the unigram language model's advantages are that it can be learned from raw sentences and that it can tokenize sentences in a non-segmented language such as chinese and japanese without requiring a word segmenter. ## proposed model: bilingual subword segmentation this section proposes "bilingual subword segmentation," which tokenizes sentences by using subword units induced from bilingual sentences. in particular, our proposed segmentation tokenizes sentences so unigram language model bilingual subword segmentation for training data: sampling from the k-best pairs on the basis of the number of tokens bilingual-subword-segmented training data: d source: { f (s) } |d| @xmath0: { ê(s) } |d| @xmath1: d nmt model character-based bilstm training from @xmath2, ê(s) ) } |d| @xmath3: as to minimize the difference between the number of a sentence's subword units and that of its translation. in segmentation of training data for nmt, a sentence can be tokenized while referring to its translation (i.e., a sentence in the other language). on the other hand, in segmentation of test data for nmt, translations cannot be given. to address the different situations, we propose a bilingual subword segmentation method for training data and test data, as illustrated in figures 1(a ) and (b), respectively. note that our proposed method does not depend on an nmt model or a training method; therefore, it can be applied only by replacing a conventional subword segmentation with our proposed subword segmentation. ## conclusion in this paper, we proposed a new subword segmentation method for nmt, "bilingual subword segmentation," which tokenizes sentences by using subword units induced from bilingual sentences. experiments on wat aspec ja-en and en-ja tasks and wmt14 en-de and de-en translation tasks show that the proposed method improves transformer nmt translation performance. through experiments and discussions, we found that translation performance improves by tokenizing sentences so as to minimize the difference between the number of subword units of a sentence and that of its translation. in future work, we would like to confirm our proposed method's effectiveness for other language pairs.