BERTS eg: BERT Based Unsupervised Subword Segmentation for Neural Machine Translation (2022)

Abstract: Existing subword segmenters are either 1) frequency-based without semantics information or 2) neural-based but trained on parallel corpora. To address this, we present BERTSeg, an unsupervised neural subword segmenter for neural machine translation, which utilizes the contextualized semantic embeddings of words from characterBERT and maximizes the generation probability of subword segmentations. Furthermore, we propose a generation probability-based regularization method that enables BERTSeg to produce multiple segmentations for one word to improve the robustness of neural machine translation. Experimental results show that BERTSeg with regularization achieves up to 8 BLEU points improvement in 9 translation directions on ALT, IWSLT15 Vi->En, WMT16 Ro->En, and WMT15 Fi->En datasets compared with BPE. In addition, BERTSeg is efficient, needing up to 5 minutes for training.

## introduction subword segmentation is the task of splitting a word into smaller n-gram character units called subwords @xcite . it alleviates the out-of-vocabulary (oov) problem in neural machine translation (nmt) @xcite @xcite by enabling an nmt system to have a fixed-size vocabulary while being able to handle all possible words regardless of their frequencies. studies in subword segmentation fall into two categories: frequency-based approaches and neural network-based approaches. frequency-based approaches @xcite @xcite adopt a greedy algorithm that generates the vocabulary with frequent subword fragments in the corpus during training and merges adjacent highfrequency fragments starting from characters recursively during inference. among these methods, bpe-dropout @xcite and senten-cepiece with regularization @xcite generate multiple segmentations by random sampling. frequency-based approaches do not consider semantic information of the subwords, therefore the generated segmentation is not linguistically motivated. for example, the word "fellowships" is segmented into "fell/ows/hip/s" by bpe whereas "fellow/ships" is a more linguistically motivated segmentation. neural approaches such as dpe @xcite implicitly considers the contextual semantic information of subwords by maximizing the generation probabilities of the target language sentences conditioned on the source language sentences. however, it trains on parallel sentences, which poses a problem for low-resource languages. dpe is slow because it calculates the probabilities of all possible sentence segmentations, therefore, not practical in high-resource scenarios. we propose bertseg, an unsupervised neural subword segmenter that leverages contextualized word representations from the pre-trained model, characterbert @xcite . it combines the advantages of frequency-based and neural he leaned closer as well, watching. especially watching my son. she had someone watching him. w a t c h i n g watching watching watching watching average embedding hidden state characterbert embedding transformer decoder w a t c h i n g wat at ch hi in watch atch ching ing optimal path @xmath0: bertseg architecture. the encoder is a characterbert that generates average embeddings for one word in different contexts. the transformer decoder takes characters as input and generates probabilities of the next subword. during training, the objective is to maximize the probabilities of all possible segmentations. during inference, the model retraces the optimal segmentation. in our task, this enables the model to generate linguistically intuitive segmentations rather than highfrequency fragments, as shown in furthermore, we propose a subword regularization method bertseg-regularization which enables the model to produce multiple segmentations based on segmentation probabilities to improve the robustness of nmt, as represented in experimental results on the low-resource alt and high-resource iwslt and wmt datasets show approximately 5 and 2 bleu points improvement over bpe with statistical significance @xmath1.001 and outperforms all other baseline methods. moreover, our method is efficient because of leveraging the word-level data. bertseg requires up to 5 minutes to train, whereas dpe requires hours to days to train and volt also costs 30 minutes to generate the optimal vocabulary. finally, analysis shows high generalizability on unseen words. ## experimental settings datasets ## related work early nmt studies apply word-level vocabulary to represent only frequent words, which causes the out-of-vocabulary (oov) problem @xcite . to address this, character-based @xcite @xcite , hybrid word-character based @xcite , or utf-8 based (shaham and levy, 2021) nmt models were proposed. however, the resultant long input/output sequences increase the model and computational complexity. subword segmentation methods address the oov problem by segmenting words into subwords that are in a fixed vocabulary of character n-grams. bpe @xcite generates the subword vocabulary by first splitting all the sentences into characters, then iteratively saving the most frequent adjacent pairs into the vocabulary and merging them, until reaching the desired size. each test sentence is segmented similarly. word-piece @xcite and senten-cepiece @xcite are another two widely-used subword methods. among the subword methods, bpe @xcite does not model the input sequence whereas sentencepiece @xcite applies a unigram model to output probabilities of each segmentation. based on sequence modeling via segmentations theory @xcite , the generation probability of a target sentence can be calculated by the sum of probabilities of all its possible segmentations. dpe @xcite ) models the whole target sentence conditioned on the source sentence. however, we show that modeling words conditioned on their semantic embedding is a more efficient way. regularization as data augmentation can boost performance. bpe-dropout @xcite randomly drops subword merge operation. spmregularization @xcite generates multiple segmentations with their probabilities. leveraging the dynamic programming algorithm, we retrace the global best-n segmentations with the highest probabilities in polynomial time. ## conclusion and future work we proposed bertseg, an unsupervised neural subword segmenter for nmt, together with a regularization algorithm. mt results showed significant improvement over frequency-based and neural network-based methods. the training is efficient even compared with non-neural methods. to address the limitations shown in appendix a, future works include eliminating the dependency on the bpe vocabulary, extending to a multilingual segmenter with mbert @xcite embeddings, and applying it to other generation tasks.