Optimizing Word Alignments with Better Subword Tokenization (2021)

Abstract: Word alignment identify translational correspondences between words in a parallel sentence pair and are used and for example and to train statistical machine translation and learn bilingual dictionaries or to perform quality estimation. Subword tokenization has become a standard preprocessing step for a large number of applications and notably for state-of-the-art open vocabulary machine translation systems. In this paper and we thoroughly study how this preprocessing step interacts with the word alignment task and propose several tokenization strategies to obtain well-segmented parallel corpora. Using these new techniques and we were able to improve baseline word-based alignment models for six language pairs.

## introduction word alignment is a basic task in multilingual natural language processing (nlp) and is used, for instance, to learn bilingual dictionaries, to train statistical machine translation (smt) systems @xcite , to filter out noise from translation memories @xcite or in quality estimation applications @xcite . word alignment can also serve to explain mt decisions @xcite . given pairs associating a sentence in a source language and a translation in a target language, word alignment aims to identify translational equivalences at the level of individual word tokens and has been initially approached with generative probabilistic models learning alignment in an unsupervised manner @xcite . with rapid advances in neural based nlp, word alignment has recently regained some traction @xcite and improvements of the state of the art for multiple language pairs have been reported thanks to neuralized generative models @xcite @xcite , pre-trained multilingual embeddings @xcite @xcite or more powerful architectures based on the transformer translation model of @xcite , as reported for instance by @xcite ; @xcite and @xcite . in addition to using neural architectures, these new models differ from past approaches in that they compute alignments based on a decomposition into subword units @xcite , which makes it possible to easily accommodate open-ended vocabularies and mitigate issues related to the alignment of unknown words, which has always been a challenge for discrete models. another interesting property of subword units in the context of word alignment is that (a) they ease the generation of many-to-one / one-to-many links, which are difficult to handle in standard asymmetric models such as ibm-1 and ibm-4 @xcite @xcite ; (b) they also enable to actively manipulate the lengths of the source and target sentences so as to make them more even, arguably a facilitating factor for alignment and translation models @xcite . in this work, we take a closer look at the interaction between alignment and subword tokenization and try to address the following research questions: how much of the reported improvements in alignment performance can be linked to subword splitting? which issue(s) of basic alignment models do they mitigate? is it possible to design more active segmentation strategies that would target the alignment problem for specific language pairs? our conclusions rests on the analysis of a systematic study of word alignment for 6 language pairs from multiple language families. we notably show that subword tokenization also help discrete alignment models. we also study techniques aimed at optimizing tokenization, which enable us to further improve the alignment accuracy and mitigate the problems cause by rare / unaligned words. this paper is organized as follows: in § 2 we review the pitfalls of generative word alignment models, and analyse in § 3 how their performance vary with changing subword tokenizations. these analyses help to understand why such preprocessing actually improves word based models. our main proposals are sketched in § 4, where we show how to optimize subword tokenization for better alignments. in § 5, we then briefly review related work, before concluding in § 6. ## pitfalls and limitations of word alignments models in this section, we experiment with well-known word alignment packages (fastalign @xcite , giza++ @xcite , eflomal @xcite as well as simalign (jalili sabet et al., 2020) foot_0 ), outlining difficult issues for word alignment models such as the prediction of null links, of many-to-one links, as well as the alignment of rare words. detailed analyses are in @xcite . asymmetric alignment models associate each source word with exactly one target word; such alignments are denoted as @xmath0, when english is the source language. as a preamble, we start with our data condition. ## optimizing subword tokenization in this section, we build on the intuition that pairs of sentences which differ in length are difficult to align @xcite , suggesting that subword splitting should be used to make the algorithm 1 finding the vocabulary pair minimizing the average length differences require: α: source side vocabulary size; β: target side vocabulary size ε: search space radius (@xmath1; ρ: step size (@xmath2; ensure: 1000 ≤ α, β ≤ 50000 while ε ≥ 100 do for ν ∈ {α -ε, α, α + ε}, µ ∈ {β -ε, β, β + ε} do if f (ν, µ) < f (α, β) then α = ν; β = µ; = 2000 end if end for if α and β remain the same then ε = ε -ρ end if end while ## related work subword segmentation is introduced in the context of neural translation in @xcite , using a reimplementation foot_4 of the byte pair encoding algorithm of @xcite . bpe is a greedy, bottom up algorithm that recursively aggregates frequent bigrams into new symbols, and is thoroughly analyzed in @xcite . the main alternative is sentencepiece introduced in @xcite , which implements a form of variable-length probabilistic unigram model, which can be traced back to @xcite . with bpe/unigram subtokenization becoming a standard for many applications, several studies have started to investigate more closely the impact on these preprocessing decisions on the final performance. the implementation of sentencepiece foot_5 reports a large number of mt experiments aimed to compare bpe and unigram in multiple conditions, concluding that both yield comparable bleu scores across the board when used with a fixed tokenization in words. the shortcomings of bpe/unigram segmentations have been the subject of several studies, reporting comparisons with (a) linguistic segmentations @xcite @xcite ) and (b) alternative preprocessing schemes such as character-based models ( eg. in sennrich (2017); sajjad et al. (2017); cherry et al. (2018) ). @xcite conduct a systematic exploration considering a large numbers of vocabulary sizes to better understand its impact on nmt performance, comparing several nmt architectures such as shallow/deep-transformer, tiny/shallow/deep-lstm. @xcite evaluate the impact of tokenization on language model pre-training. they conclude that tokenization encodes a surprising amount of inductive bias and that lm-based tokenization produces subword units that qualitatively align with morphology much better than those produced by bpe, suggesting that the latter is better than the former for pretrained models. the work of @xcite is our main inspiration, and explore ways to optimize the subword segmentation, using, as we do, sampling techniques and length-based heuristics to chose the most appropriate target for each source, and observing gains in translation performance. ## conclusion and outlook in this work, we have studied the interaction between word alignment and word segmentation based on two algorithms (bpe and unigram) and multiple word aligners. using smaller units notably mitigate issues with rare/unknown words; shorter units also help to retrieve more correct links for non-canonical (one-to-many, many-to-one) alignment links. based on these observations, we have thoroughly analyzed the variation of alignment scores with respect to vocabulary sizes, showing that the word-based segmentation was less than optimal. we have finally explored various ways to actively optimize the subword tokenization; promising results in this direction have been obtained with the unigram algorithm, owing to its ability to generate multiple high-probability segmentations. we have notably found that adjusting length differences in source and target was a reasonable heuristic to progress towards better joint tokenizations, even though (a) the relationship between length difference and alignment quality was not as clear as one may have wished; (b) inconsistencies have been observed between unigram and bpe. in the future, we will continue to explore inexpensive ways to identify promising joint segmentations and improve the alignment between subword units.