Subword Segmental Machine Translation: Unifying Segmentation and Target Sentence Generation (2023)

Abstract: Subword segmenters like BPE operate as a preprocessing step in neural machine translation and other (conditional) language models. They are applied to datasets before training, so translation or text generation quality relies on the quality of segmentations. We propose a departure from this paradigm, called subword segmental machine translation (SSMT). SSMT unifies subword segmentation and MT in a single trainable model. It learns to segment target sentence words while jointly learning to generate target sentences. To use SSMT during inference we propose dynamic decoding, a text generation algorithm that adapts segmentations as it generates translations. Experiments across 6 translation directions show that SSMT improves chrF scores for morphologically rich agglutinative languages. Gains are strongest in the very low-resource scenario. SSMT also learns subwords that are closer to morphemes compared to baselines and proves more robust on a test set constructed for evaluating morphological compositional generalisation.

## introduction the continued success of neural machine translation (nmt) can be partially attributed to effective subword segmenters. algorithms like byte-pair encoding (bpe) @xcite and unigram lm (ulm) @xcite are computationally efficient preprocessing steps that enable smaller vocabularies and open-vocabulary models. these methods have proved quite effective, but fall short in certain contexts. for morphologically complex languages they are sub-optimal @xcite and inconsistent @xcite . this is amplified in low-resource settings @xcite @xcite , where handling rare words is crucial. these issues can be partially attributed to the fact that subword segmentation is separated from model training. bpe and ulm are applied to the training train i do understand. test do you understand? → u-ya-qonda? i am busy. in this paper we go one step further by fully unifying nmt and subword segmentation. we propose subword segmental machine translation (ssmt), an end-to-end nmt model that learns subword segmentation during training and can be used directly for inference. it is trained with a dynamic programming algorithm that enables learning subword segmentations that optimise its mt training objective. the architecture is a transformerbased adaptation of the subword segmental language model (sslm) @xcite for the joint task of mt and target-side segmentation. we also propose dynamic decoding, a decoding algorithm for subword segmental models that dynamically adapts subword segmentations as it generates translations. the fact that our model can be used directly to generate translations sets it apart from existing segmenters. ssmt is not a preprocessing step in any sense -it is single model that learns how to translate and how to segment words, and it can be used to generate translations. we evaluate on @xmath0, zulu, swati, finnish, tswana, afrikaans). as shown in we analyse the linguistic plausibility of ssmt by applying it to unsupervised morphological segmentation. ssmt subwords are closer to morphemes than our baselines. lastly, we adapt the methods of @xcite to construct an mt test set for morphological compositional generalisation -the ability to generalise to previously unseen combinations of morphemes. the performance of all models degrade on the more challenging test set, but ssmt exhibits the greatest robustness. we posit that ssmt's performance gains on morphologically complex languages are due to its morphologically consistent segmentations and its superior modelling of morphological composition. foot_0 2 related work subword segmentation has been widely adopted in nlp. several algorithms have been proposed, with bpe @xcite and ulm @xcite among the most popular. bpe starts with an initial vocabulary of characters and iteratively adds frequently co-occuring subwords. ulm starts with a large initial vocabulary and iteratively discards subwords based on the unigram language model likelihood. both of these exemplify the dominant there are downsides to relegating subword segmentation to the domain of preprocessing. the algorithms are task-agnostic. bpe is essentially a compression algorithm @xcite , while ulm assumes independence between subword occurrences. neither of these strategies are in any way connected to the task for which the subwords will eventually be used (in our case machine translation). ideally subword segmentation should be part of the learnable parameters of a model, so that it can be adjusted to optimise the training objective. there has been some research on unifying subword segmentation and machine translation. following recent character-based language models @xcite , there has been work on character-level nmt models that learn latent subword representations @xcite . however, @xcite found that subword nmt models still outperform their character-level counterparts. @xcite learn source sentence segmentation during training and find that models prefer character-level segmentations. dpe @xcite learns target sentence segmentation during training and is then applied as a subword segmenter. this line of work is related to a more general approach known as segmental sequence modelling, where sequence segmentation is viewed as a latent variable to be marginalised over during training. it was initially proposed for tasks like handwriting recognition @xcite and speech recognition @xcite . subsequently segmental language models (slms) have been proposed for unsupervised chinese word segmentation @xcite @xcite . this was adapted for subword segmentation by @xcite , who proposed subword segmental language modelling (sslm). this is the line of work we build on in this paper, adapting subword segmental modelling for nmt. our model contrasts with dpe in a few ways. firstly, our lexicon consists of the v most frequent character n-grams, so unlike dpe we don't rely on bpe to build the vocabulary. secondly, we supplement our subword model with a character decoder, which is capable of generating out-of-vocabulary subwords. lastly, through our proposed dynamic decoding we use ssmt directly to generate translations, instead of having to train an additional nmt model from scratch on our segmentations. ## subword segmental machine translation (ssmt) the subword probability of equation 2 is based on a mixture (shown on the right in figure 1 ), ## dynamic decoding for standard subword models, beam search over the subword vocabulary is the de facto approach. however, the ssmt mixture model (equation 3 ) has two vocabularies, a character vocabulary and a subword lexicon. beam search can be applied to either one. however, to approximate finding the highest scoring translation, subword prediction should be based on the full mixture distribution. during training ssmt considers all possible segmentations of the target sentence with dynamic programming. we would like to consider different segmentations during decoding as well, instead of being limited to the subword boundaries dictated by greedy prediction. doing this requires retaining part of the dynamic program during decoding, similar to yu et al. ( 2016 ) who modelled the latent alignment between (multi-word) segments. in this section we outline dynamic decoding, an algorithm that (1) incorporates both the character and lexicon models and (2) dynamically adjusts subword segmentation during generation. ## machine translation experiments we train mt models from english to 6 languages. as shown in ## unsupervised morphological segmentation morphemes are the primary linguistic units in agglutinative languages. we can analyse to what extent ssmt subwords resemble morphemes by applying it as a segmenter to the task of unsupervised morphological segmentation. the task is fully unsupervised, since our baselines and ssmt models are tuned to optimise validation mt performance and never have access to morphological annotations (they are trained on raw text). the task amounts to evaluating whether these subword segmenters "discover" morphemes as linguistic units. we evaluate our models on data from the sadilar-ii project @xcite . the dataset contains 146 parallel sentences in english and 3 of the agglutinating conjunctive languages for which we train mt models (xhosa, zulu, swati). the dataset provides morphological segmentations for all words in the parallel sentences. we apply the preprocessing scripts of @xcite to extract surface segmentations. to apply ssmt as a segmenter we use the viterbi algorithm to compute the highest scoring subword segmentation of a target sentence given the source sentence. we compare ssmt subwords to the baseline segmenters from our mt experiments. ## morphological compositional generalization ssmt learns morphological segmentation better than standard segmenters, but is it also learning to compose the meanings of words from their constituent morphemes? to investigate this we design an experiment aimed at testing morphological compositional generalisation. compositional generalisation is the ability to compose novel combinations from known parts @xcite . recent works have investigated whether neural models are able to achieve such generalisation @xcite @xcite . for example, @xcite test whether models can handle novel syntactic combinations of known semantic phrases. they construct train/test splits with similar phrase distributions, but divergent syntactic compound distributions. we adapt their approach to construct a test set with a similar morpheme distribution to the train set, but a divergent word distribution. this evaluates whether models can handle novel combinations of known morphemes (previously unseen words consisting of previously seen morphemes). to compute morpheme distributions we segment our train and test sets into morphemes with the trained morphological segmenters of @xcite . following @xcite , we refer to morphemes as atoms and words as compounds. for a dataset t , we compute the distribution of its compounds f c (t ) as the relative word frequencies and the distribution of its atoms f a (t ) as the relative morpheme frequencies. for a train set v and test set w we compute compound divergence d c (v ||w ) and atom divergence d a (v ||w ), respectively quantifying how different the word and morpheme distributions of the train and test sets are (larger divergence implies greater difference). we use the definitions of compound and atom divergence proposed by @xcite and include these in appendix c. we implement a procedure (also outlined in appendix c) for extracting a subset of the test set such that d c can be specified and d a is held as low as possible, producing a test set that requires models trained on v to generalise to new morphological compositions. ## conclusion ssmt unifies subword segmentation, mt training, and decoding in a single model. our results show that it improves translation over existing segmenters when the target language is agglutinative and conjunctively written. it also produces subwords that are closer to morphemes and learns subword-to-word composition that more closely resembles morphological composition. in future work our dynamic decoding algorithm could be used to generate text with subword segmental models for text generation tasks other than mt.