Structural Biases for Improving Transformers on Translation into Morphologically Rich Languages (2021)

Abstract: Machine translation has seen rapid progress with the advent of Transformer-based models. These models have no explicit linguistic structure built into them, yet they may still implicitly learn structured relationships by attending to relevant tokens. We hypothesize that this structural learning could be made more robust by explicitly endowing Transformers with a structural bias, and we investigate two methods for building in such a bias. One method, the TP-Transformer, augments the traditional Transformer architecture to include an additional component to represent structure. The second method imbues structure at the data level by segmenting the data with morphological tokenization. We test these methods on translating from English into morphologically rich languages, Turkish and Inuktitut, and consider both automatic metrics and human evaluations. We find that each of these two approaches allows the network to achieve better performance, but this improvement is dependent on the size of the dataset. In sum, structural encoding methods make Transformers more sample-efficient, enabling them to perform better from smaller amounts of data.

## introduction the task of machine translation has seen major progress in recent times with the advent of large-scale transformer-based models (e.g., @xcite @xcite . however, there has been less progress on language pairs that specifically involve morphologically rich languages. moreover, although there has been previous work that builds linguistic structure into translation models to deal with morphological complexity @xcite @xcite , to the best to our knowledge there has not been work that applies such strategies to large-scale transformer-based models. we hypothesize that providing transformers access to structured linguistic representations can significantly boost their performance on translation into languages with complex morphology that encodes linguistic structure. in this work, we investigate two methods for introducing such structural bias into transformer-based models. in the first method, we use the tp-transformer (tpt) @xcite , in which a traditional transformer is augmented with tensor product representations (tprs) @xcite ) ( § 2). at a high level, tprs use a composition of roles ⇤ work partially done while at microsoft research. proceedings of the 18th biennial machine translation summit, virtual usa, august @xcite 4th workshop on technologies for mt of low resource languages english: i want people to raise their hands who are in favour of the motion to report progress. (17) turkish: ilerleme raporunun talep edilmesinden yana olanların el kaldırmalarını istiyorum. (9) inuktitut: isaaquvaksi taikkua nangmaksaqtut pigiaqtitausimajumut nuqqarumaliqtu. (5) in the english to turkish translation task, we find that the tp-transformer beats the transformer when evaluated for nuances such as morphology, word-order and subject/objectverb agreement. tpt provides a significant improvement on small datasets segmented with language agnostic bpe (⇠ 1 bleu for open subtitles 1.4m and ⇠ 2.5 bleu for hansard) and a more modest improvement on larger datasets (0.16 bleu for open subtitles 5m and 0.36 bleu for open subtitles 36m). using morphologically segmented data helps substantially with models that are trained on small datasets. this is true for both pre-training (open subtitles 1.4m and inuktitut hansard), as well as models that are trained on large datasets and later finetuned using a smaller dataset (setimes). this suggests that the method of encoding structure directly in the training data helps substantially with sample efficiency and transfer learning. in order to better understand our models, we conduct detailed analysis, including error analysis, on sample outputs from different model variations (appendix g). we also separate results out into different bins as defined by the morphological density of the target outputs to understand how results vary with morphological complexity §6. we find that morphological tokenization is strongly correlated with improved performance on complex sentences. ## using the tp-transformer the tp-transformer (tpt) was introduced by @xcite to improve performance on mathematical problem solving, a highly symbolic task. the model introduces an additional component to the attention mechanism which represents relational structure. in addition to the standard key k, query q, and value v vectors used in attention, they introduce the role vector r. let the input for token i 2 1, .., n at layer l be represented as x l i . for head h, the vectors are: the output of soft attention v lh i is: in a tensor product representation, role vectors are bound to their corresponding filler vectors by the tensor product ⌦ or some compression of it: in the tpt, we use the compression of discarding the off-diagonal elements, resulting in the elementwise or hadamard product . the query q lh i is interpreted as probing for a filler for the role r lh i , so the output of attention v lh i is taken to be the filler of that role; thus for the original tpt, this yielded: the role vector r is intended to act as a structural encoding independent of that structure's content (which is encoded in v ). we hypothesize that, by disentangling structure and content in this way, we can improve the model's ability to place familiar linguistic units in novel structures (e.g., using a suffix with a word stem that never had that suffix during training). such structural flexibility is crucial for morphologically-rich languages. we make two modifications to the tpt used in @xcite foot_0 . first, we use relative position embeddings @xcite . we also use a residual connection to produce gradients that are not zero; v r is a multiplicative interaction, so values of r near 0 will produce activation values and gradients of 0. this is similar to the model detail in @xcite section 7.2. a schematic of our attention is shown in figure 1 . the rest of the architecture follows the standard residual connections and encoder-decoder architecture defined in @xcite 3 using morphological segmentation our target languages, turkish and inuktitut, both exhibit a high degree of morphological complexity. words in both languages consist of a root followed by potentially many suffixes, each of which may have multiple surface forms. language segmented word turkish anla-t-ma-yacak gloss understand-caus-neg-fut english will not tell bpe anlat-mayacak inuktitut miv-vi-liar-uma-lauq-tur-uuq gloss land-place-go-want-past-3s-say.3s english he said he wanted to go to the landing strip. bpe mivvi-lia-ruma-lau-qturuuq we used two methods of subword tokenization: one utilizing a type of characterlevel byte-pair encoding @xcite , and one incorporating morphological parsing plus byte-pair encoding. the first method (which we label 'bpe') used sentencepiece @xcite , a tokenizer that builds subword tokens using a combination of bytepair encoding and unigram language modeling. bpe relies only on character frequencies and incorporates no morphological information. the second method (which we call 'morphological tokenization') incorporated morphological information by parsing all words (i.e. breaking them up into their composite morphemes) in our morphologically complex target languages before tokenizing them. for turkish, we used the morphological parser from zemberek @xcite , an open-source turkish nlp toolkit. zemberek uses sentence-level disambiguation to produce the most likely parse of each word given its sentential context. for inuktitut, we used the morphological parsing method adopted by @xcite , incorporating a symbolic parser with a neural parser backoff. see appendix d for implementational details on morphological segmentation. the differences in how these tokenizers divide multi-morphemic turkish and inuktitut words into subwords are illustrated in ## experimental results we aim to answer the following research questions (rq) through our experimentation: ## morphological density analysis given the rich morphology of the target languages, we are interested in whether either structural bias or morphological segmentation improves performance on more morphologically complex sentences. to answer this question, we used our turkish morphological segmenter on sequences from the test set and binned sentences based on the average morphemes per word in a sentence. for example, a long sentence with simple words that are all a single morpheme would have an average morpheme per word of 1, whereas a sentence that is made of complex words would have a larger average morpheme per word. we then calculated the bleu score for each of these buckets so that we could see if our models performed better on sentences that are morphologically complex. the results are shown in figure 3 . on the 36m training set (top row), both of our methods provide an improvement at almost every morpheme density. comparing tpt to a standard transformer, figure 3a shows a relatively consistent improvement of around 0.4 bleu with a large increase for simple sentences. comparing standard transformers with morphological parsing against bpe, figure 3b shows that as the morphological complexity of sequences increases, the model using morphological tokenization improves over bpe tokenization. the same trend is visible when comparing tpt with morphological tokenization with a standard transformer using bpe tokenization (figure 3c ), except the magnitude of the increase is greater. the morphological analysis on the 5m training set (bottom row) is less conclusive. tpt does not appear to have any impact as the morphological density increases (figure 3d ). morphological tokenization shows a similar upward trend as on the 36m dataset, but this improvement disappears suddenly at 3.0 morphemes per word (figure 3e ). as the morphological density increases, the number of samples for each bucket on the test set decreases, so it is possible that the sudden drop is the result of too few samples. our results also show some correspondence with the overall morphological complexity of the dataset. we computed a modified version of the c d measure (the "relative entropy of word structure") from @xcite , as we found it to be the most robust to the meaning variations between corpora (supplementary materials section 4). higher values of this measure correspond to more regular structure/information in words, and thus, greater morphological complexity. we computed the measure over the first 100,000 characters of the test set of each dataset. we computed c d as 1.89 for the hansard dataset, while the turkish datasets ranged from c d 1.45-1.49. this corresponds to the relatively large increase in bleu seen for inuktitut. using tprs tprs have gained traction recently with the interest in neurosymbolic computation to achieve out-of-domain generalization. they have been used in a variety of domains, including mathematical problem solving @xcite , reasoning @xcite , image captioning @xcite , question-answering @xcite , and program synthesis @xcite . a separate line of work uses tprs as an interpretation tool to understand representations in networks that do not explicitly use tprs @xcite . ## conclusion we investigated two methods for improving translation into morphologically rich languages with transformers. the tp-transformer adds an additional component to transformer attention to represent relational structure. this model had the largest improvement on smaller datasets and modest improvement on larger datasets. we also investigated morphological tokenization which had substantial improvements on small datasets and transfer learning. the models were analyzed by human evaluators to tease apart different dimensions along which our models excel; tp-transformer had fewer morphological, word-order, and agreement issues. we analyzed the performance of our networks under varying morphological complexity and found that morphological tokenization provides a large benefit for more complex sentences. transformer tp-transformer 1.4m .351 ±.005 .365 ±.004 1.4m morph .438 ±.001 .440 ±.001 5m .461 .463 5m morph .467 .469 36m .486 .488 36m morph .490 .492