Pointer-Generator Networks for Low-Resource Machine Translation: Don’t Copy That! (2024)

Abstract: While Transformer-based neural machine translation (NMT) is very effective in high-resource settings, many languages lack the necessary large parallel corpora to benefit from it. In the context of low-resource (LR) MT between two closely-related languages, a natural intuition is to seek benefits from structural “shortcuts”, such as copying subwords from the source to the target, given that such language pairs often share a considerable number of identical words, cognates, and borrowings. We test Pointer-Generator Networks for this purpose for six language pairs over a variety of resource ranges, and find weak improvements for most settings. However, analysis shows that the model does not show greater improvements for closely-related vs. more distant language pairs, or for lower resource ranges, and that the models do not exhibit the expected usage of the mechanism for shared subwords. Our discussion of the reasons for this behaviour highlights several general challenges for LR NMT, such as modern tokenization strategies, noisy real-world conditions, and linguistic complexities. We call for better scrutiny of linguistically motivated improvements to NMT given the blackbox nature of Transformer models, as well as for a focus on the above problems in the field.

## introduction and motivation while state-of-the-art (sota) transformer models @xcite for nmt work well for high-resource language pairs, their performance degrades in low-resource situations @xcite @xcite ; this means that most languages in the world cannot benefit from mainstream advances and models @xcite . there is therefore a clear appeal to developing simple architectural mechanisms for these models that are targeted at yielding improvements in data-scarce scenarios, while interfering minimally with mainstream preprocessing, tokenization, and training pipelines. in the context of a low-resource language (lrl), we are often interested in translation to and from a closely related hrl, which possibly has linguistic genealogical, regional, and cultural ties with the lrl, foot_0 in order to make the abundant content in hrls available in related lrls. we expect that closely related languages share considerable overlap at the subword level from cognates, borrowings and shared vocabulary (see examples in figure 1 ). given the absence of large parallel corpora for our language pair, we aim to leverage this shared knowledge across source and target, intuitively, to provide "easier" routes for our mt model from source to target sentence. pointer generator networks (pgns; @xcite ) are a mechanism which allow the model, for every output token produced, to either copy some token from the input ("point") or "generate" a token as per usual from the vocabulary. pgns have been used for a variety of problems, described in section 2, often targeted at repeated spans of text in the input and output; however, as we far as we know, this is the first work to study its applicability to lr nmt. in this case, we hypothesise that the pointing mechanism will show advantages for rare shared subwords, for which the best strategy may be to copy them to the output. we introduce a pgn mechanism into a transformer-based nmt architecture, and test its performance for 6 language pairs over 4 low-resource training ranges. we work with hindi-bhojpuri (hi-bh), spanish-catalan (es-ca), and french-occitan (fr-oc), representing closelyrelated pairs, hindi-marathi (hi-mr), a relatively more distant pair, 2 and spanish-english (es-en) and french-german (fr-de), representing further distant pairs. we expect that pgn will help most for (1) lower-resource scenarios (2) more closely-related language pairs (3) sentence pairs with higher subword overlap. while pgn shows improvements in certain settings, our comparative analysis of the benefits of pgns across the three dimensions above shows clear lack of evidence for the these hypotheses. further, our visualizations of the pgn mechanism also indicate that observed benefits do not come from intended sources. we discuss various factors that contribute to this failure, highlighting fundamental challenges for lr nmt, such as noisy datasets, mainstream tokenization practices best suited for high-resource scenarios, as well as linguistic and orthographic complexities that may obfuscate underlying source-target similarities. 3 ## related work pointer networks were introduced to solve problems that involved permuting the input, such as the traveling salesman problem and the complex hull problem @xcite . their use in nlp has been largely been for monolingual summarization, where the target may naturally contain identical spans from the source. cheng and lapata (2016) present a complex hierarchical lstmbased model for summarization, which directly extracts sentences from the text and words from sentences. @xcite use pointer networks in rnn-based sequence-to-sequence models for summarization and machine translation, training their model explicitly to use the pointing mechanism for uncommon words. @xcite and @xcite also incorporate variants of pointer-generator networks into rnn-based 2 hindi, bhojpuri, and marathi belong to the indic branch of the indo-european family. hindi and bhojpuri further belong to the shaurasenic sub-branch and are closer lexically and grammatically to each other and other languages on or close to the hindi belt such as punjabi, rajasthani, haryanvi, and maithili, than hindi is to marathic languages and dialects; this is supported by lexical and other studies of cross-lingual similarity @xcite @xcite . see glottolog ( https: //glottolog.org/resource/languoid/id/cont1248 ) for the phylogenetic tree. our work is the first to examine the applicability of pgns as facilitators in the low-resource mt scenario, looking to exploit linguistic relationships between the source and target in the absence of external resources. we work with transformerbased nmt, and make no changes to standard bpe tokenization schemes or training objectives (unlike @xcite and @xcite ). this is so that our findings are most relevant in today's paradigm of generalized strategies for endto-end multilingual mt; our mechanism can be easily plugged into and trained with any modern (multilingual) mt pipeline. ## model the pgn model provides two routes to the model for predicting any target token: copying from the source or generating from the vocabulary @xcite . copy and generate distributions at step t are mixed using a learned parameter p t copy , to obtain the final probability distribution p t for the target token. here, c t is the context vector, calculated as c @xmath0, where a t represents cross-attention vector, and e t contains the encoder hidden states. d t and s t contain the decoder's final hidden states and input respectively, ⊕ denotes concatenation, and w and b are learned weights and a bias vector respectively. p t c and p t g represent the copy and generate distributions (softmaxed logits) respectively at step t. we use cross-attention weights over source tokens for the copy logits, and standard decoder outputs for generate logits. ## results and discussion improvement patterns see results in controlled test sets we test the motivating hypothesis that pgn models is likely to benefit sentence pairs with higher subword overlap. we rank sentence pairs in our test set by percentage of shared subwords in source and target, and construct test subsets with low and high shared-subword density from the top and bottom 500 sentences respectively. however, in usage of the copy mechanism we record values of p copy to track the model's usage of the copy mechanism. while p copy values are relatively high foot_5 for copied subwords, numerals, and proper nouns, we often see that they they are also high for seemingly random subwords. 9 we also do not see a relationship between the p copy value of a target token and the entropy of the cross-attention distribution for that token. a reasonable intuition about pgn training generalization is that in the absence of any information, the model will default to copying, since this is likely to do better on average than guesses over the entire vocabulary, and that eventually, it will learn to generate language-specific subwords, memorising the relevant strategy for given subwords in encoder states (used to calculate p copy as shown in section 3). however, our visualizations of crossattention and p copy usage throughout training show no evidence of this generalization strategy. it's possible that since initial cross-attention distributions are noisy, and most subwords are not direct copies, the model is discouraged from copying early on; it's also possible that the model finds it easier or trivial to encode copied source-target equivalents via the "generate" mechanism and does not need an explicit copier, given that it must additionally learn which subwords should be copied. we discuss potential reasons for this below. in general, it appears that the model uses the copy mechanism to encode a task that is not easily interpretable, possibly resulting in the observed small improvements over some datasets. tokenization in theory, the copier would learn best if the tokenizer behaved in a morphologically principled manner. 10 however, bpe tokenization generally results in subword splits that may not reflect shared stems in word equivalents @xcite . 11 a natural idea here may be an investigation of morphologically inspired tokenizers @xcite @xcite . however, we generally see inconclusive, at best marginal, benefits of such tokenizers over bpe in modern neural mt @xcite @xcite , especially those relying on unsupervised morphological segmentation, e.g., with morfessor @xcite in the absence of morphological analysers. these ideas have not been incorporated into mainstream tokenization strategies. recent work attempts to solve this general problem by looking at maximisation of shared subwords in multilingual tokenizers @xcite @xcite ; it's possible that such strategies will dovetail well with pgn mechanisms if widely adopted in the future. ## challenges for lr nmt incorporating knowledge of linguistic relationships among closely related data-imbalanced language pairs offers a natural strategy for mitigating data scarcity in mainstream nmt between regional languages, and it is crucial to understand the challenges in this realm. we show that while the pgn mechanism offers an intuitive theoretical shortcut for translation between closely related languages, its performance in practice is limited, potentially by the combined effect of noisy real-world datasets containing non-literal translations, the behaviour of standard tokenizers, as well as linguistic complexities beyond simplified ideas of shared vocabulary and cognates. these are inevitable hurdles to any project that attempts to use structural linguistic knowledge to benefit nmt performance. further, we show that despite showing benefits in certain settings over the entire test set, the pgn mechanism does not perform as expected on target phenomena. the generalization mechanisms of blackbox transformer-models are not well understood and may not be easily guided by linguistic intuition: we underline the importance of verifying that improvements are coming from the intended places rather than good starts or extra parameters. finally, our analysis hints that pgn-like shortcuts may not be worth offering in the first place: "easy" equivalences, a natural target of linguistic interventions, may not be the bottleneck for lr nmt. instead, it is more likely that the true bottlenecks are handling precisely the above challenges, i.e. non-systematic differences, one-off phenomena, and real-world noise in low-resource conditions. ## limitations while we show that our particular flavour of nmt incorporating pgns does not provide fundamental benefits for low-resource nmt, this is naturally not to say that an improved variant of this idea would not work better. there are several potential ways forward arising from our discussion of the reasons for the failure of our method in section 5: for example, using morphological segmentation for tokenization to increase subword overlap, or using priors for p copy so that it is encouraged for shared subwords. previous work provides different kinds of help to the copier: for example, @xcite explicitly train the copier to copy unknown words with a separate training objective. however, as we mention in section 1, our motivation lay in designing a simple architectural mechanism that can be easily integrated into mainstream (multilingual) nmt pipelines to make them more capable for low-resource mt, without requiring much additional language-pair specific attention to training paradigms or tools such as morphological analysers and bilingual lexicons, which are in any case of poor quality for low-resource languages. we restrict our negative result to the scope set up by this motivation. further, our results are limited to the 6 language pairs that we experimented on. while we simulate identical low-resource conditions for all our language pairs, we clearly see the difference in absolute performances on hi-mr or hi-bh as compared to the high-resource language pairs: the data for the latter are simply of much better quality. this demonstrates the need to experiment and present further results on non-simulated truly low-resource conditions, such as the hi-bh language pair studied here. finally, this discussion is only relevant to translation between closely-related languages that share a script (although this is the predominant case), allowing for lexical similarity to be reflected by shared subwords. ## conclusion in this work, we investigate the applicability of pointer-generator networks in nmt, hypothesizing that an explicit copy mechanism will provide benefits for low-resource translation between closely related languages. we show that while we do observe weak improvements, these are not higher for closer-related languages, sentence pairs with higher overlap, or lower resource ranges, contrary to intuition. our discussion of potential reasons for the failure of this approach highlights several general challenges for low-resource nmt, such as mainstream tokenization strategies, noisy data, and non-systematic linguistic differences.