H inted BT : A ugmenting B ack- T ranslation with Quality and Transliteration Hints (2021)

Abstract: Back-translation (BT) of target monolingual corpora is a widely used data augmentation strategy for neural machine translation (NMT), especially for low-resource language pairs. To improve effectiveness of the available BT data, we introduce HintedBT—a family of techniques which provides hints (through tags) to the encoder and decoder. First, we propose a novel method of using both high and low quality BT data by providing hints (as source tags on the encoder) to the model about the quality of each source-target pair. We don’t filter out low quality data but instead show that these hints enable the model to learn effectively from noisy data. Second, we address the problem of predicting whether a source token needs to be translated or transliterated to the target language, which is common in cross-script translation tasks (i.e., where source and target do not share the written script). For such cases, we propose training the model with additional hints (as target tags on the decoder) that provide information about the operation required on the source (translation or both translation and transliteration). We conduct experiments and detailed analyses on standard WMT benchmarks for three cross-script low/medium-resource language pairs: Hindi,Gujarati,Tamil-to-English. Our methods compare favorably with five strong and well established baselines. We show that using these hints, both separately and together, significantly improves translation quality and leads to state-of-the-art performance in all three language pairs in corresponding bilingual settings.

## introduction neural machine translation (nmt) @xcite @xcite @xcite models have become the state-of-the-art approach to machine translation. however, nmt * *equal contribution figure 1 : examples from the wmt 2014 @xmath0. the top example is a case of only translation, and the bottom one is a case where some words in the source (a named-entity "coulson", and english words 'phone', 'hacking' written in hindi) need to be transliterated. models are data hungry and have been shown to under-perform in low-resource scenarios @xcite . various supervised and unsupervised techniques @xcite have been proposed to address the paucity of high-quality parallel data in such cases. back-translation @xcite ) is one such widely used data augmentation technique in which synthetic parallel data is created by translating monolingual data in the target language to the source language using a baseline system. however, in order to get high quality parallel back-translated (bt) data, we need a high quality @xmath1. this in turn depends on having a substantial amount of high quality parallel (bitext) data already available. for low-resource languages, both the quantity and quality of bitext data is limited, leading to poor back-translation models. existing methods either use all bt data available @xcite , or use various cleaning techniques to identify and filter out lower quality bt data @xcite . however, filtering reduces the amount of data available for training in a scenario which is already lowresource. how to efficiently use back-translation data in a situation where data is both scarce and of varied quality is the first key challenge we tackle in this paper. the second challenge that arises increasingly often in low-resource mt is that of cross-script nmt: translation tasks where the source and target languages do not share the same script. crossscript nmt tasks have been steadily increasing in the wmt shared news translation tasks 1 over the past few years (28% of tasks in 2017 and 2018, 44% in 2019, and 63% in 2020). cross-script nmt models must implicitly predict whether a source token needs to be translated or transliterated (see example in figure 1 ). lack of shared vocabulary coupled with low data quantity and quality makes cross-script nmt in low-resource settings a very challenging task. in this work, we propose hintedbt, a family of techniques that provide hints to the model to make the limited bt data even more effective. we present results on three cross-script wmt datasets: hindi(hi)/gujarati(gu)/tamil(ta)→english(en). in our first proposed hintedbt method, quality tagging, we use tags to provide hints to the model about the quality of each source-target bt pair. in the second method, translit tagging, we use tags to address the cross-script nmt challenge described above: we force the decoder to predict the operation that needs to be done on the source -only translation (or) both translation + transliteration, in addition to predicting the translated sentence. the correct operation is provided as an additional tag during training. we make the following contributions in this paper: leveraging monolingual data for nmt: initial efforts in this space focused on using target-side language models @xcite . recently, back-translation, first introduced for phrase-based models @xcite and popularized for nmt by @xcite , has been widely used. it has been shown that the quality of the back-translated data matters @xcite . given this finding, several works have performed filtering using sentence-level similarity metrics on the round-trip translated target and the original target @xcite , or cross-entropy scores (junczys-dowmunt, 2018). several works have looked into iterative back-translation for supervised and unsupervised mt @xcite @xcite @xcite . ## quality-based tagging of the bt data in low-resource scenarios, where bitext data is low in quantity and quality, bt data will likely contain pairs with varying quality. so far, there have been two broad approaches to deal with bt data: (a) full-bt: use all the bt data without considering the quality of the bt pairs @xcite ) (b) topk-bt: use only high quality bt pairs by introducing some notion of quality between the source and target @xcite . the full-bt method suffers from the disadvantage that it mixes the good and bad quality data, hence confusing the model. this was one of the primary motivations for introducing topk-bt models. however topk-bt models, while being quality-aware, filter away a substantial chunk of the parallel data which could be harmful in low-resource settings. in this work, we introduce a third type of using bt data called quality tagging. this approach uses all the bt data by utilizing quality information about each instance. our method extends the tagged bt approach @xcite that uses "tags" or markers on the source to differentiate between bitext and bt data. we attach multiple tags to the bt data, where each tag corresponds to a quality bin. the quality bin provides a hint of the quality of the bt pair being tagged. we use labse @xcite , a bert-based languageagnostic cross-lingual model to compute sentence embeddings. the cosine similarity between these source and target embeddings is treated as the quality score of the bt pair. bt pairs are then binned into k groups based on the quality score, and the bin-id is used as a tag in the source while training (cf. examples in figure 2 ). we explore three design choices in quality tagging below: a) bin assignment: how to assign a particular bt pair to a bin? b) number of bins to use c) bitext quality tagging. design choice 1 -bin assignment: we have two direct options: equal width binning or equal volume binning. in equal width binning, we divide quality score range into k intervals of equal size. each interval then corresponds to a bin and each bt pair is assigned to the bin which contains its quality score. in equal volume binning we sort the n data points by their quality score and divide points into k equally sized groups. each group then corresponds to a bin. we see that equal width binning (and other size-agnostic approaches like k-means) can cause severely size-unbalanced bins, with the lowest bin(s) not adding any signal at all. this is primarily because the cosine similarity used as quality score is language-pair agnostic and not calibrated to well separated quality bins. equal volume binning addresses this concern while also providing sufficiently inherent quality-based clusters with a good choice of k. design choice 2 -number of bins: we experimented with different number of bins (see detailed results in appendix e). from the dev-bleu scores, we found that for @xmath2, four bins provide the best performance, while for @xmath3. we uniformly use four bins for the sake of simplicity and point out that deeper analysis of the interplay between bitext, bt quality and number of bins is an interesting area of future work. design choice 3 -bitext quality tagging: we have three choices for this question: a) bitext is left untagged. b)bitext is always tagged with the highest quality bin. c) a bitext pair is also scored using labse and assigned to a bin just as a bt pair would be. we discuss this design choice further in section 5.5. ## translit tagging of the bt data when the source and target are written in different scripts, certain words in the source explicitly need to be transliterated to the target language, such as entities, or target language words written in the source script (see example in figure 1 ). in such cases, the model needs to identify which source words should be translated to the target language, and which need to be transliterated. to understand the prevalence of this pattern, we split the test data into two categories: {txn, both}. 'txn' means the target sentence requires translating every source word and 'both' means the a mix of translation and transliteration is needed to generate the target from the source words. then we compare the percentage of sentence pairs in each category for the hi/gu/@xmath4. for each word in the source sentence, we use fst transliteration models @xcite to generate 10 english (i.e., the target language) transliterations. if any of these transliterations are present in the corresponding target, we categorize the pair as both, else as txn. from to utilize this information about cross-script data in training, we propose a novel method: translit tagging. we use the aforementioned methodology to split the train data into two categories: {txn, both} as before. we then convert this information into tags, which we prepend to the target sentence (refer figure 2 for an example). this method teaches the model to predict if the transliteration operation is required or not for the given source sentence, hence the name 'translit' tagging. during inference, the model first produces the translit tag on the output, before producing the rest of the translated text. another option is to present translit tags on the source side while training. this method does not perform as well and also has practical challenges that we describe in detail in appendix f. ## experiment analysis in this section, we analyse a few key aspects of the experiments described in the previous section. ## conclusion in this work, we propose hintedbt, a family of techniques that adds hints to back-translation data to improve their effectiveness. we first propose quality tagging wherein we add tags to the source which indicate the quality of the source-target pair. we then propose translit tagging which uses tags on the target side corresponding to the translation/transliteration operations that are required on the source. we present strong experimental results over competitive baselines and demonstrate that models trained with our tagged data are competent with state-of-the-art systems for all three language pairs. the application of our techniques to multilingual models and to other generation techniques for back-translation (such as noised beam @xcite ) are interesting avenues for future work. the respective bitext models. however, further reduction of model size (reducing the number of attention heads, hidden dimension etc.) caused the models to underfit. the @xmath6. training was done using tensorflow-lingvo @xcite . note: for @xmath7, we randomly pick 200 pairs from each train source (from