Translating Short Segments with NMT : A Case Study in E nglish-to- H indi (2018)

Abstract: This paper presents a case study in translating short image captions of the Visual Genome dataset from English into Hindi using out-of-domain data sets of varying size. We experiment with three NMT models: the shallow and deep sequence-tosequence and the Transformer model as implemented in Marian toolkit. Phrase-based Moses serves as the baseline. The results indicate that the Transformer model outperforms others in the large data setting in a number of automatic metrics and manual evaluation, and it also produces the fewest truncated sentences. Transformer training is however very sensitive to the hyperparameters, so it requires more experimenting. The deep sequence-to-sequence model produced more flawless outputs in the small data setting and it was generally more stable, at the cost of more training iterations.

## introduction in recent years, neural machine translation (nmt) systems have been gaining more popularity due to their improved accuracy and even more fluency compared with "classical" statistical machine translation systems such as phrase-based mt (pbmt), see e.g. the shared tasks of wmt and iwslt @xcite . the major advantages of nmt include the consideration of the entire sentence, capturing similarity of words, and the capacity to learn complex relationships between languages. at the same time, it has been observed that nmt is more sensitive to the shortage of or noise in the parallel training data @xcite . our goal is to create the hindi version of visual genome @xcite . 1 hindi, with 260 million speakers, is the fourth most widely spoken language on the planet (after chinese, spanish and english). hindi is a morphologically rich language (mrl), with e.g. the gender category being reflected in the forms of nouns, verbs and also adjectives @xcite . the structural and morphological differences between english and hindi result in translation difficulties @xcite . visual genome is a dataset of images, captions and relations. as such, it is potentially useful for many nlp and image processing applications. the hindi version would allow to exploit this dataset e.g. to create hindi image labellers or other practical tools. the textual part of visual genome consists primarily of short sentences or noun phrases that were manually attached to rectangular regions in an input image. in the current version, visual genome contains 108k distinct images with 5.4 million such labelled regions in total. on average, an image is thus associated with 50 text segments. text segments can repeat across images and indeed, when de-duplicated, the set of unique strings reduces to 3.15 million unique segments. even with this de-duplication, this set remains too big to be translated manually. it is thus natural to attempt to translate this dataset automatically and in this paper, we are trying to find the best base-line translation. in the future, we want to include also information available in the context of each of the labels: either the text descriptions of nearby regions or directly the visual information in a form of multi-modal translation @xcite @xcite . the paper is organized as follows. section 2 reviews related work on neural mt and english-hindi translation. section 3 describes our experimental setting: data, models and their parameters. section 4 provides automatic and manual evaluation of the translations and section 5 discusses the results in closer detail. we conclude in section 6. wang et al. (2017) use the encoder-decoder framework with attention @xcite for their submission to the workshop on asian translation (wat) 2017 shared task and observe considerable gains for english-to-hindi compared to pbmt. similarly to other works, they benefit from subword units @xcite and back-translation @xcite , as well as model ensembling. agrawal and misra sharma (2017) evaluate english-hindi translation quality using several variants of rnn-based neural network architecture and basic units (lstms, @xcite @xcite , including the attention mechanism by @xcite and more layers in the encoder and decoder. the bi-directional lstm model with four layers and attention performs best. the early models of nmt have suffered from lower translation quality for long sentences, see e.g. cho et al. (2014a) and bahdanau et al. (2015). a recent experiment by @xcite has however suggested that nmt can perform worse than pbmt also for short segments (insignificantly). it is thus natural to evaluate the effect in our particular setting. we note that monolingual data plays an important role in boosting the performance of the translation in both pbmt @xcite and nmt @xcite . we leave these experiments for future work because we would first need to find or select hindi texts closely matching to the domain of visual genome texts. ## experiments the overall framework of our work is shown in figure 1 . the targeted dataset is english text descriptions from visual genome but no similar or related data is available in hindi. so far, we thus used visual genome only to select the development and the test set. we experimented with two parallel corpora as our training data, hindencorp and iitb corpus (see section 3.1), three nmt models and the pbmt baseline (section 3.2). ## results figure 2 presents the learning curves for all the models evaluated on the development set using the bleu score @xcite . (pbmt training is displayed in terms of mert iterations on the secondary x axis.) for nmt, we validated the model every 10000 iterations and ran the training until the crossentropy has not improved for 10 consecutive validations. for each model, we selected the iteration where the highest bleu score was reached and translated the test set with this model. ## analysis and discussion we assumed that pbmt may perform better on short segments. in order to test this assumption, we divided the 1000 test segments into 5 groups based on the source segment length. group boundaries were chosen to achieve reasonably balance distribution and at least a minimal size for automatic scoring: source length: 1-3 4 5 6 7-12 segment count: 73 380 282 165 100 flawless: a car on a street सडक पर एक कार gloss: a car on a street a white and yellow passenger car एक सफे द और पीला यात ◌् र कार gloss: a white and yellow passenger car white part of the chair कु र ◌् सी का सफे द भाग gloss: white part of the chair partly correct: a man wearing white shorts एक आदमी सफे द शॉर ◌् ट पहनना gloss: a man put on white short (output does not convey the intended meaning in the target language) dog in a lake इस झील म कु त ◌् ते gloss: dogs in this lake (grammar error: dog vs. dogs) ambiguity: faucet is above sink फे सबु क स क से ऊपर है gloss: facebook is above sink (bad translation of the word "faucet') green bean in soup आत ◌् मा म हरा gloss: spirit in green (mis-translated words "bean", and "soup") then on sentences over 7 words. in other cases, s2s wins. with the iitb training corpus, transformer wins and pbmt loses across all lengths. a generally interesting property of nmt is its ability to correctly predict the sentence length @xcite . we take a look at this by considering both the relation of our candidate translations with the source and with the reference. figure 6 plots the length of the translation for individual source segments sorted by length. we see that the target length varies a lot across segments and also different nmt models. in general, outputs are longer than sources but the length of the source is not really followed by any of the models. we observed on the hindencorp training data that some of the nmt models tended to cut off sentences too short in early iterations. to examine this, we checked the difference in length be- ## conclusion we have applied the state-of-the-art neural machine translation models and the phrase-based baseline to english-to-hindi translation. our target domain were relatively short segments appearing in descriptions of image regions in the visual genome. the results indicate that with smaller data (274k parallel segments, 3.8m english tokens), the deep sequence-to-sequence attentional model is the best choice, although the pbmt baseline seemed to perform well in two of the tested automatic metrics, character and per. with large parallel data available, transformer should be preferred and all nmt models clearly outperform pbmt. we have not yet explored the effect of adding monolingual data. a deeper analysis has not revealed any difference in performance for shorter or longer segments, but the manual annotation suggested that the performance of nmt models varies across individual segments. the overall performance is thus perhaps too crude and it would be suboptimal to decide for a single model. in the future, we will focus on the possibilities of multi-modal translation @xcite @xcite to improve translation quality using the visual genome images or other contextual information available. our ultimate plan is to release a machine-translated hindi version of visual genome.