Low Resource Multimodal Neural Machine Translation of E nglish- H indi in News Domain (2021)

Abstract: Incorporating multiple input modalities in a machine translation (MT) system is gaining popularity among MT researchers. Unlike the publicly available dataset for Multimodal Machine Translation (MMT) tasks, where the captions are short image descriptions, the news captions provide a more detailed description of the contents of the images. As a result, numerous named entities relating to specific persons, locations, etc., are found. In this paper, we acquire two monolingual news datasets reported in English and Hindi paired with the images to generate a synthetic English-Hindi parallel corpus. The parallel corpus is used to train the English-Hindi Neural Machine Translation (NMT) and an English-Hindi MMT system by incorporating the image feature paired with the corresponding parallel corpus. We also conduct a systematic analysis to evaluate the English-Hindi MT systems with 1) more synthetic data and 2) by adding back-translated data. Our finding shows improvement in terms of BLEU scores for both the NMT (+8.05) and MMT (+11.03) systems.

## introduction with the implementation of encoderdecoder ar chitecture @xcite , mt systems have under gone quality enhancement. instead of using text as the only input in an mt system, the current trend has also started exploring multimodal ma chine translation (mmt), where multiple input modalities such as visual modality are incorpo rated along with the text as an input to the mt sys tem. using the mmt system has shown improve ment in the translated text output as compared to the nmt system @xcite . to analyse the benefits of using multiple modalities of input, various shared tasks are organized (wat2019 multimodal translation task foot_0 , wmt2018 foot_1 , vmt challenge foot_2 ). however, the image descriptions in the majority of current datasets are made up of usercaptioned or created by crowdsourcing. on the other hand, the cap tions present in the news details the contents of the image with better clarity, and as a result, contain many named entities relating to specific individu als, locations, organizations, etc. for example, in figure 1 , the caption "the old looking ship is sail ing at sunset" correctly depicts the image on some levels, yet it fails to portray the picture's higher level scenario as described in the caption on the left. limited data resources set back the development of a machine learningbased system. the lack of highquality training parallel dataset poses a con siderable challenge in developing an mt system for low resource languages. for an extremely low resource language pair, training with nmt, which is a datadriven approach, often reports to poor performance of the mt system (singh and hu jon, 2020; singh and singh, 2020). as such, re searchers have investigated various models to aug ment the dataset using the monolingual corpus, such as backtranslation @xcite , incorporating language model train on monolin gual dataset @xcite , etc. the ap proach reported in @xcite acquire an additional training dataset by backtranslating from a monolingual tar get dataset. in this paper, we acquire monolin gual news datasets reported in english and hindi, which are used to generate a synthetic parallel cor pus. @xmath0. we train @xmath1. we also conduct a systematic analysis to evaluate the mt systems by training with 1) more synthetic data and 2) adding backtranslated data. belonging to the same language family, indoeuropean, en glish, and hindi follow different word orders: sub ject verb object (svo) and subject object verb (sov). the remainder of this paper is organized as fol lows: section 2 discuss the previous related works followed by the framework of our model in sec tion 3. section 4 details our system set up and section 5 illustrates the analysis of our result. sec tion 6 sums up the conclusion and future works. ## related works a review of the machine translationrelated works is discussed in this section. based on the encoder decoder model of the nmt model, various archi tectures are built to improve the performance of mt systems @xcite . for both the encoder and the de coder, @xcite stacked numerous layers of an rnn with a long shortterm mem ory (lstm) @xcite hidden unit. @xcite introduced an attention mechanism where the decoder attends to various sections of the source text at each step of generation of the output. while the model en hances the translation of long sentences due to its sequential nature, each hidden state depends on the output of the previous hidden state resulting in a large consumption of computational power. @xcite introduced a method for data com pression, bpe, which iteratively substitutes sin gle, unused bytes for common bytes pairs in a se quence. @xcite proposed a method for word segmentation to deal with an open vo cabulary problem. instead of common byte pairs, the method combines characters or character se quences. @xcite achieves a bet ter mt system by introducing a dropout in bpe @xcite , the bpedropout excludes some merges randomly, resulting in the same word with different segmentation. vinyals et al. (2015) introduced a neural and probabilistic framework to generate image cap tions. the model comprises a vision convolution neural network (cnn), which is followed by a re current neural network (rnn) to generate a lan guage. extracting global features from an image to incorporate into attentionbased nmt, calixto et al. (2017b) introduced various multimodal neu ral machine translation models. using the features of an image to initialize the encoder hidden state is reported to be the best performing among other models. using hindi visual genome @xcite carried out an mmt for the englishhindi language pair. the au thor reported that the use of multiple modalities as an input improves the mt system. various approaches to data augmentation are ap plied to mitigate the scarcity of parallel training datasets for mt tasks. @xcite used a language model train on a monolingual dataset, achieving an improvement of up to 1.96 bleu on turkishenglish, a low resource language pair. the author also reported that domain similarity be tween monolingual dataset and target task was the key factor to use an external language model to improve the mt system. @xcite carried out the backtranslation of monolingual tar get text into the source text, thereby generating an additional training dataset. the author reported that even the limited amount of backtranslated indomain monolingual datasets could be utilized efficiently for domain adaptation. @xcite used a textonly nmt model train on multi30k @xcite dataset (german english), without images to backtranslate german descriptions in the multi30k into english and in cluded it as additional training data. ## methodology in our experiment, news articles reported in en glish and hindi along with the corresponding im ages in the articles are collected. after subjecting to a preprocessing step, the collected dataset is sentences images enhi st 80900 80900 hien st 42400 42400 ## conclusion and future work the lack of a highquality parallel dataset for the mt tasks is one of the major challenges, espe cially for low resource languages. in this work, we collected two monolingual news datasets re an analysis of the dataset augmentation with the lack of a parallel dataset is carried out. we ob serve an improvement in the mt systems in bleu scores for both the nmt and mmt systems with the data augmentation approach. our results also show that when the training dataset is comprised of a synthetic dataset from both @xmath3, the backtranslated dataset in t all setting is more effective in the mmt system as compared to the nmt system. in the fu ture, we would like to incorporate multiple modal ities to improve the mt system.