Experiences of Adapting Multimodal Machine Translation Techniques for H indi (2021)

Abstract: Multimodal Neural Machine Translation (MNMT) is an interesting task in natural language processing (NLP) where we use visual modalities along with a source sentence to aid the source to target translation process. Recently, there has been a lot of works in MNMT frameworks to boost the performance of standalone Machine Translation tasks. Most of the prior works in MNMT tried to perform translation between two widely known languages (e.g. English-to-German, English-to-French ). In this paper, We explore the effectiveness of different state-of-the-art MNMT methods, which use various data oriented techniques including multimodal pre-training, for low resource languages. Although the existing methods works well on high resource languages, usability of those methods on low-resource languages is unknown. In this paper, we evaluate the existing methods on Hindi and report our findings.

## introduction machine translation (mt) has been a challenging task in natural language processing (nlp). in the last few years, there have been significant progress in mt due to easier accessibility to data, computation and discovery of new deep learning based mt techniques. the argument over using text only dataset for mt is limited when viewed with respect to how human performs translation between two languages. we, as humans comprehend the world and perform action by combining different input modalities (e.g, text, image etc). similarly the same argument can be used to develop machines that can process different input modalities to perform downstream tasks. in this paper we specifically focus on the task of mt combined with visual inputs. we call this task as multimodal mt (w.r.t the fact that we use visual modalities combined with classical text-only mt framework). we specifically aim to address the translation task between english and a low resource language (hindi) and illustrate that by combining classical mt methods with visual information and explore how this could improve the task of translation. there are technical motivation too for performing multimodal translation tasks. text-based systems are unable to translate sentences that are ambiguous, which can have different target translations depending upon the situation. while words surrounding the ambiguous word can clear ambiguity up to some extent, it might be helpful to introduce related information from other modalities. following these motivation, we describe several baseline approaches well known in the field of multimodal mt and adapt them for a low-resource language. specifically, we focus on applying the benchmark methodologies for translating between english and hindi and viceversa with the help of visual information. ## related work several systems attempted to use extra information to translate text. unimodal systems include document-level nmt, @xcite which utilises document as context, sentence-level nmt with contextual information @xcite , etc. among multimodal systems, @xcite used an object detection system to extract local and global image features. thereafter, they used those image features as additional inputs to encoder and decoder. @xcite used attention mechanism on visual inputs for the source hidden states. @xcite used dynamic context-guided capsule network @xcite for iterative extraction of related visual features. @xcite demonstrated an unsupervised method based on the language translation cycle consistency loss conditioned on figure 1 : an example of the multimodal dataset the image. this is done to learn the bidirectional multi-modal translation simultaneously. moreover, @xcite showed that jointly learning textimage interaction instead of modeling them separately using attention networks is more useful. there exists a few multimodal machine translation (mmt) methods for english-hindi language pair. @xcite proposed to enhance the textual input by bringing the visual information to a textual domain by extracting object tags from the image. for pre-training, they used iit corpus @xcite . other methods include usage of doubly attentive decoders as mentioned in section 4.2. ## dataset description & pre-processing we use hindi visual genome 1.1 dataset @xcite introduced in the 8-th workshop on asian translation (wat) consisting of a total of 32, 922 sentences splited between 28929 train, 998 valid, 1595 test and 1400 challenge sentences. each sentence represents a caption/description of a rectangular portion of an image, associated with the sentence. coordinates of the rectangular portion is available with the dataset. we convert all the datasets into lowercase. then, we combine hindencorp @xcite and visual genome 1.1 training set and learn byte-pairencoding @xcite with 20, 000 oper-ations using subword-nmt foot_0 . for image processing, we use the following two methods: we extract pre-trained resnet50 features from the images as resnet50 is used for image features in most of the existing methods. ## evaluation results for every method, we train the model with 100 epochs and pick the model with best validation set result to generate translations. we keep the same hyper-parameters as reported in the respected paper. we translate all text from english to hindi by using beam size of 5. for bleu and ribes score, we tokenize all texts with indic-tokenizer and calculate bleu with moses multi-bleu.perl and ribes v1.02.4 foot_1 . we calculate bleu scores using sacre-bleu @xcite using two different tokenizers foot_2 . we report our results in the challenge set was created by searching for (particularly) ambiguous english words based on the embedding similarity and manually selecting those where the image and surrounding text help to resolve the ambiguity. object tag based system performs excellently compared to the other methods and achieves 52.02 bleu points. most of the systems perform better than the text-only mnmt. thus, image features have significant contribution in the performance of the challenge set. it is to be noted that, while image features add extra information to clarify ambiguous sentences, it also drops translation results in non-ambiguous sentences, acting as noise. in future, it will be important to build generalized system that can handle both of these properties. furthermore, we notice that cropped images helped to generate better results than that of full images. ## research directions multimodal machine translation approach proposed by @xcite ) (multimodal transformer) used cross-attention mechanism in place of standard self attention to model the interdependence between textual and visual modalities. cross attention mechanism is used as a generalization of self-attention mechanism inside transformer models where the input query (q) is different compared to same key (k) and value (v) vectors. the standard mechanism used in multimodal transformer was to input concatenated representation of textual and visual tokens as query vector whereas visual tokens are input as both key and value vectors. we propose the following directions for future experiments. we aim to do these experiments in the next iterations of the paper. ## conclusion neural machine translation is a very challenging task as one sentence can be translated into multiple ways. multimodal translation is used to introduce information from different modalities to assist the system to translate. there are several systems that translate multimodal information from one language to the other. these methods are proven to be helpful to generate better translation on high resource languages including english, german, french, etc. however, it was unknown how useful these methods are, specifically on low resource languages like hindi. we show comparisons of different state-of-the-art mnmt systems. we observe that multimodal information is useful to translate ambiguous sentences on hindi. furthermore, we found that multimodal information can act as noise and may not be worth using in case of non-ambiguous sentences. we hope those will serve as reference for future mnmt systems on low resource settings. in future, we would like to propose different mnmt methods for low resource languages maximizing multimodal information to improve translation while minimizing the same acting as noise. furthermore, we would like to extend our work on more existing methods and adapt them to more low resource languages.