E nglish to B engali Multimodal Neural Machine Translation using Transliteration-based Phrase Pairs Augmentation (2022)

Abstract: Automatic translation of one natural language to another is a popular task of natural language processing. Although the deep learning-based technique known as neural machine translation (NMT) is a widely accepted machine translation approach, it needs an adequate amount of training data, which is a challenging issue for low-resource pair translation. Moreover, the multimodal concept utilizes text and visual features to improve low-resource pair translation. WAT2022 (Workshop on Asian Translation 2022) organizes (hosted by the COLING 2022) English to Bengali multimodal translation task where we have participated as a team named CNLP-NITS-PP in two tracks: 1) text-only and 2) multimodal translation. Herein, we have proposed a transliteration-based phrase pairs augmentation approach which shows improvement in the multimodal translation task and achieved benchmark results on Bengali Visual Genome 1.0 dataset. We have attained the best results on the challenge and evaluation test set for English to Bengali multimodal translation with BLEU scores of 28.70, 43.90 and RIBES scores of 0.688931, 0.780669, respectively.

## introduction in recent years, multimodel approaches have shown remarkable contributions in various nlp applications such as machine translation, caption generation, etc. especially in machine translation, multiple input modalities, like text, image, or audio/speech, integrate with nmt, known as multimodal nmt (mnmt), attempts to improve lowresource pair translation by merging visual features in addition to textual features @xcite . the attention-based encoder-decoder architecture of nmt handles various issues of long-term dependency and variable-length phrases via sequenceto-sequence learning and attains a state-of-the-art technique of machine translation (mt) @xcite . also, nmt shows remarkable performance for low-resource indian languages @xcite @xcite @xcite . further, to handle the data scarcity problem, the authors @xcite augmenting phrase pairs and the source language transliterationbased @xcite approach to enhance text-only based for low-resource pair translation. this paper aims to investigate english to bengali multimodal translation task in wat2022 with a proposed transliteration-based phrase pairs augmentation approach. the rest of the paper is organized as follows: section 2 presents the review of related works. the system description is briefly discussed in section 3. section 4 reports the results and section 5 concludes the paper with future scope. ## related work in the literature survey, there is minimal existing work, particularly on the english to bengali multimodal translation task @xcite . in @xcite , they used bengali visual genome 1.0 @xcite adopted vita @xcite ) approach where they extracted object tags from the image and utilized mbart model @xcite for encoding english sentences with the object tags and decoding to generate the bengali translation. the obtained bleu scores were 43.5 and 26.8 on the evaluation and challenge test sets, respectively. moreover, the related existing works are available on english to hindi multimodal translation task @xcite @xcite @xcite . the authors @xcite used hindi visual genome 1.1 and adopts rnn-based mnmt model @xcite with advantages pre-trained word embeddings on monolingual corpus, achieved bleu scores of 39.28, 39.46 on challenge and evaluation test set respectively. this work investigates transliteration-based phrase pairs augmentation to improve the multimodal translation task of english to bengali. ## system description we have carried out four operations: transliterationbased phrase pairs augmentation, data preprocessing, model training, and testing. the opennmt-py @xcite tool is utilized to build multimodal and text-only models separately. ## result and analysis the wat2022 shared task organizer @xcite published the evaluation result foot_3 of the multimodal translation task for english to bengali, where our team achieves the first position in multimodal submission for both challenge and evaluation test set. herein, we have participated with a team named cnlp-nits-pp in the multimodal and text-only submission tracks, where a total of three teams participated. the automatic evaluation metrics, bleu @xcite , ribes @xcite are used for evaluation of results. ## conclusion and future work in this work, we have proposed a transliterationbased phrase pairs augmentation approach which has been introduced in the wat2022 multimodal translation task of english to bengali. the multimodal nmt attains a higher score than the textonly nmt model and other existing works. furthermore, the designed multilingual-based approach will be investigated to improve the translational performance of low-resource multimodal nmt.