TMEKU System for the WAT 2021 Multimodal Translation Task (2021)

Abstract: We introduce our TMEKU system submitted to the English-Japanese Multimodal Translation Task for WAT 2021. We participated in the Flickr30kEnt-JP task and Ambiguous MSCOCO Multimodal task under the constrained condition using only the officially provided datasets. Our proposed system employs soft alignment of word-region for multimodal neural machine translation (MNMT). The experimental results evaluated on the BLEU metric provided by the WAT 2021 evaluation site show that the TMEKU system has achieved the best performance among all the participated systems. Further analysis of the case study demonstrates that leveraging word-region alignment between the textual and visual modalities is the key to performance enhancement in our TMEKU system, which leads to better visual information use.

## introduction neural machine translation (nmt) @xcite has achieved state-of-the-art translation performance. however, there remain numerous situations where textual context alone is insufficient for correct translation, such as in the presence of ambiguous words and grammatical gender. therefore, researchers in this field have established multimodal neural machine translation (mnmt) tasks @xcite @xcite , which translates sentences paired with images into a target language. due to the lack of multimodal datasets, multimodal tasks on the @xmath0. since the year 2020, as the multimodal dataset on the 1 tmeku is the abbreviation of the combination of the tokyo metropolitan university, the ehime university and the kyoto university. @xmath1, the multimodal machine translation (mmt) tasks on the @xmath2. some studies @xcite have started to focus on incorporating multimodal contents, particularly images, to improve the translation performance on the @xmath3. in this study, we apply our system @xcite for the mmt task on the @xmath4, which is called tmeku system. this system is designed to translate a source word into a target word, focusing on a relevant image region. to guide the model to translate certain words based on certain image regions, explicit alignment over source words and image regions is needed. we propose to generate soft alignment of word-region based on cosine similarity between source words and visual concepts. while encoding, textual and visual modalities are represented interactively by leveraging the word-region alignment, which is associating image regions with respective source words. the contributions of this study are as follows: ## case study we show two cases in figure 3 , and improvement is highlighted in green. we perform two types of visualization for each case: (1) we visualize the source-target word alignment of the text-attention. (2) we visualize the region-target alignment of the image-attention at a time step that generates a certain target word along with attending to the most heavily weighted image region feature. in the case shown on the left, our tmeku system translates "entering" to "entrant," but the baselines under-translate. by visualization, the textattention and image-attention assign the highest weights to the word and region that are semantically relevant at that time step of generating "entrant." this example shows that translation quality improvement is due to the simultaneous attentions of semantically related image regions and words. in the case shown on the right, our tmeku system correctly translates "backyard" to a compound noun of "arrière-cour." but the baselines mistranslates it to "cour," which means "yard" in english. through visualization, we find that the text-attention and image-attention focus on the features that are semantically relevant at that time step. this example shows that the image region feature associated with its semantically relevant textual feature can overcome the deficiency, where the object attribute cannot be specifically represented by only the image region feature. ## conclusion we presented our tmeku system to the @xmath5, which is designed to simultaneously consider relevant textual and visual features during translation. by integrating the explicit word-region alignment, the object-level regional features can be further specified with respective source textual features. this leads the two attention mechanisms to understand the semantic relationships between textual objects and visual concepts. experimental results show that our tmeku system exceeded baselines by a large margin and achieved the best performance among all the participated systems. we also performed analysis of case study to demonstrate the specific improvements resulting from related modalities. in the future, we plan to propose a more efficient integration method to make modalities interactive with each other.