E nglish-to- J apanese Multimodal Machine Translation Based on Image-Text Matching of Lecture Videos (2024)

Abstract: We work on a multimodal machine translation of the audio contained in English lecture videos to generate Japanese subtitles. Image-guided multimodal machine translation is promising for error correction in speech recognition and for text disambiguation. In our situation, lecture videos provide a variety of images. Images of presentation materials can complement information not available from audio and may help improve translation quality. However, images of speakers or audiences would not directly affect the translation quality. We construct a multimodal parallel corpus with automatic speech recognition text and multiple images for a transcribed parallel corpus of lecture videos, and propose a method to select the most relevant ones from the multiple images with the speech text for improving the performance of image-guided multimodal machine translation. Experimental results on translating automatic speech recognition or transcribed English text into Japanese show the effectiveness of our method to select a relevant image.

## introduction multimodal machine translation @xcite is a machine translation (mt) approach that combines information from modalities other than text, such as audio and images. since images provide visual information that is not included in audio or text, it is expected to improve translation quality by correcting errors in automatic speech recognition (asr) or by complementing information in ambiguous text. this study tackles the task of translating english audio or subtitles from lecture videos into japanese. in such situations, since useful information can be obtained from the images in the presentation materials, image-guided mt can improve translation quality over text-only mt. however, some of the images derived from lecture videos are when you do that here's what i can promise you're going to be the 800pound gorilla in the forest asr text (english) so when you do that, here's what i can promise: you're going to be the 800 pound gorilla in the forest. ## tail corpus for english-to-japanese multimodal mt of lecture subtitles, we construct a corpus consisting of five sets of images, audio in english, asr sentences in english, transcribed sentences in english, and reference translations in japanese for lecture videos from ted. 2 since an english-japanese parallel corpus consisting of transcribed sentences for ted lecture videos has been released in the iwslt2017 competition @xcite , we annotate it with images, audio, and asr sentences, as shown in figure 2 . ## image-guided machine translation in this study, as shown in figure 1 , we are given an english sentence that has been automatically recognized or manually transcribed from a lecture subtitle as well as three images that correspond in time to the text. the image-guided machine translation that we are working on is the task of inputting one image selected from among three images along with its english text and translating it into japanese subtitles. to select the image related to a given english sentence, we estimate the semantic similarity between vision and language. both of the following two proposed methods are based on blip @xcite , a pre-trained multimodal model. this model is a 4-layer, 128-dimensional transformer @xcite combined with image features from the vision transformer (vit_tiny_patch16_384) @xcite . radam @xcite was used for optimization and trained with a batch size of 4, 096 tokens and a learning rate of 1e -4. training was terminated when the cross-entropy loss in the validation dataset was not updated 10 times. ## conclusion in this study, we constructed a multimodal parallel corpus of images, audio in english, asr sentences in english, transcribed sentences in english, and reference translations in japanese of approximately 75k sentence pairs to generate crosslingual subtitles from lecture videos. experimental results reveal that our embedding-based imagetext matching method contributes to improved performance of image-guided machine translation. our future work includes further improvement of translation quality by combining multiple images.