Enhancing Video Translation Context with Object Labels (2023)

Abstract: We present a simple yet efficient method to enhance the quality of machine translation models trained on multimodal corpora by augmenting the training text with labels of detected objects in the corresponding video segments. We then test the effects of label augmentation in both baseline and two automatic speech recognition (ASR) conditions. In contrast with multimodal techniques that merge visual and textual features, our modular method is easy to implement and the results are more interpretable. Comparisons are made with Transformer translation architectures trained with baseline and augmented labels, showing improvements of up to +1.0 BLEU on the How2 dataset.

## introduction video streams are rich sources of content and the application of machine translation to videos present open research challenges. specifically, we are interested in translating the speech content present in videos, using the visual modality as auxiliary input to improve translation quality. intuitively, visual signals may help disambiguate under-specified words or correct speech recognition errors. there has been much research in speech translation, which focuses on speech input, and multimodal machine translation, which focuses on visual and textual inputs; this work combines aspects of both areas. we assume a cascaded pipeline, where the speech in a video input is first passed to a speech recognition component, then the text transcripts together with the video frames are passed to a multimodal machine translation (mmt) system. our contribution is a mmt system that augments text-based training data with labels obtained from a computer vision object detector (fig. 1 ). in contrast to more complex multimodal fusion techniques that combine vision and translation neural networks into end-to-end models, our modular approach is simple to implement, requiring no toolkit changes, and allows for easier interpretation of results. on the how2 dataset @xcite , we experiment with using clean transcripts and automatic speech recognition transcripts of varying quality as input to our translation systems. this tests the effectiveness of our multimodal approach in noisy conditions, beneficial in real-world use cases. results show gains of +0.4 to +1.0 bleu on the how2 held-out test set. src: and then you're going to stir it so have your stirrer available. person cup bottle tgt: e então você vai mexer, então tenha seu agitador disponível. ## object class label augmentation when considering the translation of instructional videos, the speaker's narration may use ambiguous language when describing the steps to the task as the viewer may be able to infer the intent through objects or actions in the scene. if mt systems are trained on the speaker's words and translations, these cues from the scene are not present. we proposed to address this omission by analyzing clips of the video and augmenting the text data with objects found in that clip. ## how2 dataset the how2 @xcite dataset is a collection of instructional videos hosted on youtube that are paired with spoken utterances, english subtitles and a set of crowdsourced portuguese translations. additional metadata such as video descriptions and summaries are also available. the dataset contains upwards of 2,000 hours of videos, but only a 300 this portion consists of 13,493 videos consisting of a total run-time of 305.1 hours from which 189,276 utterances are extracted. these videos and segments are then segregated into training, validation and test sets as shown in ## experiments to gauge the effectiveness of the label augmentation approach, we train baseline and object-label augmented systems in marian (junczys-dowmunt et al., 2018) with a transformer-base @xcite architecture. we also replicate the baseline and image feature augmented shallow recurrent neural network (rnn systems) described in @xcite for comparison. ## related work perhaps most closely related to our approach is vita @xcite , which adds object labels extracted from images in an image captioning translation task. while the motivation of adding object labels are similar, there are important differences with our setup: 1) we work on video narration of an author's task demonstration where objects appear at different points in the clip, which differs significantly from static image captions. 2) our work focuses on training mt systems from scratch as opposed to fine-tuning existing models. as part of the work in @xcite , the authors project static image features into the ## future work having proven our object label augmentation technique on how2, future work includes applying label augmentation to other datasets such as the vatex @xcite video description and visa @xcite ambiguous subtitles datasets. further research into the effects of asr degraded speech and examining task-agnostic image-language models such as clip @xcite for label augmentation may also be useful. ## conclusion we present a straight-forward method to improve mt context quality by augmenting training data with objects detected in corresponding video clips. using these augmented corpora, we realize gains of up to +1.0 bleu over baselines without changes to the underlying mt toolkits used to build models. we additionally show improvements of up to +0.7 bleu with object label augmentation when substituting asr speech for gold standard inputs.