Translation Deserves Better: Analyzing Translation Artifacts in Cross-lingual Visual Question Answering (2024)

Abstract: Building a reliable visual question answering (VQA) system across different languages is a challenging problem, primarily due to the lack of abundant samples for training. To address this challenge, recent studies have employed machine translation systems for the cross-lingual VQA task. This involves translating the evaluation samples into a source language (usually English) and using monolingual models (i.e., translate-test). However, our analysis reveals that translated texts contain unique characteristics distinct from human-written ones, referred to as translation artifacts. We find that these artifacts can significantly affect the models, confirmed by extensive experiments across diverse models, languages, and translation processes. In light of this, we present a simple data augmentation strategy that can alleviate the adverse impacts of translation artifacts.

## introduction visual question answering (vqa) aims to answer an open-ended question by reasoning about a given image @xcite . despite recent advances in vision-language (vl) modeling, building proficient models across various languages is still challenging. this issue primarily arises from the limited availability of annotated datasets, which are predominantly in high-resource languages such as english. although recent efforts in developing multilingual vl models can address this issue to some extent @xcite @xcite , training on datasets in the target languages is still crucial for enhanced model performance in those languages. to mitigate the data scarcity issue, cross-lingual transfer learning focuses on utilizing extensive datasets in a source language (typically english) to ## translation artifacts in cross-lingual visual question answering in this work, we analyze the impact of machine translation on cross-lingual vqa tasks, especially on the translate-test approach. to this end, we vary the origin of training datasets into human and a machine translation (mt) system and then observe how this change affects the model behavior. we use roundtrip (rt) translation to generate machine-translated training samples from the source language-english. 2 ## reducing the effect of translation artifact on cross-lingual vqa our findings demonstrate that training vqa models on translated texts induces higher accuracy in language transfer through the translate-test approach. despite such gains, translated texts inevitably contain wrongly translated information due to the imperfection of mt systems. moreover, as translations are known to be different from the naturally written human texts @xcite , training models solely on the translated texts may degrade overall performance. these problems can be observed in our previous results; in to resolve this, we leverage a simple data augmentation technique that uses both rt-translated texts and the original human-written texts for model training (merge). furthermore, following marie et al. (2020), we also adopt the ap-proach that includes special tagging tokens in front of translated texts in both training and evaluation phases (tag). as merge and tag double the number of training examples, we reduce the total training steps to half for a fair comparison across methods. results with data augmentation methods are in ## conclusion in this work, we analyze the impacts of translation artifacts presented in machine-translated english texts for cross-lingual vqa. through extensive experiments, we find that current vl models usually suffer from distributional shifts caused by translation artifacts during cross-lingual transfer, resulting in undesirable performance degradation. as a remedy, we conduct experiments with simple data augmentation strategies and observe consistent performance gains. our work focuses on translations that are semantically similar but written differently from human texts. in future work, we will explore mistranslation arising from context-free translation, where image information is not considered during a translation process. to this end, recently advanced multimodal translation systems can be utilized @xcite . other important directions include considering a variance among different translations generated from a single text and devising an advanced training strategy to consider translation artifacts.