Translated Benchmarks Can Be Misleading: the Case of E stonian Question Answering (2023)

Abstract: Translated test datasets are a popular and cheaper alternative to native test datasets. However, one of the properties of translated data is the existence of cultural knowledge unfamiliar to the target language speakers. This can make translated test datasets differ significantly from native target datasets. As a result, we might inaccurately estimate the performance of the models in the target language. In this paper, we use both native and translated Estonian QA datasets to study this topic more closely. We discover that relying on the translated test dataset results in an overestimation of the model’s performance on native Estonian data.

## introduction translating test datasets to the target language has become a popular alternative to creating datasets from scratch in the target language @xcite @xcite . the main reason for this is that translating data, either manually or automatically, and reannotating it is easier than hiring data annotators to annotate the data. in addition, to ensure the quality of the newly created dataset, the authors often go through an exhaustive process of verifying the data quality, making creating new datasets even more expensive. on the other hand, existing datasets are already established in the nlp community. another benefit of translated datasets is that they make evaluating cross-lingual transfer learning easier, as the identical datasets make the results directly comparable across languages. however, in case only a translated test dataset exists for a specific task in a specific language, it is also likely true that there is probably no task-specific native training data available in that language. if there was native training data available, then a small subset of it could have been used to create a test dataset. creating only a training dataset with no target test dataset available would also provide no benefit to the creators. the existence of (translated) test dataset in some specific language, together with the nonexistence of training data in the same language, has created an interesting situation where translated datasets have been mostly employed to advance cross-lingual transfer learning or related methods (e.g. translate-test). foot_0 however, this contradicts the idea of these methods, which is to generalize to languages where training data for the task is unavailable. with translated test datasets, the training data is usually available 2 ; it is just in another (source) language. in fact, it is most likely used to train the model, which will be evaluated with the translated test dataset. because of this, there is a danger that evaluation results become artificially inflated and overestimate the model's performance on native data. this paper aims to study the concerns of using translated test datasets more closely. we use english as a source language and estonian as a target language and evaluate models trained on the source language with native and translated target datasets to see how the results on translated dataset compare to the results on the native dataset. we opt for translate-test setup because it can be generalized more easily to different tasks as only a model trained in english is needed. in addition, it is competitive or even better at solving estonian language understanding tasks than cross-lingual transfer methods (see dataset task metric trte trtr cl native sota estqa (käver, 2021) extractive qa f1 73.0 79.9 73.4 49.20 82.4 news stories (härm and alumäe, 2022) abstractive summarization rouge-1 17.22 17.0 -16.22 trte xcopa et (ponti et al., 2020) commonsense reasoning accuracy 81.0 57.4 † 79.0 ‡ -trte/81.0 ‡ meanwhile, translate-train has also been shown to outperform cross-lingual transfer learning methods and can compete with translate-test @xcite . the success of machine translation-based methods has motivated researchers to improve these methods even more. @xcite shows that translate-train can be improved by learning a mapping from originals to translationese that is applied during test time to the originals of the target language. dutta chowdhury et al. (2022) employs a bias-removal technique to remove translationese signals from the classifier. @xcite proposes translate-all -a method that uses both techniques simultaneously. their model is trained both on data in the source language and source data translated to the target language. during inference, the two predictions, one on the target dataset and another on the target dataset translated to the source language, are ensembled. @xcite shows that even if a training dataset is available in the target language, it might still be beneficial to translate both training and test datasets to english to employ pre-trained english language models instead of native language models. @xcite draws attention to the fact that even human-translated datasets can contain artifacts that can hurt the performance of the model when compared to the native english datasets. he shows that the performance drop is indeed caused by the fact that training is done on the original data while testing is done on translated data. ## methodology our goal is to compare evaluation results obtained with native and translated estonian questionanswering datasets in a translate-test setting where the data is machine translated to english and fed to a model also trained on english. we hypothesize that translated test dataset will overestimate results on the native test dataset. ## results ## conclusion we compared the performance of an english extractive qa model on native and translated estonian test datasets in translate-test setting to see how results on the translated dataset compare to the results on the native dataset. our experiments showed that results on the translated dataset overestimate the results on the native dataset.