Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset (2022)

Abstract: Research in massively multilingual image captioning has been severely hampered by a lack of high-quality evaluation datasets. In this paper we present the Crossmodal-3600 dataset (XM3600 in short), a geographically diverse set of 3600 images annotated with human-generated reference captions in 36 languages. The images were selected from across the world, covering regions where the 36 languages are spoken, and annotated with captions that achieve consistency in terms of style across all languages, while avoiding annotation artifacts due to direct translation. We apply this benchmark to model selection for massively multilingual image captioning models, and show superior correlation results with human evaluations when using XM3600 as golden references for automatic metrics.

## introduction image captioning is the task of automatically generating a fluent natural language description for a given image. this task is important for enabling accessibility for visually impaired users, and is a core task in multimodal research encompassing both vision and language modeling. however, datasets for this task are primarily available in english @xcite @xcite @xcite . beyond english, there are a few datasets such as multi30k with captions in german @xcite , french @xcite and czech @xcite , but they are limited to only a few languages that cover a small fraction of the world's population, while featuring images that severely under-represent the richness and diversity of cultures from across the globe. these aspects have hindered research on image captioning for a wide variety of languages, and directly hamper deploying accessibility solutions for a large potential audience around the world. creating large training and evaluation datasets in multiple languages is a resource-intensive endeavor. recent works @xcite have shown that it is feasible to build multilingual image captioning models trained on machine-translated data (with english captions as the starting point). this work also shows that the effectiveness of some of the most reliable automatic metrics for image captioning, such as cider foot_0 @xcite is severely diminished when applied to translated evaluation sets, resulting in poorer agreement with human evaluations compared to the english case. as such, the current situation is that trustworthy model evaluation can only be based on extensive and expensive human evaluations. however, such evaluations cannot usually be replicated across different research efforts, and therefore do not offer a fast and robust mechanism for model hill-climbing and comparison of multiple lines of research. the proposed xm3600 image captioning evaluation dataset provides a robust benchmark for multilingual image captioning, and can be reliably used to compare research contributions in this emerging field. our contributions are as follows: (i) for human caption annotations, we have devised a protocol that allows annotators for a specific target language to produce image captions in a style that is consistent across languages; this protocol results in image-caption annotations that are free of direct translation artefacts, an issue that has plagued machine translation research for many years and is now well understood @xcite ; (ii) for image selection, we have devised an algorithmic approach to sample a set of 3600 geographically-diverse images from the open images dataset @xcite , aimed at creating a representative set of images from across the world; (iii) for the resulting xm3600 bench-figure 1 : sample captions in three different languages (out of 36 -see full list of captions in appendix a), showcasing the creation of annotations that are consistent in style across languages, while being free of directtranslation artefacts (e.g. the spanish "number 42" or the thai "convertibles" would not be possible when directly translating from the english versions). mark, we empirically measure its ability to rank image captioning model variations, and show that it provides high levels of agreement with human judgements, therefore validating its usefulness as a benchmark and alleviating the need for human judgement in the future. fig. 1 shows a few sample captions for an image in xm3600 that exemplify point (i) above, and fig. 2 shows the variety of cultural aspects captured by the image sampling approach from point (ii). we provide detailed explanations and results for each of the points above in the rest of the paper. we have released xm3600 under a cc-by4.0 license at https://google.github.io/crossmodal-3600/ . ## the xm3600 dataset in this section, we describe the heuristics used for language and image selection, the design of the caption annotation process, caption statistics including quality, and annotator details. ## model comparison using xm3600 in this section, we detail our experiments for comparing several models using human evaluations, and also using xm3600 annotations as gold foot_4 references for automated metrics. for model comparison, we train several multilingual image captioning models with different sizes over different datasets, and compare them on xm3600. as our main result, we show a high level of correlation between model rankings based on human-evaluation scores and the scores obtained using cider @xcite with xm3600 annotations as gold references. ## conclusions we introduce the xm3600 dataset as a benchmark for evaluating the performance of multilingual image captioning models. the images in the dataset are geographically diverse, covering all inhabited continents and a large fraction of the world population. we believe this benchmark has the potential to positively impact both the research and the applications of this technology, and enable (among other things) better accessibility for visually-impaired users across the world, including speakers of lowresource languages. the main appeal of this benchmark is that it alleviates the need for extensive human evaluation, which is difficult to achieve across multiple languages and hinders direct comparison between different research ideas and results. we show significant improvements in correlation with human judgements when using the xm3600 dataset as references for automatic metrics, and therefore hope that the adoption of this dataset as a standard benchmark will facilitate faster progress and better comparisons among competing ideas. our empirical observations are primarily on the full set of side-by-side comparisons over english and three other languages (spanish, hindi, chinese). due to the similarity in the data collection and the quality control process, we expect similar results to hold for all the other languages as well; we validated this expectation with additional empirical observations covering an additional 28 languages.