No Culture Left Behind: A rt EL ingo-28, a Benchmark of W iki A rt with Captions in 28 Languages (2024)

Abstract: Research in vision and language has made considerable progress thanks to benchmarks such as COCO. COCO captions focused on unambiguous facts in English; ArtEmis introduced subjective emotions and ArtELingo introduced some multilinguality (Chinese and Arabic). However we believe there should be more multilinguality. Hence, we present ArtELingo-28, a vision-language benchmark that spans 28 languages and encompasses approximately 200,000 annotations (140 annotations per image). Traditionally, vision research focused on unambiguous class labels, whereas ArtELingo-28 emphasizes diversity of opinions over languages and cultures. The challenge is to build machine learning systems that assign emotional captions to images. Baseline results will be presented for three novel conditions: Zero-Shot, Few-Shot and One-vs-All Zero-Shot. We find that cross-lingual transfer is more successful for culturally-related languages. Data and code will be made publicly available.

## introduction a quick review of recent surveys on multimodal ai @xcite @xcite , reveals just how much the literature is focused on english. the literature on benchmarking @xcite provides an astoundingly similar story. with the pervasiveness of ai technology in our societies, it is essential to make the technology accessible to a wider population. although english is widely spoken as a first language or a second language, most of the world (75% per capita) does not speak english. 1 figure 1 shows some annotations from artelingo-28. for 2000 images from wikiart, we have ∼140 emotion labels per image, as well as captions from annotators with diverse backgrounds covering 28 languages. unlike captions in traditional benchmarks such as coco @xcite and visual genome @xcite which emphasize unambiguous class labels, the captions in figure 1 emphasize subjective opinions over objective facts, and diversity over languages and cultures. figure 1 shows 5 annotations from 5 languages for 4 wikiart images. compare, for example, the captions for the first image in burmese, malay, korean and setswana. there are differences of opinion in both labels and captions: emotion labels: disgust (burmese), awe (malay) captions: focus on chest (burmese & malay); focus on face and hair @xcite to advance the field beyond objective facts and unambiguous class labels, it is critical to embrace diversity and subjective differences of opinion. traditionally, vision research has focused on classifying objects in the image in an objective way, but we prefer to view art as a form of communication between the artist and the audience, where there is more room for subjectivity and diversity. communication depends on much more than just the pixels in the image such as the cultural backgrounds of the participants. 2 to add 25 new languages to artelingo-28 required considerable effort. amazon mechanical turk works well for a few languages, but less so for many of the 25 languages. artelingo-28 consumed 6.25k hours of work, performed by 220 annotators from 23 countries. compared to artelingo which added just 3 languages, our dataset required significantly more management and coordination; artelingo-28 was managed by a team of 32 coordinators who contributed more than 2.5k hours. to cover many practical situations, we utilize artelingo-28 to build 3 evaluation setups: zero-shot, few-shot, and one-vs-all zero-shot. the main task evaluates the performance of the generation of affective explanations. in the zero-shot setup, we train a model on a large-scale training dataset in a few high-resource languages. we then evaluate that model on languages that do not appear in the the training data. the few-shot setup addresses the situation where we have a few training examples in low-resource languages, in addition to the large-scale training dataset from the zero-shot setup. we fine-tune the models from the zero-shot setup on the few-shot low-resource data and then evaluate them on the rest of the samples. finally, in the one-vs-all zero-shot setup, we have the large-scale training dataset as well as small-scale data in one language (one). after fine-tuning, we evaluate on the unseen languages (all). this setup is designed to shed light on pairwise interactions between languages, highlighting cultural effects. we observe clusters (cultural groups) forming from our trained models. these groups go beyond writing systems (scripts), capturing cultural connections between languages. additionally, we observe that the multilingual setup is challenging for vision and language models, partly because of the massive vocabulary. we address this challenge by utilizing pretrained multilingual llms such as bloomz. in short, our contributions are: multimodal benchmarks: benchmarks have always driven the development of many breakthroughs. imagenet @xcite being a perfect example, it led to the development of alexnet @xcite which sparked the fire of the deep learning era. recent benchmarks are moving towards multimodality. in particular, vision and language understanding datasets such as coco @xcite , conceptual captions @xcite , laion @xcite , vqa @xcite , visual commonsense reasoning @xcite , and gqa @xcite have pushed the frontiers of what is possible. they allowed developing models that can perform complex tasks such as visual grounding, image captioning, text-to-image generation, guided segmentation, and more. although these datasets are framed as benchmarks to develop vision and language, they mainly cover english language. multilingual datasets: however, "english is not a synonym for natural language processing," bender @xcite . cultural background has a great influence on perception. the marvl dataset @xcite collected concepts and images that are specified by native speakers with diverse backgrounds. their dataset revealed a major gap in models trained with english-biased datasets such as imagenet. in addition to marvl, other multilingual datasets were proposed such as multi30k @xcite , a translated version of flicker30k @xcite , as well as translated versions of coco @xcite @xcite . however, translated datasets have much less diversity compared to natively collected data as shown in the marvl dataset @xcite . affective datasets: the aforementioned datasets describe the facts and objective reality of the visual input. a recent line of work moved beyond factual captions. artemis (achlioptas et al., 2021; mohamed et al., 2022b) collected emotion labels and affective explanations to 80k wikiart artworks. artemis embellished the facts in images with emotions and commentary. the emotional captions revealed new associations that are ignored in factual captions. the subjective captions in artemis exposed the diversity of human responses. artelingo (mohamed et al., 2022a) improved the diversity of affective captioning by including arabic and chinese. they showed how the different cultures respond differently to the same visual stimulus. in this work, we embrace the cultural differences by introducing 25 more languages. apart from the affective image captioning line of work, many other emotions-related datasets were introduced. unimodal datasets that study emotional responses to single input modality such as text @xcite @xcite , image @xcite , audio @xcite , and multimodal @xcite , however, they are all small scale english datasets. emotions shape how humans perceive and process external stimuli, and then act upon those signals. work in the psychology literature has explored the effect of cultural background on shaping emotional responses @xcite @xcite . they provide concrete evidence that people from different parts of the world, speaking different languages, perceive the world differently and hence respond differently. artelingo-28 is set apart by the inclusion of many languages and hence covering more diverse views of the world. llms: recently, large language models (llms) have become popular due to a number of major successes, driven in large part by the availability of more data and computational power. the power of llms became apparent with gpt3 (brown et al., 2020), a major breakthrough over its predecessors. gpt3 can solve unseen tasks in zero-shot settings. since then, more and more large language models have been developed; bloom and opt @xcite were developed as open-source alternatives to gpt3; chinchilla @xcite , palm @xcite , megatron-turing nlg @xcite are proprietary llms with even more parameters (more than 175b parameters in gpt). notably, llama @xcite is an opensource model with fewer parameters than gpt3. however, it was trained with more than a trillion tokens. recently, more powerful models have been developed; llama2/3 @xcite , @xcite . ## artelingo-28 ## models we are interested in models able to perform vision and language understanding in many languages. unfortunately, most open-source state-of-the-art models are heavily biased toward english. hence, we adapt sota general vision and language models to a multilingual setting. the drastic increase in the vocabulary size make such adaptation a challenging task. bloomz tokenizer has a vocabulary size of 250680 tokens compared to only 32000 for llama2. hence, the embedding layer of bloomz is much bigger making it harder to align visual features with the language embedding space. ## experiments this section provides baselines for the task of multilingual affective image captioning. this task takes three inputs: image, emotion, language. the model is trained to generate an affective caption in the desired language explaining why the image evokes the desired emotion. the next three subsections discuss three experimental setups to evaluate baseline models over a variety of practical situations. in all three setups, we report standard metrics; bleu-4 @xcite , meteor @xcite , rouge @xcite and cider @xcite . we report summary scores averaged over evaluation languages; due to page limitations, detailed results by language are reported in appendix d.3. finally, we provide results for emotion label prediction in section 5.4. ## conclusion in summary, artelingo-28 addresses a critical gap in evaluating large-scale multilingual affective vision and language understanding. by adding 25 languages and 200k high-quality annotations, including low-resource languages, our dataset embraces the cultural differences. our evaluation setups -zero-shot, few-shot, and one vs all zero-shot -assess affective explanation generation across diverse linguistic contexts. the one vs all zero-shot setup extends evaluation to languages beyond the training dataset, revealing cultural connections through cross-lingual transfer performance. in this work, we introduced a dataset, proposed a benchmark, and adapted four vision and language models, overcoming current limitations in multilingual ai evaluation. artelingo-28 sets a benchmark for bridging linguistic and cultural gaps in affective vision and language understanding.