A rt EL ingo: A Million Emotion Annotations of W iki A rt with Emphasis on Diversity over Language and Culture (2022)

Abstract: This paper introduces ArtELingo, a new benchmark and dataset, designed to encourage work on diversity across languages and cultures. Following ArtEmis, a collection of 80k artworks from WikiArt with 0.45M emotion labels and English-only captions, ArtELingo adds another 0.79M annotations in Arabic and Chinese, plus 4.8K in Spanish to evaluate “cultural-transfer” performance. 51K artworks have 5 annotations or more in 3 languages. This diversity makes it possible to study similarities and differences across languages and cultures. Further, we investigate captioning tasks, and find diversity improves the performance of baseline models. ArtELingo is publicly available at ‘www.artelingo.org‘ with standard splits and baseline models. We hope our work will help ease future research on multilinguality and culturally-aware AI.

## introduction figure 1 compares and contrasts annotations on wikiart across language/culture. we believe these differences are interesting and important, and far from random. one might suggest using machine translation to translate english captions to many other languages, but we believe that doing so would miss much of the opportunity. building humancompatible ai that is more aware of our emotional being is important for increasing the social acceptance of ai. artemis @xcite is an important step in this direction, introducing a collection of 0.45m emotion labels and affective language explanations in english on more than 80,000 artworks from wikiart. however, by design, artemis is limited to english, lacking coverage of other cultures and languages. cultural differences are a major source of diversity @xcite . the customs, social values, lifestyles, and history of different countries and cultures greatly influence human behavior. emotional experiences are no exception; people from different countries respond differently to similar scenarios. for example, a person born and raised in a nordic country would be more comfortable in a lush forest than in a desert, but a bedouin may be more comfortable in a desert than in a forest. consider figure 1c , where an arabic annotator assigned the image the label contentment, but the other two annotators used the label: sadness. captions are useful for diving deeper into these differences. the sadness annotations mention death 2 and disasters, 3 in contrast with the contentment annotation that ends with: feeling of satisfaction. there can be interesting differences between languages/cultures even when annotators use the same label. consider figure 1b , where all three labels are contentment. although the three captions agree on the label, two of the captions imply that some/all of the girls are sisters, but there is no such implication in the english caption. we believe deep nets will be viewed as more culturally aware, if they can capture linguistic/cultural patterns such as these. emotions are based on past experience, and play an integral role in determining human behavior. not only they reflect our internal state but also directly effect how we perceive, interpret external stimuli @xcite , and how to act based on them @xcite . hence, studying emotions is essential to exploring a confounding aspect of human intelligence. in summary, our contributions are: the rest of the paper is organized as follows: related work is discussed in §2, followed by our main motivation in §3, and data collection in §4. ## opportunities for improvement many of the resources mentioned above have advanced our understanding of the relationship between emotion and various stimuli, through there are always opportunities for improvement. we are particularly interested in three such opportunities: scale, multimodality and multilinguality/muliculturalism. as for scale, demand for larger training sets is expected to continue to increase, given the rise of large scale foundation models @xcite . as for multimodality, although most benchmarks mentioned above focus on a single modality, there are a few multimodal exceptions such as iemo-cap @xcite , coco and artemis. iemocap collected speech and facial and hand movements of 10 actors. unfortunately, this approach may be expensive to scale up. the use of amazon mechanical turk in artemis is easier for scaling, however, artemis is limited to english. artelingo addresses multilinguality/multi-culturalism by adding arabic and chinese annotations. we use languages as a proxy to reflect different cultures. english is a representative sample of the west, and chinese is a representative sample of the east, and arabic is a representative sample of the middle east. ## artelingo following artemis, we employ amazon mechanical turk (amt) platform to collect our data using interfaces ( see figures 8 , 9 , 10 in the appendix). we faced a lack of arabic and chinese speaking annotators on amt which led us to devise different strategies to recruit annotators. arabic speakers were recruited by advertising the task in middle eastern universities encouraging students and their families to join our data collection efforts. whereas chinese speakers were recruited through baidu who we'd like to thank. annotators are asked to carefully examine each artwork before selecting the dominant emotion induced by it from a list of four positive, four negative emotions, and other to indicate a different emotion. annotators are then asked to write captions that reflects the content of the artwork and explains their choice of emotion. similar to artemis, we collect annotations from five annotators for each artwork. for a better cultural representation in artelingo, we restrict the collection of different languages annotations to countries with large numbers of native speakers. chinese data is collected from china. for arabic, we collect our data mainly from saudi arabia and egypt. finally, spanish is collected from latin america and spain. figure 4 shows that most of the annotations are from a long tail of workers who annotated less than 1000 artworks ensuring a diverse representation of cultures. quality control. annotations were rejected if they are too short, or if they are too similar to captions for other artworks. in addition, a manual review was conducted by multiple reviewers, ensuring captions reflect the selected emotion label and the details of the artwork. ## dataset analysis 5.1 qualitative there are some interesting similarities and differences between language and culture, as discussed in figure 1 . there is a considerable inter-annotator agreement (iaa) in the dataset, and there are also some interesting disagreements. there is agreement in figure 2a that a mother's love is universally warm and pleasant. it is an instinct for mothers to be loving, caring and protective of their children. 14 on the other hand, there is a difference in figure 1a . all three annotators agree to observe a waterfall though some mention energy and growth, while others saw horses and wedding veils. ## emotion label prediction baseline models for two tasks, emotion label prediction and caption generation, will be discussed in this section and the following section. these discussions assume familiarity with deep nets including fine-tuning bert @xcite and cross language models xlm @xcite , as well as huggingface @xcite . emotion classification. given an input caption, c, we wish to predict an output emotion label, ê, where ê is one of the 9 emotions. the model starts with a pretrained language model, lm , and a tokenizer. the tokenizer converts c into a sequence of l tokens x. the language model converts x into more useful representation, lm (x) ∈ r l×d , where d is the number of hidden dimensions (a property of the lm). finally, we feed lm (x) into a linear layer to predict the emotion label, ê. majority baseline. we use the majority emotion label for each artwork as the predicted emotion for all captions belonging to that artwork. concretely, each artwork, i, has a set of caption-emotion pairs, s. the majority classifier outputs the most frequent emotion, ê, in the set s for all of the captions in the set, @xmath0, language models. we finetune 3 models based on bert (bert-e, bert-a and bert-c), where bert-e is tuned for english, and bert-a is tuned for arabic and bert-c is tuned for chinese. section 11.2 discusses more pretraining and finetuning details. we also finetune 4 models based on cross language models, xlm-roberta @xcite , where xlm-e, xlm-a and xlm-c correspond to english, arabic, and chinese languages, as before. in addition, we create xlm-ace by training on the combination of all 3 languages. 3-headed transformer. finally, we create a model with xlm-r backbone but replace the single classifier head with 3 classifier heads, one for each of the 3 languages. while training, we feed the captions from each language to the shared backbone and then use the corresponding head to predict an emotion that would ultimately reflect the culture of that language. geva et al. (2021) analyzed similar multi-headed transformers and showed how the non-target heads can be used to interpret the results of the target head. similarly, our 3-headed transformer can be used to predict 3 different emotions each one reflecting the culture norms represented in each language. we can then use these predictions to better understand the similarities and differences combines arabic (a), chinese (c), and english (e). "m" stands for mode where the majority vote between the 3 heads is used. for spanish we evaluate the models without any finetuning @xcite . we are interested in large off-diagonal values in figure 7 , especially between positive and negative emotions. for example, arabic disgust is often confused with english amusement. upon further investigation, we found nude paintings contributed ∼15% of these confusions. explicit content and alcohol are frowned upon in some arabic speaking communities, as illustrated by the second and third rows of religious symbols are also associated with large off-diagonal values in confusion matrices. the first row in transformer head input caption (gloss) e a c [a] a beautiful girl holding a jesus cross stomping on the devil awe awe fear [e] the woman on the ground isn't wearing any clothes amu. dis. amu. cont. cont. cont. the input is a caption in arabic (a), chinese (c) or english (e). the first column shows the language and a gloss. the last three columns show predictions for each head (with interesting differences across heads). zero-shot evaluation. we use spanish annotations in artelingo to evaluate models mentioned above in a zero-shot setting. the last column in the first relation suggests that 3-heads may not perform as well as xlm when there is plenty of data, but 3-heads may have advantages in low-resource and zero-shot settings. 3-heads are better for capturing interactions between languages. the second relation suggests that language transfer may be more effective across some language pairs than others. historically, spanish and english are both relatively close indo-european languages, 20 compared to semitic languages such as arabic. there has been much less contact @xcite between those languages and chinese. ## affective caption generation the previous section described baseline models for the first task: label prediction. this section will describe baseline models for the second task: affective caption generation. to this end, we follow achlioptas et al. (2021) and train two affective captioning models: show, attend, and tell (sat) @xcite and meshed memory transformer (m 2 ) @xcite . we use affective captioning models to refer to captioning models that generate affective captions. these captions connect the dots between input paintings and emotions. sat is a lstm @xcite based captioning model with an attention module, it consists of a visual encoder and a text decoder. the visual encoder extracts visual features from an input image. the decoder then uses a stack of an attention module and lstm recurrent unit to generate a caption autoregressively. m 2 is a transformer based model @xcite which utilizes a pretrained faster-rcnn @xcite object detector to extract visual region features. these features are used as an input sequence to a multi-layer attention based encoder. m 2 differs from basic transformers by feeding the encoded features from all encoder layers to the cross attention module in each decoder's layer. in order to include emotion and language grounding, we use a simple embedding layer to convert the emotion and language labels into feature vectors and then concatenate them to the visual features. experimental setup. for both models, we use the default parameters proposed in @xcite . we train four different versions of each model, three versions are trained on english, arabic, and chinese only datasets, while the fourth version is trained on the three languages combined. we then test all the models on all the languages. in results. we report the results of our baseline models in ## conclusion this paper introduced artelingo, a multilingual dataset and benchmark on wikiart images with more than 1.2m captions and emotion labels. the benchmark has diverse emotional experiences constructed over different cultures, and communicated in four languages (english, chinese, arabic, and spanish). we found more agreement for some genres such as landscapes and more disagreement for other genres such as sketches. these differences are interesting and important, and far from random. annotations for trees in figure 1c are labeled as sadness in english and chinese but contentment in arabic. people are likely to feel more comfortable with what they know. people raised in countries with lush forests are likely to prefer that, whereas people brought up in less humid environments are likely to prefer that. towards building more socially and multiculturally aware ai, we created baseline models for two tasks on artelingo: (1) emotion label prediction and (2) affective caption generation. for emotion label prediction, our best baseline model trained xlm on a combination of training data from all three languages (xlm-ace). we also created 3-headed transformers, training three heads for three languages (arabic, chinese, and english) at the same time. the performance of this model is close to xml-ace, but generalizes better in a zero-shot experiment on spanish. for the caption generation task, we trained two models on artelingo, sat and m 2 . for english and arabic, models on all three languages have a similar performance to language specific models, but for chinese, it is best to train without the other languages since the performance drop is significant. we hope our benchmark and baselines will help ease future research in visually-grounded language models that can communicate affectively with us. in addition, artelingo can provide empirical examples of cross-cultural similarities and differences. sociologists and cultural psychologists may formulate hypotheses and conduct field studies based on artelingo. data, code, and models are publicly available at www.artelingo.org/ .