F eeling B lue: A Corpus for Understanding the Emotional Connotation of Color in Context (2023)

Abstract: While the link between color and emotion has been widely studied, how context-based changes in color impact the intensity of perceived emotions is not well understood. In this work, we present a new multimodal dataset for exploring the emotional connotation of color as mediated by line, stroke, texture, shape, and language. Our dataset, FeelingBlue, is a collection of 19,788 4-tuples of abstract art ranked by annotators according to their evoked emotions and paired with rationales for those annotations. Using this corpus, we present a baseline for a new task: Justified Affect Transformation. Given an image I, the task is to 1) recolor I to enhance a specified emotion e and 2) provide a textual justification for the change in e. Our model is an ensemble of deep neural networks which takes I, generates an emotionally transformed color palette p conditioned on I, applies p to I, and then justifies the color transformation in text via a visual-linguistic model. Experimental results shed light on the emotional connotation of color in context, demonstrating both the promise of our approach on this challenging task and the considerable potential for future investigations enabled by our corpus.1

## introduction color is a powerful tool for conveying emotion across cultures, a connection apparent in both language and art @xcite . metaphoric language frequently uses color as a vehicle for emotion: familiar english metaphors include ''feeling blue'' or ''green with envy''. similarly, artists often pick specific colors in order to convey particular emotions in their work, while viewer perceptions of a piece of art are affected by its color palette @xcite . previous 1 our dataset, code, and models are available at https:// github.com/amith-ananthram/feelingblue . our work is the first to explore these questions. we present feelingblue, a new corpus of relative emotion labels for abstract art paired with english rationales for the emotion labels (see figure 1 and section 3). a challenge with such annotations is the extreme subjectivity inherent to emotion. in contrast to existing likert-based corpora, we employ a best-worst scaling (bws) annotation scheme that is more consistent and replicable @xcite . moreover, as our focus is color in context (colors and their form), we restrict our corpus to abstract art, a genre where color is often the focus of the experience, mitigating the effect of confounding factors like facial expressions and recognizable objects on perceived emotions (as observed in @xcite @xcite . to demonstrate feelingblue's usefulness in explorations of the emotional connotation of color in context, we introduce a novel task, justified affect transformation-conditional on an input image i o and an emotion e, the task is 1) to recolor i o to produce an image i e that evokes e more intensely than i o and 2) to provide justifications for why i o evokes e less intensely and why i e evokes e more intensely. using feelingblue, we figure 1 : representative examples spanning feelingblue's emotion subsets. each image in an emotion subset has a @xmath0, 1] derived from its best-worst scaling annotations. images selected as the ''least''/''most'' emotional in a 4-tuple (not shown here) have rationales explaining why they are ''less''/''more'' emotional than the rest. information about these works can be found at https://github.com/amith-ananthram /feelingblue . build a baseline system for two subtasks: image recoloring and rationale retrieval (section 4). we conduct a thorough human evaluation that confirms both the promise of our approach on this challenging new task and the opportunity for future investigations enabled by feelingblue. our results reveal regularities between context-based changes in color and emotion while also demonstrating the potential of linguistic framing to mold this subjective human experience (section 5). our dataset, code, and models are available at https://github.com/amith-ananthram /feelingblue . ## related work while the body of work studying the relationship between color and emotion is quite large, almost all of it has focused on identifying categorical relationships in text to produce association lexicons (notable works include @xcite @xcite . in the domain of affective image analysis, previous work has mostly explored classifying the emotional content of images. machajdik and image recoloring is a very small subset of work in the field of image transformation and has mostly focused on palettes. palettenet recolors a source image given a target image palette @xcite while @xcite semantically generate palettes for coloring gray-scaled images. no previous work has examined recoloring images to change their perceived emotional content. similarly, the field of text-to-image synthesis, which has seen major progress recently with models like dall-e 2 @xcite , has centered on generating images de novo or on in-painting. there has been no work that recolors an image while preserving its original structure. ulinski et al. (2012 3 dataset our dataset, feelingblue, is a collection of abstract art ranked by the emotions they most evoke with accompanying textual justifications (see figure 1 ). it contains 5 overlapping subsets of images (one each for anger, disgust, fear, happiness, and sadness) with continuous-value scores that measure the intensity of the respective emotion in each image compared to the other images in the subset. while wikiart (mohammad and kiritchenko, 2018), deviantart foot_1 @xcite , and other emotion corpora contain images with multi-label continuous emotion scores, these scores were collected for each image in isolation without accompanying rationales. they reflect how often annotators believed that a particular emotion fit an image resulting in a measure of the presence of the emotion rather than its intensity. as such, their scores are not a suitable way to order these images by the strength of the emotion they evoke. in contrast, our annotations were collected by asking annotators to 1) rank groups of images according to a specified emotion and 2) justify their choice. below, we detail how we compiled this corpus. ## justified affect transformation we define a new task to serve as a vehicle for exploring the emotional connotation of color in context enabled by our corpus: justified affect transformation. given an image i o and a target emotion @xmath1, the task is: to solve this task, we propose a two step approach: 1) an image recoloring component that takes as input an image i o and a target emotion @xmath2, a version of i o recolored to better evoke e (section 4.1); and 2) a rationale retrieval component that takes as input two images, i o and i e , an emotion @xmath3, and a large set of candidate rationales r, and outputs a ranked list of rationales r less that justify why i o evokes e less than i e and a ranked list of rationales r more that justify why i e evokes e more than i o (section 4.2). ## results and discussion to understand our approach's strengths, we evaluate our image recoloring model and rationale retrieval model both separately and together. evaluation data for imaging recoloring. the evaluation set for our image recoloring model contains 100 images-30 are randomly selected from our validation set and the remaining 70 are unseen images from wikiart. we generate 5 recolorized versions i e of each image i o corresponding to each of our 5 emotions, resulting in 1000 recolored variants for evaluation (see section 5.1). ## conclusion we introduce feelingblue, a new corpus of abstract art with relative emotion labels and english rationales. enabled by this dataset, we present a baseline system for justified affect transformation, the novel task of 1) recoloring an image to enhance a specific emotion and 2) providing a textual rationale for the recoloring. our results reveal insights into the emotional connotation of color in context: its potential is constrained by its form and effective justifications of its effects can range from the general to the specific. they also suggest an interesting direction for future work-how much is our emotional response to color affected by linguistic framing? we hope that feelingblue will enable such future inquiries.