Impressions: Visual Semiotics and Aesthetic Impact Understanding (2023)

Abstract: Is aesthetic impact different from beauty? Is visual salience a reflection of its capacity for effective communication? We present Impressions, a novel dataset through which to investigate the semiotics of images, and how specific visual features and design choices can elicit specific emotions, thoughts and beliefs. We posit that the impactfulness of an image extends beyond formal definitions of aesthetics, to its success as a communicative act, where style contributes as much to meaning formation as the subject matter. We also acknowledge that existing Image Captioning datasets are not designed to empower state-of-the-art architectures to model potential human impressions or interpretations of images. To fill this need, we design an annotation task heavily inspired by image analysis techniques in the Visual Arts to collect 1,440 image-caption pairs and 4,320 unique annotations exploring impact, pragmatic image description, impressions and aesthetic design choices. We show that existing multimodal image captioning and conditional generation models struggle to simulate plausible human responses to images. However, this dataset significantly improves their ability to model impressions and aesthetic evaluations of images through fine-tuning and few-shot adaptation.

## introduction the image captioning datasets used to train multimodal architectures such as blip @xcite , m2 transformer @xcite , and flamingo @xcite , contain terse and reductive captions that describe their visual counterparts. this is because annotators were encouraged to write single-sentence captions that focused on the most concrete elements of an image: objects, entities, shapes, colors, etc. for tasks such as object detection and scene segmentation, this is the intuitive approach. but as transformer-based multimodal architectures grow more proficient, training solely on datasets such as coco captions @xcite , conceptual captions @xcite , and wit @xcite will inhibit models' ability to reason about the semiotics of images. this motivates our impressions dataset, which captions images with semiotic and aesthetic elements that ground viewers' subjective perceptions (see figure 2 ) as highlighted by @xcite , an observer cannot view an image without imposing upon it values, beliefs, and expectations that are highly grounded in their understanding of cultural norms and social cues. humans instinctively make assumptions about emotions, actions, and relationships -setting, space, and circumstance -past, present, and future. furthermore, these inferences are not solely dependent on the concrete objects and entities in an image. photojournalists often manipulate composition, lighting, exposure, and camera angles to direct an audience's perception of the subject matter. aesthetic design choices shape the meaning of an image, and can communicate just as much as the concrete elements depicted. to better understand this phenomenon, @xcite applied the principles of denotation and connotation to visual symbols in photography. denotation is the meaning carried by the literal depictions in the image. whereas connotation is the additional meaning assigned to a signifier that can be dependent on (1) rules and conventions the observer is familiar with, (2) visual and aesthetic elements of the image, or (3) cultural implication of the depiction as opposed to what is really there. few datasets have ventured to encourage pragmatic inferences on visual scenes and collect audience interpretations and impressions of images. yet an understanding of an audience's likely interpretations of visual media could empower multimodal dialogue systems, where images can be used in combination with text to communicate. furthermore, a better understanding of the connotation of visual symbols can enable text-to-image architectures to capture more nuance in language, thus rendering more salient compositions and styles in the output. to this end, we contribute the following: visual semiotics and communication this work is heavily inspired by the discussion of the semiotics of visual media in @xcite , specifically how connotation and denotation of signifiers are extended from linguistics to visual studies. additionally, in focusing on gathering human impressions we analyze the perlocutionary force of images (their meaning as perceived by an audience). @xcite , artemis @xcite , and ava @xcite datasets in respective order on the left. each visualization is taken directly from the cited works. in contrast to existing benchmarks, impressions contains rich commentary on viewer perception that is not constrained to emotion, and a discussion of aesthetic elements that is grounded in the impressions that inspire. perlocution is among the types of speech acts, or communicative acts, identified by @xcite . image captioning most image captioning tasks are framed with literal descriptions of the constituent objects, such as their behaviors, quantities and attributes. in the popular coco @xcite , wit @xcite , and conceptual captions @xcite benchmarks, these descriptions have a neutral, objective tone. flickr30k @xcite takes it a step further, presenting crowd-sourced annotations of images depicting people going about everyday activities in an effort to study the visual denotation of linguistic expressions. more recent work conditions image caption generation on non-neutral stylistic features like humor @xcite , romance @xcite , sentiment @xcite , or personality traits @xcite . in a similar effort to our own, achlioptas et al. (2021) built a dataset for affective captions to media in the visual arts domain. the most important difference between this work and our own is that we consider additional impressions beyond affect. not every image is going to give the viewer an emotive response but could alternatively inspire them to think or believe something. most image captioning systems lack knowledge of symbolic iconography, cultural conventions, etc. @xcite . by opening the question of human impression in this way, we aim to resolve a better understanding of social queues, cultural norms, and popular connotations of visual symbols through annotator responses. a breath of prior work has endeavored to generate pragmatic image captioning through context-agnostic supervision @xcite , listener and speaker models @xcite , relaxation of language encoding @xcite , and explicit image-caption pair annotation @xcite . interestingly, datasets such as flickr30k have been shown to unintentionally resolve annotator inference @xcite , yielding more pragmatic and biased captions than intended. ## impressions dataset the impressions dataset consists of 1,440 images and 4,320 distinct impression annotations. each annotation is composed of three free-form responses to questions addressing (1) image description, (2) viewers' impression of the image, and (3) aesthetic image evaluation grounded in the viewers' impression. additionally, we present aesthetic impact scores (in a range of 1 to 4) for 3,450 images. in the section below we will describe our process for collecting, annotating, and analyzing the dataset. ## evaluation to demonstrate the impressions dataset's ability to enable state-of-the-art image-captioning and conditional-generation models to simulate human impressions and resolve impactful aesthetic elements of images, we design the following experiment. git @xcite , blip @xcite , openflamingo @xcite , and llava @xcite are fine-tuned / few-shotadapted using a training set of 1,340 images and 4,020 impression annotations (each containing 3 unique captions). git and blip were fine-tuned for image-captioning separately on each of the different impressions data types (description, perception, aesthetic evaluation). openflamingo was similarly few-shot adapted for vqa using 16 examples of one data type at a time. llava was fine-tuned for vqa on all annotations simultaneously via an instruction-tuning training task. automatic evaluation human evaluation each fine-tuned/few-shot adapted model is evaluated on 300 image-prompt pairs. the prompts are divided evenly between questions targeting image description, perception, and aesthetic evaluation. for the models fine-tuned for image-captioning, 100 captions are generated per annotation category. this same process is repeated on the base pretrained git, blip and llava architectures, and in the zero-shot setting with openflamingo. this produced a final set of 300 generation pairs, where each pair contains one caption from the fine-tuned or adapted model and one from the base. we submitted the 300 generation pairs to upwork and requested that annotators identify which caption better simulates a potential human impression and identifies impactful aesthetic elements (see appendix i). for each generation pair we collect 3 unique annotator votes and select the majority vote as the final evaluation. the results of this human evaluation task are displayed in the results in image description generations on image description were least improved by fine-tuning on the impressions dataset across all architectures explored in this study. however, as a plethora of datasets have been created for this purpose, this result was expected. the greatest improvement on descriptive generations was by fine-tuning git to achieve 75% preferential improvement over the base model. this was followed by fine-tuned blip, which was preferred 69% over blip base. this indicates that the impression dataset facilitated model learning of captions that most closely aligned with viewers' perspectives. ## persona-specific generation to investigate the variation in human perceptions of images captured by impressions, we design a set of experiments exploring the distinctive gen-eration qualities that may emerge when training multi-modal models on annotations created by individuals belonging to different personality or demographic groups. prior to beginning the image annotation task, annotators completed two surveys on personality traits and demographic information through amazon mechanical turk (see appendix e). to build each persona-specific llava model of image perception, we fine-tune on a random sample of 500 annotations from the respective personality or demographic group: introvert vs. extrovert, agreeable vs. disagreeable, business-oriented occupation vs. creative occupation, and no art experience vs. 3+ years of art experience. an evaluation set of 100 images is leveraged to produce image perceptions with each of the eight llava-7b-v0 models fine-tuned on personaspecific data. we then compare distributions of sen- the image showcases a city skyline with a body of water in the foreground, featuring boats and a small waterway. the sun is shining, casting warm light on the scene. this picturesque view makes me feel a sense of awe for the combination of urban architecture and natural elements. overall, this captivating view evokes positive feelings and a desire to explore the city further. ## conclusion impressions was designed with inspiration from visual arts and media studies to evaluate and enhance multimodal models' ability to reason about visual connotation. we show that, through finetuning and few-shot learning, this dataset enabled an array of architectures to resolve potential human impressions and discuss impactful aesthetic elements. state-of-the-art vision models are proficient enough at resolving entities and stylistic attributes to support such a task, but the weakness existed on the language side. this work highlights that targeted prompts modeled after image analysis techniques succeed in teasing out complex commentary on perlocation and the aesthetic elements it is grounded in. these prompts can be used by future works such as vqa and instruction-tuning, with applications in multimodal dialogue systems, text-to-image generation, and engagement prediction in advertising. 12281 perhaps the most noticeable limitation of the impressions dataset is its size. due to resources constraints, the benchmark contains only 1,440 images with 3 unique annotations each. an increase in images would have allowed for a wider exploration of visual symbols, and an increase in annotations per image would have better resolved the natural variation in audience impression. additionally, we acknowledge that by welcoming inference in the annotation task, we also risk introducing harmful, biased, and discriminatory ideas. although the authors have not observed any such content in data quality checks, this dataset would benefit from an exploration on potential bias resolution.