E mo G ist: Efficient In-Context Learning for Visual Emotion Understanding (2025)

Abstract: In this paper, we introduce EmoGist, a training-free, in-context learning method for performing visual emotion classification with LVLMs. The key intuition of our approach is that context-dependent definition of emotion labels could allow more accurate predictions of emotions, as the ways in which emotions manifest within images are highly context dependent and nuanced. EmoGist pre-generates multiple descriptions of emotion labels, by analyzing the clusters of example images belonging to each label. At test time, we retrieve a version of description based on the cosine similarity of test image to cluster centroids, and feed it together with the test image to a fast LVLM for classification. Through our experiments, we show that EmoGist allows up to 12 points improvement in micro F1 scores with the multi-label Memotion dataset, and up to 8 points in macro F1 in the multi-class FI dataset.

## introduction automated classification of visual emotion @xcite @xcite is an extremely challenging problem, as the ways in which emotions are embedded within images are inherently nuanced. hence, even large vision-language models (lvlms) that are extensively trained for reasoning over visual inputs struggle in detecting these emotions @xcite , as their training may not necessarily involve the ability to understand such nuanced patterns. in this paper, we introduce emogist, a trainingfree, in-context learning method for performing visual emotion classification with lvlms. the key intuition of our approach is that the real meaning of different emotion labels could be dependent on the image's context. for example, we could intuitively imagine that the way the emotion of 'excitement' for sporting events could be significantly different from the 'excitement' of the academics for an upcoming conference. hence, guiding lvlms with such context dependent definition of emotion labels could allow the models to better focus on the nuanced patterns of the image. emogist automatically pre-generates nuanced, context-specific descriptions of emotion labels, by analyzing the clusters of example images belonging to each label. at test time, we retrieve a version of description based on the cosine similarity of test image to cluster centroids, and feed it together with the test image to a fast lvlm for classification. through our experiments, we show that em-ogist allows up to 12 points improvement in micro f1 scores with the multi-label emotion classification, and up to 8 points improvement in macro f1 for the multi-class case. we also demonstrate that emogist could achieve improvements in smaller lvlms with 2 billion parameters. 1 2 related work while visual emotions has been extensively studied in the many related fields including computer vision and psychology @xcite @xcite @xcite , we are only starting to see the efforts to exploit large vision-language models for automated understanding of visual emotions @xcite @xcite @xcite . visual in-context learning (icl) has seen considerable amount of interest in recent literature @xcite @xcite , where many work have investigated effective strategies for choosing visual icl examples given the test instance. however, we believe that our work is first to investigate icl strategies with lvlms in detail for evoked emotion classification. ## emogist we describe the major components of emogist, our in-context learning method with lvlms for emotion classification. instead of naively retrieving individual examples based on the visual similarity of the image, the key idea is to obtain an nuanced description of emotion labels, which could effectively serve as the decision boundary for the lvlm to make its predictions on. because the same emotion could manifest in many different ways for across different images, we develop a strategy where we utilize stronger lvlms to pre-generate multiple descriptions of different emotion labels, by analyzing the clusters of example images belonging to each category. ## experiments we used two datasets: one is the memotion 1.0 dataset @xcite , a collection of meme images from social media where we perform a multi-label classification across 4 labels: sarcastic, humorous, offensive, and motivational. the second dataset is the fi dataset @xcite , a collection of everyday images across the internet tagged based on the ekman model @xcite of 8 classes: amusement, anger, awe, contentment, disgust, excitement, fear, and sadness. foot_3 ## conclusion and future work in this paper, we introduced emogist, a trainingfree in-cotext learning method for visual emotion understanding with large vision-language models. we observe a significant amount of improvements over the zero-shot and naive icl baselines across sota lvlms of 2 and 7 billion parameters. in particular, we find that emogist e , the variant of our method with simple ensembling, achieves robust performance improvements with higher precision. in future work, we'd like to explore more deeply into the label descriptions generated by the strong lvlms and investigate various reasoning strategies for obtaining emotion label descriptions that are more transferrable across different domains.