Visual Prompting in LLM s for Enhancing Emotion Recognition (2024)

Abstract: Vision Large Language Models (VLLMs) are transforming the intersection of computer vision and natural language processing; however, the potential of using visual prompts for emotion recognition in these models remains largely unexplored and untapped. Traditional methods in VLLMs struggle with spatial localization and often discard valuable global context. We propose a novel Set-of-Vision prompting (SoV) approach that enhances zero-shot emotion recognition by using spatial information, such as bounding boxes and facial landmarks, to mark targets precisely. SoV improves accuracy in face count and emotion categorization while preserving the enriched image context. Through comprehensive experimentation and analysis of recent commercial or open-source VLLMs, we evaluate the SoV model’s ability to comprehend facial expressions in natural environments. Our findings demonstrate the effectiveness of integrating spatial visual prompts into VLLMs for improving emotion recognition performance.

## introduction as the integration of computer vision and natural language processing progresses, vllms @xcite are revolutionizing the way machines interpret visual and textual data. emotion recognition is gaining considerable interest across multiple disciplines and presents distinct challenges @xcite . it requires the decoding of emotions from nuanced indicators like facial expressions, body language, and contextual details. previous methods, such as those in @xcite and @xcite , enhance in-context emotion classification by training transformerbased models or clip to generate descriptions of emotions in visual contexts. however, these methods overlook the spatial relationships between different people and facial features within a single face. these spatial relationships can be labeled with numbers and bounding boxes in our sov prompts to guide vllms. a person nearby may have similar facial expressions, while a person farther away may show different facial expressions. additionally, the relationships between the eyes, mouth, and nose features can be highlighted by facial landmarks in sov prompts to guide vllms. recent studies @xcite @xcite have explored visual prompting, a technique employed in image-language tasks to guide llms by incorporating markers such as colorful boxes or circles to emphasize specific targets within an image. reclip @xcite adds colorful boxes directly onto an image to highlight specific targets and blurs other irrelevant areas to reduce the performance gap with supervised models on both real and synthetic datasets. additionally, redcircle @xcite employs visual prompt engineering, specifically drawing a red circle around an object in an image, to direct a vision-language model's attention to that region and enhance its performance in tasks like zero-shot keypoint localization. however, both of these approaches focus on local objects and ignore spatial context information. yang et al. @xcite propose using fine-grained visual prompts, such as segmentation masks, and enhancing focus on relevant areas with a 'blur reverse mask' that blurs regions outside the target mask to minimize distractions and maintain spatial context. although visual prompting techniques have garnered interest, their full potential remains unexplored for emotion recognition tasks. current approaches rely solely on coarse markers like colorful boxes, circles, or masks, which can introduce ambiguity, blur the face images, and pose challenges for accurate recognition tasks. this paper addresses these issues by systematically organizing and investigating various forms of visual prompting. furthermore, we propose a new prompting approach called set-of-vision prompting (sov) in fig. 1 , which utilizes spatial information such as numbers, bounding boxes, and facial landmarks to precisely mark each target while maintaining background context, thereby enhancing the zero-shot performance of facial expression recognition. the top of fig. 2 , shows an approach where specific vision prompts are not used. as a result, the analysis inaccurately counts 22 visible faces and misclassifies persons' emotions into incorrect categories, with a number of faces labeled under 'neutral emotion' and fewer under 'mildly positive emotion' and 'happy'. this misclassification and miscount demonstrate the limitations when detailed visual cues are not utilized in the analysis. in the bottom of fig. 2 , the use of sov prompts, such as numbering each face, placing bounding boxes, and identifying facial landmarks, allows for a more precise analysis. the correct number of faces is identified (18), and the emotions are accurately categorized into more nuanced groups: 'neutral emotion', 'mildly positive emotion', and 'smiling or happy'. this method provides a clearer and more detailed breakdown of each individual's emotional state based on visible facial expressions. this comparison highlights the importance and effectiveness of integrating visual prompts in vllms analysis for more accurate and detailed recognition and categorization of human emotions in images. to summarize, our main contributions are: (1) the paper introduces a novel visual prompting method (sov) that highlights facial regions directly within the entire image. this preserves background context, enhancing the ability of vllms to perform accurate emotion recognition without the need for cropping faces, thus maintaining the holistic view of the image. (2) the proposed face overlap handling algorithm effectively addresses conflicts arising from overlapping face detections, especially in images with dense face clusters. by prioritizing larger faces and iteratively checking for overlaps, the algorithm ensures that non-occluded faces are retained for subsequent emotion analysis. (3) our results show that incorporating spatial visual prompts (sovs) into vllms can enhance their performance in recognizing emotions. ## limitations we suggest the implementation of set-of-vision prompting to bridge visual and textual prompts. however, a challenge arises as it is difficult to precisely describe visual prompts, such as the shape, location, or color of a bounding box and facial landmarks, in language. this issue might require encoding the visual prompts and fine-tuning the entire model for better accuracy. moreover, this method is computationally intensive, potentially limiting its scalability and practicality in real-time applications, especially when handling large datasets or streaming videos such as tracking one person's emotion in different video frames. ## conclusion in conclusion, our set-of-vision prompting (sov) approach significantly advances the field of emotion recognition within vllms by addressing critical challenges in spatial localization and global context preservation. by leveraging spatial information such as bounding boxes and facial landmarks, sov enhances zero-shot emotion recognition accuracy, ensuring precise face count and emotion categorization. our face overlap handling algorithm and combined text-vision prompting strategy further refine the recognition process, highlighting the efficacy of integrating visual prompts in vllms for more accurate and detailed emotion analysis. this approach not only preserves the enriched image context but also offers a solution for detailed and nuanced emotion recognition, underscoring its potential impact on various applications within computer vision and natural language processing.