---
license: mit
---


# Contextual Emotion Recognition with LLaVA

This repository contains the checkpoints of fine-tuned LLaVA for the contextual emotion recognition task as described in our paper. 
For more information about LLaVA, visit [LLaVA Official Website](https://llava-vl.github.io).

## Fine-tuning

We finetuned LLaVA on the contextual emotion recognition task using the EMOTIC [EMOTIC Dataset](https://s3.sunai.uoc.edu/emotic/index.html) datasets:

1. **EMOTIC Train Set:** 
   - Precision: 54.27
   - F1 Score: 22.73

2. **EMOTIC Validation Set with Augmentation:**
   - F1 Score: 36.83
   - Precision: 38.71


We provided the LoRA weights for our fine-tuned models. The base model used for fine-tuning is `llava-v1.5-13b` which is available on the Hugging Face Model Hub. You can find the model [here](https://huggingface.co/liuhaotian/llava-v1.5-13b/tree/main).

## Usage

To perform contextual emotion recognition using our fine-tuned model, follow these steps:

1. Prepare your input:
   - An image with a bounding box of the target individual.
   - Text prompt: From suffering, pain, aversion, disapproval, anger, fear, annoyance, fatigue, disquietment, doubt/confusion, embarrassment, disconnection, affection, confidence, engagement, happiness, peace, pleasure, esteem, excitement, anticipation, yearning, sensitivity, surprise, sadness, and sympathy, pick the top labels that the person in the red bounding box is feeling at the same time.

2. Run LLaVA using our provided LoRA weights and the base model.

3. Receive the output which includes the emotion labels that the target individual is feeling.