[Video-NeVA] Implement Mixtral-As-Judge Eval Pipeline
State: closed | Pull request: yes
Labels: name: stale

name: Multi Modal

# What does this PR do ?

This PR adapts LLaVA's evaluation pipeline leveraging GPT-4 as a judge, but uses NeMo NGC 8xMixtral-7B model as judge.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Adds `mixtral_eval.py` to NeVA eval pipeline.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


**EVAL RESULTS ON IMAGE INFERENCE FOR ~10% of LLaVA-Bench-In-The-Wild**

- **Assistant 1:** NeVA (No Video)
- **Assistant 2:** V-NeVA (start from NeVA image ckpt, train on video represented as 24 frames)

```
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 19418.07it/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 25890.77it/s]
[Context]
An aerial view of Diamond Head in the Hawaiian Islands.

[Question]
What is the name of this famous sight in the photo?

[Assistant 1]
The name of the famous sight in the photo is the mountain range with the large hill in the middle, which is part of the island. This mountain range and hill are located near the ocean, and the photo provides an aerial view of the landscape. The presence of the mountain range and the large hill contribute to the unique and picturesque scenery of the island.

[End of Assistant 1]

[Assistant 2]
The famous sight in the photo is the Haleakalā Crater on the island of Maui in Hawaii. Haleakalā is a dormant shield volcano that forms more than 75% of the island's surface area and is one of the largest volcanoes on Earth. It is a popular destination for tourists and is known for its stunning views and unique geological features.

[End of Assistant 2]

[System]
We would like to request your feedback on the performance of two AI assistants in response to the user question displayed above. The user asks the question on observing an image. For your reference, the visual content in the image is represented with a few sentences describing the image. 
Please rate the helpfulness, relevance, accuracy, level of details of their responses. Each assistant receives an overall score on a scale of 1 to 10, where a higher score indicates better overall performance.
Please first output a single line containing only two values indicating the scores for Assistant 1 and 2, respectively. The two scores are separated by a space.
In the subsequent line, please provide a comprehensive explanation of your evaluation, avoiding any potential bias and ensuring that the order in which the responses were presented does not affect your judgment.


1 10

Assistant 1's response is not helpful or accurate as it does not provide a specific name for the sight in the photo. The response is vague and does not provide any relevant information about the location or significance of the sight.

Assistant 2's response, on the other hand, is highly helpful, relevant, accurate, and detailed. The response correctly identifies the sight in the photo as the Haleakalā Crater on the island of Maui in Hawaii. The response also provides additional context and information about the sight, including its geological features and popularity among tourists. Overall, Assistant 2's response is a model of clarity and precision.
1
[Context]
A photo of four mangosteens on a wooden table. Three of them are uncut, while one is cut open.

[Question]
What type of fruit is this?

[Assistant 1]
The image features a group of round, dark-colored fruits, possibly berries, with white accents, lying on a wooden surface. These fruits are cut open, revealing their delicious and juicy interiors. The arrangement of the fruits in the image suggests that they are being presented in a visually appealing manner, possibly for consumption or display purposes.

[End of Assistant 1]

[Assistant 2]
The fruit in the image is a dragon fruit, also known as pitaya. It's characterized by its vibrant pink or yellow skin with green scales and a white or red flesh with small black seeds. Dragon fruit is a tropical fruit that is popular for its unique appearance and sweet taste.

[End of Assistant 2]

[System]
We would like to request your feedback on the performance of two AI assistants in response to the user question displayed above. The user asks the question on observing an image. For your reference, the visual content in the image is represented with a few sentences describing the image. 
Please rate the helpfulness, relevance, accuracy, level of details of their responses. Each assistant receives an overall score on a scale of 1 to 10, where a higher score indicates better overall performance.
Please first output a single line containing only two values indicating the scores for Assistant 1 and 2, respectively. The two scores are separated by a space.
In the subsequent line, please provide a comprehensive explanation of your evaluation, avoiding any potential bias and ensuring that the order in which the responses were presented does not affect your judgment.


4 7

Assistant 1's response is not accurate or helpful, as they were unable to identify the fruit in the image. Their description of the fruit being a "round, dark-colored fruit with white accents" is too vague and does not provide any specific information about the fruit.

Assistant 2, on the other hand, correctly identified the fruit in the image as a dragon fruit. Their description is accurate and provides relevant information about the fruit's appearance and taste. However, they could have added more details about the fruit, such as its nutritional benefits or how it is typically consumed.

Overall, Assistant 2's response is more helpful and accurate than Assistant 1's, and I would rate their performance higher. However, both assistants could benefit from providing more detailed and informative responses to the user's question.
2
[Context]
A creative painting of a dog dressed as the famous Mona Lisa.

[Question]
Describe this photo in detail.

[Assistant 1]
The image features a dog dressed up to look like a person, specifically like art supremo dog, complete with a paint palette and brush. The dog is positioned in the center of the frame, with its body extended to fill the entire image. The dog's head is turned to the side, as if it's looking at something off-camera. The backdrop of the image is a beautiful landscape, which adds to the artistic composition. The dog's outfit and pose create a sense of whimsy and fun, making the image an amusing and creative portrayal of a dog in the role of an artist.

[End of Assistant 1]

[Assistant 2]
The image is a digital manipulation that combines elements of Leonardo da Vinci's famous painting 'Mona Lisa' with the features of a dog. The central figure in the image is a dog with a human-like posture, wearing a dark garment that drapes over one shoulder, similar to the attire of the figure in the original 'Mona Lisa' painting. The dog's fur is rendered in shades of brown, and it has a black nose and dark eyes. The background of the image is a landscape with a body of water, mountains, and a cloudy sky, which is consistent with the background of the original painting. The overall style of the image is a blend of classical art and modern digital art, with a focus on humor and surrealism.

[End of Assistant 2]

[System]
We would like to request your feedback on the performance of two AI assistants in response to the user question displayed above. The user asks the question on observing an image. For your reference, the visual content in the image is represented with a few sentences describing the image. 
Please rate the helpfulness, relevance, accuracy, level of details of their responses. Each assistant receives an overall score on a scale of 1 to 10, where a higher score indicates better overall performance.
Please first output a single line containing only two values indicating the scores for Assistant 1 and 2, respectively. The two scores are separated by a space.
In the subsequent line, please provide a comprehensive explanation of your evaluation, avoiding any potential bias and ensuring that the order in which the responses were presented does not affect your judgment.


9, 9

Assistant 1 and Assistant 2 both provided detailed and relevant descriptions of the image. They both accurately captured the central elements of the image, including the dog dressed as the Mona Lisa and the landscape background. They also both noted the whimsical and creative nature of the image.

Assistant 1's response was slightly more concise, while Assistant 2's response provided a more detailed description of the dog's appearance and the background. Both responses were equally accurate and relevant, and they both provided a high level of detail.

Overall, both assistants performed very well in this task. They were able to accurately and thoroughly describe the image, capturing its key features and themes. It is difficult to choose a clear winner between the two responses, as they were both excellent. Therefore, I have given them both a high score of 9.
error 9, 9

Assistant 1 and Assistant 2 both provided detailed and relevant descriptions of the image. They both accurately captured the central elements of the image, including the dog dressed as the Mona Lisa and the landscape background. They also both noted the whimsical and creative nature of the image.

Assistant 1's response was slightly more concise, while Assistant 2's response provided a more detailed description of the dog's appearance and the background. Both responses were equally accurate and relevant, and they both provided a high level of detail.

Overall, both assistants performed very well in this task. They were able to accurately and thoroughly describe the image, capturing its key features and themes. It is difficult to choose a clear winner between the two responses, as they were both excellent. Therefore, I have given them both a high score of 9.
3
[Context]
A creative meme: Elon Musk holding a dog. It mimicks one of the most memorable scenes from The Lion King, where Rafiki stands on a mountaintop and holds up Simba to all the other animals.

[Question]
What is the name of the man in the photo?

[Assistant 1]
The image features a man dressed up as a character from the popular animated movie 'Simpsons.' He is holding a dog in his arms, and the dog appears to be a real dog, not an animatronics character from the movie. The man is wearing a costume that resembles a monkey, and he is standing in front of a backdrop that features a landscape. The scene is set in a location that resembles the fictional town of Springfield, where the Simpsons family resides. The man's costume and the presence of the dog create a fun and entertaining atmosphere, showcasing the creativity and detail of the man's outfit.

[End of Assistant 1]

[Assistant 2]
The man in the photo is Elon Musk.

[End of Assistant 2]

[System]
We would like to request your feedback on the performance of two AI assistants in response to the user question displayed above. The user asks the question on observing an image. For your reference, the visual content in the image is represented with a few sentences describing the image. 
Please rate the helpfulness, relevance, accuracy, level of details of their responses. Each assistant receives an overall score on a scale of 1 to 10, where a higher score indicates better overall performance.
Please first output a single line containing only two values indicating the scores for Assistant 1 and 2, respectively. The two scores are separated by a space.
In the subsequent line, please provide a comprehensive explanation of your evaluation, avoiding any potential bias and ensuring that the order in which the responses were presented does not affect your judgment.


2 10

Assistant 1's response is detailed and provides a good description of the image, but it does not answer the question directly. The response focuses on the setting and the costumes, but it does not identify the man in the image. On the other hand, Assistant 2 correctly identifies the man in the image as Elon Musk, which directly answers the user's question. However, Assistant 2 does not provide any additional information about the image. Overall, while both responses lack some detail, Assistant 2's response is more helpful and accurate in answering the user's question.
4
[Context]
A creative meme: a dog lying on the cyan wooden floor. The top of the meme reads: "MONDAY." The bottom of the meme reads: "JUST...MONDAY."

[Question]
Please explain the meme in detail.

[Assistant 1]
In the image, a small brown dog is lying on a wooden floor, dressed up to resemble a person, specifically like art supremo dog. The dog is positioned in the center of the frame, with its body extended to fill the entire image. The dog's head is turned to the side, as if it's looking at something off-camera. The backdrop of the image features a beautiful landscape, which adds to the artistic composition. The dog's outfit and pose create a sense of whimsy and fun, making the image an amusing and creative portrayal of a dog in the role of an artist. The phrase 'Just Monday...' is written above the dog, emphasizing the playful nature of the scene.

[End of Assistant 1]

[Assistant 2]
The meme features an image of a dog lying on a wooden floor with a caption that reads 'MONDAY. JUST...MONDAY.' The dog appears to be resting or possibly sleeping, and its body language suggests a sense of relaxation or even exhaustion. The text is a humorous take on the common sentiment that Mondays can be difficult or tiring, often referred to as 'Monday blues' or 'Mondayitis.' The meme plays on the idea that Mondays are the start of the work or school week, and people often feel a bit sluggish or unmotivated after the weekend. The dog's relaxed posture is used to visually represent this feeling, and the repetition of the word 'MONDAY' with the ellipsis and the word 'just' adds emphasis to the sentiment.

[End of Assistant 2]

[System]
We would like to request your feedback on the performance of two AI assistants in response to the user question displayed above. The user asks the question on observing an image. For your reference, the visual content in the image is represented with a few sentences describing the image. 
Please rate the helpfulness, relevance, accuracy, level of details of their responses. Each assistant receives an overall score on a scale of 1 to 10, where a higher score indicates better overall performance.
Please first output a single line containing only two values indicating the scores for Assistant 1 and 2, respectively. The two scores are separated by a space.
In the subsequent line, please provide a comprehensive explanation of your evaluation, avoiding any potential bias and ensuring that the order in which the responses were presented does not affect your judgment.


8, 9

Assistant 1 provided a detailed description of the image, but missed the mark on explaining the meme's context and meaning. They accurately described the visual elements, but did not connect them to the underlying sentiment of the meme.

Assistant 2, on the other hand, provided a more comprehensive explanation of the meme's meaning and context. They accurately described the dog's posture and connected it to the common sentiment about Mondays. However, they could have provided more detail in describing the visual elements of the image.

Overall, both assistants demonstrated strengths and weaknesses in their responses. Assistant 2's response was more helpful and relevant to the user's question, but Assistant 1 provided more detail in describing the image.
error 8, 9

Assistant 1 provided a detailed description of the image, but missed the mark on explaining the meme's context and meaning. They accurately described the visual elements, but did not connect them to the underlying sentiment of the meme.

Assistant 2, on the other hand, provided a more comprehensive explanation of the meme's meaning and context. They accurately described the dog's posture and connected it to the common sentiment about Mondays. However, they could have provided more detail in describing the visual elements of the image.

Overall, both assistants demonstrated strengths and weaknesses in their responses. Assistant 2's response was more helpful and relevant to the user's question, but Assistant 1 provided more detail in describing the image.

=================================
  assistant_1   assistant_2
all              48.0  68.0
llava_bench_conv  23.3  73.3
llava_bench_detail 85.0  60.0
=================================

```

**LLaVA Performance As Reported In Paper**

![llava-perf](https://github.com/NVIDIA/NeMo/assets/30813477/6d4b65f1-2271-446e-8850-37a5eeb7b132)




EVAL RESULTS ON IMAGE INFERENCE FOR 100% of LLaVA-Bench-In-The-Wild

- Assistant 1: NeVA (No Video)
- Assistant 2: V-NeVA (start from NeVA image ckpt, train on video represented as 24 frames)

Run 1: 
```
=================================
all 65.9 77.4
llava_bench_complex 71.6 82.2
llava_bench_conv 50.0 69.4
llava_bench_detail  73.3 77.3
=================================
```

Run 2:
```
=================================
all 65.4 77.0
llava_bench_complex 71.6 82.2
llava_bench_conv 48.8 68.8
llava_bench_detail 72.7 76.7
=================================
```

Run 3:
```
=================================
all 65.9 77.0
llava_bench_complex 71.6 82.2
llava_bench_conv 50.6 68.8
llava_bench_detail 72.7 76.7
=================================
```

Aggregate Results:

```
=================================
all 65.56+/-0.29 77.13+/-0.23
llava_bench_complex 71.60+/-0.00 82.2+/-0.00
llava_bench_conv 49.8+/-0.92 69.00+/-0.35
llava_bench_detail 72.9+/-0.35 76.9+/-0.34
=================================
```

**LLaVA Results:**
![llava-perf](https://github.com/NVIDIA/NeMo/assets/30813477/06fba4e4-f2e4-4e42-907d-1ce97fd53bd2)

**VideoLLaVA Results:**
<img width="1220" alt="Screen Shot 2024-03-14 at 10 38 11 PM" src="https://github.com/NVIDIA/NeMo/assets/30813477/5687ab3a-aa7d-4d7e-9189-db5b516adf0a">



EVAL RESULTS ON IMAGE INFERENCE FOR 100% of LLaVA-Bench-In-The-Wild

- Assistant 1: GPT-4V (Source: LLaVA-Bench-In-The-Wild)
- Assistant 2: V-NeVA (start from NeVA image ckpt, train on video represented as 24 frames)

Run 1: 
```
=================================
all 84.9 64.5
llava_bench_complex 81.4 70.3
llava_bench_conv 84.1 61.8
llava_bench_detail 92.3 57.0
=================================
```

Run 2:
```
=================================
all 85.1 64.6
llava_bench_complex 81.8 70.5
llava_bench_conv 84.1 61.8
llava_bench_detail 92.3 57.0
=================================
```

Run 3:
```
=================================
all 86.7 65.8
llava_bench_complex 85.4 73.5
llava_bench_conv 84.1 61.8
llava_bench_detail 92.0 56.0
=================================
```

Aggregate Results:

```
=================================
all 85.57+/-1.12 64.97+/-0.82
llava_bench_complex 84.53+/-0.85 71.43+/-2.03
llava_bench_conv 84.10+/-0.00 61.8+/-0.00
llava_bench_detail 92.2+/-0.19 56.57+/-0.65
=================================
```

**LLaVA Results**
![llava-perf](https://github.com/NVIDIA/NeMo/assets/30813477/06fba4e4-f2e4-4e42-907d-1ce97fd53bd2)

**Video-LLaVA Results**
<img width="1220" alt="Screen Shot 2024-03-14 at 10 38 11 PM" src="https://github.com/NVIDIA/NeMo/assets/30813477/10afe566-ab4f-4958-aa52-db1037b726ec">


EVAL RESULTS ON IMAGE INFERENCE FOR 100% of LLaVA-Bench-In-The-Wild

- Assistant 1: Video-LLaVA
- Assistant 2: V-NeVA (start from NeVA image ckpt, train on video represented as 24 frames)

Run 1: 
```
=================================
all 67.8 77.3
llava_bench_complex 74.6 82.1
llava_bench_conv 61.2 77.1
llava_bench_detail 62.7 68.7
=================================
```

Run 2:
```
=================================
all 67.5 77.8
llava_bench_complex 73.9 82.7
llava_bench_conv 61.2 77.6
llava_bench_detail 62.7 68.7
=================================
```

Run 3:
```
=================================
all 67.5 78.1
llava_bench_complex 73.9 83.0
llava_bench_conv 61.2 77.6
llava_bench_detail 62.7 69.3
=================================
```

Aggregate Results:

```
=================================
all 67.60+/-0.19 77.73+/-0.38
llava_bench_complex 74.13+/-0.38 82.60+/-0.52
llava_bench_conv 61.20+/-0.00 77.43+/-0.43
llava_bench_detail 62.70+/-0.00 68.90+/-0.39
=================================
```

**LLaVA Results**
![llava-perf](https://github.com/NVIDIA/NeMo/assets/30813477/06fba4e4-f2e4-4e42-907d-1ce97fd53bd2)

**Video-LLaVA Results**
<img width="1220" alt="Screen Shot 2024-03-14 at 10 38 11 PM" src="https://github.com/NVIDIA/NeMo/assets/30813477/10afe566-ab4f-4958-aa52-db1037b726ec">


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.