Repository: open-compass/VLMEvalKit
Issue: Questions regarding the metrics for SEED bench
Root cause: Documentation Deficiency

Hi,

Thanks for putting up the benchmark and releasing the eval tool. I'm running some experiments on both MMBench and the SEED bench, where I'm having some confusion regarding the metrics in the SEED leaderboard and would appreciate any inputs.

![image](https://github.com/open-compass/VLMEvalKit/assets/19763611/39621d74-eea6-4dfe-87d4-1fe96da9a1c0)

Specifically, I have three questions.

1. What does "heuristic matching" mean in `ExactMatchRate`?
2. I'm not fully understanding the definition of `MatchedAcc` and `ExactMatchAcc` (and the difference between them). Would you mind explaining it with a concrete example?
3. It is mentioned, for the official SEED leaderboard, that `For models with limited instruction following capabilities (including qwen_base, MiniGPT-4, InstructBLIP, flamingov2), the performance gap between generation-based evaluation and PPL-based evaluation is significant`. I understand what `PPL-based evaluation` means (ranking options by perplexity), but what does `generation-based evaluation` mean here?

Thank you in advance for your help.

1. 'Heuristic Matching' means we try to match the prediction with an option using pre-defined procedure. We try to match: 1. The option label (A, B, C, D) or the option content with the VLM prediction. Codes: in: https://github.com/open-compass/VLMEvalKit/blob/7057afc9c68e7eea890033b27b8220f61109d08d/vlmeval/utils/matching_util.py#L58

2. Here is an example with 3 samples:
    1. Answer: A; Prediction: blahblah
    2. Answer: B; Prediction: B
    4. Answer: C; Prediction: A

    Among three samples, sample 2, 3 are matched (the prediction can be matched to an option), while sample 1 is not matched. 
    1. MatchedAcc = 1 / 2 (all matched cases) = 50%
    2. ExactMatchAcc = 1 / 3 (all cases) = 33%

3. `generation-based evaluation` is the one used in VLMEvalKit, during which we feed question and options to the VLM and calculate the acc based on VLM-generated outputs. 

@kennymckormick Thank you for the reply; it's super helpful. One last follow-up question:
![image](https://github.com/open-compass/VLMEvalKit/assets/19763611/704a15c2-2cdf-4f2b-9a1c-367a47c889c7)
Here is the result I got by evaluating on SEEDBench_IMG. Is the `acc` here `LLMMatchAcc` or `ExactMatchAcc`?

Another question here (I'm new to this field; need to learn a lot...)

I was evaluating on `scene understanding` category only and got `73.115896` accuracy, which is not exactly the same as the `73.527549` that I got in the above screenshot (where I evaluated on all categories). Is this expected, or that I did something wrong? I thought that across different evaluation runs the result should be consistent.

In case this information helps, I was creating a new tsv file `SEEDBench_IMG_SceneUnderstanding.tsv` from the original `SEEDBench_IMG.tsv` by filtering only the questions whose category is `scene understanding`, on which I performed the evaluation with the provided eval toolkit.

> @kennymckormick Thank you for the reply; it's super helpful. One last follow-up question: ![image](https://private-user-images.githubusercontent.com/19763611/292242236-704a15c2-2cdf-4f2b-9a1c-367a47c889c7.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTEiLCJleHAiOjE3MDMyMzY0OTMsIm5iZiI6MTcwMzIzNjE5MywicGF0aCI6Ii8xOTc2MzYxMS8yOTIyNDIyMzYtNzA0YTE1YzItMmNkZi00ZjJiLTlhMWMtMzY3YTQ3Yzg4OWM3LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFJV05KWUFYNENTVkVINTNBJTJGMjAyMzEyMjIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjMxMjIyVDA5MDk1M1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWE1NTVmMGI5NzVjOTQxOTZmNDkyOTdmMzE2NTRmNjc5ZmIzNmI3ZGZhODA0YzY3Y2QxYTBkZGY4MjhkMzNhY2ImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.FeA-fsWdxEa3RKJDUIrad2SUrSiVsl87s8huUzIXq-U) Here is the result I got by evaluating on SEEDBench_IMG. Is the `acc` here `LLMMatchAcc` or `ExactMatchAcc`?

In this table, the `acc` here is `MatchedAcc`. 

> Another question here (I'm new to this field; need to learn a lot...)
> 
> I was evaluating on `scene understanding` category only and got `73.115896` accuracy, which is not exactly the same as the `73.527549` that I got in the above screenshot (where I evaluated on all categories). Is this expected, or that I did something wrong? I thought that across different evaluation runs the result should be consistent.
> 
> In case this information helps, I was creating a new tsv file `SEEDBench_IMG_SceneUnderstanding.tsv` from the original `SEEDBench_IMG.tsv` by filtering only the questions whose category is `scene understanding`, on which I performed the evaluation with the provided eval toolkit.

I'm not that sure, but it sounds like some randomness in your model during evaluation. You can run the evaluation again to check if the accuracy number further changes. 

I see. Thank you again for these replies.

> @kennymckormick Thank you for the reply; it's super helpful. One last follow-up question: ![image](https://private-user-images.githubusercontent.com/19763611/292242236-704a15c2-2cdf-4f2b-9a1c-367a47c889c7.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTgzMzIxNjEsIm5iZiI6MTcxODMzMTg2MSwicGF0aCI6Ii8xOTc2MzYxMS8yOTIyNDIyMzYtNzA0YTE1YzItMmNkZi00ZjJiLTlhMWMtMzY3YTQ3Yzg4OWM3LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA2MTQlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNjE0VDAyMjQyMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWFhMDFiOWMwN2VkYzE2ZjJhMTZhNjg0Y2QzODQ4NmJhYmRkOGUxMmM2OTdlY2ZkZTQzYjY1OTFiYjJjMDIxODMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.B7Ihz_kr5yvfoH84MetP9RpF6HIW93lU5IPvKWHZPYo) Here is the result I got by evaluating on SEEDBench_IMG. Is the `acc` here `LLMMatchAcc` or `ExactMatchAcc`?

HI，could you tell me how did you evaluate your result，There was no response when I clicked refresh after submission of seedbench leadboard