Repository: EvolvingLMMs-Lab/lmms-eval
Issue: is there something wrong with the 000000494869.jpg in lmms-eval MME dataset ?
Root cause: Environment Incompatibility

We find the results are slightly different when evaluating the same model on the original MME benchmark and lmms-eval, we compare the generated responses and find the responses for 000000494869.jpg are different, we place them below:

---------['position/000000494869.jpg']---------------
---------['Instruction: Is the baby on the left of the dog in the image? Please answer yes or no.\nResponse:']---------------
---------['no']---------------official answer
---------['No, the baby is not on the left of the dog in the image.']---------------evaluate on original MME
---------['yes, the baby is on the left of the dog in the image. the']---------------evaluate on lmms-eval 

and we upload the corresponding image in the MME benchmark for further comparison

![000000494869](https://github.com/EvolvingLMMs-Lab/lmms-eval/assets/20677791/4bfeea5a-aca5-4488-93a7-53e850398d49)


<img width="641" alt="image" src="https://github.com/EvolvingLMMs-Lab/lmms-eval/assets/92624596/af4a80fa-b328-4289-b0c7-ae813a7ec067">
<img width="1368" alt="image" src="https://github.com/EvolvingLMMs-Lab/lmms-eval/assets/92624596/119e219d-6a5f-49a3-9486-21854618dc87">

Seems like the image are the same in our dataset. Have no idea why the answer is different. Does the original one use temperature larger than 0 or other generation parameters?

I checked the generation parameters and did not find inconsistent parameters (both temperatures are set to 0).  I think there may be something different in the images. Could you provide a position/000000494869.jpg in the lmms-eval dataset to me for testing?  baiyuting_hi@163.com

The image may be stored in JPG format during the process we convert it to huggingface dataset hub. That might have some infomation loss (although invisible to human eyes). 

May I ask for your help for us to check if the image's metadata in `lmms-lab/MME` is different with original one?

I would like to test it. My plan is very simple.
Firstly, I would like to check if we maintain the same original MME dataset. During the downloading process, some images may need to be manually collected because of reasons like the invalid download link or bad network. So, the images may be different since we manually collect images.
Then, if no problems are found,  there may be something different between the original MME dataset and the transformed llm-eval/MME dataset. 
Maybe in the first step, two MME datasets are found inconsistent and there is no need to further test. 
So, I want a position/000000494869.jpg in your maintained MME dataset to make a comparison and decide if it is necessary to further test according to the comparison results.  Could you provide a position/000000494869.jpg to me for testing? My email is baiyuting_hi@163.com.

@baiyuting Sure I have sent you an email with the image attached.

I replace the original position/000000494869.jpg with the new image and test on the same model with the same code. The result is as below.

---------['position']---------------
---------['position/000000494869.jpg']---------------
---------['Instruction: Is the baby on the left of the dog in the image? Please answer yes or no.\nResponse:']---------------
---------['Yes, the baby is on the left of the dog in the image. The']---------------evaluate on original MME with a new image

It is the same as the result evaluated by lmms-eval. 
It seems that the problem is from the image itself. We maintain different MME datasets. I think it may be helpful to check if the image is the same as the one in the official MME dataset.

To test if something is different between my MME dataset and the official MME dataset, I replaced position/000000494869.jpg with the corresponding image from the official MME dataset. The result is as follows.

---------['position']---------------
---------['position/000000494869.jpg']---------------
---------['Instruction: Is the baby on the left of the dog in the image? Please answer yes or no.\nResponse:']---------------
---------['No, the baby is not on the left of the dog in the image.']---------------evaluate on official MME dataset

So, according to the result, the official MME dataset is consistent with the MME dataset I maintained.  
So, there may be something wrong with the given image. There may be something wrong in your maintained MME dataset, which may influence the transformed lmms-eval/MME dataset and the following evaluation.