Repository: tatsu-lab/alpaca_eval
Issue: Bugs from reference model while evaluating
Root cause: Documentation Deficiency

when i use the command :
alpaca_eval --model_outputs '/home/repos/alpaca_eval/alpaca_data/Infini-Megrez-7b-20231114-v2.json' --annotators_config 'alpaca_eval_gpt4' --reference_outputs  '/home/repos/alpaca_eval/alpaca_data/chatglm2-6b.json' 
this is the output from chatglm2-6B:
![image](https://github.com/tatsu-lab/alpaca_eval/assets/79788571/0cc41694-74e8-4486-81ee-9d5889fabe9c)
this is the output from our model:
![image](https://github.com/tatsu-lab/alpaca_eval/assets/79788571/ca5f05e1-ed1f-4bd6-b74f-2c07f8aaaa92)

however, according to the JSON file, there is no output for chatglm2-6B from annotator. On the contrary, it is an output content that has not been seen before. just like this 
![image](https://github.com/tatsu-lab/alpaca_eval/assets/79788571/82cf9e40-a83c-4bf6-8d23-c5714c4d88a5)



> when i use the command : alpaca_eval --model_outputs '/home/repos/alpaca_eval/alpaca_data/Infini-Megrez-7b-20231114-v2.json' --annotators_config 'alpaca_eval_gpt4' --reference_outputs '/home/repos/alpaca_eval/alpaca_data/chatglm2-6b.json' this is the output from chatglm2-6B: ![image](https://user-images.githubusercontent.com/79788571/282758396-0cc41694-74e8-4486-81ee-9d5889fabe9c.png) this is the output from our model: ![image](https://user-images.githubusercontent.com/79788571/282758564-ca5f05e1-ed1f-4bd6-b74f-2c07f8aaaa92.png)
> 
> however, according to the JSON file, there is no output for chatglm2-6B from annotator. On the contrary, it is an output content that has not been seen before. just like this ![image](https://user-images.githubusercontent.com/79788571/282759192-82cf9e40-a83c-4bf6-8d23-c5714c4d88a5.png)

you can watch the difference with output_1 and output from chatglm2-6B

By the way, my dataset is translated from alpaca eval to Chinese 
@winglian @zfang @jondurbin @44670 thx

![image](https://github.com/tatsu-lab/alpaca_eval/assets/79788571/9ff1d2d3-b6c6-478b-a30d-35e282355e5b)
Besides, the instructions in my dataset are all in Chinese, why do they appear in English

> when i use the command : alpaca_eval --model_outputs '/home/repos/alpaca_eval/alpaca_data/Infini-Megrez-7b-20231114-v2.json' --annotators_config 'alpaca_eval_gpt4' --reference_outputs '/home/repos/alpaca_eval/alpaca_data/chatglm2-6b.json' this is the output from chatglm2-6B: ![image](https://user-images.githubusercontent.com/79788571/282758396-0cc41694-74e8-4486-81ee-9d5889fabe9c.png) this is the output from our model: ![image](https://user-images.githubusercontent.com/79788571/282758564-ca5f05e1-ed1f-4bd6-b74f-2c07f8aaaa92.png)
> 
> however, according to the JSON file, there is no output for chatglm2-6B from annotator. On the contrary, it is an output content that has not been seen before. just like this ![image](https://user-images.githubusercontent.com/79788571/282759192-82cf9e40-a83c-4bf6-8d23-c5714c4d88a5.png)

This seems correct to me, `output_1` is the reference_outputs (i.e. 'chatglm2-6b.json' here), `output_2` is the model you are evaluating (i.e. 'Infini-Megrez-7b-20231114-v2.json'). Can you explain the issue here?



> ![image](https://user-images.githubusercontent.com/79788571/282802627-9ff1d2d3-b6c6-478b-a30d-35e282355e5b.png) Besides, the instructions in my dataset are all in Chinese, why do they appear in English

This is an example in the alpaca_eval dataset, are you sure it's neither in 'Infini-Megrez-7b-20231114-v2.json' nor 'chatglm2-6b.json'?

> > when i use the command : alpaca_eval --model_outputs '/home/repos/alpaca_eval/alpaca_data/Infini-Megrez-7b-20231114-v2.json' --annotators_config 'alpaca_eval_gpt4' --reference_outputs '/home/repos/alpaca_eval/alpaca_data/chatglm2-6b.json' this is the output from chatglm2-6B: ![image](https://user-images.githubusercontent.com/79788571/282758396-0cc41694-74e8-4486-81ee-9d5889fabe9c.png) this is the output from our model: ![image](https://user-images.githubusercontent.com/79788571/282758564-ca5f05e1-ed1f-4bd6-b74f-2c07f8aaaa92.png)
> > however, according to the JSON file, there is no output for chatglm2-6B from annotator. On the contrary, it is an output content that has not been seen before. just like this ![image](https://user-images.githubusercontent.com/79788571/282759192-82cf9e40-a83c-4bf6-8d23-c5714c4d88a5.png)
> 
> This seems correct to me, `output_1` is the reference_outputs (i.e. 'chatglm2-6b.json' here), `output_2` is the model you are evaluating (i.e. 'Infini-Megrez-7b-20231114-v2.json'). Can you explain the issue here?
> 
> > ![image](https://user-images.githubusercontent.com/79788571/282802627-9ff1d2d3-b6c6-478b-a30d-35e282355e5b.png) Besides, the instructions in my dataset are all in Chinese, why do they appear in English
> 
> This is an example in the alpaca_eval dataset, are you sure it's neither in 'Infini-Megrez-7b-20231114-v2.json' nor 'chatglm2-6b.json'?

1. However, output_1  is not the same as the output value of chatglm, although theoretically the output1 comes from chatglm
2. I am very certain because the dataset I processed does not contain English

Can you send me a minimal file (both the outputs and the baseline) where these errors show up? E.g. less than 10 instructions. I can then try to understand why that happens. 

My hunch is that it is a data or a caching issue.

> Can you send me a minimal file (both the outputs and the baseline) where these errors show up? E.g. less than 10 instructions. I can then try to understand why that happens.
> 
> My hunch is that it is a data or a caching issue.

I think I should have found the problem. My command did not specify output_ Path, I thought the final evaluation file was this
![image](https://github.com/tatsu-lab/alpaca_eval/assets/79788571/02c13e9a-cd6c-4e1e-bf3b-8c8ac86a445b)
After I specified the output path, there was no problem. Thank you for your patience