Repository: open-compass/opencompass
Issue: [Feature] Improve evaluation scripts for mbpp datasets
Root cause: Validation Gap

### Describe the feature

When I evaluated the vicuna-7b-v1.5 model using the mbpp_gen script, the score was 0 and most answers showed failed. Perhaps the evaluate script did not properly format the answer.
![微信截图_20240228213314](https://github.com/open-compass/opencompass/assets/53924360/98d9f2b3-ea86-486e-82bf-ca953f4a83cc)
![微信图片_20240228213255](https://github.com/open-compass/opencompass/assets/53924360/ada58663-b800-49c9-bac9-a6e984e394e7)


### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

I met the same error.Here's the result I got on MBPP datasets when evaluating gemma-7b-it.
![image](https://github.com/open-compass/opencompass/assets/23159695/15d34066-147d-408e-9a3c-f58a220dc88c)
By the way, I want to know where to find the prediction result as you paste.Thank you for your report.

> I met the same error.Here's the result I got on MBPP datasets when evaluating gemma-7b-it. ![image](https://private-user-images.githubusercontent.com/23159695/310845147-15d34066-147d-408e-9a3c-f58a220dc88c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDk4MDg1MjYsIm5iZiI6MTcwOTgwODIyNiwicGF0aCI6Ii8yMzE1OTY5NS8zMTA4NDUxNDctMTVkMzQwNjYtMTQ3ZC00MDhlLTlhM2MtZjU4YTIyMGRjODhjLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAzMDclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMzA3VDEwNDM0NlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTJiMzcxZWYxY2I4MTEzMDUzNDAwYjNjMWM1MjcyYzQ1ZDQzZjI0YjExOGVjOWFhZWU4NTQzNTY1NTU4MGRlM2UmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.RdKcC75JhnPZqsybDNx2_ztEJVMrlI845WSCIC26oUQ) By the way, I want to know where to find the prediction result as you paste.Thank you for your report.

I find the prediction file, and each prediction is empty.
![截屏2024-03-07 下午6 46 09](https://github.com/open-compass/opencompass/assets/23159695/da8c476f-03e9-408a-9466-7c6ec3903091)
I don't know what happened here, because I used the default config for the model and the dataset.Looking forward to helpful findings.

> > I met the same error.Here's the result I got on MBPP datasets when evaluating gemma-7b-it. ![image](https://private-user-images.githubusercontent.com/23159695/310845147-15d34066-147d-408e-9a3c-f58a220dc88c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDk4MDg1MjYsIm5iZiI6MTcwOTgwODIyNiwicGF0aCI6Ii8yMzE1OTY5NS8zMTA4NDUxNDctMTVkMzQwNjYtMTQ3ZC00MDhlLTlhM2MtZjU4YTIyMGRjODhjLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAzMDclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMzA3VDEwNDM0NlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTJiMzcxZWYxY2I4MTEzMDUzNDAwYjNjMWM1MjcyYzQ1ZDQzZjI0YjExOGVjOWFhZWU4NTQzNTY1NTU4MGRlM2UmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.RdKcC75JhnPZqsybDNx2_ztEJVMrlI845WSCIC26oUQ) By the way, I want to know where to find the prediction result as you paste.Thank you for your report.
> 
> I find the prediction file, and each prediction is empty. ![截屏2024-03-07 下午6 46 09](https://private-user-images.githubusercontent.com/23159695/310848901-da8c476f-03e9-408a-9466-7c6ec3903091.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTE0NDg1ODMsIm5iZiI6MTcxMTQ0ODI4MywicGF0aCI6Ii8yMzE1OTY5NS8zMTA4NDg5MDEtZGE4YzQ3NmYtMDNlOS00MDhhLTk0NjYtN2M2ZWMzOTAzMDkxLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAzMjYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMzI2VDEwMTgwM1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWEwNzk2MjA4ZWI1YjM3NTRhOTI2MWVjNjU2YTc3ZjQyOTg0ZGMyNzM3YjRlY2FiZGI4ODZlNTY3YjUyMjFjNDUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.Y_ptpOmqPuY2aJVuzdSn6MRrXV39YrBlwl19WWgPyKg) I don't know what happened here, because I used the default config for the model and the dataset.Looking forward to helpful findings.

 same question when testing bbh_gen task, the prediction is empty. Have you fixed it? @YFCYFC 

> > > I met the same error.Here's the result I got on MBPP datasets when evaluating gemma-7b-it. ![image](https://private-user-images.githubusercontent.com/23159695/310845147-15d34066-147d-408e-9a3c-f58a220dc88c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDk4MDg1MjYsIm5iZiI6MTcwOTgwODIyNiwicGF0aCI6Ii8yMzE1OTY5NS8zMTA4NDUxNDctMTVkMzQwNjYtMTQ3ZC00MDhlLTlhM2MtZjU4YTIyMGRjODhjLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAzMDclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMzA3VDEwNDM0NlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTJiMzcxZWYxY2I4MTEzMDUzNDAwYjNjMWM1MjcyYzQ1ZDQzZjI0YjExOGVjOWFhZWU4NTQzNTY1NTU4MGRlM2UmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.RdKcC75JhnPZqsybDNx2_ztEJVMrlI845WSCIC26oUQ) By the way, I want to know where to find the prediction result as you paste.Thank you for your report.
> > 
> > 
> > I find the prediction file, and each prediction is empty. ![截屏2024-03-07 下午6 46 09](https://private-user-images.githubusercontent.com/23159695/310848901-da8c476f-03e9-408a-9466-7c6ec3903091.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTE0NDg1ODMsIm5iZiI6MTcxMTQ0ODI4MywicGF0aCI6Ii8yMzE1OTY5NS8zMTA4NDg5MDEtZGE4YzQ3NmYtMDNlOS00MDhhLTk0NjYtN2M2ZWMzOTAzMDkxLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAzMjYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMzI2VDEwMTgwM1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWEwNzk2MjA4ZWI1YjM3NTRhOTI2MWVjNjU2YTc3ZjQyOTg0ZGMyNzM3YjRlY2FiZGI4ODZlNTY3YjUyMjFjNDUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.Y_ptpOmqPuY2aJVuzdSn6MRrXV39YrBlwl19WWgPyKg) I don't know what happened here, because I used the default config for the model and the dataset.Looking forward to helpful findings.
> 
> same question when testing bbh_gen task, the prediction is empty. Have you fixed it? @YFCYFC

Sorry,I did not found the reason exactly.I just fine tuned my model once again, and the predictions were not empty, but still not the desired result, so I didn't post the update here.I suggest you finetune your model for more than 1 epochs, which may help.