Repository: open-compass/opencompass
Issue: [Feature] Discussion of Evaluation Results for HumanEval Dataset
Root cause: Documentation Deficiency

### Describe the feature

I ran an evaluation on the HumanEval dataset using the **`starcoder2-15b-instruct`** LLM. The command I used is as follows:
```
python run.py \
    --datasets humaneval_gen \
    --hf-type chat \
    --hf-path bigcode/starcoder2-15b-instruct-v0.1
```
**I believe I used the default settings with greedy decoding to evaluate the pass@1 metric, so it is not the same issue as #1011** , and here are the results:
```
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dataset           version    metric            mode      _hf
----------------  ---------  ----------------  ------  -----
openai_humaneval  8e312c     humaneval_pass@1  gen     65.85
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
```
However, I noticed a discrepancy between this result and the one (72.6) reported on **[bigcode/starcoder2-15b-instruct-v0.1](https://huggingface.co/bigcode/starcoder2-15b-instruct-v0.1)**.

Additionally, I cross-checked with the results for **`llama-3-8b-instruct-hf`** reported in the [HumanEval README](https://github.com/open-compass/opencompass/blob/main/configs/datasets/humaneval/README.md) within OpenCompass (55.49). I used the following command for evaluation:
```
python run.py \
    --datasets humaneval_gen \  
    --hf-type chat \
    --hf-path meta-llama/Meta-Llama-3-8B-Instruct
```
And obtained these results:
```
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dataset           version    metric            mode      _hf
----------------  ---------  ----------------  ------  -----
openai_humaneval  8e312c     humaneval_pass@1  gen     59.15
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
```
Notably, both the results reported in OpenCompass and my own evaluations show a gap compared to the results (62.2) reported on  **[meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct)**.

I would like to open a discussion on this issue to explore the potential reasons behind these discrepancies.

Could there be any suggestions or solutions that might help me reproduce more accurate results?

Thanks.


### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

The discrepancy arises from variations in the prompts and inherent randomness. When you perform inference using vLLM or LMDeploy, the performance might exhibit a significant fluctuation, typically within a range of 3 to 5 points.

Thanks for your reply.

Do you have any suggestions on how to stabilize the performance during evaluation or achieve relatively higher results?

I guess that LLM developers might report higher results in their findings. If possible, I'd like to explore how they achieve that.

Prompt engineering is an effective approach; you can explore different prompts. OC also offers several alternative prompts, so feel free to try them out.
Like: https://github.com/open-compass/opencompass/blob/main/configs/datasets/humaneval/humaneval_openai_sample_evals_gen_159614.py

Thanks for your reply.

I will have a try regarding the prompt engineering.