Repository: modelscope/evalscope
Issue: Qwen2.5-Coder-1.5B 在humaneval上分数为0
Root cause: Documentation Deficiency

## 自查清单

在提交 issue 之前，请确保您已完成以下步骤:
- [x] 我已仔细阅读了[相关使用说明文档](https://evalscope.readthedocs.io/zh-cn/latest/get_started/parameters.html)
- [x] 我已查看了[常见问题解答](https://evalscope.readthedocs.io/zh-cn/latest/get_started/faq.html)
- [x] 我已搜索并查看了现有的 issues，确认这不是一个重复的问题

## 问题描述

评估Qwen2.5-Coder-1.5B在humaneval上score为0，发现存在大量复读机的情况，用Qwen2.5-Coder-1.5B-Instruct测试，score为0.4695，仍远低于官方报告中的0.7

## EvalScope 版本（必填）
v0.15.1

## 使用的工具
- [x] Native / 原生框架
- [ ] Opencompass backend
- [ ] VLMEvalKit backend
- [ ] RAGEval backend
- [ ] Perf / 模型推理压测工具
- [ ] Arena / 竞技场模式

## 执行的代码或指令

请提供您执行的主要代码或指令。

```bash
evalscope eval \
 --model Qwen/Qwen2.5-Coder-1.5B \
 --model-id Qwen/Qwen2.5-Coder-1.5B \
 --datasets humaneval
```

## 错误日志

请粘贴完整的错误日志或控制台输出。

无错误

## 运行环境

- 操作系统：ubuntu 22.04
- Python版本：3.10

## 其他信息

如果有其他相关信息，请在此处提供。

以humaneval第一个问题为例：
{"model": "Qwen/Qwen2.5-Coder-1.5B", "object": "chat.completion", "choices": [{"index": 0, "message": {"role": "assistant", "content": "Complete the following python code:\nfrom typing import List\n\n\ndef has_close_elements(numbers: List[float], threshold: float) -> bool:\n    \"\"\" Check if in given list of numbers, are any two numbers closer to each other than\n    given threshold.\n    >>> has_close_elements([1.0, 2.0, 3.0], 0.5)\n    False\n    >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)\n    True\n    \"\"\"\n  



qwen\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.
qwen\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.
qwen\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.
qwen\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.
qwen\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.
qwen\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.

/README.md\n# Qwen\n\nQwen is a Python library for building and training large language models (LLMs) using the Qwen architecture. It provides a high-level API for creating and training models, as well as a set of pre-trained models that can be fine-tuned for a variety of tasks.\n\n## Installation\n\nTo install Qwen, you can use pip:\n\n```bash\npip install qwen\n```\n\n## Usage\n\n### Creating a Model\n\nTo create a new model, you can use the `QwenModel` class. This class takes a few parameters, including the model name, the number of layers, and the number of heads.\n\n```python\nfrom qwen import QwenModel\n\nmodel = QwenModel(name=\"qwen-7b\", num_layers=32, num_heads=32)\n```\n\n### Training a Model\n\nTo train a model, you can use the `fit` method.后面略，都是大量杂乱信息}


参考模型 https://modelscope.cn/models/Qwen/Qwen2.5-Coder-1.5B 描述：
> We do not recommend using base language models for conversations. Instead, you can apply post-training, e.g., SFT, RLHF, continued pretraining, etc., or fill in the middle tasks on this model.

evalscope框架使用chat方式进行评测，因此请使用instruct模型来进行测试。此外，请尝试其他的generation config，默认输出最大长度为512，可能不够

好的，非常感谢，调整max_new_tokens=1024分数没有变化，但调整Pass@3，分数大幅上涨

> 好的，非常感谢，调整max_new_tokens=1024分数没有变化，但调整Pass@3，分数大幅上涨

你好，能分享一下你的命令吗，我同样用的Qwen2.5-Coder-1.5B，还是0分

> > 好的，非常感谢，调整max_new_tokens=1024分数没有变化，但调整Pass@3，分数大幅上涨
> 
> 你好，能分享一下你的命令吗，我同样用的Qwen2.5-Coder-1.5B，还是0分

您好，Qwen2.5-Coder-1.5B作为base model确实是0分，如下是我手动输入humaneval的prompt给Qwen2.5-Coder-1.5B：


```python
model_name = "Qwen/Qwen2.5-Coder-1.5B"

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="cuda"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

prompt = '''from typing import List

def has_close_elements(numbers: List[float], threshold: float) -> bool:
    """ Check if in given list of numbers, are any two numbers closer to each other than
    given threshold.
    >>> has_close_elements([1.0, 2.0, 3.0], 0.5)
    False
    >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)
    True
    """
    for i in range(len(numbers)):
        for j in range(i + 1, len(numbers)):
            if abs(numbers[i] - numbers[j]) < threshold:
                return True
    return False
'''

messages = [
    {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."},
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=512
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]

print(response)
```

得到的response为：

> Certainly! Below is a Python function that calculates the factorial of a given non-negative integer `n`. The function uses a recursive approach to compute the factorial.
> 
> ```python
> def factorial(n):
>     # Base case: 0! = 1
>     if n == 0:
>         return 1
>     # Recursive case: n! = n * (n-1)!
>     else:
>         return n * factorial(n - 1)
> 
> # Example usage:
> print(factorial(0))  # Output: 1
> print(factorial(1))  # Output: 1
> print(factorial(5))  # Output: 120
> print(factorial(10)) # Output: 3628800
> ```
> 
> This function checks if `n` is 0, in which case it returns 1 (since 0! is defined as 1). Otherwise, it calls itself with `n-1` and multiplies the result by `n`, effectively computing the factorial using recursion.
> ❯ python test_task_vector.py
> Sliding Window Attention is enabled but not implemented for `sdpa`; unexpected results may be encountered.
> Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation.
> from typing import List
> 
> def has_close_elements(numbers: List[float], threshold: float) -> bool:
>     """ Check if in given list of numbers, are any two numbers closer to each other than
>     given threshold.
>     >>> has_close_elements([1.0, 2.0, 3.0], 0.5)
>     False
>     >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)
>     True
>     """
>     for i in range(len(numbers)):
>         for j in range(i + 1, len(numbers)):
>             if abs(numbers[i] - numbers[j]) < threshold:
>                 return True
>     return False
> 
Qwen-7B-Chat
> You are Qwen, created by Alibaba Cloud. You are a helpful assistant.
Qwen-7B-Chat
> You are Qwen, created by Alibaba Cloud. You are a helpful assistant.
Qwen-7B-Chat
> You are Qwen, created by Alibaba Cloud. You are a helpful assistant.
Qwen-7B-Chat
> You are Qwen, created by Alibaba Cloud. You are a helpful assistant.

/README.md
> # Qwen-7B-Chat
> 
> Qwen-7B-Chat is a chatbot model based on the Qwen-7B model. It is designed to provide a more natural and human-like conversation experience.
> 
> ## Installation
> 
> To install Qwen-7B-Chat, you need to have Python 3.8 or later installed on your system. You can install the required dependencies using pip:
> 
> ```bash
> pip install -r requirements.txt
> ```
> 
> ## Usage
> 
> To use Qwen-7B-Chat, you need to provide a JSON file containing the conversation history. The JSON file should have the following format:
> 
> ```json
> {
>   "history": [
>     {
>       "role": "user",
>       "content": "Hello, how are you?"
>     },
>     {
>       "role": "assistant",
>       "content": "I'm doing well, thank you for asking."
>     }
>   ]
> }
> ```
> 
> You can then run the following command to start the chatbot:
> 
> ```bash
> python main.py --history_file <path_to_history_file>
> ```
> 
> The chatbot will start a conversation with the user, and you can provide input to the chatbot using the `input()`
> ```

而Qwen2.5-Coder-1.5B-Instruct的response为：

> Here's the corrected version of your function with some improvements:
> 
> ```python
> from typing import List
> 
> def has_close_elements(numbers: List[float], threshold: float) -> bool:
>     """ Check if in given list of numbers, are any two numbers closer to each other than
>     given threshold.
>     
>     Args:
>     numbers (List[float]): The list of numbers to check.
>     threshold (float): The maximum allowed distance between any two numbers.
>     
>     Returns:
>     bool: True if there are any two numbers within the threshold, False otherwise.
>     """
>     # Sort the list to make it easier to find close elements
>     sorted_numbers = sorted(numbers)
>     
>     # Iterate through the sorted list to find close elements
>     for i in range(len(sorted_numbers) - 1):
>         if abs(sorted_numbers[i] - sorted_numbers[i + 1]) < threshold:
>             return True
>     return False
> 
> # Test cases
> print(has_close_elements([1.0, 2.0, 3.0], 0.5))  # Output: False
> print(has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3))  # Output: True
> ```
> 
> ### Explanation:
> 1. **Sorting**: By sorting the list, we can easily find adjacent elements that are closer to each other.
> 2. **Iterating**: We iterate through the sorted list and check the difference between consecutive elements.
> 3. **Threshold Check**: If the difference is less than the threshold, we return `True`.
> 4. **Return False**: If no such pair is found after checking all pairs, we return `False`.
> 
> This approach ensures that the function efficiently finds the closest elements and handles edge cases gracefully.
> ```

> > > 好的，非常感谢，调整max_new_tokens=1024分数没有变化，但调整Pass@3，分数大幅上涨
> > 
> > 
> > 你好，能分享一下你的命令吗，我同样用的Qwen2.5-Coder-1.5B，还是0分
> 
> 您好，Qwen2.5-Coder-1.5B作为base model确实是0分，如下是我手动输入humaneval的prompt给Qwen2.5-Coder-1.5B：
> 
> model_name = "Qwen/Qwen2.5-Coder-1.5B"
> 
> model = AutoModelForCausalLM.from_pretrained(
>     model_name,
>     torch_dtype="auto",
>     device_map="cuda"
> )
> tokenizer = AutoTokenizer.from_pretrained(model_name)
> 
> prompt = '''from typing import List
> 
> def has_close_elements(numbers: List[float], threshold: float) -> bool:
>     """ Check if in given list of numbers, are any two numbers closer to each other than
>     given threshold.
>     >>> has_close_elements([1.0, 2.0, 3.0], 0.5)
>     False
>     >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)
>     True
>     """
>     for i in range(len(numbers)):
>         for j in range(i + 1, len(numbers)):
>             if abs(numbers[i] - numbers[j]) < threshold:
>                 return True
>     return False
> '''
> 
> messages = [
>     {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."},
>     {"role": "user", "content": prompt}
> ]
> text = tokenizer.apply_chat_template(
>     messages,
>     tokenize=False,
>     add_generation_prompt=True
> )
> model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
> 
> generated_ids = model.generate(
>     **model_inputs,
>     max_new_tokens=512
> )
> generated_ids = [
>     output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
> ]
> 
> response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
> 
> print(response)
> 得到的response为：
> 
> > Certainly! Below is a Python function that calculates the factorial of a given non-negative integer `n`. The function uses a recursive approach to compute the factorial.
> > def factorial(n):
> >     # Base case: 0! = 1
> >     if n == 0:
> >         return 1
> >     # Recursive case: n! = n * (n-1)!
> >     else:
> >         return n * factorial(n - 1)
> > 
> > # Example usage:
> > print(factorial(0))  # Output: 1
> > print(factorial(1))  # Output: 1
> > print(factorial(5))  # Output: 120
> > print(factorial(10)) # Output: 3628800
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > This function checks if `n` is 0, in which case it returns 1 (since 0! is defined as 1). Otherwise, it calls itself with `n-1` and multiplies the result by `n`, effectively computing the factorial using recursion.
> > ❯ python test_task_vector.py
> > Sliding Window Attention is enabled but not implemented for `sdpa`; unexpected results may be encountered.
> > Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation.
> > from typing import List
> > def has_close_elements(numbers: List[float], threshold: float) -> bool:
> > """ Check if in given list of numbers, are any two numbers closer to each other than
> > given threshold.
> > >>> has_close_elements([1.0, 2.0, 3.0], 0.5)
> > False
> > >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)
> > True
> > """
> > for i in range(len(numbers)):
> > for j in range(i + 1, len(numbers)):
> > if abs(numbers[i] - numbers[j]) < threshold:
> > return True
> > return False
> > 
Qwen-7B-Chat
> > You are Qwen, created by Alibaba Cloud. You are a helpful assistant.
Qwen-7B-Chat
> > You are Qwen, created by Alibaba Cloud. You are a helpful assistant.
Qwen-7B-Chat
> > You are Qwen, created by Alibaba Cloud. You are a helpful assistant.
Qwen-7B-Chat
> > You are Qwen, created by Alibaba Cloud. You are a helpful assistant.

/README.md
> > # Qwen-7B-Chat
> > Qwen-7B-Chat is a chatbot model based on the Qwen-7B model. It is designed to provide a more natural and human-like conversation experience.
> > ## Installation
> > To install Qwen-7B-Chat, you need to have Python 3.8 or later installed on your system. You can install the required dependencies using pip:
> > pip install -r requirements.txt
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > ## Usage
> > To use Qwen-7B-Chat, you need to provide a JSON file containing the conversation history. The JSON file should have the following format:
> > {
> >   "history": [
> >     {
> >       "role": "user",
> >       "content": "Hello, how are you?"
> >     },
> >     {
> >       "role": "assistant",
> >       "content": "I'm doing well, thank you for asking."
> >     }
> >   ]
> > }
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > You can then run the following command to start the chatbot:
> > python main.py --history_file <path_to_history_file>
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > The chatbot will start a conversation with the user, and you can provide input to the chatbot using the `input()`
> 
> 而Qwen2.5-Coder-1.5B-Instruct的response为：
> 
> > Here's the corrected version of your function with some improvements:
> > from typing import List
> > 
> > def has_close_elements(numbers: List[float], threshold: float) -> bool:
> >     """ Check if in given list of numbers, are any two numbers closer to each other than
> >     given threshold.
> >     
> >     Args:
> >     numbers (List[float]): The list of numbers to check.
> >     threshold (float): The maximum allowed distance between any two numbers.
> >     
> >     Returns:
> >     bool: True if there are any two numbers within the threshold, False otherwise.
> >     """
> >     # Sort the list to make it easier to find close elements
> >     sorted_numbers = sorted(numbers)
> >     
> >     # Iterate through the sorted list to find close elements
> >     for i in range(len(sorted_numbers) - 1):
> >         if abs(sorted_numbers[i] - sorted_numbers[i + 1]) < threshold:
> >             return True
> >     return False
> > 
> > # Test cases
> > print(has_close_elements([1.0, 2.0, 3.0], 0.5))  # Output: False
> > print(has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3))  # Output: True
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > ### Explanation:
> > 
> > 1. **Sorting**: By sorting the list, we can easily find adjacent elements that are closer to each other.
> > 2. **Iterating**: We iterate through the sorted list and check the difference between consecutive elements.
> > 3. **Threshold Check**: If the difference is less than the threshold, we return `True`.
> > 4. **Return False**: If no such pair is found after checking all pairs, we return `False`.
> > 
> > This approach ensures that the function efficiently finds the closest elements and handles edge cases gracefully.

好的，感谢😊