Repository: huggingface/lighteval
Issue: [BUG] Qwen2.5-Coder-1.5B-intruct's humaneval_instruct score is low!!
Root cause: Documentation Deficiency

## Describe the bug
I test Qwen2.5-Coder-1.5B and Qwen2.5-Coder-1.5B-Instruct on humaneval and humaneval_insturct, but score is lower than their blog post(https://qwenlm.github.io/blog/qwen2.5-coder-family/).

Qwen2.5-Coder-1.5B is 39.02(blog : 43.9) on humaneval
Qwen2.5-Coder-1.5B-instruct is 53.66 on humaneval, and **17.07** on humaneval_instuct. (blog : 70.7)

I don't understand why instruction model is worse in humaneval_instruct tasks...


## To Reproduce
```
HF_ALLOW_CODE_EVAL="1" CUDA_VISIBLE_DEVICES="3,4" lm_eval --model hf \
    --model_args pretrained=Qwen/Qwen2.5-Coder-1.5B,parallelize=True \
    --tasks humaneval \
    --batch_size 1 \
    --output_path ./Qwen2.5-Coder-1.5B \
    --confirm_run_unsafe_code

HF_ALLOW_CODE_EVAL="1" CUDA_VISIBLE_DEVICES="3" lm_eval --model hf \
    --model_args pretrained=Qwen/Qwen2.5-Coder-1.5B-Instruct,parallelize=True \
    --tasks humaneval_instruct \
    --batch_size 1 \
    --output_path ./Qwen2.5-Coder-1.5B-Instruct \
    --confirm_run_unsafe_code 
```

## Expected behavior

Qwen2.5-Coder-1.5B-instruct's humaneval_insturct score is must be similar to 70 

## Version info
os : Ubuntu 22.04.4 LTS
lighteval : 0.6.0.dev0


Package                   Version 
------------------------- --------------
absl-py                   2.1.0
accelerate                1.4.0
aenum                     3.1.15
aiohappyeyeballs          2.5.0
aiohttp                   3.11.13
aiosignal                 1.3.2
annotated-types           0.7.0
antlr4-python3-runtime    4.13.2
anyio                     4.6.2.post1
argon2-cffi               23.1.0
argon2-cffi-bindings      21.2.0
arrow                     1.3.0
asttokens                 2.4.1
async-lru                 2.0.4
attrs                     24.2.0
Babel                     2.14.0
beautifulsoup4            4.12.3
bleach                    6.1.0
blis                      0.7.11
Brotli                    1.0.9
cached-property           1.5.2
catalogue                 2.0.10
certifi                   2024.8.30
cffi                      1.17.1
cfgv                      3.4.0
chardet                   5.2.0
charset-normalizer        3.4.0
click                     8.1.8
cloudpathlib              0.16.0
colorama                  0.4.6
coloredlogs               15.0.1
colorlog                  6.9.0
comm                      0.2.2
confection                0.1.5
contourpy                 1.3.0
cycler                    0.12.1
cymem                     2.0.11
DataProperty              1.1.0
datasets                  3.3.2
debugpy                   1.8.7
decorator                 5.1.1
defusedxml                0.7.1
dill                      0.3.8
distlib                   0.3.9
distro                    1.9.0
docstring_parser          0.16
editdistance              0.8.1
emoji                     2.14.1
entrypoints               0.4
evaluate                  0.4.3
exceptiongroup            1.2.2
executing                 2.1.0
fastjsonschema            2.20.0
filelock                  3.16.1
flatbuffers               24.3.25
fonttools                 4.54.1
fqdn                      1.5.1
frozenlist                1.5.0
fsspec                    2024.12.0
gitdb                     4.0.12
GitPython                 3.1.44
gmpy2                     2.1.5
h11                       0.14.0
h2                        4.1.0
hpack                     4.0.0
httpcore                  1.0.6
httpx                     0.27.2
huggingface-hub           0.29.2
humanfriendly             10.0
hyperframe                6.0.1
identify                  2.6.9
idna                      3.10
imagecodecs               2023.8.12
imageio                   2.36.0
importlib_metadata        8.5.0
importlib_resources       6.4.5
iniconfig                 2.1.0
ipykernel                 6.29.5
ipython                   8.28.0
ipywidgets                8.1.5
isoduration               20.11.0
jedi                      0.19.1
jieba                     0.42.1
Jinja2                    3.1.4
jiter                     0.9.0
joblib                    1.4.2
json5                     0.9.25
jsonargparse              4.37.0
jsonlines                 4.0.0
jsonpointer               3.0.0
jsonschema                4.23.0
jsonschema-specifications 2024.10.1
jupyter_client            8.6.3
jupyter_core              5.7.2
jupyter-events            0.10.0
jupyter-lsp               2.2.5
jupyter_server            2.14.2
jupyter_server_terminals  0.5.3
jupyterlab                4.2.5
jupyterlab_pygments       0.3.0
jupyterlab_server         2.27.3
jupyterlab_widgets        3.0.13
kiwisolver                1.4.7
langcodes                 3.5.0
langdetect                1.0.9
language_data             1.3.0
latex2sympy2_extended     1.0.6
lazy_loader               0.4
lighteval                 0.6.0.dev0     /data/donggukang/lighteval
lightning                 2.5.0.post0
lightning-utilities       0.14.1
litgpt                    0.5.8.dev1     /data/donggukang/litgpt
lm_eval                   0.4.3          /data/donggukang/lm-evaluation-harness_hf/lm-evaluation-harness
lxml                      5.3.1
marisa-trie               1.2.1
markdown-it-py            3.0.0
MarkupSafe                3.0.2
math-verify               0.5.2
matplotlib                3.9.1
matplotlib-inline         0.1.7
mbstrdecoder              1.1.4
mdurl                     0.1.2
mistune                   3.0.2
more-itertools            10.6.0
mpmath                    1.3.0
multidict                 6.1.0
multiprocess              0.70.16
munkres                   1.1.4
murmurhash                1.0.12
natto-py                  1.0.1
nbclient                  0.10.0
nbconvert                 7.16.4
nbformat                  5.10.4
nest_asyncio              1.6.0
networkx                  3.4.2
nltk                      3.9.1
nodeenv                   1.9.1
notebook_shim             0.2.4
numexpr                   2.10.2
numpy                     1.26.4
nvidia-cublas-cu12        12.4.5.8
nvidia-cuda-cupti-cu12    12.4.127
nvidia-cuda-nvrtc-cu12    12.4.127
nvidia-cuda-runtime-cu12  12.4.127
nvidia-cudnn-cu12         9.1.0.70
nvidia-cufft-cu12         11.2.1.3
nvidia-curand-cu12        10.3.5.147
nvidia-cusolver-cu12      11.6.1.9
nvidia-cusparse-cu12      12.3.1.170
nvidia-cusparselt-cu12    0.6.2
nvidia-ml-py              12.570.86
nvidia-nccl-cu12          2.21.5
nvidia-nvjitlink-cu12     12.4.127
nvidia-nvtx-cu12          12.4.127
nvitop                    1.4.2
onnx                      1.17.0
onnxconverter-common      1.14.0
onnxruntime-gpu           1.19.2
openai                    1.67.0
overrides                 7.7.0
packaging                 24.1
pandas                    2.2.3
pandocfilters             1.5.0
parso                     0.8.4
pathvalidate              3.2.3
peft                      0.14.0
pexpect                   4.9.0
pickleshare               0.7.5
Pillow                    10.0.0
pip                       24.2
pkgutil_resolve_name      1.3.10
platformdirs              4.3.6
pluggy                    1.5.0
ply                       3.11
portalocker               3.1.1
pre_commit                4.2.0
preshed                   3.0.9
prometheus_client         0.21.0
prompt_toolkit            3.0.48
propcache                 0.3.0
protobuf                  3.20.2
psutil                    6.0.0
ptyprocess                0.7.0
pure_eval                 0.2.3
pyarrow                   19.0.1
pybind11                  2.13.6
pycountry                 24.6.1
pycparser                 2.22
pydantic                  2.10.6
pydantic_core             2.27.2
Pygments                  2.18.0
pyparsing                 3.2.0
PyQt5                     5.15.9
PyQt5-sip                 12.12.2
PySocks                   1.7.1
pytablewriter             1.2.1
pytest                    7.4.0
pythainlp                 5.1.0
python-crfsuite           0.9.11
python-dateutil           2.9.0
python-json-logger        2.0.7
pytorch-lightning         2.5.0.post0
pytz                      2024.1
pyvi                      0.1.1
PyWavelets                1.7.0
PyYAML                    6.0.2
pyzmq                     26.2.0
referencing               0.35.1
regex                     2024.11.6
requests                  2.32.3
rfc3339-validator         0.1.4
rfc3986-validator         0.1.1
rich                      13.9.4
rouge_score               0.1.2
rpds-py                   0.20.0
ruff                      0.2.2
sacrebleu                 2.5.1
safetensors               0.5.3
scikit-image              0.24.0
scikit-learn              1.5.2
scipy                     1.14.1
Send2Trash                1.8.3
sentencepiece             0.2.0
setuptools                75.1.0
sip                       6.7.12
six                       1.16.0
skl2onnx                  1.17.0
sklearn-crfsuite          0.5.0
smart-open                6.4.0
smmap                     5.0.2
sniffio                   1.3.1
soupsieve                 2.5
spacy                     3.7.2
spacy-legacy              3.0.12
spacy-loggers             1.0.5
sqlitedict                2.1.0
srsly                     2.5.1
stack-data                0.6.2
stanza                    1.10.1
SudachiDict-core          20250129
SudachiPy                 0.6.10
sympy                     1.13.1
tabledata                 1.3.4
tabulate                  0.9.0
tcolorpy                  0.1.7
termcolor                 2.3.0
terminado                 0.18.1
tf2onnx                   1.16.1
thinc                     8.2.5
threadpoolctl             3.5.0
tifffile                  2024.9.20
tiktoken                  0.9.0
tinycss2                  1.3.0
tokenizers                0.21.0
toml                      0.10.2
tomli                     2.0.2
torch                     2.6.0
torchaudio                2.4.1
torchmetrics              1.6.3
torchvision               0.21.0
tornado                   6.4.1
tqdm                      4.67.1
tqdm-multiprocess         0.0.11
traitlets                 5.14.3
transformers              4.49.0
triton                    3.2.0
typepy                    1.3.4
typer                     0.9.4
types-python-dateutil     2.9.0.20241003
typeshed_client           2.7.0
typing_extensions         4.12.2
typing-utils              0.1.0
tzdata                    2024.2
unicodedata2              15.1.0
uri-template              1.3.0
urllib3                   2.2.3
virtualenv                20.29.3
wasabi                    1.1.3
wcwidth                   0.2.13
weasel                    0.3.4
webcolors                 24.8.0
webencodings              0.5.1
websocket-client          1.8.0
wheel                     0.44.0
widgetsnbextension        4.0.13
word2number               1.1
xxhash                    3.5.0
yarl                      1.18.3
zipp                      3.20.2
zstandard                 0.23.0


Hi! You need to use `--use_chat_template` when evaluating chat models