Repository: EleutherAI/lm-evaluation-harness
Issue: Llama 3.2 Fails to Recognize Chat Template: ValueError on Tokenizer Configuration
Root cause: External Dependency Breakage

Dear Developers,

Hello. I am using your framework, and it is very useful. However, when using the YAML file to provide a chat template, Llama 3 can recognize the chat template correctly, but Llama 3.2 throws an error. 
The YAML file is as follows:
```yaml
task: CSQA_llama
dataset_path: here1swqw/CSQA_Text_Only
training_split: train
validation_split: validation
test_split: validation
output_type: generate_until
doc_to_text: 'Given the following problem, give a final answer to the problem. Your response should end with "The answer is (option)." where The "option" is the option that represents the correct answer, for example, "The answer is (A).".
Question: {{question}}
Answer:'
doc_to_target: '{{answer.split(''####'')[-1].strip() if answer is defined else target}}'
repeats: 1
metric_list:
  - metric: exact_match
    aggregation: mean
    higher_is_better: true
    ignore_case: true
    ignore_punctuation: false
    regexes_to_ignore:
      - ","
      - "\\$"
      - "(?s).*#### "
      - "\\.$"
generation_kwargs:
  until:
  - '
'
  - '
'
  - '
'
  - 

  - 

filter_list:
- name: "score-first-strict"
  filter:
    - function: "regex"
      regex_pattern: '[Tt]he answer is \((\w)\)'
    - function: "take_first"
- name: "score-first-flexible" #
  filter:
    - function: "regex"
      regex_pattern: '\((\w)\)'
    - function: "take_first"
```
And the error message is as follows:

```
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/mnt/d/Aw/lm-evaluation-harness/lm_eval/__main__.py", line 461, in <module>
    cli_evaluate()
  File "/mnt/d/Aw/lm-evaluation-harness/lm_eval/__main__.py", line 382, in cli_evaluate
    results = evaluator.simple_evaluate(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/d/Aw/lm-evaluation-harness/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/mnt/d/Aw/lm-evaluation-harness/lm_eval/evaluator.py", line 301, in simple_evaluate
    results = evaluate(
              ^^^^^^^^^
  File "/mnt/d/Aw/lm-evaluation-harness/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/mnt/d/Aw/lm-evaluation-harness/lm_eval/evaluator.py", line 444, in evaluate
    task.build_all_requests(
  File "/mnt/d/Aw/lm-evaluation-harness/lm_eval/api/task.py", line 438, in build_all_requests
    fewshot_ctx = self.fewshot_context(
                  ^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/d/Aw/lm-evaluation-harness/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/mnt/d/Aw/lm-evaluation-harness/lm_eval/api/task.py", line 1115, in fewshot_context
    return chat_template(labeled_examples)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/d/Aw/lm-evaluation-harness/lm_eval/models/huggingface.py", line 1325, in apply_chat_template
    return self.tokenizer.apply_chat_template(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/A w/miniconda3/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1803, in apply_chat_template
    chat_template = self.get_chat_template(chat_template, tools)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/Aw/miniconda3/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1967, in get_chat_template
    raise ValueError(
ValueError: Cannot use chat template functions because tokenizer.chat_template is not set and no template argument was passed! For information about writing templates and setting the tokenizer.chat_template attribute, please see the documentation at https://huggingface.co/docs/transformers/main/en/chat_templating.
```

Hi! Are you using an instruct 3.2 model rather than the base?

> Hi! Are you using an instruct 3.2 model rather than the base?

3.2-instruct and 3 instruct. llama 3 instruct will be fine, but when it comes to 3.2 instruct, it failed

> > Hi! Are you using an instruct 3.2 model rather than the base?
> 
> 3.2-instruct and 3 instruct. llama 3 instruct will be fine, but when it comes to 3.2 instruct, it failed

hmm. what model is it specifically?

> > > Hi! Are you using an instruct 3.2 model rather than the base?
> > 
> > 
> > 3.2-instruct and 3 instruct. llama 3 instruct will be fine, but when it comes to 3.2 instruct, it failed
> 
> hmm. what model is it specifically?

Haha, maybe I should think it over again. Thank you for your response. If I have any questions, I’ll reach out to you. Many thanks!