Repository: EleutherAI/lm-evaluation-harness
Issue: `meta-llama/Llama-2-7b-hf` doesn't work with `squadv2` (and I think other python-defined tasks)
Root cause: Validation Gap

Steps to reproduce the problem:

```
%env HF_TOKEN=<..>
!lm_eval --model hf-auto --model_args pretrained=meta-llama/Llama-2-7b-hf,revision=main,parallelize=True --tasks squadv2 --device cuda --batch_size 1 --num_fewshot 0

2024-02-09:18:16:22,147 INFO     [utils.py:160] NumExpr defaulting to 8 threads.
2024-02-09:18:16:22,330 INFO     [config.py:58] PyTorch version 2.1.0+cu121 available.
2024-02-09:18:16:22,331 INFO     [config.py:95] TensorFlow version 2.15.0 available.
2024-02-09:18:16:22,332 INFO     [config.py:108] JAX version 0.4.23 available.
2024-02-09 18:16:22.917604: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-02-09 18:16:22.917653: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-02-09 18:16:22.919007: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-02-09 18:16:24.052113: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-02-09:18:16:27,157 INFO     [__main__.py:162] Verbosity set to INFO
2024-02-09:18:16:27,157 INFO     [__init__.py:358] lm_eval.tasks.initialize_tasks() is deprecated and no longer necessary. It will be removed in v0.4.2 release. TaskManager will instead be used.
2024-02-09:18:16:31,122 INFO     [__main__.py:238] Selected Tasks: ['squadv2']
2024-02-09:18:16:31,122 INFO     [__main__.py:239] Loading selected tasks...
2024-02-09:18:16:32,000 INFO     [modeling.py:940] We will use 90% of the memory on device 0 for storing the model, and 10% for the buffer to avoid OOM. You can set `max_memory` in to a higher value to use more memory (at your own risk).
Loading checkpoint shards: 100% 2/2 [00:04<00:00,  2.18s/it]
2024-02-09:18:16:37,249 INFO     [evaluator.py:139] get_task_dict has been updated to accept an optional argument, `task_manager`Read more here: https://github.com/EleutherAI/lm-evaluation-harness/blob/recursive-groups/docs/interface.md#external-library-usage
Downloading readme: 100% 8.18k/8.18k [00:00<00:00, 30.5MB/s]
Downloading data: 100% 16.4M/16.4M [00:00<00:00, 24.0MB/s]
Downloading data: 100% 1.35M/1.35M [00:00<00:00, 5.01MB/s]
Generating train split: 100% 130319/130319 [00:00<00:00, 489339.41 examples/s]
Generating validation split: 100% 11873/11873 [00:00<00:00, 567742.57 examples/s]
2024-02-09:18:16:53,839 WARNING  [evaluator.py:171] Overwriting default num_fewshot of squadv2 from None to 0
2024-02-09:18:16:53,839 INFO     [task.py:360] Building contexts for task on rank 0...
2024-02-09:18:16:55,137 INFO     [evaluator.py:362] Running generate_until requests
  0% 0/11873 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/usr/local/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/content/lm-evaluation-harness/lm_eval/__main__.py", line 241, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/content/lm-evaluation-harness/lm_eval/utils.py", line 415, in _wrapper
    return fn(*args, **kwargs)
  File "/content/lm-evaluation-harness/lm_eval/evaluator.py", line 179, in simple_evaluate
    results = evaluate(
  File "/content/lm-evaluation-harness/lm_eval/utils.py", line 415, in _wrapper
    return fn(*args, **kwargs)
  File "/content/lm-evaluation-harness/lm_eval/evaluator.py", line 373, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)
  File "/content/lm-evaluation-harness/lm_eval/models/huggingface.py", line 1143, in generate_until
    cont = self._model_generate(
  File "/content/lm-evaluation-harness/lm_eval/models/huggingface.py", line 736, in _model_generate
    return self.model.generate(
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py", line 1514, in generate
    logits_warper = self._get_logits_warper(generation_config)
  File "/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py", line 695, in _get_logits_warper
    warpers.append(TemperatureLogitsWarper(generation_config.temperature))
  File "/usr/local/lib/python3.10/dist-packages/transformers/generation/logits_process.py", line 278, in __init__
    raise ValueError(except_msg)
ValueError: `temperature` (=0.0) has to be a strictly positive float, otherwise your next token scores will be invalid. If you're looking for greedy decoding strategies, set `do_sample=False`.
  0% 0/11873 [00:07<?, ?it/s]
```

Thanks for reporting this! what’s your `transformers` version?

4.37.2! (the latest on pip)

I'm wondering if an easy fix might be to just unset `temperature` if `do_sample=False`

Yep, this would suffice for now! (and is what we’ll probably have to merge as a fix)

Considering this has been fine with just a warning for us until now this may be a change to how HF treats these kwargs though.

@haileyschoelkopf wait why is this not kicking in? https://github.com/EleutherAI/lm-evaluation-harness/blob/ab4dba8fa4d23b38f2cb03eb747b505f93f33f1c/lm_eval/models/huggingface.py#L730

Update -- it seems like, in some cases, `do_sample` is undefined and `temperature == 0.0`; I think we can safely set `do_sample = True` when this happens! Doing a pull request right now.

Sorted -- for `squadv2` `, do_sample` was undefined; I added a check in `huggingface.py` and, if `temperature==0.0` and `do_sample` is None or undefined, then `do_sample` is set to `False` to avoid sampling with zero temperature.

Pull request: https://github.com/EleutherAI/lm-evaluation-harness/pull/1418