Repository: EleutherAI/lm-evaluation-harness
Issue: Problems when testing Mistral-7B
Root cause: Configuration Error

I tested Mistral-7B-v0.1 on 5 different benchmarks using the platform, but I encountered some issues.
When testing Squadv2 and 2shots, there was no problem with VLLM during generation, but it reported an error of OOM during the later calculation loglikelihhod stage (even setting bs to 1 );
When testing bbh-3shot, nq_open, triviaqa, mmlu -5shots, the entire platform froze. This situation will occur whether I use a single card or multiple cards.
I also tried to accelerate, but the results were very similar and I was unable to successfully run the task.
The environment I am using is vllm 0.3.2, transformers 4.38.1, Python 3.11, cuda 12.1, 6000ada 48G

2024-03-07:11:17:37,806 INFO     [task.py:386] Building contexts for None on rank 0...
100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 11873/11873 [00:00<00:00, 17424.55it/s]
2024-03-07:11:17:39,075 INFO     [evaluator.py:359] Running generate_until requests
Processed prompts: 100%|██████████████████████████████████████████████████████████████████████████████████| 11873/11873 [15:35<00:00, 12.69it/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 11873/11873 [15:39<00:00, 12.64it/s]
2024-03-07:11:33:33,265 INFO     [evaluator.py:359] Running loglikelihood requests
  0%|                                                                                                                 | 0/11873 [00:00<?, ?it/sT
  File "/opt/conda/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/gemini/data-3/lm-evaluation-harness-main/lm_eval/__main__.py", line 279, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/gemini/data-3/lm-evaluation-harness-main/lm_eval/utils.py", line 288, in _wrapper
    return fn(*args, **kwargs)
  File "/gemini/data-3/lm-evaluation-harness-main/lm_eval/evaluator.py", line 233, in simple_evaluate
    results = evaluate(
  File "/gemini/data-3/lm-evaluation-harness-main/lm_eval/utils.py", line 288, in _wrapper
    return fn(*args, **kwargs)
  File "/gemini/data-3/lm-evaluation-harness-main/lm_eval/evaluator.py", line 370, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)
  File "/gemini/data-3/lm-evaluation-harness-main/lm_eval/models/vllm_causallms.py", line 226, in loglikelihood
    return self._loglikelihood_tokens(new_reqs)
  File "/gemini/data-3/lm-evaluation-harness-main/lm_eval/models/vllm_causallms.py", line 372, in _loglikelihood_tokens
    outputs = self._model_generate(requests=inputs, generate=False)
  File "/gemini/data-3/lm-evaluation-harness-main/lm_eval/models/vllm_causallms.py", line 190, in _model_generate
    outputs = self.model.generate(
  File "/opt/conda/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 182, in generate
    return self._run_engine(use_tqdm)
  File "/opt/conda/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 208, in _run_engine
    step_outputs = self.llm_engine.step()
  File "/opt/conda/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 838, in step
    all_outputs = self._run_workers(
  File "/opt/conda/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 1041, in _run_workers
    driver_worker_output = getattr(self.driver_worker,
  File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/vllm/worker/worker.py", line 223, in execute_model
    output = self.model_runner.execute_model(seq_group_metadata_list,
  File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 590, in execute_model
    output = self.model.sample(
  File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 346, in sample
    next_tokens = self.sampler(self.lm_head.weight, hidden_states,
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/layers/sampler.py", line 109, in forward
    probs = torch.softmax(logits, dim=-1, dtype=torch.float)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.89 GiB. GPU 0 has a total capacty of 47.51 GiB of which 774.62 MiB is free. Process 710726 has 46.73 GiB memory in use. Of the allocated memory 43.10 GiB is allocated by PyTorch, and 1.32 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Processed prompts:   0%|▎                                                                                    | 42/11873 [00:12<58:44,  3.36it/s]
  0%|                                                                                                                 | 0/11873 [00:16<?, ?it/s]

Mistral has too big context available to fill, so pass max_length inside. Something like (check documentation yourself):
`--model_args pretrained=mistralai/Mistral-7B-v0.1,dtype=auto,max_length=11000`


> Mistral has too big context available to fill, so pass max_length inside. Something like (check documentation yourself): `--model_args pretrained=mistralai/Mistral-7B-v0.1,dtype=auto,max_length=11000`

Thanks! After setting max_length to 4096, it works for squadv2 2shots. However, I still get stuck at testing bbh-3shot, nq_open, triviaqa, mmlu for 5shots. The whole program just stuck there and both cpu and gpu utilization is nearly 0%.
<img width="1003" alt="image" src="https://github.com/EleutherAI/lm-evaluation-harness/assets/57589246/e5fe2a75-716b-4c21-9e81-ccccb9e1f0ca">


I don't understand it, but it seems after changing datasets back to version 2.14.1, everything back to normal suddenly