Repository: bigcode-project/bigcode-evaluation-harness
Issue: Code-Llama-7B-Python 4 Bit Error on HumanEval
Root cause: External Dependency Breakage

Hi All,

I've been trying to run the HumanEval benchmark on some CodeLlama models. However, I get an error when using this code. Any idea what's causing the error? I also tested this code on "meta-llama/CodeLlama-7b-hf" (still 4-bit) and had no errors.

**Code**
accelerate launch  main.py \
  --model "meta-llama/CodeLlama-7b-Python-hf" \
  --max_length_generation 1024 \
  --tasks humaneval \
  --temperature 0.2 \
  --n_samples 200 \
  --batch_size 100 \
  --allow_code_execution \
  --load_in_4bit \
  --save_generations \
  --save_generations_path /scratch/wpy2004/coding_compression_llms/eval_save/CodeLlama-7b-Python-hf-humaneval_bnb4.json \
  --metric_output_path /scratch/wpy2004/coding_compression_llms/result_save/CodeLlama-7b-Python-hf-humaneval_bnb4.json \
  --limit 10

**Error**
The following values were not passed to `accelerate launch` and had defaults used instead:
        `--num_processes` was set to a value of `1`
        `--num_machines` was set to a value of `1`
        `--mixed_precision` was set to a value of `'no'`
        `--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
Selected Tasks: ['humaneval']
Loading model in 4bit
Loading checkpoint shards: 100%|██████████████| 2/2 [00:16<00:00,  8.42s/it]
number of problems for this task is 10
 10%|████                                    | 2/20 [00:27<04:09, 13.87s/it]
Traceback (most recent call last):
  File "/scratch/wpy2004/coding_compression_llms/bigcode-evaluation-harness/main.py", line 414, in <module>
    main()
  File "/scratch/wpy2004/coding_compression_llms/bigcode-evaluation-harness/main.py", line 398, in main
    results[task] = evaluator.evaluate(
                    ^^^^^^^^^^^^^^^^^^^
  File "/scratch/wpy2004/coding_compression_llms/bigcode-evaluation-harness/bigcode_eval/evaluator.py", line 95, in evaluate
    generations, references = self.generate_text(task_name, intermediate_generations=intermediate_generations)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/wpy2004/coding_compression_llms/bigcode-evaluation-harness/bigcode_eval/evaluator.py", line 69, in generate_text
    generations = parallel_generations(
                  ^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/wpy2004/coding_compression_llms/bigcode-evaluation-harness/bigcode_eval/generation.py", line 141, in parallel_generations
    generations = complete_code(
                  ^^^^^^^^^^^^^^
  File "/scratch/wpy2004/coding_compression_llms/bigcode-evaluation-harness/bigcode_eval/utils.py", line 294, in complete_code
    generated_tokens = accelerator.unwrap_model(model).generate(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ext3/miniconda3/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/ext3/miniconda3/lib/python3.11/site-packages/transformers/generation/utils.py", line 2223, in generate
    result = self._sample(
             ^^^^^^^^^^^^^
  File "/ext3/miniconda3/lib/python3.11/site-packages/transformers/generation/utils.py", line 3257, in _sample
    next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: probability tensor contains either `inf`, `nan` or element < 0
Traceback (most recent call last):
  File "/ext3/miniconda3/bin/accelerate", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/ext3/miniconda3/lib/python3.11/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main
    args.func(args)
  File "/ext3/miniconda3/lib/python3.11/site-packages/accelerate/commands/launch.py", line 1097, in launch_command
    simple_launcher(args)
  File "/ext3/miniconda3/lib/python3.11/site-packages/accelerate/commands/launch.py", line 703, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/ext3/miniconda3/bin/python', 'main.py', '--model', 'meta-llama/CodeLlama-7b-Python-hf', '--max_length_generation', '1024', '--tasks', 'humaneval', '--temperature', '0.2', '--n_samples', '200', '--batch_size', '100', '--allow_code_execution', '--load_in_4bit', '--save_generations', '--save_generations_path', '/scratch/wpy2004/coding_compression_llms/eval_save/CodeLlama-7b-Python-hf-humaneval_bnb4.json', '--metric_output_path', '/scratch/wpy2004/coding_compression_llms/result_save/CodeLlama-7b-Python-hf-humaneval_bnb4.json', '--limit', '10']' returned non-zero exit status 1.