Repository: tatsu-lab/alpaca_eval
Issue: error while using guanaco_33b as evaluator
Root cause: Validation Gap

Command:
alpaca_eval evaluate --model_outputs=results/vicuna-7b-v1.3/model_outputs.json --reference_outputs=results/gpt4/model_outputs.json --annotators_config=guanaco_33b/configs.yaml

Error:
ValueError: Error in inference. Full error: Input validation error: `inputs` must have less than 1024 tokens. Given: 1233

I switched to local vs api, still with guanaco_33b, with the config sligthly modified:
```
guanaco_33b:
  prompt_template: "guanaco_33b/basic_prompt.txt"
  fn_completions: "huggingface_local_completions" # use "huggingface_local_completions" if you want local serving
  completions_kwargs:
    model_name: "timdettmers/guanaco-33b-merged"
    model_kwargs:
      trust_remote_code: True
      load_in_4bit: True  
    max_new_tokens: 50
    is_fast_tokenizer: False # needed for local
  completion_parser_kwargs:
    outputs_to_match:
      1: '(?:^|\n) ?Output \(a\)'
      2: '(?:^|\n) ?Output \(b\)'
  batch_size: 1
```
As you can see I am using `load_in_4bit: true` to be able to fit the model on my 2 24GB GPU.

Now it works during a certain period of time until I get:
1) some warnings:
```
WARNING:root:Found 2 annotations in:'''
 Output (a) is more informative and detailed, providing a clear explanation of the game and its rules. It also includes tips on improving skills and safety considerations.

Output (b) is more concise and to the point,
''' but expected 1. We are setting all annotations to None.
```
and
2) an error:
```
Traceback (most recent call last):
  File "/home/vincent/miniconda3/envs/pytorch1.14/bin/alpaca_eval", line 33, in <module>
    sys.exit(load_entry_point('alpaca-eval==0.3.0', 'console_scripts', 'alpaca_eval')())
  File "/home/vincent/miniconda3/envs/pytorch1.14/lib/python3.10/site-packages/alpaca_eval-0.3.0-py3.10.egg/alpaca_eval/main.py", line 539, in main
    fire.Fire(ALL_FUNCTIONS)
  File "/home/vincent/miniconda3/envs/pytorch1.14/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/vincent/miniconda3/envs/pytorch1.14/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/vincent/miniconda3/envs/pytorch1.14/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/vincent/miniconda3/envs/pytorch1.14/lib/python3.10/site-packages/alpaca_eval-0.3.0-py3.10.egg/alpaca_eval/main.py", line 132, in evaluate
    annotations = annotator.annotate_head2head(
  File "/home/vincent/miniconda3/envs/pytorch1.14/lib/python3.10/site-packages/alpaca_eval-0.3.0-py3.10.egg/alpaca_eval/annotators/pairwise_evaluator.py", line 237, in annotate_head2head
    out = self.__call__(df_to_annotate, **decoding_kwargs)
  File "/home/vincent/miniconda3/envs/pytorch1.14/lib/python3.10/site-packages/alpaca_eval-0.3.0-py3.10.egg/alpaca_eval/annotators/base.py", line 171, in __call__
    df_annotated = self._annotate(curr_df_to_annotate, **decoding_kwargs)
  File "/home/vincent/miniconda3/envs/pytorch1.14/lib/python3.10/site-packages/alpaca_eval-0.3.0-py3.10.egg/alpaca_eval/annotators/base.py", line 258, in _annotate
    curr_annotated = self.annotators[annotator](
  File "/home/vincent/miniconda3/envs/pytorch1.14/lib/python3.10/site-packages/alpaca_eval-0.3.0-py3.10.egg/alpaca_eval/annotators/base.py", line 566, in __call__
    completions = self.fn_completions(prompts=prompts, **self.completions_kwargs, **decoding_kwargs)
  File "/home/vincent/miniconda3/envs/pytorch1.14/lib/python3.10/site-packages/alpaca_eval-0.3.0-py3.10.egg/alpaca_eval/decoders/huggingface_local.py", line 91, in huggingface_local_completions
    model = AutoModelForCausalLM.from_pretrained(model_name, cache_dir=cache_dir, **model_kwargs).eval()
  File "/home/vincent/miniconda3/envs/pytorch1.14/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 493, in from_pretrained
    return model_class.from_pretrained(
  File "/home/vincent/miniconda3/envs/pytorch1.14/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2842, in from_pretrained
    raise ValueError(
ValueError: 
                        Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit
                        the quantized model. If you want to dispatch the model on the CPU or the disk while keeping
                        these modules in 32-bit, you need to set `load_in_8bit_fp32_cpu_offload=True` and pass a custom
                        `device_map` to `from_pretrained`. Check
                        https://huggingface.co/docs/transformers/main/en/main_classes/quantization#offload-between-cpu-and-gpu
                        for more details.
```

However I don't want to use cpu offload sine I want to keep all weight in 4bit on GPU. Unsure how to fix this.