Repository: tatsu-lab/alpaca_eval
Issue: llama 13b as evaluate_from_model does not use GPU
Root cause: External Dependency Breakage

###
The code is running on 1 * A100, I am using nvidia-smi to check and find that the GPU is not used. After running `alpaca_eval evaluate_from_model --model_outputs $PWD/qa.json --annotators_config 'alpaca_eval_gpt4' --model_configs $PWD/llama` for a few minutes reports an error. May I ask what the cause is? Where am I not set up properly ?
```
ded_voc
    return self.unk_token_id
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1155, in unk_token_id
    return self.convert_tokens_to_ids(self.unk_token)
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 250, in convert_tokens_to_ids
    return self._convert_token_to_id_with_added_voc(tokens)
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 257, in _convert_token_to_id_with_added_voc
    return self.unk_token_id
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1155, in unk_token_id
    return self.convert_tokens_to_ids(self.unk_token)
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 250, in convert_tokens_to_ids
    return self._convert_token_to_id_with_added_voc(tokens)
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 257, in _convert_token_to_id_with_added_voc
    return self.unk_token_id
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1155, in unk_token_id
    return self.convert_tokens_to_ids(self.unk_token)
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 250, in convert_tokens_to_ids
    return self._convert_token_to_id_with_added_voc(tokens)
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 257, in _convert_token_to_id_with_added_voc
    return self.unk_token_id
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1155, in unk_token_id
    return self.convert_tokens_to_ids(self.unk_token)
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 250, in convert_tokens_to_ids
    return self._convert_token_to_id_with_added_voc(tokens)
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 257, in _convert_token_to_id_with_added_voc
    return self.unk_token_id
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1155, in unk_token_id
    return self.convert_tokens_to_ids(self.unk_token)
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 250, in convert_tokens_to_ids
    return self._convert_token_to_id_with_added_voc(tokens)
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 257, in _convert_token_to_id_with_added_voc
    return self.unk_token_id
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1155, in unk_token_id
    return self.convert_tokens_to_ids(self.unk_token)
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 250, in convert_tokens_to_ids
    return self._convert_token_to_id_with_added_voc(tokens)
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 257, in _convert_token_to_id_with_added_voc
    return self.unk_token_id
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1155, in unk_token_id
    return self.convert_tokens_to_ids(self.unk_token)
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 250, in convert_tokens_to_ids
    return self._convert_token_to_id_with_added_voc(tokens)
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 257, in _convert_token_to_id_with_added_voc
    return self.unk_token_id
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1155, in unk_token_id
    return self.convert_tokens_to_ids(self.unk_token)
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 250, in convert_tokens_to_ids
    return self._convert_token_to_id_with_added_voc(tokens)
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 257, in _convert_token_to_id_with_added_voc
    return self.unk_token_id
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1155, in unk_token_id
    return self.convert_tokens_to_ids(self.unk_token)
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 250, in convert_tokens_to_ids
    return self._convert_token_to_id_with_added_voc(tokens)
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 257, in _convert_token_to_id_with_added_voc
    return self.unk_token_id
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1155, in unk_token_id
    return self.convert_tokens_to_ids(self.unk_token)
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 250, in convert_tokens_to_ids
    return self._convert_token_to_id_with_added_voc(tokens)
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 257, in _convert_token_to_id_with_added_voc
    return self.unk_token_id
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1155, in unk_token_id
    return self.convert_tokens_to_ids(self.unk_token)
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 250, in convert_tokens_to_ids
    return self._convert_token_to_id_with_added_voc(tokens)
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 257, in _convert_token_to_id_with_added_voc
    return self.unk_token_id
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1155, in unk_token_id
    return self.convert_tokens_to_ids(self.unk_token)
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 250, in convert_tokens_to_ids
    return self._convert_token_to_id_with_added_voc(tokens)
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 257, in _convert_token_to_id_with_added_voc
    return self.unk_token_id
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1155, in unk_token_id
    return self.convert_tokens_to_ids(self.unk_token)
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 250, in convert_tokens_to_ids
    return self._convert_token_to_id_with_added_voc(tokens)
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 257, in _convert_token_to_id_with_added_voc
    return self.unk_token_id
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1155, in unk_token_id
    return self.convert_tokens_to_ids(self.unk_token)
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 250, in convert_tokens_to_ids
    return self._convert_token_to_id_with_added_voc(tokens)
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 257, in _convert_token_to_id_with_added_voc
    return self.unk_token_id
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1155, in unk_token_id
    return self.convert_tokens_to_ids(self.unk_token)
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 250, in convert_tokens_to_ids
    return self._convert_token_to_id_with_added_voc(tokens)
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 257, in _convert_token_to_id_with_added_voc
    return self.unk_token_id
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1155, in unk_token_id
    return self.convert_tokens_to_ids(self.unk_token)
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 250, in convert_tokens_to_ids
    return self._convert_token_to_id_with_added_voc(tokens)
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 257, in _convert_token_to_id_with_added_voc
    return self.unk_token_id
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1155, in unk_token_id
    return self.convert_tokens_to_ids(self.unk_token)
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 250, in convert_tokens_to_ids
    return self._convert_token_to_id_with_added_voc(tokens)
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 257, in _convert_token_to_id_with_added_voc
    return self.unk_token_id
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1155, in unk_token_id
    return self.convert_tokens_to_ids(self.unk_token)
  File "/root/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1035, in unk_token
    return str(self._unk_token)
RecursionError: maximum recursion depth exceeded while calling a Python object
```
### Run command
`alpaca_eval evaluate_from_model --model_outputs $PWD/qa.json --annotators_config 'alpaca_eval_gpt4' --model_configs $PWD/llama`

### model config
```yaml
# configs.yaml
knowlm-13b:
  prompt_template: "/root/model-eval/llama/prompt.txt"
  fn_completions: "huggingface_local_completions"
  completions_kwargs:
    model_name: "/root/.cache/LLAMA/" # LLAMA 13B
    model_kwargs:
      torch_dtype: 'float32'
    max_new_tokens: 2000
    temperature: 0.7
    top_p: 1.0
    do_sample: True
  pretty_name: "LLAMA 13B"
  link: "https://example.com/"
```
### env
```
> python --version
Python 3.10.11

>>> import torch
>>> print(torch.cuda.is_available())
True
>>> print(torch.__version__)
2.0.1+cu117

> tree /root/.cache/LLAMA/
|-- config.json
|-- generation_config.json
|-- model-00002-of-00006.safetensors
|-- pytorch_model-00001-of-00006.bin
|-- pytorch_model-00002-of-00006.bin
|-- pytorch_model-00003-of-00006.bin
|-- pytorch_model-00004-of-00006.bin
|-- pytorch_model-00005-of-00006.bin
|-- pytorch_model-00006-of-00006.bin
|-- pytorch_model.bin.index.json
|-- special_tokens_map.json
|-- tokenizer.model
|-- tokenizer_config.json
```


Hi @RealTong ,
This seems to be a runtime issue instead of a gpu not in use issue. The error stacktrace seems to suggest a recursion error during tokenization. What is your transformers version? Could you upgrade to `transformers>=4.29.2` and try again?

I've tried transformers>=4.29.2 and it still doesn't work. It starts reporting RecursionError: maximum recursion depth exceeded while calling a Python object after 3 minutes of running.

Is there any other useful information I can provide?

I see. The issue could be due to some issue in your local checkpoint, which would be hard for me to help you debug. As a sanity check, could you try running a model that we've already integrated/validated? For example, `falcon-7b-instruct`? 
```
alpaca_eval evaluate_from_model --model_configs 'falcon-7b-instruct'
```

```yaml
falcon-7b-instruct:
  prompt_template: "/root/code/eval/models/falcon-7B/prompt.txt"
  fn_completions: "huggingface_local_completions"
  completions_kwargs:
    model_name: "/root/.cache/LLM-Repo/model/falcon-7b-instruct/"
    model_kwargs:
      torch_dtype: 'bfloat16'
    max_new_tokens: 2048
    temperature: 0.7
    top_p: 1.0
    do_sample: True
    trust_remote_code: True
  pretty_name: "Falcon 7B Instruct"
  link: ""
```

I run `alpaca_eval evaluate_from_model --model_configs 'falcon-7b-instruct'` and I am prompted with `ValueError: Loading /root/.cache/LLM-Repo/model/falcon-7b -instruct/ requires you to execute the configuration file in that repo on your local machine. Make sure you have read the code there to avoid malicious use. Then set the option `trust_remote_code=True` to remove this error.`

Why do I load the local model and execute the code from the network. I have added `trust_remote_code: True` to the proxy configs.yaml, but it still gives me this error, how should I fix it?



Ah great point! Apologize for the oversight. Just patched the configs with the proper flag. Could you try again?

Not working :).

Reported error: TypeError: transformers.pipelines.base.infer_framework_load_model() got multiple values for keyword argument 'trust_remote_code'

Here is my configuration
```yaml
falcon-7b-instruct:
  prompt_template: "/root/code/eval/models/falcon-7B/prompt.txt"
  fn_completions: "huggingface_local_completions"
  completions_kwargs:
    model_name: "/root/.cache/LLM-Repo/model/falcon-7b-instruct/"
    model_kwargs:
      torch_dtype: 'bfloat16'
      trust_remote_code: True
    max_new_tokens: 2048
    temperature: 0.7
    top_p: 1.0
    do_sample: True
  pretty_name: "Falcon 7B Instruct"
  link: ""
```

Ok very interesting, I'll need to investigate this issue more when I get my GPU machines back. Currently our cluster is down for the week, so I'll provide an update next week.

Since this issue is specific to falcon, let's try another model that doesn't require the `trust_remode_code` Flag. Can you try
```
alpaca_eval evaluate_from_model --model_configs 'oasst-sft-pythia-12b'
```
Apologize for the back and forth on resolving this.

Hi @RealTong, just following up on this.
I've fixed the issue with falcon https://github.com/tatsu-lab/alpaca_eval/pull/47 and it's now able to run decoding (there still be a small issue https://github.com/tatsu-lab/alpaca_eval/issues/46 we're looking into).

Regardless, could you please pull the latest from main and see if your original GPU issue has been resolved? Just to clarify, I've tested `alpaca_eval evaluate_from_model --model_configs 'falcon-7b-instruct'` on my system and have verified that the GPU is being utilized.
If you're still running into a GPU issue, please feel free to open another issue and post the relevant details.