Repository: EleutherAI/lm-evaluation-harness
Issue: RecursionError: maximum recursion depth exceeded while calling a Python object at _create_auto_tokenizer
Root cause: External Dependency Breakage

I am attempting to run lm-evaluation-harness using ```decapoda-research/llama-7b-hf``` and ```tloen/alpaca-lora-7b``` peft but getting a recursion error at the ```self.AUTO_TOKENIZER_CLASS.from_pretrained``` method call. 

```--model_args="pretrained=decapoda-research/llama-7b-hf,peft=tloen/alpaca-lora-7b"```

I am using the master branch of lm-evaluation-harness and the following package versions for ```protobuf``` and ```sentencepiece```:
```
protobuf==3.19.0
sentencepiece==0.1.99
```

Would appreciate any help as I am not sure where is the error or how to fix it. 

```python
def _create_auto_tokenizer(
        self,
        *,
        pretrained: str,
        revision: str,
        subfolder: str,
        tokenizer: Optional[str] = None,
        trust_remote_code: Optional[bool] = False,
    ) -> transformers.PreTrainedTokenizer:
        """Returns a pre-trained tokenizer from a pre-trained tokenizer configuration."""
        tokenizer = self.AUTO_TOKENIZER_CLASS.from_pretrained(
            pretrained if tokenizer is None else tokenizer,
            revision=revision + ("/" + subfolder if subfolder is not None else ""),
            trust_remote_code=trust_remote_code,
        )
        tokenizer.pad_token = tokenizer.eos_token
        return tokenizer
``` 
The last few lines of the error:
```
  File "/home/adrian/miniconda3/envs/flm-eval-env/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1155, in unk_token_id
    return self.convert_tokens_to_ids(self.unk_token)
  File "/home/adrian/miniconda3/envs/flm-eval-env/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 250, in convert_tokens_to_ids
    return self._convert_token_to_id_with_added_voc(tokens)
  File "/home/adrian/miniconda3/envs/flm-eval-env/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 257, in _convert_token_to_id_with_added_voc
    return self.unk_token_id
  File "/home/adrian/miniconda3/envs/flm-eval-env/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1155, in unk_token_id
    return self.convert_tokens_to_ids(self.unk_token)
  File "/home/adrian/miniconda3/envs/flm-eval-env/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 250, in convert_tokens_to_ids
    return self._convert_token_to_id_with_added_voc(tokens)
  File "/home/adrian/miniconda3/envs/flm-eval-env/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 257, in _convert_token_to_id_with_added_voc
    return self.unk_token_id
  File "/home/adrian/miniconda3/envs/flm-eval-env/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1155, in unk_token_id
    return self.convert_tokens_to_ids(self.unk_token)
  File "/home/adrian/miniconda3/envs/flm-eval-env/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1026, in unk_token
    @property
  File "_pydevd_bundle/pydevd_cython.pyx", line 1457, in _pydevd_bundle.pydevd_cython.SafeCallWrapper.__call__
RecursionError: maximum recursion depth exceeded while calling a Python object
```



Hi! Thanks for reporting this error. This might be the same thing that was cropping up in #442 .

I think that if you add, where 

```python 
tokenizer.pad_token = tokenizer.eos_token
```
, the additional line
```python
tokenizer.unk_token = tokenizer.eos_token
```
There's a chance this would be fixed--or alternatively, setting `tokenizer.pad_token` and `unk_token` to 0 or some other hardcoded value.

I'd really like to get to the bottom of this rare error though, so if you're able it would be a great help to know:

1. What is the exact command (and commit) you're invoking here?
2. Do you have the full error paste on hand? It'd be great to know at what line in our codebase the tokenizer's unk token is required, I would not expect that to be called at any point ideally.

I am getting this issue . I am using the huggingface llama1-7b model from decapoda-research. I tried adding pad and unk tokens as you suggested. It is not getting solved.  This is the command I am running:
 python main.py \
    --model hf-causal-experimental \
    --model_args pretrained=/path/to/model,use_accelerate=True,trust_remote_code=True \
    --tasks hellaswag \
    --device cuda:0 \
    --num_fewshot 10 \
    --no_cache
    
  But I am not getting this error for llama-2. I am using the repo without making any changes. What could be the reason?

This issue no longer appears on the most recent v0.4.x versions, to the best of my knowledge.