Repository: EleutherAI/lm-evaluation-harness
Issue: Error occur when evaluating local model: transformer sentencepiece piece id out of range
Root cause: Validation Gap

I have a finetuned model saved locally and want to evaluate the model using lm-evaluation-harness. 

I passed in this command:
python ./../main.py \
        --model hf-causal \
	--model_args pretrained=$1,tokenizer=$1 \
        --tasks gsm8k \ 
	--num_fewshot 5 \
	--batch_size 1 \
        --device cuda:0  \
	--no_cache \
        --write_out  \
        --output_base_path ./output
NOTE: I also modified the DATASET_PATH in gsm8k to be a local path where gsm8k is stored.

However, the following error persists (same error occurred for MATH eval as well):
Traceback (most recent call last):
  File "/models/anqi/lm-evaluation-harness/finetuned_llamav2-7b/./../main.py", line 105, in <module>
    main()
  File "/models/anqi/lm-evaluation-harness/finetuned_llamav2-7b/./../main.py", line 69, in main
    results = evaluator.simple_evaluate(
  File "/models/anqi/lm-evaluation-harness/lm_eval/utils.py", line 243, in _wrapper
    return fn(*args, **kwargs)
  File "/models/anqi/lm-evaluation-harness/lm_eval/evaluator.py", line 110, in simple_evaluate
    results = evaluate(
  File "/models/anqi/lm-evaluation-harness/lm_eval/utils.py", line 243, in _wrapper
    return fn(*args, **kwargs)
  File "/models/anqi/lm-evaluation-harness/lm_eval/evaluator.py", line 312, in evaluate
    resps = getattr(lm, reqtype)([req.args for req in reqs])
  File "/models/anqi/lm-evaluation-harness/lm_eval/base.py", line 460, in greedy_until
    s = self.tok_decode(cont[0].tolist()[context_enc.shape[1] :])
  File "/models/anqi/lm-evaluation-harness/lm_eval/models/gpt2.py", line 159, in tok_decode
    return self.tokenizer.decode(tokens)
  File "/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py", line 3746, in decode
    return self._decode(
  File "/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils.py", line 1001, in _decode
    filtered_tokens = self.convert_ids_to_tokens(token_ids, skip_special_tokens=skip_special_tokens)
  File "/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils.py", line 982, in convert_ids_to_tokens
    tokens.append(self._convert_id_to_token(index))
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/llama/tokenization_llama.py", line 280, in _convert_id_to_token
    token = self.sp_model.IdToPiece(index)
  File "/usr/local/lib/python3.10/dist-packages/sentencepiece/__init__.py", line 1045, in _batched_func
    return _func(self, arg)
  File "/usr/local/lib/python3.10/dist-packages/sentencepiece/__init__.py", line 1038, in _func
    raise IndexError('piece id is out of range.')

Hi! Can you print what token ids are being passed to self.tokenizer.decode?

Did you add any special tokens to your finetuned model?

Sorry for the late reply. I didn't add any special tokens. 
The package is working fine with all models directly downloaded from HF but not with finetuned models (I used DeepSpeed+LoRA). I think it has something to do with the output dimension of the final fully-connected layer, which is greater than the vocab size. I got it solved by adding a condition to recognize the padding token exceeding the range.

Closing because this should be fixed via our updates to the setting of pad tokens. Please reopen this if you are still experiencing the issue on the most recent codebase version (`main` branch).