Repository: EleutherAI/lm-evaluation-harness
Issue: How to evaluate local model with local-completions? 
Root cause: Validation Gap

I installed lm_eval on my laptop and would like to evaluate local model running on another server. Could anyone help me on how to run the command? Any parameters I set in wrong? Or any other information I should provide? Thanks a lot!

The command I ran like below:
lm_eval --model local-completions --tasks mmlu_college_biology --model_args model=Meta-Llama-3.1-8B-Instruct,base_url=http://<server IP>:<port>/v1/completions,num_concurrent=1,max_retries=3,tokenized_requests=False

While I got below output:

2024-10-14:11:53:33,690 INFO     [__main__.py:279] Verbosity set to INFO
2024-10-14:11:54:14,581 INFO     [__main__.py:376] Selected Tasks: ['mmlu_college_biology']
2024-10-14:11:54:14,639 INFO     [evaluator.py:164] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2024-10-14:11:54:14,640 INFO     [evaluator.py:201] Initializing local-completions model, with arguments: {'model': 'Meta-Llama-3.1-8B-Instruct', 'base_url': 'http://10.232.19.223:9997/v1/completions', 'num_concurrent': 1, 'max_retries': 3, 'tokenized_requests': False}
2024-10-14:11:54:14,640 INFO     [api_models.py:111] Using max length 2048 - 1
2024-10-14:11:54:14,640 INFO     [api_models.py:114] Concurrent requests are disabled. To enable concurrent requests, set `num_concurrent` > 1.
2024-10-14:11:54:14,640 INFO     [api_models.py:124] Using tokenizer huggingface
Traceback (most recent call last):
  File "C:\Users\zhuotao.liu\AppData\Local\miniforge3\envs\EAI_env\Lib\site-packages\huggingface_hub\utils\_http.py", line 406, in hf_raise_for_status
    response.raise_for_status()
  File "C:\Users\zhuotao.liu\AppData\Local\miniforge3\envs\EAI_env\Lib\site-packages\requests\models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://huggingface.co/Meta-Llama-3.1-8B-Instruct/resolve/main/tokenizer_config.json

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\zhuotao.liu\AppData\Local\miniforge3\envs\EAI_env\Lib\site-packages\transformers\utils\hub.py", line 402, in cached_file
    resolved_file = hf_hub_download(
                    ^^^^^^^^^^^^^^^^
  File "C:\Users\zhuotao.liu\AppData\Local\miniforge3\envs\EAI_env\Lib\site-packages\huggingface_hub\utils\_deprecation.py", line 101, in inner_f
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "C:\Users\zhuotao.liu\AppData\Local\miniforge3\envs\EAI_env\Lib\site-packages\huggingface_hub\utils\_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\zhuotao.liu\AppData\Local\miniforge3\envs\EAI_env\Lib\site-packages\huggingface_hub\file_download.py", line 1232, in hf_hub_download
    return _hf_hub_download_to_cache_dir(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\zhuotao.liu\AppData\Local\miniforge3\envs\EAI_env\Lib\site-packages\huggingface_hub\file_download.py", line 1339, in _hf_hub_download_to_cache_dir
    _raise_on_head_call_error(head_call_error, force_download, local_files_only)
  File "C:\Users\zhuotao.liu\AppData\Local\miniforge3\envs\EAI_env\Lib\site-packages\huggingface_hub\file_download.py", line 1854, in _raise_on_head_call_error
    raise head_call_error
  File "C:\Users\zhuotao.liu\AppData\Local\miniforge3\envs\EAI_env\Lib\site-packages\huggingface_hub\file_download.py", line 1746, in _get_metadata_or_catch_error
    metadata = get_hf_file_metadata(
               ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\zhuotao.liu\AppData\Local\miniforge3\envs\EAI_env\Lib\site-packages\huggingface_hub\utils\_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\zhuotao.liu\AppData\Local\miniforge3\envs\EAI_env\Lib\site-packages\huggingface_hub\file_download.py", line 1666, in get_hf_file_metadata
    r = _request_wrapper(
        ^^^^^^^^^^^^^^^^^
  File "C:\Users\zhuotao.liu\AppData\Local\miniforge3\envs\EAI_env\Lib\site-packages\huggingface_hub\file_download.py", line 364, in _request_wrapper
    response = _request_wrapper(
               ^^^^^^^^^^^^^^^^^
  File "C:\Users\zhuotao.liu\AppData\Local\miniforge3\envs\EAI_env\Lib\site-packages\huggingface_hub\file_download.py", line 388, in _request_wrapper
    hf_raise_for_status(response)
  File "C:\Users\zhuotao.liu\AppData\Local\miniforge3\envs\EAI_env\Lib\site-packages\huggingface_hub\utils\_http.py", line 454, in hf_raise_for_status
    raise _format(RepositoryNotFoundError, message, response) from e
huggingface_hub.errors.RepositoryNotFoundError: 401 Client Error. (Request ID: Root=1-670c95e9-16fbdf657452e8740053bb10;b1f41d63-9a3c-4e63-ac5c-0a51d0718275)

Repository Not Found for url: https://huggingface.co/Meta-Llama-3.1-8B-Instruct/resolve/main/tokenizer_config.json.
Please make sure you specified the correct `repo_id` and `repo_type`.
If you are trying to access a private or gated repo, make sure you are authenticated.
Invalid username or password.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\zhuotao.liu\AppData\Local\miniforge3\envs\EAI_env\Lib\site-packages\transformers\utils\hub.py", line 425, in cached_file
    raise EnvironmentError(
OSError: Meta-Llama-3.1-8B-Instruct is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo either by logging in with `huggingface-cli login` or by passing `token=<your_token>`
python-BaseException

Hi! It uses the model name to look up the tokenizer from huggingface, where the repo address is `meta-llama/Llama-3.1-8B-Instruct`. You can also provide a different one by passing in  `tokenizer=...`, if your API requires the model name in a particular format. 

@baberabb Thanks a lot for the answer. And I've tried while met another issue wish you can help.

Let me clarify my evironment. We have a server A on which the model "Meta-Llama-3.1-8B-Instruct" is loaded through Xinference. And I set up lm_eval on my laptop. I wish using lm_eval on my labtop to evaluate the model on server A through openai completions API.

Now I use below command (any parameters wrong?):
lm_eval --model local-completions --tasks mmlu_college_biology --model_args model=Meta-Llama-3.1-8B-Instruct,tokenizer="d:/llm/model/Meta-Llama-3.1-8B-Instruct",base_url=http://<server A IP>:<port>/v1/completions

And get below output:
2024-10-21:17:49:43,923 INFO     [__main__.py:279] Verbosity set to INFO
2024-10-21:17:50:26,636 INFO     [__main__.py:376] Selected Tasks: ['mmlu_college_biology']
2024-10-21:17:50:26,702 INFO     [evaluator.py:164] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2024-10-21:17:50:26,702 INFO     [evaluator.py:201] Initializing local-completions model, with arguments: {'model': 'Meta-Llama-3.1-8B-Instruct', 'tokenizer': 'd:/llm/model/Meta-Llama-3.1-8B-Instruct', 'base_url': 'http://10.232.19.223:9997/v1/completions'}
2024-10-21:17:50:26,702 INFO     [api_models.py:111] Using max length 2048 - 1
2024-10-21:17:50:26,702 INFO     [api_models.py:114] Concurrent requests are disabled. To enable concurrent requests, set `num_concurrent` > 1.
2024-10-21:17:50:26,702 INFO     [api_models.py:124] Using tokenizer huggingface
2024-10-21:17:50:31,040 WARNING  [model.py:422] model.chat_template was called with the chat_template set to False or None. Therefore no chat template will be applied. Make sure this is an intended behavior.
2024-10-21:17:50:31,045 INFO     [task.py:415] Building contexts for mmlu_college_biology on rank 0...
100%|██████████| 144/144 [00:00<00:00, 221.67it/s]
2024-10-21:17:50:31,715 INFO     [evaluator.py:489] Running loglikelihood requests
Requesting API:   0%|          | 0/576 [00:00<?, ?it/s]2024-10-21:17:50:32,327 WARNING  [api_models.py:347] API request failed with error message: Internal Server Error: 1 validation error for CreateCompletionRequest
prompt
  str type expected (type=type_error.str). Retrying...
2024-10-21:17:50:34,457 WARNING  [api_models.py:347] API request failed with error message: Internal Server Error: 1 validation error for CreateCompletionRequest
prompt
  str type expected (type=type_error.str). Retrying...
2024-10-21:17:50:36,686 WARNING  [api_models.py:347] API request failed with error message: Internal Server Error: 1 validation error for CreateCompletionRequest
prompt
  str type expected (type=type_error.str). Retrying...
Traceback (most recent call last):
  File "C:\Users\zhuotao.liu\AppData\Local\miniforge3\envs\EAI_env\Lib\concurrent\futures\_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\zhuotao.liu\AppData\Local\miniforge3\envs\EAI_env\Lib\concurrent\futures\_base.py", line 401, in __get_result
    raise self._exception
  File "C:\Users\zhuotao.liu\AppData\Local\miniforge3\envs\EAI_env\Lib\site-packages\tenacity\__init__.py", line 478, in __call__
    result = fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^
  File "D:\pythonProject\EleutherAI\lm-evaluation-harness\lm_eval\models\api_models.py", line 350, in model_call
    response.raise_for_status()
  File "C:\Users\zhuotao.liu\AppData\Local\miniforge3\envs\EAI_env\Lib\site-packages\requests\models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://10.232.19.223:9997/v1/completions
python-BaseException
Requesting API:   0%|          | 0/576 [04:00<?, ?it/s]

I checked that the REST API request.body is like below. It's not wrapped into the requested openai completions API format.

{"model": "Meta-Llama-3.1-8B-Instruct", "prompt": [[791, 2768, 527, 5361, 5873, 4860, 320, 4291, 11503, 8, 922, 7926, 34458, 382, 32, 19646, 1481, 5620, 98486, 706, 17551, 1202, 19335, 304, 279, 4219, 315, 264, 220, 20, 4771, 6418, 7732, 52346, 11, 323, 279, 19335, 617, 15196, 67, 1139, 83861, 13, 763, 6300, 2015, 320, 42450, 311, 4994, 705, 1148, 39881, 323, 20438, 13918, 1053, 279, 83861, 13123, 439, 814, 8343, 872, 1648, 9017, 279, 4994, 315, 279, 52346, 30, 320, 2520, 40075, 11, 9855, 430, 279, 4248, 1814, 34965, 706, 2736, 21536, 1022, 6724, 29275, 32, 13, 393, 411, 26, 6156, 865, 982, 76, 26, 14580, 865, 982, 76, 26, 64603, 25113, 2411, 26, 6156, 1343, 385, 336, 26, 14580, 1343, 385, 336, 26, 49370, 198, 33, 13, 393, 411, 26, 14580, 865, 982, 76, 26, 6156, 865, 982, 76, 26, 64603, 25113, 2411, 26, 14580, 1343, 385, 336, 26, 6156, 1343, 385, 336, 26, 49370, 198, 34, 13, 393, 411, 26, 14580, 865, 982, 76, 26, 6156, 865, 982, 76, 26, 64603, 25113, 2411, 26, 6156, 1343, 385, 336, 26, 14580, 1343, 385, 336, 26, 49370, 198, 35, 13, 393, 411, 26, 6156, 865, 982, 76, 26, 14580, 865, 982, 76, 26, 64603, 25113, 2411, 26, 14580, 1343, 385, 336, 26, 6156, 1343, 385, 336, 26, 49370, 198, 16533, 25, 356]], "temperature": 0, "max_tokens": 1, "logprobs": 1, "seed": 1234, "echo": true}'

looks like your API requires a str `prompt`. You can set `tokenized_requests=False` in the model args (the tokens will be decoded back to strings). Also might want to add `--apply_chat_template` (optionally with `--fewshot_as_multiturn`) to wrap the prompt in a chat template. Some more details [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/API_guide.md)

why lm_eval needs tokenizer? 

Up ? 
I'm trying to hit an OpenAI compatible local API and am getting the same error, despite 
* `tokenized_requests=False`
* `tokenizer=none`

me meet it too.  I JUST want to evalut the model with http（open ai api style).   anyone solve it？

Following

Same problem... Trying to evaluate vllm with the following, it only works if I actually give it a tokenizer:

Fails without tokenizer:
```console
lm_eval \
   --model local-completions \
   --tasks gsm8k \
   --model_args model=MiniMax-M2-AWQ,base_url=http://127.0.0.1:8000/v1/completions,tokenized_requests=False,trust_remote_code=True,num_concurrent=10 \
   --output_path ./results
```

Works with tokenizer:
```console
lm_eval \
   --model local-completions \
   --tasks gsm8k \
   --model_args model=MiniMax-M2-AWQ,base_url=http://127.0.0.1:8000/v1/completions,tokenizer=QuantTrio/MiniMax-M2-AWQ,trust_remote_code=True,num_concurrent=10 \
   --output_path ./results
```