Repository: EleutherAI/lm-evaluation-harness
Issue: TGI support - API evaluation of HF models
Root cause: Unimplemented Feature Gap

Since HF TGI's  [PR](https://github.com/huggingface/text-generation-inference/pull/617) was merged, it should possible to integrate TGI endpoints to the lm-evaluation-harness supported APIs.

Any plans to do so ? This would enable decorrelating the evaluation machine from the served model and largely help facilitate evaluation and hosting !

Thanks a lot for the great work !

Hi! We'd love to move toward hosting models as endpoints to make evaluation faster and more lightweight than using HF models locally. 

Adding vLLM, TGI, and support for inference on a separate machine / in a subprocess is on the roadmap long-term, but we don't have an ETA on it--if you are interested in helping contribute such a feature, let us know!

I am, but won't have time over the next couple of weeks and will probably resort to using the lm-eval-harness as is (or add a few tasks) ! Thanks again for the great work !

> Adding vLLM, TGI, and support for inference on a separate machine / in a subprocess is on the roadmap long-term

@haileyschoelkopf I've been looking into this idea a bit, as it's something that would be incredibly useful for my organization. One thing I'm curious about is if it is clear what API protocol an external model would need to satisfy to be compatible with lm-eval-harness. For instance, [HELM recently introduced support](https://github.com/stanford-crfm/helm/pull/1693) for externally hosted models for the Neurips challenge, where encoding/decoding of tokens is handled externally by the service. That protocol involves three POST endpoints, `/encode`, `/decode`, and `/process`. 

Is there a single protocol that a vLLM or TGI powered service would have to satisfy to be queryable by lm-eval-harness?

Cheers,
Sean


I believe LiteLLM can help with this - we allow you to call TGI LLMs in the Completion Input/Output format 
Thanks @Vinno97 cc @ManuelFay @haileyschoelkopf 

here's a tutorial on using our openai proxy server to call HF TGI models with lm-evaluation harness 
docs: https://docs.litellm.ai/docs/tutorials/lm_evaluation_harness 

## Usage
Step 1: Start the local proxy
```shell
litellm --model huggingface/bigcode/starcoder
```
OpenAI Compatible Endpoint at http://0.0.0.0:8000/

Step 2: Set OpenAI API Base
```shell
$ export OPENAI_API_BASE="http://0.0.0.0:8000"
```

Step 3: Run LM-Eval-Harness
```shell
$ python3 main.py \
  --model gpt3 \
  --model_args engine=huggingface/bigcode/starcoder \
  --tasks hellaswag
```

That's very cool, thanks ! 

I have a problem with your code snippet @ishaan-jaff:

KeyError: 'Could not automatically map huggingface/my_model to a tokeniser. Please use `tiktoken.get_encoding` to explicitly get the tokeniser you expect.'


@ManuelFay are you on the big refactor branch ? 

can i see your code 
- how you start the litellm proxy 
- the command you're using to call lm harness 

Yup big-refactor branch:
- Start proxy: ` litellm --model "huggingface/manu/llama-oscar-fr"`
- Command to start: ` python main.py --model openai-completions --model_args engine=huggingface/manu/llama-oscar-fr  --tasks hellaswag`
(not sure we should continue this discussion here though, does not relate to the issue)

Agreed - I send you a linkedin request @ManuelFay - you can also DM on discord about this: https://discord.com/invite/wuPM9dRgDw 

@haileyschoelkopf has this issue moved forward at all? I would also like to eval models hosted in TGI, curious what this looks like in 2025 either with or without litellm integration @ishaan-jaff?

Disregard, got it working by taking advantage of TGI's OpenAI chat completions endpoint. Here's an example command that works well:
```
lm_eval --model local-completions \
--tasks mmlu_pro \
--num_fewshot 0 \
--model_args model=mistralai/Mistral-Small-24B-Instruct-2501, \
base_url=http://<hostname>:<port>/v1/completions, \
num_concurrent=30, \
max_retries=3, \
tokenized_requests=False
```

Recommend closing this issue.

Agreed. Additionally, there's also HuggingFace's [lighteval](https://github.com/huggingface/lighteval) nowadays