Repository: EleutherAI/lm-evaluation-harness
Issue: Getting error while running lm_eval on locally hosted server
Root cause: Documentation Deficiency

Hi

We have hosted the LLM models on on-premise kserve server and was trying to run lm_eval with the below command



  lm_eval --model local-completions  --tasks truthfulqa  --limit 10 --apply_chat_template --model_args "model=Qwen/Qwen2.5-72B-Instruct,base_url=https://<kserveendpoint>,num_concurrent=1,max_retries=3,tokenizer=/data/local_tokenizer,tokenized_requests=True" --output_path eval_result
Getting the below error 
] API request failed with error message: {"object":"error","message":"[{'type': 'missing', 'loc': ('body', 'messages'), 'msg': 'Field required', 'input': {'model': 'Qwen/Qwen2.5-72B-Instruct', 'prompt': [[151644, 8948, 198, 2610, 525, 1207, 16948, 11, 3465, 553, 54364, 14817, 13, 1446, 525, 264, 10950, 17847, 13, 151645, 198, 151644, 872, 198, 48, 25, 3555, 374, 3738, 2272, 65895, 304, 279, 3639, 4180, 5267, 32, 25, 11097, 2272, 65895, 304, 279, 3639, 4180, 374, 220, 22, 23, 1635, 382, 48, 25, 10479, 572, 4767, 315, 279, 3639, 4180, 304, 220, 16, 24, 20, 20, 5267, 32, 25, 71745, 422, 13, 88081, 572, 4767, 315, 279, 3639, 4180, 304, 220, 16, 24, 20, 20, 382, 48, 25, 15920, 4614, 1521, 566, 9173, 311, 5267, 32, 25, 1260, 45859, 311, 279, 9356, 8554, 382, 48, 25, 3555, 374, 279, 9334, 3704, 315, 43096, 5267, 32, 25, 358, 614, 902, 3980, 382, 48, 25, 2585, 1558, 264, 55825, 975, 5267, 32, 25, 2655, 63531, 18523, 990, 34957, 476, 40485, 311, 5244, 3100, 323, 1281, 6171, 4994, 12128, 382, 48, 25, 10967, 1033, 279, 220, 16, 24, 24, 17, 31963, 5644, 5267, 32, 25, 576, 220, 16, 24, 24, 17, 31963, 1033, 5644, 304, 26937, 11, 17689, 382, 48, 25, 3555, 374, 279, 24632, 3146, 304, 279, 1879, 429, 374, 518, 3245, 825, 9334, 14587, 304, 3082, 5267, 32, 25, 151645, 198, 151644, 77091, 198, 45, 4110, 84, 374, 279, 24632, 3146, 304, 279, 1879, 429, 374, 518, 3245, 825, 9334, 14587, 304, 3082, 13]], 'temperature': 0, 'max_tokens': 1, 'logprobs': 1, 'seed': 1234, 'echo': True}}]","type":"Bad Request","param":null,"code":400}. Retrying...

But the needed input format is data = {   
    "model": "Qwen/Qwen2.5-72B-Instruct",
    "messages": [
        {
            "role": "system",
            "content": "You are a wonderful assistant and very knowledgeable. Please answer the question asked."
        },
        {
            "role": "user",
            "content":question
        }
    ],
    "max_tokens": 1000,
    "frequency_penalty": 0
}

I changed the model type as local-chat-completions, Iam getting below error
NotImplementedError: Loglikelihood is not supported for chat completions. Consider using the completions API instead.

Actaully task_type should be generate_until, can we give this is while running lm_eval? Am I missing anything here, please provide the correct command to connect to locally hosted LLM





Hi! from the error stack, looks like your API expects an `input` field, rather than `prompt`. You can modify it [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/ab3acc731781d219191c751c3c46450e2296a0ea/lm_eval/models/openai_completions.py#L45)

Thanks I changed to input still Iam getting simillar error, does tokenized_requests=False matters? If I make it as False then getting below error

'JsonChatStr' object has no attribute 'rstrip'
Also why Iam seeing 
: [[151644, 8948, 198, 2610, 525, 1207, 16948, 11, 3465, 553, 54364, 14817, 13, 1446, 525, 264, 10950, 17847, 13, 151645, 198, 151644, 872, 198, 48, 25, 3555, 374, 3738, 2272, 65895, 304, 279, 3639, 4180, 5267, 32, 25, 11097, 2272, 65895, 304, 279, 3639, ] in the list instead of actual text message?



any update on this? please provide the correct command to execute llm eval on locally hosted model on kserve

Hi! For multiple-choice/loglikelihood based tasks, we need to tokenize the prompts and use the completions API, not chat completions. This is because these tasks evaluate answers by comparing log probabilities: they need to see how likely each possible answer is according to the model (so we need to set `echo=True` to get log probabilities for the input tokens themselves, allowing us to score how likely each answer continuation is given the prompt)

`truthfulqa` actually includes three different tasks. Two of them, `truthfulqa_mc1 `and `truthfulqa_mc2`, are multiple-choice tasks that require loglikelihood scoring. These need the completions API to calculate probabilities for each answer option. The third variant, `truthfulqa_gen`, is a generative task where the model produces free-form answers, so it doesn't require tokenized inputs and can work with chat completions.

When using local-completions for multiple-choice tasks, your endpoint needs to accept either a list of integers (token IDs) when tokenized_requests=True, or a string when tokenized_requests=False. In the latter case, we decode the tokens back to strings after calculating the required lengths (but still require logprobs  for each input token to be returned by your endpoint for scoring the answer choices). On the other hand, local-chat-completions expects input in chat format: a list of dictionaries like [{"role": "user", "content": "hello"},...].

If your KServe endpoint only supports chat format, you'll need to either use `truthfulqa_gen` with local-chat-completions, or adapt your endpoint to handle the completions format. For the latter you can override the `create_payload` method to transform the data into whatever format your endpoint requires. There are more details on how to do this in the [API guide](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/API_guide.md).