Repository: EleutherAI/lm-evaluation-harness
Issue: RuntimeError: 500 Server Error for URL During LM-Eval with gguf Model
Root cause: External Dependency Breakage

 When running lm-eval with the gguf model on a locally hosted endpoint (http://localhost:8080), the process repeatedly fails with a 500 Server Error: Internal Server Error from /v1/completions. After several retries, lm-eval raises a RuntimeError indicating it could not get a valid response.

Below is a snippet of the command used and the relevant logs:

```
lm-eval \
  --model gguf \
  --model_args "model=/home/test/nanogpt-chat-q8_0.gguf,base_url=http://localhost:8080,mode=chat,apply_chat_template=True" \
  --tasks hellaswag
```

Logs & Error Traceback:

```
2025-04-09:09:48:45 INFO     [__main__:440] Selected Tasks: ['hellaswag']
2025-04-09:09:48:45 INFO     [evaluator:185] Setting random seed to 0 ...
2025-04-09:09:48:45 INFO     [evaluator:223] Initializing gguf model, with arguments: ...
...
2025-04-09:09:54:04 INFO     [evaluator:559] Running loglikelihood requests
  0%|                                                                | 0/40168 [00:00<?, ?it/s]
2025-04-09:09:54:04 ERROR    [models.gguf:68] RequestException: 500 Server Error: Internal Server Error for url: http://localhost:8080/v1/completions
...
Traceback (most recent call last):
  File "/home/test/lm_evaluation_harness/env/bin/lm-eval", line 8, in <module>
    sys.exit(cli_evaluate())
  ...
  File "/home/test/lm_evaluation_harness/env/lib/python3.12/site-packages/lm_eval/models/gguf.py", line 71, in gguf_completion
    raise RuntimeError(
RuntimeError: Failed to get a valid response after 3 retries.
```

Steps to Reproduce:

    Install the lm-eval tool and required dependencies.

    Launch a local LLM endpoint at http://localhost:8080.

    Run the command:

   ```
 lm-eval \
      --model gguf \
      --model_args "model=/home/test/nanogpt-chat-q8_0.gguf,base_url=http://localhost:8080,mode=chat,apply_chat_template=True" \
      --tasks hellaswag
```

    Observe the error logs indicating 500 Server Error and eventual RuntimeError.

Expected Behavior:

    The evaluation should run to completion, returning the hellaswag results without internal server errors.

Actual Behavior:

    The process fails during loglikelihood requests with an internal server error from the local endpoint. After three retries, the script raises a RuntimeError.

Additional Context:

    The .gguf model is located at /home/test/nanogpt-chat-q8_0.gguf.

    The environment includes:

        Python 3.12

        lm_eval (latest from PyPI or GitHub)

        Other dependencies installed in a virtual environment

    The error happens consistently at the same stage (loglikelihood requests).

Please let me know if more details are needed or if there are specific debug flags I can enable to provide additional logs. Any guidance on what might cause the endpoint to return a 500 would be greatly appreciated!

First of all, you have to get rid of the apply_chat_template, as that is not supported from my understanding. Furthermore, this might help to get it working: https://github.com/EleutherAI/lm-evaluation-harness/pull/2856. Had a similiar issue there, where the echo parameter was not supported but the output from your llama.cpp server would help to debug.

It appears that the GGUF code (2 yrs. old) uses an invalid JSON data parameter "echo" which the llama.cpp code explicitly returns a 500 error in Fe. 2025 when using the OpenAPI "completions" endpoint.

lm-eval call
```
lm-eval --model gguf --model_args base_url=http://127.0.0.1:8080 --tasks hellaswag
ERROR    [models.gguf:68] RequestException: 500 Server Error: Internal Server Error for url: http://127.0.0.1:8080/v1/completions
```

llama-server log:
```
srv  log_server_r: request: POST /v1/completions 127.0.0.1 500
srv  log_server_r: request:  {"prompt": "Roof shingle removal: A man is sitting on a roof. He is using wrap to wrap a pair of skis.", "logprobs": 10, "temperature": 0.0, "max_tokens": 1, "echo": true}
srv  log_server_r: response: {"error":{"code":500,"message":"Only no echo is supported","type":"server_error"}}
```