Repository: EleutherAI/lm-evaluation-harness
Issue: GGUF Local Model
Root cause: Documentation Deficiency

Is there examples of lm_eval for gruff models hosted locally?

lm_eval --model gguf \
    --model_args pretrained=Llama-2-7b-chat-hf-Q4_K_M.gguf,
    --tasks hellaswag \
    --device mps

Getting AssertionError: must pass `base_url` to use GGUF LM!

Could I host on Ollama?

I can't seem to get it to work for even a model on the HF hub. For example, running
```python
!lm_eval --model gguf \
--model_args base_url=https://huggingface.co/TheBloke/LLaMa-7B-GGML \
--tasks hellaswag --limit 10
```

Results in
```bash

2024-01-08:14:15:24,615 INFO     [task.py:355] Building contexts for task on rank 0...
2024-01-08:14:15:24,628 INFO     [evaluator.py:319] Running loglikelihood requests
  0% 0/40 [00:00<?, ?it/s]2024-01-08:14:15:24,817 ERROR    [gguf.py:65] RequestException: 404 Client Error: Not Found for url: https://huggingface.co/TheBloke/LLaMa-7B-GGML/v1/completions
2024-01-08:14:15:30,159 ERROR    [gguf.py:65] RequestException: 404 Client Error: Not Found for url: https://huggingface.co/TheBloke/LLaMa-7B-GGML/v1/completions
2024-01-08:14:15:35,487 ERROR    [gguf.py:65] RequestException: 404 Client Error: Not Found for url: https://huggingface.co/TheBloke/LLaMa-7B-GGML/v1/completions
  0% 0/40 [00:15<?, ?it/s]
Traceback (most recent call last):
  File "/usr/local/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/__main__.py", line 207, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/utils.py", line 402, in _wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/evaluator.py", line 152, in simple_evaluate
    results = evaluate(
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/utils.py", line 402, in _wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/evaluator.py", line 330, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/models/gguf.py", line 75, in loglikelihood
    response = self.gguf_completion(context=context, continuation=continuation)
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/models/gguf.py", line 68, in gguf_completion
    raise Exception(f"Failed to get a valid response after {retries} retries.")
Exception: Failed to get a valid response after 3 retries.
```

The steps to run a GGUF model in our library are:

1. install Llama-cpp-python: https://github.com/abetlen/llama-cpp-python?tab=readme-ov-file#openai-compatible-web-server
2. Launch the server, via e.g. `python3 -m llama_cpp.server --model models/7B/llama-model.gguf`
3. Run the library, passing `base_url=http://localhost:8000` or whatever is the address the local server can be reached at , e.g.

```
lm_eval --model gguf --model_args base_url=http://localhost:8000 --tasks lambada_openai
```

Will describe this more clearly in the readme.

@haileyschoelkopf ah, I see. And `python3 -m llama_cpp.server --model models/7B/llama-model.gguf` accepts both urls (e.g., to huggingface) and local paths? Or do you need to separately download the model locally?

@haileyschoelkopf thanks this fix works. One change on the Mac I had to add quotes on the pip >
pip install 'llama-cpp-python[server]'

server now hosts my quantized model locally
python3 -m llama_cpp.server --model models/Llama-2-7b-chat-hf-Q4_K_M.gguf

and lm-eval is progressing (switching to CUDA GPU is probably better)
lm_eval --model gguf --model_args base_url=http://localhost:8000 --tasks hellaswag --device mps

@kolbeuk note that, just like when you run with OAI's api, the `--device` flag is ignored. Whether Llama-CPP will use CUDA depends on how you install it, and 

@StellaAthena it currently requires you to download the model locally. (e.g. downloading the correct file from a HF repo: https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF/tree/main ). As far as I'm aware this is how people typically go about using Llama-cpp? Though now that the ecosystem has progressed a bit it may be that things like Ollama are now also popular.

@haileyschoelkopf 
Hello, Hailey. I succeeded to run the gguf file as you recommended. However, I can only use the cpu, not gpus. Can you me the example for gpu setup when we use the llama_cpp.server?


> @haileyschoelkopf Hello, Hailey. I succeeded to run the gguf file as you recommended. However, I can only use the cpu, not gpus. Can you me the example for gpu setup when we use the llama_cpp.server?

I just ran into the same problem, if you want to use CUDA acceleration with llama-cpp-python, you have to enable it when you install llama-cpp-python[server] through pip.

On linux the commands would be:
export CMAKE_ARGS="-DLLAMA_CUBLAS=on"
pip install llama-cpp-python[server] --upgrade --force-reinstall --no-cache-dir

Then you run the server with --n_gpu_layer set to what your vram can handle:
python3 -m llama_cpp.server --model some_model.gguf --n_gpu_layer 128

hope this helps.

on a separate issue, when everything is working properly I'm getting some kind of rate limit. It's generating over 150 tokens per second in my phi-2 test but it's taking over 2 seconds per iteration

```
llama_print_timings:        load time =     265.33 ms
llama_print_timings:      sample time =       0.58 ms /     1 runs   (    0.58 ms per token,  1730.10 tokens per second)
llama_print_timings: prompt eval time =      94.38 ms /    14 tokens (    6.74 ms per token,   148.34 tokens per second)
llama_print_timings:        eval time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_print_timings:       total time =     135.06 ms
INFO:     127.0.0.1:32858 - "POST /v1/completions HTTP/1.1" 200 OK
  1%|█▏                                                                                                                                                   | 327/40168 [12:00<23:36:43,  2.13s/it]Llama.generate: prefix-match hit

llama_print_timings:        load time =     265.33 ms
llama_print_timings:      sample time =       0.58 ms /     1 runs   (    0.58 ms per token,  1721.17 tokens per second)
llama_print_timings: prompt eval time =     139.20 ms /    22 tokens (    6.33 ms per token,   158.05 tokens per second)
llama_print_timings:        eval time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_print_timings:       total time =     201.73 ms
INFO:     127.0.0.1:32860 - "POST /v1/completions HTTP/1.1" 200 OK
  1%|█▏                                                                                                                                                   | 328/40168 [12:02<24:48:57,  2.24s/it]Llama.generate: prefix-match hit

llama_print_timings:        load time =     265.33 ms
llama_print_timings:      sample time =       0.62 ms /     1 runs   (    0.62 ms per token,  1618.12 tokens per second)
llama_print_timings: prompt eval time =     187.34 ms /    55 tokens (    3.41 ms per token,   293.59 tokens per second)
llama_print_timings:        eval time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_print_timings:       total time =     348.88 ms
INFO:     127.0.0.1:32864 - "POST /v1/completions HTTP/1.1" 200 OK
  1%|█▏                                                                                                                                                   | 329/40168 [12:06<28:04:42,  2.54s/it]Llama.generate: prefix-match hit
```

Thanks @adennen for answering this one, had typed out those instructions but must have failed to hit send.


Do you think this slowness is due to our implementation / could it be overcome by batching on our end if that’s something supported by Llama-cpp-python? Else, probably this issue is better suited for raising on their project.

Hi @haileyschoelkopf, @kolbeuk, can this also run on jupyter environment? Thank you!

hi, can i use llama.cpp without server? because i guess the communication time will affect my progress. how can I accelerate the progress?

I run this
```
lm_eval --model gguf --model_args base_url=http://localhost:8000 --tasks hellaswag --device mps
```
on deepseek r1 
```
python3 -m llama_cpp.server --model DeepSeek-R1-Distill-Qwen-1.5B-Q4_K_M.gguf
```
after 8 iteration get this error  in llama_cpp
```
ValueError: Out of range float values are not JSON compliant
```