File size: 1,398 Bytes
eec676d |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 |
# llama.cpp
llama.cpp is the best backend in two important scenarios:
1) You don't have a GPU.
2) You want to run a model that doesn't fit into your GPU.
## Setting up the models
#### Pre-converted
Download the GGUF or GGML models directly into your `text-generation-webui/models` folder. It will be a single file.
* For GGUF models, make sure its name contains `.gguf`.
* For GGML models, make sure its name contains `ggml` and ends in `.bin`.
`q4_K_M` quantization is recommended.
#### Convert Llama yourself
Follow the instructions in the llama.cpp README to generate a ggml: https://github.com/ggerganov/llama.cpp#prepare-data--run
## GPU acceleration
Enabled with the `--n-gpu-layers` parameter.
* If you have enough VRAM, use a high number like `--n-gpu-layers 1000` to offload all layers to the GPU.
* Otherwise, start with a low number like `--n-gpu-layers 10` and then gradually increase it until you run out of memory.
This feature works out of the box for NVIDIA GPUs on Linux (amd64) or Windows. For other GPUs, you need to uninstall `llama-cpp-python` with
```
pip uninstall -y llama-cpp-python
```
and then recompile it using the commands here: https://pypi.org/project/llama-cpp-python/
#### macOS
For macOS, these are the commands:
```
pip uninstall -y llama-cpp-python
CMAKE_ARGS="-DLLAMA_METAL=on" FORCE_CMAKE=1 pip install llama-cpp-python --no-cache-dir
```
|