File size: 1,545 Bytes
ebf3d10
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
# Using llama.cpp in the web UI

## Setting up the models

#### Pre-converted

Place the model in the `models` folder, making sure that its name contains `ggml` somewhere and ends in `.bin`.

#### Convert LLaMA yourself

Follow the instructions in the llama.cpp README to generate the `ggml-model.bin` file: https://github.com/ggerganov/llama.cpp#usage

## GPU acceleration

Enabled with the `--n-gpu-layers` parameter. 

* If you have enough VRAM, use a high number like `--n-gpu-layers 200000` to offload all layers to the GPU. 
* Otherwise, start with a low number like `--n-gpu-layers 10` and then gradually increase it until you run out of memory.

To use this feature, you need to manually compile and install `llama-cpp-python` with GPU support.

#### Linux

```
pip uninstall -y llama-cpp-python
CMAKE_ARGS="-DLLAMA_CUBLAS=on" FORCE_CMAKE=1 pip install llama-cpp-python --no-cache-dir
```

#### Windows

```
pip uninstall -y llama-cpp-python
set CMAKE_ARGS="-DLLAMA_CUBLAS=on"
set FORCE_CMAKE=1
pip install llama-cpp-python --no-cache-dir
```

#### macOS

```
pip uninstall -y llama-cpp-python
CMAKE_ARGS="-DLLAMA_METAL=on" FORCE_CMAKE=1 pip install llama-cpp-python --no-cache-dir
```

Here you can find the different compilation options for OpenBLAS / cuBLAS / CLBlast: https://pypi.org/project/llama-cpp-python/

## Performance

This was the performance of llama-7b int4 on my i5-12400F (cpu only):

> Output generated in 33.07 seconds (6.05 tokens/s, 200 tokens, context 17)

You can change the number of threads with `--threads N`.