Running Local LLMs, CPU vs. GPU - a Quick Speed Test
Tags: ai, llm, chatgpt, machinelearning
Published: 2024-03-11 13:50:50

**May 12 Update**

Putting together a table with all the results from the comments. Putting at the top own measurements where I had control over the environment and have more confidence in measurement consistency (e.g. using the right model, similar size messages, ensuring settings consistency etc.).

| Spec                                  | Result     |
| --------------------------------------| ---------- |
| Apple M1 Pro **CPU**                  | 14.8 tok/s |
| Apple M1 Pro **GPU**                  | 19.4 tok/s |
| AMD Ryzen 7 7840U **CPU**             | 7.3 tok/s  |
| AMD Radeon 780M **iGPU**              | 5.0 tok/s  |
| AMD Ryzen 5 7535HS **CPU**            | 7.4 tok/s  |
| GeForce RTX 4060 Mobile OC **GPU**    | 37.9 tok/s |
| GeForce RTX 4060 Mobile OC FA **GPU** | 39.7 tok/s |
| GeForce RTX 4090 OC (+180 Core, +1500 Mem) **GPU** | 108.5 tok/s |
| GeForce RTX 4090 OC FA (+180 Core, +1500 Mem) **GPU** | 119.1 tok/s |
|  --- Contributed by commenters ---    | ---        |
| M3 Pro 12-core CPU 18GB **CPU**       | 17.9 tok/s |
| M3 Pro 12-core CPU 18GB **GPU**       | 21.1 tok/s |
| iPad Pro M1 256GB, using LLM Farm     | 12.1 tok/s |
| AMD Ryzen 7 7800x3d **CPU**           | 9.7 tok/s  |
| Intel i7 14700k  **CPU**              | 9.8 tok/s  |
| ROG Ally Ryzen Z1 Extreme, 25W, **CPU**| 5.3 tok/s |
| ROG Ally Ryzen Z1 Extreme, 15W, **CPU**| 5.05 tok/s |
| GeForce RTX 4080 OC **GPU**           | 78.1 tok/s |
| Zotac Trinity non-OC 4080 Super **GPU** | 71.6 tok/s |
| RTX 4070 TI Super **GPU**             | 62 tok/s   |
| RTX 4070 Super **GPU**                | 58.2 tok/s |
| AMD 7900 XTX **GPU**                  | 70.1 tok/s |
| AMD RX 6800XT 16GB **GPU**            | 52.9 tok/s |
| Razer Blade 2021, RTX 3070 TI **GPU** | 41.8 tok/s |
| Razer Blade 2021, Ryzen 5900HX **CPU** | 7.0 tok/s |

A small observation, overclocking RTX 4060 and 4090 I noticed that LM Studio/llama.cpp doesn't benefit from core speeds yet gains from memory frequency.

**Updated on March 14, more configs tested**

Today, tools like [LM Studio](https://lmstudio.ai) make it easy to find, download, and run large language models on consumer-grade hardware. A typical quantized 7B model (a model with 7 billion parameters which are squeezed into 8 bits each or even smaller) would require 4-7GB of RAM/VRAM which is something an average laptop has.

LM Studio allows you to pick whether to run the model using CPU and RAM **or** using GPU and VRAM. It also shows the tok/s metric at the bottom of the chat dialog

![LM Studio, perf metric](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ke2ogtokvh57epm02axh.png)

I have used [this 5.94GB version](https://huggingface.co/MaziyarPanahi/mistral-7b-finetuned-orca-dpo-v2-Mistral-7B-Instruct-v0.2-slerp-GGUF/blob/main/mistral-7b-finetuned-orca-dpo-v2-Mistral-7B-Instruct-v0.2-slerp.Q6_K.gguf) of fine-tuned Mistral 7B and did a quick test of both options (CPU vs GPU) and here're the results. I have also added a few cases with [Flash Attention (FA)](https://github.com/Dao-AILab/flash-attention/tree/main) enabled (added in recent versions of LM Studio under "Model initialisation" category).

## Tokens/second

| Spec                                  | Result     |
| --------------------------------------| ---------- |
| Apple M1 Pro **CPU**                  | 14.8 tok/s |
| Apple M1 Pro **GPU**                  | 19.4 tok/s |
| AMD Ryzen 7 7840U **CPU**             | 7.3 tok/s  |
| AMD Radeon 780M **iGPU**              | 5.0 tok/s  |
| AMD Ryzen 5 7535HS **CPU**            | 7.4 tok/s  |
| GeForce RTX 4060 Mobile OC **GPU**    | 37.9 tok/s |
| AMD Ryzen 7 7800x3d **CPU**     | 9.7 tok/s  |
| GeForce RTX 4080 OC **GPU**     | 78.1 tok/s |


## Hardware Specs

1. 2021 M1 Mac Book Pro, 10-core CPU(8 performance and 2 efficiency), 16-core iGPU, 16GB of RAM

2. 2023 AOKZEO A1 Pro gaming handheld, AMD Ryzen 7 7840U CPU (8 cores, 16 threads), 32 GB LPDDR5X RAM, Radeon 780M iGPU (using system RAM as VRAM), TDP at 30W
  - 3D Mark TimeSpy **GPU** Score 3000
  - 3D Mark TimeSpy **CPU** Score 7300

3. 2023 MSI Bravo C7VF-039XRU laptop, AMD Ryzen 5 7535HS CPU (6 cores, 12 threads, 54W), 16GB DDR RAM, GeForce RTX 4060 (8GB VRAM, 105W)
  - GPU was slightly undervalued/overlocked, 3D Mark TimeSpy **GPU** Score 11300
  - 3D Mark TimeSpy **CPU** Score 7600

4. Desktop PC, AMD Ryzen 7 7800x3d (8 cores 16 threads, 78w during test), 6200 DDR5, GeForce RTX 4080 16GB VRAM (slightly overclocked, 228w during test)

## Screenshots

### Mac

![M1 CPU](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sav73b8s9lk3lgagmukw.png)

![M1 GPU](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/z2oca4lor7egcxdevoig.png)

### AOKZOE

![7840U](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ocvnbq6khjmrwkjvmguz.png)

![780M](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dvnuh5z39ztrvjurhbhx.png)

### MSI

![7535HS](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fkptxz7ym3jclozsgrgg.png)
![RTX 4060](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dujghsazba5gw834nu5z.png)

### Desktop PC

![7800x3d](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/oz0nhfboadchgbxiqz4g.png)

![RTX 4080](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ujeasla9yynjefnaur4j.png)

## P.S>

![Typing Test](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/c9pel0p1ygfetfzn4p8z.png)

It just hit me that while an average persons types 30~40 words per minute, RTX 4060 at 38 tokens/second (roughly 30 words per second) achieves 1800 WPM.

## P.P.S>

Thanks to [Sergey Zinchenko](https://github.com/sergey-zinchenko) added the 4th config (
7800x3d + GeForce RTX 4080)