Fixed 5am spelling mistakes
Browse files
README.md
CHANGED
@@ -4,8 +4,8 @@ base_model: [deepseek-ai/DeepSeek-Coder-V2-Instruct]
|
|
4 |
|
5 |
#### Custom quantizations of deepseek-coder-v2-instruct optimized for cpu inference.
|
6 |
|
7 |
-
###
|
8 |
-
### While it required custom code to make, it is
|
9 |
|
10 |
>[!TIP]
|
11 |
>The following 4bit version is the one I use myself, it gets 17tps on 64 arm cores.
|
@@ -45,8 +45,9 @@ aria2c -x 8 -o deepseek_coder_v2_cpu_iq4xm.gguf-00003-of-00004.gguf \
|
|
45 |
aria2c -x 8 -o deepseek_coder_v2_cpu_iq4xm.gguf-00004-of-00004.gguf \
|
46 |
https://huggingface.co/nisten/deepseek-coder-v2-inst-cpu-optimized-gguf/resolve/main/deepseek_coder_v2_cpu_iq4xm.gguf-00004-of-00004.gguf
|
47 |
```
|
|
|
48 |
>[!TIP]
|
49 |
-
> ### And for downloading the Q8_0 version converted in the most lossless way possible from hf bf16
|
50 |
>
|
51 |
```verilog
|
52 |
|
@@ -69,6 +70,6 @@ aria2c -x 8 -o deepseek_coder_v2_cpu_q8_0-00006-of-00006.gguf \
|
|
69 |
https://huggingface.co/nisten/deepseek-coder-v2-inst-cpu-optimized-gguf/resolve/main/deepseek_coder_v2_cpu_q8_0-00006-of-00006.gguf
|
70 |
```
|
71 |
|
72 |
-
Enjoy and
|
73 |
|
74 |
-Nisten
|
|
|
4 |
|
5 |
#### Custom quantizations of deepseek-coder-v2-instruct optimized for cpu inference.
|
6 |
|
7 |
+
### This iq4xm one uses GGML TYPE IQ_4_XS 4bit in combination with q8_0 bit so it runs fast with minimal loss and takes advantage of int8 optimizations on most newer server cpus.
|
8 |
+
### While it required custom code to make, it is compatible with standard llama.cpp from github or just search nisten in lmstudio.
|
9 |
|
10 |
>[!TIP]
|
11 |
>The following 4bit version is the one I use myself, it gets 17tps on 64 arm cores.
|
|
|
45 |
aria2c -x 8 -o deepseek_coder_v2_cpu_iq4xm.gguf-00004-of-00004.gguf \
|
46 |
https://huggingface.co/nisten/deepseek-coder-v2-inst-cpu-optimized-gguf/resolve/main/deepseek_coder_v2_cpu_iq4xm.gguf-00004-of-00004.gguf
|
47 |
```
|
48 |
+
|
49 |
>[!TIP]
|
50 |
+
> ### And for downloading the Q8_0 version converted in the most lossless way possible from hf bf16 download these:
|
51 |
>
|
52 |
```verilog
|
53 |
|
|
|
70 |
https://huggingface.co/nisten/deepseek-coder-v2-inst-cpu-optimized-gguf/resolve/main/deepseek_coder_v2_cpu_q8_0-00006-of-00006.gguf
|
71 |
```
|
72 |
|
73 |
+
Enjoy and remember to accelerate!
|
74 |
|
75 |
-Nisten
|