Commit
•
c216b7b
1
Parent(s):
8f7724c
Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
@@ -14,7 +14,6 @@ tags:
|
|
14 |
- imat
|
15 |
- imatrix
|
16 |
- static
|
17 |
-
- 8bit
|
18 |
- 6bit
|
19 |
- 5bit
|
20 |
- 4bit
|
@@ -55,7 +54,6 @@ Link: [here](https://huggingface.co/legraphista/DeepSeek-Coder-V2-Instruct-IMat-
|
|
55 |
### Common Quants
|
56 |
| Filename | Quant type | File Size | Status | Uses IMatrix | Is Split |
|
57 |
| -------- | ---------- | --------- | ------ | ------------ | -------- |
|
58 |
-
| DeepSeek-Coder-V2-Instruct.Q8_0 | Q8_0 | - | ⏳ Processing | ⚪ Static | -
|
59 |
| DeepSeek-Coder-V2-Instruct.Q6_K | Q6_K | - | ⏳ Processing | ⚪ Static | -
|
60 |
| DeepSeek-Coder-V2-Instruct.Q4_K | Q4_K | - | ⏳ Processing | 🟢 IMatrix | -
|
61 |
| DeepSeek-Coder-V2-Instruct.Q3_K | Q3_K | - | ⏳ Processing | 🟢 IMatrix | -
|
@@ -65,7 +63,6 @@ Link: [here](https://huggingface.co/legraphista/DeepSeek-Coder-V2-Instruct-IMat-
|
|
65 |
### All Quants
|
66 |
| Filename | Quant type | File Size | Status | Uses IMatrix | Is Split |
|
67 |
| -------- | ---------- | --------- | ------ | ------------ | -------- |
|
68 |
-
| DeepSeek-Coder-V2-Instruct.Q8_0 | Q8_0 | - | ⏳ Processing | ⚪ Static | -
|
69 |
| DeepSeek-Coder-V2-Instruct.Q6_K | Q6_K | - | ⏳ Processing | ⚪ Static | -
|
70 |
| DeepSeek-Coder-V2-Instruct.Q5_K | Q5_K | - | ⏳ Processing | ⚪ Static | -
|
71 |
| DeepSeek-Coder-V2-Instruct.Q5_K_S | Q5_K_S | - | ⏳ Processing | ⚪ Static | -
|
@@ -97,11 +94,11 @@ pip install -U "huggingface_hub[cli]"
|
|
97 |
```
|
98 |
Download the specific file you want:
|
99 |
```
|
100 |
-
huggingface-cli download legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF --include "DeepSeek-Coder-V2-Instruct.
|
101 |
```
|
102 |
If the model file is big, it has been split into multiple files. In order to download them all to a local folder, run:
|
103 |
```
|
104 |
-
huggingface-cli download legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF --include "DeepSeek-Coder-V2-Instruct.
|
105 |
# see FAQ for merging GGUF's
|
106 |
```
|
107 |
|
@@ -131,7 +128,7 @@ Assistant: {assistant_response}<|end▁of▁sentence|>User: {next_user_promp
|
|
131 |
|
132 |
### Llama.cpp
|
133 |
```
|
134 |
-
llama.cpp/main -m DeepSeek-Coder-V2-Instruct.
|
135 |
```
|
136 |
|
137 |
---
|
@@ -146,8 +143,8 @@ According to [this investigation](https://www.reddit.com/r/LocalLLaMA/comments/1
|
|
146 |
- To get hold of `gguf-split`, navigate to https://github.com/ggerganov/llama.cpp/releases
|
147 |
- Download the appropriate zip for your system from the latest release
|
148 |
- Unzip the archive and you should be able to find `gguf-split`
|
149 |
-
2. Locate your GGUF chunks folder (ex: `DeepSeek-Coder-V2-Instruct.
|
150 |
-
3. Run `gguf-split --merge DeepSeek-Coder-V2-Instruct.
|
151 |
- Make sure to point `gguf-split` to the first chunk of the split.
|
152 |
|
153 |
---
|
|
|
14 |
- imat
|
15 |
- imatrix
|
16 |
- static
|
|
|
17 |
- 6bit
|
18 |
- 5bit
|
19 |
- 4bit
|
|
|
54 |
### Common Quants
|
55 |
| Filename | Quant type | File Size | Status | Uses IMatrix | Is Split |
|
56 |
| -------- | ---------- | --------- | ------ | ------------ | -------- |
|
|
|
57 |
| DeepSeek-Coder-V2-Instruct.Q6_K | Q6_K | - | ⏳ Processing | ⚪ Static | -
|
58 |
| DeepSeek-Coder-V2-Instruct.Q4_K | Q4_K | - | ⏳ Processing | 🟢 IMatrix | -
|
59 |
| DeepSeek-Coder-V2-Instruct.Q3_K | Q3_K | - | ⏳ Processing | 🟢 IMatrix | -
|
|
|
63 |
### All Quants
|
64 |
| Filename | Quant type | File Size | Status | Uses IMatrix | Is Split |
|
65 |
| -------- | ---------- | --------- | ------ | ------------ | -------- |
|
|
|
66 |
| DeepSeek-Coder-V2-Instruct.Q6_K | Q6_K | - | ⏳ Processing | ⚪ Static | -
|
67 |
| DeepSeek-Coder-V2-Instruct.Q5_K | Q5_K | - | ⏳ Processing | ⚪ Static | -
|
68 |
| DeepSeek-Coder-V2-Instruct.Q5_K_S | Q5_K_S | - | ⏳ Processing | ⚪ Static | -
|
|
|
94 |
```
|
95 |
Download the specific file you want:
|
96 |
```
|
97 |
+
huggingface-cli download legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF --include "DeepSeek-Coder-V2-Instruct.Q6_K.gguf" --local-dir ./
|
98 |
```
|
99 |
If the model file is big, it has been split into multiple files. In order to download them all to a local folder, run:
|
100 |
```
|
101 |
+
huggingface-cli download legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF --include "DeepSeek-Coder-V2-Instruct.Q6_K/*" --local-dir ./
|
102 |
# see FAQ for merging GGUF's
|
103 |
```
|
104 |
|
|
|
128 |
|
129 |
### Llama.cpp
|
130 |
```
|
131 |
+
llama.cpp/main -m DeepSeek-Coder-V2-Instruct.Q6_K.gguf --color -i -p "prompt here (according to the chat template)"
|
132 |
```
|
133 |
|
134 |
---
|
|
|
143 |
- To get hold of `gguf-split`, navigate to https://github.com/ggerganov/llama.cpp/releases
|
144 |
- Download the appropriate zip for your system from the latest release
|
145 |
- Unzip the archive and you should be able to find `gguf-split`
|
146 |
+
2. Locate your GGUF chunks folder (ex: `DeepSeek-Coder-V2-Instruct.Q6_K`)
|
147 |
+
3. Run `gguf-split --merge DeepSeek-Coder-V2-Instruct.Q6_K/DeepSeek-Coder-V2-Instruct.Q6_K-00001-of-XXXXX.gguf DeepSeek-Coder-V2-Instruct.Q6_K.gguf`
|
148 |
- Make sure to point `gguf-split` to the first chunk of the split.
|
149 |
|
150 |
---
|