legraphista
commited on
Commit
•
874460d
1
Parent(s):
9035716
Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
@@ -14,6 +14,7 @@ tags:
|
|
14 |
- imat
|
15 |
- imatrix
|
16 |
- static
|
|
|
17 |
- 6bit
|
18 |
- 5bit
|
19 |
- 4bit
|
@@ -54,6 +55,7 @@ Link: [here](https://huggingface.co/legraphista/DeepSeek-Coder-V2-Instruct-IMat-
|
|
54 |
### Common Quants
|
55 |
| Filename | Quant type | File Size | Status | Uses IMatrix | Is Split |
|
56 |
| -------- | ---------- | --------- | ------ | ------------ | -------- |
|
|
|
57 |
| [DeepSeek-Coder-V2-Instruct.Q6_K/*](https://huggingface.co/legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF/tree/main/DeepSeek-Coder-V2-Instruct.Q6_K) | Q6_K | 193.54GB | ✅ Available | ⚪ Static | ✂ Yes
|
58 |
| [DeepSeek-Coder-V2-Instruct.Q4_K/*](https://huggingface.co/legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF/tree/main/DeepSeek-Coder-V2-Instruct.Q4_K) | Q4_K | 142.45GB | ✅ Available | 🟢 IMatrix | ✂ Yes
|
59 |
| [DeepSeek-Coder-V2-Instruct.Q3_K/*](https://huggingface.co/legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF/tree/main/DeepSeek-Coder-V2-Instruct.Q3_K) | Q3_K | 112.67GB | ✅ Available | 🟢 IMatrix | ✂ Yes
|
@@ -63,6 +65,7 @@ Link: [here](https://huggingface.co/legraphista/DeepSeek-Coder-V2-Instruct-IMat-
|
|
63 |
### All Quants
|
64 |
| Filename | Quant type | File Size | Status | Uses IMatrix | Is Split |
|
65 |
| -------- | ---------- | --------- | ------ | ------------ | -------- |
|
|
|
66 |
| [DeepSeek-Coder-V2-Instruct.Q6_K/*](https://huggingface.co/legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF/tree/main/DeepSeek-Coder-V2-Instruct.Q6_K) | Q6_K | 193.54GB | ✅ Available | ⚪ Static | ✂ Yes
|
67 |
| [DeepSeek-Coder-V2-Instruct.Q5_K/*](https://huggingface.co/legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF/tree/main/DeepSeek-Coder-V2-Instruct.Q5_K) | Q5_K | 167.22GB | ✅ Available | ⚪ Static | ✂ Yes
|
68 |
| [DeepSeek-Coder-V2-Instruct.Q5_K_S/*](https://huggingface.co/legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF/tree/main/DeepSeek-Coder-V2-Instruct.Q5_K_S) | Q5_K_S | 162.31GB | ✅ Available | ⚪ Static | ✂ Yes
|
@@ -94,11 +97,11 @@ pip install -U "huggingface_hub[cli]"
|
|
94 |
```
|
95 |
Download the specific file you want:
|
96 |
```
|
97 |
-
huggingface-cli download legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF --include "DeepSeek-Coder-V2-Instruct.
|
98 |
```
|
99 |
If the model file is big, it has been split into multiple files. In order to download them all to a local folder, run:
|
100 |
```
|
101 |
-
huggingface-cli download legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF --include "DeepSeek-Coder-V2-Instruct.
|
102 |
# see FAQ for merging GGUF's
|
103 |
```
|
104 |
|
@@ -128,7 +131,7 @@ Assistant: {assistant_response}<|end▁of▁sentence|>User: {next_user_promp
|
|
128 |
|
129 |
### Llama.cpp
|
130 |
```
|
131 |
-
llama.cpp/main -m DeepSeek-Coder-V2-Instruct.
|
132 |
```
|
133 |
|
134 |
---
|
@@ -143,8 +146,8 @@ According to [this investigation](https://www.reddit.com/r/LocalLLaMA/comments/1
|
|
143 |
- To get hold of `gguf-split`, navigate to https://github.com/ggerganov/llama.cpp/releases
|
144 |
- Download the appropriate zip for your system from the latest release
|
145 |
- Unzip the archive and you should be able to find `gguf-split`
|
146 |
-
2. Locate your GGUF chunks folder (ex: `DeepSeek-Coder-V2-Instruct.
|
147 |
-
3. Run `gguf-split --merge DeepSeek-Coder-V2-Instruct.
|
148 |
- Make sure to point `gguf-split` to the first chunk of the split.
|
149 |
|
150 |
---
|
|
|
14 |
- imat
|
15 |
- imatrix
|
16 |
- static
|
17 |
+
- 8bit
|
18 |
- 6bit
|
19 |
- 5bit
|
20 |
- 4bit
|
|
|
55 |
### Common Quants
|
56 |
| Filename | Quant type | File Size | Status | Uses IMatrix | Is Split |
|
57 |
| -------- | ---------- | --------- | ------ | ------------ | -------- |
|
58 |
+
| [DeepSeek-Coder-V2-Instruct.Q8_0/*](https://huggingface.co/legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF/tree/main/DeepSeek-Coder-V2-Instruct.Q8_0) | Q8_0 | 250.62GB | ✅ Available | ⚪ Static | ✂ Yes
|
59 |
| [DeepSeek-Coder-V2-Instruct.Q6_K/*](https://huggingface.co/legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF/tree/main/DeepSeek-Coder-V2-Instruct.Q6_K) | Q6_K | 193.54GB | ✅ Available | ⚪ Static | ✂ Yes
|
60 |
| [DeepSeek-Coder-V2-Instruct.Q4_K/*](https://huggingface.co/legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF/tree/main/DeepSeek-Coder-V2-Instruct.Q4_K) | Q4_K | 142.45GB | ✅ Available | 🟢 IMatrix | ✂ Yes
|
61 |
| [DeepSeek-Coder-V2-Instruct.Q3_K/*](https://huggingface.co/legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF/tree/main/DeepSeek-Coder-V2-Instruct.Q3_K) | Q3_K | 112.67GB | ✅ Available | 🟢 IMatrix | ✂ Yes
|
|
|
65 |
### All Quants
|
66 |
| Filename | Quant type | File Size | Status | Uses IMatrix | Is Split |
|
67 |
| -------- | ---------- | --------- | ------ | ------------ | -------- |
|
68 |
+
| [DeepSeek-Coder-V2-Instruct.Q8_0/*](https://huggingface.co/legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF/tree/main/DeepSeek-Coder-V2-Instruct.Q8_0) | Q8_0 | 250.62GB | ✅ Available | ⚪ Static | ✂ Yes
|
69 |
| [DeepSeek-Coder-V2-Instruct.Q6_K/*](https://huggingface.co/legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF/tree/main/DeepSeek-Coder-V2-Instruct.Q6_K) | Q6_K | 193.54GB | ✅ Available | ⚪ Static | ✂ Yes
|
70 |
| [DeepSeek-Coder-V2-Instruct.Q5_K/*](https://huggingface.co/legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF/tree/main/DeepSeek-Coder-V2-Instruct.Q5_K) | Q5_K | 167.22GB | ✅ Available | ⚪ Static | ✂ Yes
|
71 |
| [DeepSeek-Coder-V2-Instruct.Q5_K_S/*](https://huggingface.co/legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF/tree/main/DeepSeek-Coder-V2-Instruct.Q5_K_S) | Q5_K_S | 162.31GB | ✅ Available | ⚪ Static | ✂ Yes
|
|
|
97 |
```
|
98 |
Download the specific file you want:
|
99 |
```
|
100 |
+
huggingface-cli download legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF --include "DeepSeek-Coder-V2-Instruct.Q8_0.gguf" --local-dir ./
|
101 |
```
|
102 |
If the model file is big, it has been split into multiple files. In order to download them all to a local folder, run:
|
103 |
```
|
104 |
+
huggingface-cli download legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF --include "DeepSeek-Coder-V2-Instruct.Q8_0/*" --local-dir ./
|
105 |
# see FAQ for merging GGUF's
|
106 |
```
|
107 |
|
|
|
131 |
|
132 |
### Llama.cpp
|
133 |
```
|
134 |
+
llama.cpp/main -m DeepSeek-Coder-V2-Instruct.Q8_0.gguf --color -i -p "prompt here (according to the chat template)"
|
135 |
```
|
136 |
|
137 |
---
|
|
|
146 |
- To get hold of `gguf-split`, navigate to https://github.com/ggerganov/llama.cpp/releases
|
147 |
- Download the appropriate zip for your system from the latest release
|
148 |
- Unzip the archive and you should be able to find `gguf-split`
|
149 |
+
2. Locate your GGUF chunks folder (ex: `DeepSeek-Coder-V2-Instruct.Q8_0`)
|
150 |
+
3. Run `gguf-split --merge DeepSeek-Coder-V2-Instruct.Q8_0/DeepSeek-Coder-V2-Instruct.Q8_0-00001-of-XXXXX.gguf DeepSeek-Coder-V2-Instruct.Q8_0.gguf`
|
151 |
- Make sure to point `gguf-split` to the first chunk of the split.
|
152 |
|
153 |
---
|