legraphista commited on
Commit
874460d
1 Parent(s): 9035716

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +8 -5
README.md CHANGED
@@ -14,6 +14,7 @@ tags:
14
  - imat
15
  - imatrix
16
  - static
 
17
  - 6bit
18
  - 5bit
19
  - 4bit
@@ -54,6 +55,7 @@ Link: [here](https://huggingface.co/legraphista/DeepSeek-Coder-V2-Instruct-IMat-
54
  ### Common Quants
55
  | Filename | Quant type | File Size | Status | Uses IMatrix | Is Split |
56
  | -------- | ---------- | --------- | ------ | ------------ | -------- |
 
57
  | [DeepSeek-Coder-V2-Instruct.Q6_K/*](https://huggingface.co/legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF/tree/main/DeepSeek-Coder-V2-Instruct.Q6_K) | Q6_K | 193.54GB | ✅ Available | ⚪ Static | ✂ Yes
58
  | [DeepSeek-Coder-V2-Instruct.Q4_K/*](https://huggingface.co/legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF/tree/main/DeepSeek-Coder-V2-Instruct.Q4_K) | Q4_K | 142.45GB | ✅ Available | 🟢 IMatrix | ✂ Yes
59
  | [DeepSeek-Coder-V2-Instruct.Q3_K/*](https://huggingface.co/legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF/tree/main/DeepSeek-Coder-V2-Instruct.Q3_K) | Q3_K | 112.67GB | ✅ Available | 🟢 IMatrix | ✂ Yes
@@ -63,6 +65,7 @@ Link: [here](https://huggingface.co/legraphista/DeepSeek-Coder-V2-Instruct-IMat-
63
  ### All Quants
64
  | Filename | Quant type | File Size | Status | Uses IMatrix | Is Split |
65
  | -------- | ---------- | --------- | ------ | ------------ | -------- |
 
66
  | [DeepSeek-Coder-V2-Instruct.Q6_K/*](https://huggingface.co/legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF/tree/main/DeepSeek-Coder-V2-Instruct.Q6_K) | Q6_K | 193.54GB | ✅ Available | ⚪ Static | ✂ Yes
67
  | [DeepSeek-Coder-V2-Instruct.Q5_K/*](https://huggingface.co/legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF/tree/main/DeepSeek-Coder-V2-Instruct.Q5_K) | Q5_K | 167.22GB | ✅ Available | ⚪ Static | ✂ Yes
68
  | [DeepSeek-Coder-V2-Instruct.Q5_K_S/*](https://huggingface.co/legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF/tree/main/DeepSeek-Coder-V2-Instruct.Q5_K_S) | Q5_K_S | 162.31GB | ✅ Available | ⚪ Static | ✂ Yes
@@ -94,11 +97,11 @@ pip install -U "huggingface_hub[cli]"
94
  ```
95
  Download the specific file you want:
96
  ```
97
- huggingface-cli download legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF --include "DeepSeek-Coder-V2-Instruct.Q6_K.gguf" --local-dir ./
98
  ```
99
  If the model file is big, it has been split into multiple files. In order to download them all to a local folder, run:
100
  ```
101
- huggingface-cli download legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF --include "DeepSeek-Coder-V2-Instruct.Q6_K/*" --local-dir ./
102
  # see FAQ for merging GGUF's
103
  ```
104
 
@@ -128,7 +131,7 @@ Assistant: {assistant_response}<|end▁of▁sentence|>User: {next_user_promp
128
 
129
  ### Llama.cpp
130
  ```
131
- llama.cpp/main -m DeepSeek-Coder-V2-Instruct.Q6_K.gguf --color -i -p "prompt here (according to the chat template)"
132
  ```
133
 
134
  ---
@@ -143,8 +146,8 @@ According to [this investigation](https://www.reddit.com/r/LocalLLaMA/comments/1
143
  - To get hold of `gguf-split`, navigate to https://github.com/ggerganov/llama.cpp/releases
144
  - Download the appropriate zip for your system from the latest release
145
  - Unzip the archive and you should be able to find `gguf-split`
146
- 2. Locate your GGUF chunks folder (ex: `DeepSeek-Coder-V2-Instruct.Q6_K`)
147
- 3. Run `gguf-split --merge DeepSeek-Coder-V2-Instruct.Q6_K/DeepSeek-Coder-V2-Instruct.Q6_K-00001-of-XXXXX.gguf DeepSeek-Coder-V2-Instruct.Q6_K.gguf`
148
  - Make sure to point `gguf-split` to the first chunk of the split.
149
 
150
  ---
 
14
  - imat
15
  - imatrix
16
  - static
17
+ - 8bit
18
  - 6bit
19
  - 5bit
20
  - 4bit
 
55
  ### Common Quants
56
  | Filename | Quant type | File Size | Status | Uses IMatrix | Is Split |
57
  | -------- | ---------- | --------- | ------ | ------------ | -------- |
58
+ | [DeepSeek-Coder-V2-Instruct.Q8_0/*](https://huggingface.co/legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF/tree/main/DeepSeek-Coder-V2-Instruct.Q8_0) | Q8_0 | 250.62GB | ✅ Available | ⚪ Static | ✂ Yes
59
  | [DeepSeek-Coder-V2-Instruct.Q6_K/*](https://huggingface.co/legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF/tree/main/DeepSeek-Coder-V2-Instruct.Q6_K) | Q6_K | 193.54GB | ✅ Available | ⚪ Static | ✂ Yes
60
  | [DeepSeek-Coder-V2-Instruct.Q4_K/*](https://huggingface.co/legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF/tree/main/DeepSeek-Coder-V2-Instruct.Q4_K) | Q4_K | 142.45GB | ✅ Available | 🟢 IMatrix | ✂ Yes
61
  | [DeepSeek-Coder-V2-Instruct.Q3_K/*](https://huggingface.co/legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF/tree/main/DeepSeek-Coder-V2-Instruct.Q3_K) | Q3_K | 112.67GB | ✅ Available | 🟢 IMatrix | ✂ Yes
 
65
  ### All Quants
66
  | Filename | Quant type | File Size | Status | Uses IMatrix | Is Split |
67
  | -------- | ---------- | --------- | ------ | ------------ | -------- |
68
+ | [DeepSeek-Coder-V2-Instruct.Q8_0/*](https://huggingface.co/legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF/tree/main/DeepSeek-Coder-V2-Instruct.Q8_0) | Q8_0 | 250.62GB | ✅ Available | ⚪ Static | ✂ Yes
69
  | [DeepSeek-Coder-V2-Instruct.Q6_K/*](https://huggingface.co/legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF/tree/main/DeepSeek-Coder-V2-Instruct.Q6_K) | Q6_K | 193.54GB | ✅ Available | ⚪ Static | ✂ Yes
70
  | [DeepSeek-Coder-V2-Instruct.Q5_K/*](https://huggingface.co/legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF/tree/main/DeepSeek-Coder-V2-Instruct.Q5_K) | Q5_K | 167.22GB | ✅ Available | ⚪ Static | ✂ Yes
71
  | [DeepSeek-Coder-V2-Instruct.Q5_K_S/*](https://huggingface.co/legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF/tree/main/DeepSeek-Coder-V2-Instruct.Q5_K_S) | Q5_K_S | 162.31GB | ✅ Available | ⚪ Static | ✂ Yes
 
97
  ```
98
  Download the specific file you want:
99
  ```
100
+ huggingface-cli download legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF --include "DeepSeek-Coder-V2-Instruct.Q8_0.gguf" --local-dir ./
101
  ```
102
  If the model file is big, it has been split into multiple files. In order to download them all to a local folder, run:
103
  ```
104
+ huggingface-cli download legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF --include "DeepSeek-Coder-V2-Instruct.Q8_0/*" --local-dir ./
105
  # see FAQ for merging GGUF's
106
  ```
107
 
 
131
 
132
  ### Llama.cpp
133
  ```
134
+ llama.cpp/main -m DeepSeek-Coder-V2-Instruct.Q8_0.gguf --color -i -p "prompt here (according to the chat template)"
135
  ```
136
 
137
  ---
 
146
  - To get hold of `gguf-split`, navigate to https://github.com/ggerganov/llama.cpp/releases
147
  - Download the appropriate zip for your system from the latest release
148
  - Unzip the archive and you should be able to find `gguf-split`
149
+ 2. Locate your GGUF chunks folder (ex: `DeepSeek-Coder-V2-Instruct.Q8_0`)
150
+ 3. Run `gguf-split --merge DeepSeek-Coder-V2-Instruct.Q8_0/DeepSeek-Coder-V2-Instruct.Q8_0-00001-of-XXXXX.gguf DeepSeek-Coder-V2-Instruct.Q8_0.gguf`
151
  - Make sure to point `gguf-split` to the first chunk of the split.
152
 
153
  ---