Commit
•
d41827f
1
Parent(s):
3a050f6
Update README.md
Browse files
README.md
CHANGED
|
@@ -80,7 +80,7 @@ tags:
|
|
| 80 |
--prompt-template llama-3-chat \
|
| 81 |
--ctx-size 128000
|
| 82 |
```
|
| 83 |
-
|
| 84 |
## Quantized GGUF Models
|
| 85 |
|
| 86 |
| Name | Quant method | Bits | Size | Use case |
|
|
@@ -91,7 +91,7 @@ tags:
|
|
| 91 |
| [Meta-Llama-3.1-70B-Instruct-Q3_K_S.gguf](https://huggingface.co/second-state/Meta-Llama-3.1-70B-Instruct-GGUF/blob/main/Meta-Llama-3.1-70B-Instruct-Q3_K_S.gguf) | Q3_K_S | 3 | 30.9 GB| very small, high quality loss |
|
| 92 |
| [Meta-Llama-3.1-70B-Instruct-Q4_0.gguf](https://huggingface.co/second-state/Meta-Llama-3.1-70B-Instruct-GGUF/blob/main/Meta-Llama-3.1-70B-Instruct-Q4_0.gguf) | Q4_0 | 4 | 40 GB| legacy; small, very high quality loss - prefer using Q3_K_M |
|
| 93 |
| [Meta-Llama-3.1-70B-Instruct-Q4_K_M.gguf](https://huggingface.co/second-state/Meta-Llama-3.1-70B-Instruct-GGUF/blob/main/Meta-Llama-3.1-70B-Instruct-Q4_K_M.gguf) | Q4_K_M | 4 | 42.5 GB| medium, balanced quality - recommended |
|
| 94 |
-
| [Meta-Llama-3.1-70B-Instruct-Q5_0.gguf](https://huggingface.co/second-state/Meta-Llama-3.1-70B-Instruct-GGUF/blob/main/Meta-Llama-3.1-70B-Instruct-Q5_0.gguf) | Q5_0 | 5 | 48.7 GB| legacy; medium, balanced quality - prefer using Q4_K_M |
|
| 95 |
| [Meta-Llama-3.1-70B-Instruct-Q5_K_M.gguf](https://huggingface.co/second-state/Meta-Llama-3.1-70B-Instruct-GGUF/blob/main/Meta-Llama-3.1-70B-Instruct-Q5_K_M.gguf) | Q5_K_M | 5 | 50 GB| large, very low quality loss - recommended |
|
| 96 |
| [Meta-Llama-3.1-70B-Instruct-Q5_K_S.gguf](https://huggingface.co/second-state/Meta-Llama-3.1-70B-Instruct-GGUF/blob/main/Meta-Llama-3.1-70B-Instruct-Q5_K_S.gguf) | Q5_K_S | 5 | 48.7 GB| large, low quality loss - recommended |
|
| 97 |
| [Meta-Llama-3.1-70B-Instruct-Q6_K-00001-of-00002.gguf](https://huggingface.co/second-state/Meta-Llama-3.1-70B-Instruct-GGUF/blob/main/Meta-Llama-3.1-70B-Instruct-Q6_K-00001-of-00002.gguf) | Q6_K | 6 | 32.1 GB| very large, extremely low quality loss |
|
|
|
|
| 80 |
--prompt-template llama-3-chat \
|
| 81 |
--ctx-size 128000
|
| 82 |
```
|
| 83 |
+
|
| 84 |
## Quantized GGUF Models
|
| 85 |
|
| 86 |
| Name | Quant method | Bits | Size | Use case |
|
|
|
|
| 91 |
| [Meta-Llama-3.1-70B-Instruct-Q3_K_S.gguf](https://huggingface.co/second-state/Meta-Llama-3.1-70B-Instruct-GGUF/blob/main/Meta-Llama-3.1-70B-Instruct-Q3_K_S.gguf) | Q3_K_S | 3 | 30.9 GB| very small, high quality loss |
|
| 92 |
| [Meta-Llama-3.1-70B-Instruct-Q4_0.gguf](https://huggingface.co/second-state/Meta-Llama-3.1-70B-Instruct-GGUF/blob/main/Meta-Llama-3.1-70B-Instruct-Q4_0.gguf) | Q4_0 | 4 | 40 GB| legacy; small, very high quality loss - prefer using Q3_K_M |
|
| 93 |
| [Meta-Llama-3.1-70B-Instruct-Q4_K_M.gguf](https://huggingface.co/second-state/Meta-Llama-3.1-70B-Instruct-GGUF/blob/main/Meta-Llama-3.1-70B-Instruct-Q4_K_M.gguf) | Q4_K_M | 4 | 42.5 GB| medium, balanced quality - recommended |
|
| 94 |
+
<!-- | [Meta-Llama-3.1-70B-Instruct-Q5_0.gguf](https://huggingface.co/second-state/Meta-Llama-3.1-70B-Instruct-GGUF/blob/main/Meta-Llama-3.1-70B-Instruct-Q5_0.gguf) | Q5_0 | 5 | 48.7 GB| legacy; medium, balanced quality - prefer using Q4_K_M |
|
| 95 |
| [Meta-Llama-3.1-70B-Instruct-Q5_K_M.gguf](https://huggingface.co/second-state/Meta-Llama-3.1-70B-Instruct-GGUF/blob/main/Meta-Llama-3.1-70B-Instruct-Q5_K_M.gguf) | Q5_K_M | 5 | 50 GB| large, very low quality loss - recommended |
|
| 96 |
| [Meta-Llama-3.1-70B-Instruct-Q5_K_S.gguf](https://huggingface.co/second-state/Meta-Llama-3.1-70B-Instruct-GGUF/blob/main/Meta-Llama-3.1-70B-Instruct-Q5_K_S.gguf) | Q5_K_S | 5 | 48.7 GB| large, low quality loss - recommended |
|
| 97 |
| [Meta-Llama-3.1-70B-Instruct-Q6_K-00001-of-00002.gguf](https://huggingface.co/second-state/Meta-Llama-3.1-70B-Instruct-GGUF/blob/main/Meta-Llama-3.1-70B-Instruct-Q6_K-00001-of-00002.gguf) | Q6_K | 6 | 32.1 GB| very large, extremely low quality loss |
|