Transformers
GGUF
English
rag
context obedient
TroyDoesAI
Mermaid
Flow
Diagram
Sequence
Map
Context
Accurate
Summarization
Story
Code
Coder
Architecture
Retrieval
Augmented
Generation
AI
LLM
Mistral
LLama
Large Language Model
Retrieval Augmented Generation
Troy Andrew Schultz
LookingForWork
OpenForHire
IdoCoolStuff
Knowledge Graph
Knowledge
Graph
Accelerator
Enthusiast
Chatbot
Personal Assistant
Copilot
lol
tags
Pruned
efficient
smaller
small
local
open
source
open source
quant
quantize
ablated
Ablation
uncensored
unaligned
bad
alignment
Inference Endpoints
File size: 4,229 Bytes
fa02a97 f44698e fa02a97 312f1b0 fa02a97 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 |
---
base_model: TroyDoesAI/Codestral-21B-Pruned
language:
- en
library_name: transformers
license: apache-2.0
quantized_by: mradermacher
tags:
- rag
- context obedient
- TroyDoesAI
- Mermaid
- Flow
- Diagram
- Sequence
- Map
- Context
- Accurate
- Summarization
- Story
- Code
- Coder
- Architecture
- Retrieval
- Augmented
- Generation
- AI
- LLM
- Mistral
- LLama
- Large Language Model
- Retrieval Augmented Generation
- Troy Andrew Schultz
- LookingForWork
- OpenForHire
- IdoCoolStuff
- Knowledge Graph
- Knowledge
- Graph
- Accelerator
- Enthusiast
- Chatbot
- Personal Assistant
- Copilot
- lol
- tags
- Pruned
- efficient
- smaller
- small
- local
- open
- source
- open source
- quant
- quantize
- ablated
- Ablation
- 'uncensored '
- unaligned
- 'bad '
- alignment
---
## About
<!-- ### quantize_version: 2 -->
<!-- ### output_tensor_quantised: 1 -->
<!-- ### convert_type: hf -->
<!-- ### vocab_type: -->
<!-- ### tags: -->
static quants of https://huggingface.co/TroyDoesAI/Codestral-21B-Pruned
<!-- provided-files -->
weighted/imatrix quants are available at https://huggingface.co/mradermacher/Codestral-21B-Pruned-i1-GGUF
## Usage
If you are unsure how to use GGUF files, refer to one of [TheBloke's
READMEs](https://huggingface.co/TheBloke/KafkaLM-70B-German-V0.1-GGUF) for
more details, including on how to concatenate multi-part files.
## Provided Quants
(sorted by size, not necessarily quality. IQ-quants are often preferable over similar sized non-IQ quants)
| Link | Type | Size/GB | Notes |
|:-----|:-----|--------:|:------|
| [GGUF](https://huggingface.co/mradermacher/Codestral-21B-Pruned-GGUF/resolve/main/Codestral-21B-Pruned.Q2_K.gguf) | Q2_K | 8.1 | |
| [GGUF](https://huggingface.co/mradermacher/Codestral-21B-Pruned-GGUF/resolve/main/Codestral-21B-Pruned.IQ3_XS.gguf) | IQ3_XS | 9.0 | |
| [GGUF](https://huggingface.co/mradermacher/Codestral-21B-Pruned-GGUF/resolve/main/Codestral-21B-Pruned.Q3_K_S.gguf) | Q3_K_S | 9.4 | |
| [GGUF](https://huggingface.co/mradermacher/Codestral-21B-Pruned-GGUF/resolve/main/Codestral-21B-Pruned.IQ3_S.gguf) | IQ3_S | 9.5 | beats Q3_K* |
| [GGUF](https://huggingface.co/mradermacher/Codestral-21B-Pruned-GGUF/resolve/main/Codestral-21B-Pruned.IQ3_M.gguf) | IQ3_M | 9.8 | |
| [GGUF](https://huggingface.co/mradermacher/Codestral-21B-Pruned-GGUF/resolve/main/Codestral-21B-Pruned.Q3_K_M.gguf) | Q3_K_M | 10.5 | lower quality |
| [GGUF](https://huggingface.co/mradermacher/Codestral-21B-Pruned-GGUF/resolve/main/Codestral-21B-Pruned.Q3_K_L.gguf) | Q3_K_L | 11.4 | |
| [GGUF](https://huggingface.co/mradermacher/Codestral-21B-Pruned-GGUF/resolve/main/Codestral-21B-Pruned.IQ4_XS.gguf) | IQ4_XS | 11.7 | |
| [GGUF](https://huggingface.co/mradermacher/Codestral-21B-Pruned-GGUF/resolve/main/Codestral-21B-Pruned.Q4_K_S.gguf) | Q4_K_S | 12.3 | fast, recommended |
| [GGUF](https://huggingface.co/mradermacher/Codestral-21B-Pruned-GGUF/resolve/main/Codestral-21B-Pruned.Q4_K_M.gguf) | Q4_K_M | 12.9 | fast, recommended |
| [GGUF](https://huggingface.co/mradermacher/Codestral-21B-Pruned-GGUF/resolve/main/Codestral-21B-Pruned.Q5_K_S.gguf) | Q5_K_S | 14.9 | |
| [GGUF](https://huggingface.co/mradermacher/Codestral-21B-Pruned-GGUF/resolve/main/Codestral-21B-Pruned.Q5_K_M.gguf) | Q5_K_M | 15.3 | |
| [GGUF](https://huggingface.co/mradermacher/Codestral-21B-Pruned-GGUF/resolve/main/Codestral-21B-Pruned.Q6_K.gguf) | Q6_K | 17.7 | very good quality |
| [GGUF](https://huggingface.co/mradermacher/Codestral-21B-Pruned-GGUF/resolve/main/Codestral-21B-Pruned.Q8_0.gguf) | Q8_0 | 22.9 | fast, best quality |
Here is a handy graph by ikawrakow comparing some lower-quality quant
types (lower is better):
And here are Artefact2's thoughts on the matter:
https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9
## FAQ / Model Request
See https://huggingface.co/mradermacher/model_requests for some answers to
questions you might have and/or if you want some other model quantized.
## Thanks
I thank my company, [nethype GmbH](https://www.nethype.de/), for letting
me use its servers and providing upgrades to my workstation to enable
this work in my free time.
<!-- end -->
|