Add instructions to quantize model (#1)
Browse files- Add instructions to quantize model (e40158bf0962d8ab7acd6ef5388b1f1f4e550ea8)
README.md
CHANGED
@@ -11,3 +11,11 @@ tags:
|
|
11 |
|
12 |
Quantized weights of [coedit](https://github.com/vipulraheja/coedit) for inference with [candle](https://github.com/huggingface/candle/tree/main/candle-examples/examples/quantized-t5).
|
13 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
11 |
|
12 |
Quantized weights of [coedit](https://github.com/vipulraheja/coedit) for inference with [candle](https://github.com/huggingface/candle/tree/main/candle-examples/examples/quantized-t5).
|
13 |
|
14 |
+
Conversion command, using candle:
|
15 |
+
|
16 |
+
```shell
|
17 |
+
cargo run --example tensor-tools --release -- quantize \
|
18 |
+
--quantization q6k \
|
19 |
+
/path/to/coedit-<version>/model.safetensors \
|
20 |
+
--out-file model<version>.gguf
|
21 |
+
```
|