Quantized candle weights for the CoEdIT model
Quantized weights of CoEdIT for inference with candle.
Usage
You can run the smaller models directly from the browser using this space.
Clone candle, and run the quantized-t5
example:
$ cargo run --example quantized-t5 --release -- \
--model-id "jbochi/candle-coedit-quantized" \
--prompt "Make this text coherent: Their flight is weak. They run quickly through the tree canopy." \
--temperature 0
...
Although their flight is weak, they run quickly through the tree canopy.
By default, it will use CoEdIT-large with q6k quantization (770M params, 643 MB).
To use CoEdIT-xl (3B params, 2.34 GB), or any other provided model, specify the weight-file and config-file:
$ cargo run --example quantized-t5 --release -- \
--model-id "jbochi/candle-coedit-quantized" \
--weight-file "model-xl.gguf" \
--config-file "config-xl.json" \
--prompt "Rewrite to make this easier to understand: Note that a storm surge is what forecasters consider a hurricane's most treacherous aspect." \
--temperature 0
...
Note that a storm surge is what forecasters consider a hurricane's most dangerous part.
Models available
These are all the available formats. Weight file is named {model}.gguf
and the config file is config-{base_model}.json
Model | Base model | Quantization | # Params | Size |
---|---|---|---|---|
- | small (unofficial) | None | 77M | 308 MB |
model-small | small | 6k | 77M | 78.2 MB |
model-small-q4k | small | 4k | 77M | 59.6 MB |
model-small-q4_0 | small | 4_0 | 77M | 43.4 MB |
- | base (unofficial) | None | 248M | 990 MB |
model-base | base | 6k | 248M | 194M |
model-base-q4k | base | 4k | 248M | 133M |
model-base-q4_0 | base | 4_0 | 248M | 133M |
- | large | None | 770M | 3.13 GB |
model | large | 6k | 770M | 643 MB |
model-q4k | large | 4k | 770M | 441 MB |
model-q4_0 | large | 4_0 | 770M | 441 MB |
- | xl | None | 3B | 11.4 GB |
model-xl | xl | 6k | 3B | 2.34 GB |
model-xl-q4k | xl | 4k | 3B | 1.6 GB |
model-xl-q4_0 | xl | 4_0 | 3B | 1.6 GB |
- | xxl | None | 11B | 44.5 GB |
model-xxl | xxl | 6k | 11B | 9.14 GB |
model-xxl-q4k | xxl | 4k | 11B | 6.27 GB |
model-xxl-q4_0 | xxl | 4_0 | 11B | 6.27 GB |
Model generation
The weights were quantized using candle:
cargo run --example tensor-tools --release -- quantize \
--quantization q6k \
/path/to/coedit-<version>/model.safetensors \
--out-file model<version>.gguf
- Downloads last month
- 803
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.