Aya-101-GGUF
This repo contains GGUF format model files for Cohere's Aya-101 model
Quantized using Huggingface's candle framework
How to use with Candle's quantized T5 example
Visit the candle T5 example for more detailed instruction
- Clone candle repo:
git clone https://github.com/huggingface/candle.git
cd candle/candle-examples
- Run the following command:
cargo run --example quantized-t5 --release -- \
--model-id "kcoopermiller/aya-101-GGUF" \
--weight-file "aya-101.Q2_K.gguf" \
--config-file "config.json" \
--prompt "भारत में इतनी सारी भाषाएँ क्यों हैं?" \
--temperature 0
Available weight files:
- aya-101.Q2_K.gguf
- aya-101.Q3_K.gguf
- aya-101.Q4_0.gguf
- aya-101.Q4_1.gguf
- aya-101.Q4_K.gguf
- aya-101.Q5_0.gguf
- aya-101.Q5_1.gguf
- aya-101.Q5_K.gguf
- aya-101.Q6_K.gguf
- aya-101.Q8_0.gguf
- aya-101.Q8_1.gguf (not supported on candle yet)
- aya-101.Q8_K.gguf (not supported on candle yet)
- Downloads last month
- 357
Inference API (serverless) has been turned off for this model.