Aya-101-GGUF

This repo contains GGUF format model files for Cohere's Aya-101 model

Quantized using Huggingface's candle framework

How to use with Candle's quantized T5 example

Visit the candle T5 example for more detailed instruction

  1. Clone candle repo:
git clone https://github.com/huggingface/candle.git
cd candle/candle-examples
  1. Run the following command:
cargo run --example quantized-t5 --release  -- \
  --model-id "kcoopermiller/aya-101-GGUF" \
  --weight-file "aya-101.Q2_K.gguf" \
  --config-file "config.json" \
  --prompt "भारत में इतनी सारी भाषाएँ क्यों हैं?" \
  --temperature 0

Available weight files:

  • aya-101.Q2_K.gguf
  • aya-101.Q3_K.gguf
  • aya-101.Q4_0.gguf
  • aya-101.Q4_1.gguf
  • aya-101.Q4_K.gguf
  • aya-101.Q5_0.gguf
  • aya-101.Q5_1.gguf
  • aya-101.Q5_K.gguf
  • aya-101.Q6_K.gguf
  • aya-101.Q8_0.gguf
  • aya-101.Q8_1.gguf (not supported on candle yet)
  • aya-101.Q8_K.gguf (not supported on candle yet)
Downloads last month
357
GGUF
Model size
12.9B params
Architecture
undefined

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

Inference Examples
Inference API (serverless) has been turned off for this model.

Datasets used to train kcoopermiller/aya-101-GGUF