README.md

metadata

tags:
  - text-generation
  - transformer
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation

Saanvi-C0-12B 🤖⚡

A next-generation 12B LLM optimized for speed, efficiency, and contextual accuracy.
Powered by RAG-based enhancements • 4-bit quantization • Flash Attention 2 • bfloat16 • 128k context window

---

🚀 Why Upgrade to Saanvi-C0-12B?

Saanvi-C0-12B brings a huge leap in capability over smaller models, maintaining efficiency while significantly improving reasoning, fluency, and task completion and math!

Feature	Benefit
⚡ Flash Attention 2	Up to 2.7× faster inference
🧠 4-bit Quantization	Runs on 8GB VRAM GPUs
🎯 Instruction-Tuned	Better task performance
🔥 RAG-Enhanced	More precise contextual retrieval
➗ Math-Expert	Precise Mathematics knowledge

🖥️ Optimized for Mid-Tier GPUs

• Runs on mid-range GPUs with 8GB+ VRAM (RTX 3050, RTX 2060, etc.).
• More robust than our 3B model with better contextual retention and instruction-following.
• 4-bit quantization minimizes VRAM usage without sacrificing quality.

---

⚡ Quick Start

from transformers import AutoModelForCausalLM, AutoTokenizer

model_path = "riple-saanvi-lab/Saanvi-C0-12B"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(model_path, torch_dtype="bfloat16", device_map="auto")

while True:
    user_input = input("\n👤 You: ").strip()
    if user_input.lower() == "exit":
        break
    inputs = tokenizer(user_input, return_tensors="pt").to(model.device)
    output = model.generate(**inputs, max_length=2048, do_sample=True)
    print("🤖 AI:", tokenizer.decode(output[0], skip_special_tokens=True))

---

📦 Installation

pip install torch transformers

---

📊 Benchmarks

A100-40GB Performance

Batch Size	Throughput	Latency	VRAM Usage
1	42 tok/sec	85ms	8.2GB
8	218 tok/sec	430ms	12.5GB

🚀 On Mid-Tier GPUs (RTX 3050, RTX 2060, RTX 3060 12GB)

• VRAM Usage: ~8.2GB (single batch)
• Speed: ~10-15 tok/sec
• Best Practices: Stick to smaller batch sizes for best performance.

---

📜 License

Licensed under the Apache 2.0 License. See the LICENSE file for details.

💡 Pro Tip: For maximum efficiency, use torch.compile() and CUDA graphs on high-end GPUs!

---