|
--- |
|
tags: |
|
- text-generation |
|
- transformer |
|
license: apache-2.0 |
|
library_name: transformers |
|
pipeline_tag: text-generation |
|
--- |
|
|
|
|
|
# Saanvi-C0-12B 🤖⚡ |
|
|
|
 |
|
 |
|
 |
|
|
|
**A next-generation 12B LLM optimized for speed, efficiency, and contextual accuracy.** |
|
_Powered by RAG-based enhancements • 4-bit quantization • Flash Attention 2 • bfloat16 • 128k context window_ |
|
|
|
--- |
|
|
|
## 🚀 Why Upgrade to Saanvi-C0-12B? |
|
|
|
Saanvi-C0-12B brings a **huge leap in capability** over smaller models, maintaining efficiency while significantly improving reasoning, fluency, and task completion and math! |
|
|
|
| Feature | Benefit | |
|
| --------------------- | --------------------------- | |
|
| ⚡ Flash Attention 2 | Up to **2.7× faster** inference | |
|
| 🧠 4-bit Quantization | **Runs on 8GB VRAM** GPUs | |
|
| 🎯 Instruction-Tuned | **Better task performance** | |
|
| 🔥 RAG-Enhanced | **More precise contextual retrieval** | |
|
| ➗ Math-Expert | **Precise Mathematics knowledge** | |
|
|
|
|
|
### 🖥️ Optimized for Mid-Tier GPUs |
|
- **Runs on mid-range GPUs with 8GB+ VRAM** (RTX 3050, RTX 2060, etc.). |
|
- **More robust than our 3B model** with better contextual retention and instruction-following. |
|
- **4-bit quantization** minimizes VRAM usage without sacrificing quality. |
|
|
|
--- |
|
|
|
## ⚡ Quick Start |
|
|
|
```python |
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
|
model_path = "riple-saanvi-lab/Saanvi-C0-12B" |
|
tokenizer = AutoTokenizer.from_pretrained(model_path) |
|
model = AutoModelForCausalLM.from_pretrained(model_path, torch_dtype="bfloat16", device_map="auto") |
|
|
|
while True: |
|
user_input = input("\n👤 You: ").strip() |
|
if user_input.lower() == "exit": |
|
break |
|
inputs = tokenizer(user_input, return_tensors="pt").to(model.device) |
|
output = model.generate(**inputs, max_length=2048, do_sample=True) |
|
print("🤖 AI:", tokenizer.decode(output[0], skip_special_tokens=True)) |
|
``` |
|
|
|
--- |
|
|
|
## 📦 Installation |
|
|
|
```bash |
|
pip install torch transformers |
|
``` |
|
|
|
--- |
|
|
|
## 📊 Benchmarks |
|
|
|
**A100-40GB Performance** |
|
|
|
| Batch Size | Throughput | Latency | VRAM Usage | |
|
| ---------- | ----------- | ------- | ---------- | |
|
| 1 | 42 tok/sec | 85ms | 8.2GB | |
|
| 8 | 218 tok/sec | 430ms | 12.5GB | |
|
|
|
**🚀 On Mid-Tier GPUs (RTX 3050, RTX 2060, RTX 3060 12GB)** |
|
- **VRAM Usage**: ~8.2GB (single batch) |
|
- **Speed**: ~10-15 tok/sec |
|
- **Best Practices**: Stick to **smaller batch sizes** for best performance. |
|
|
|
--- |
|
|
|
## 📜 License |
|
|
|
Licensed under the [Apache 2.0 License](LICENSE). See the [LICENSE](LICENSE) file for details. |
|
|
|
💡 **Pro Tip**: For **maximum efficiency**, use `torch.compile()` and CUDA graphs on high-end GPUs! |
|
|
|
--- |
