llcuda 2.0.6
Fast LLM inference on Tesla T4 GPUs with CUDA 12. Bundled binaries, PyTorch-style Tensor API, FlashAttention, Google Colab optimized.
Requires Python: >=3.11
Keywords: llm, cuda, inference, unsloth, t4, tesla-t4, tensor-cores, flashattention, nf4, quantization, gguf, pytorch, deep-learning, gpu, colab, tensor-api, google-colab

# llcuda v2.0.6: CUDA Inference Backend for Unsloth on Tesla T4

![Version](https://img.shields.io/badge/version-2.0.6-blue.svg)
![Python](https://img.shields.io/badge/python-3.11+-green.svg)
![CUDA](https://img.shields.io/badge/CUDA-12.x-orange.svg)
![GPU](https://img.shields.io/badge/GPU-Tesla%20T4%20ONLY-green.svg)
![License](https://img.shields.io/badge/license-MIT-blue.svg)

**Fast LLM inference on Google Colab Tesla T4 GPUs. CUDA 12 binaries bundled. One-step installation, instant import.**

---

## 📦 What is llcuda v2.0.6?

**llcuda v2.0.6** is a production-ready CUDA inference backend **exclusively designed for Tesla T4 GPUs** (Google Colab standard). It provides:

- ✅ **Bundled CUDA 12 Binaries** (~270 MB) - no runtime downloads
- ✅ **Native Tensor API** - PyTorch-style GPU operations with custom CUDA kernels
- ✅ **Tensor Core Optimization** - SM 7.5 targeting for T4 maximum performance
- ✅ **FlashAttention Support** - 2-3x faster attention for long contexts
- ✅ **GGUF Model Support** - Compatible with llama.cpp models
- ✅ **Unsloth Integration** - Direct loading of NF4-quantized fine-tuned models

---

## 🚀 Quick Start on Google Colab

### 1. Install

```bash
pip install llcuda
```

**Requirements:**
- Python 3.11+
- **Google Colab with Tesla T4 GPU** (SM 7.5)
- CUDA 12.x runtime (pre-installed in Colab)

### 2. Verify GPU

```python
import llcuda
from llcuda.core import get_device_properties

props = get_device_properties(0)
print(f"GPU: {props.name}")
print(f"Compute: SM {props.compute_capability_major}.{props.compute_capability_minor}")
```

### 3. Run Inference

```python
import llcuda

# HTTP Server API with GGUF models
engine = llcuda.InferenceEngine()
engine.load_model("gemma-3-1b-Q4_K_M", silent=True)
result = engine.infer("What is artificial intelligence?", max_tokens=100)
print(result.text)
print(f"Speed: {result.tokens_per_sec:.1f} tokens/sec")
```

### 4. Custom Tensor Operations

```python
from llcuda.core import Tensor, DType

# Create tensors on GPU
A = Tensor.zeros([2048, 2048], dtype=DType.Float16, device=0)
B = Tensor.zeros([2048, 2048], dtype=DType.Float16, device=0)

# Matrix multiplication (uses Tensor Cores on T4)
C = A @ B
print(f"Result shape: {C.shape}")
```

---

## 🎮 Performance Benchmarks (Tesla T4, CUDA 12)

| Model | Quantization | Speed (tok/s) | VRAM | Latency |
|-------|--------------|---------------|------|---------|
| Gemma 3-1B | Q4_K_M | 45 | 1.2 GB | 22 ms |
| Llama 3.2-3B | Q4_K_M | 30 | 2.0 GB | 33 ms |
| Qwen 2.5-7B | Q4_K_M | 18 | 5.0 GB | 56 ms |
| Llama 3.1-8B | Q4_K_M | 15 | 5.5 GB | 67 ms |

**Note:** FlashAttention provides 2-3x speedup for contexts > 2048 tokens.

---

## 📋 Version Info

- **Version:** 2.0.6
- **Release Date:** January 8, 2026
- **Target GPU:** Tesla T4 ONLY (SM 7.5)
- **CUDA Version:** 12.x
- **Python:** 3.11+
- **Platform:** Google Colab (primary), compatible Linux with T4

---

## ⚠️ Supported GPU ONLY

**llcuda v2.0.6 works exclusively on Tesla T4 GPU.**

✅ **Supported:**
- Google Colab Tesla T4
- On-premise Tesla T4 with CUDA 12.x

❌ **Not Supported:**
- A100, H100, L4, RTX GPUs
- Older Tesla GPUs (K80, P100)
- CPU-only systems

For other GPUs, use **llcuda v1.2.2** (less optimized).

---

## 📦 What's Included in v2.0.6

### Bundled CUDA 12 Binaries (~270 MB)

All binaries are included in the PyPI package - no runtime downloads:

- **llama-server** - Inference server for GGUF models
- **llama-cli** - Command-line interface
- **libllama.so** - Llama core library with CUDA support
- **libggml-cuda.so** - GGML CUDA kernels with FlashAttention
- **libggml-base.so** - GGML base library
- **libggml-cpu.so** - GGML CPU fallback
- **libmtmd.so** - Multithreading library

### Features

- ✅ FlashAttention (GGML_CUDA_FA=ON)
- ✅ CUDA Graphs (GGML_CUDA_GRAPHS=ON)
- ✅ All quantization types (INT4, INT8, FP16)
- ✅ SM 7.5 code generation (Tesla T4 optimized)
- ✅ Tensor Cores support

---

---

## 🛠️ V2.0 Native API Reference

### Device Management

## 🛠️ API Reference

### Device Management

```python
from llcuda.core import get_device_count, get_device_properties

# Get GPU count
num_gpus = get_device_count()

# Get device info
props = get_device_properties(0)
print(f"Device: {props.name}")
print(f"Compute: SM {props.compute_capability_major}.{props.compute_capability_minor}")
```

### Tensor Operations

```python
from llcuda.core import Tensor, DType

# Create tensors on GPU
A = Tensor.zeros([2048, 2048], dtype=DType.Float16, device=0)
B = Tensor.zeros([2048, 2048], dtype=DType.Float16, device=0)

# Matrix multiplication with Tensor Cores
C = A @ B
```

---

## 📚 Documentation

- **GitHub:** https://github.com/waqasm86/llcuda
- **PyPI:** https://pypi.org/project/llcuda/
- **Issues:** https://github.com/waqasm86/llcuda/issues

---

## 🔗 Related Projects

- [llama.cpp](https://github.com/ggml-org/llama.cpp) - Core inference engine
- [Unsloth](https://github.com/unslothai/unsloth) - Efficient fine-tuning
- [FlashAttention](https://github.com/Dao-AILab/flash-attention) - Optimized attention kernels

---

## 📄 License

MIT License - see [LICENSE](LICENSE) file

---

## 📝 Citation

If you use llcuda in your research, please cite:

```bibtex
@software{llcuda2024,
  author = {Waqas Muhammad},
  title = {llcuda: CUDA Inference Backend for Unsloth},
  year = {2024},
  url = {https://github.com/waqasm86/llcuda}
}
```

---

**llcuda v2.0.6* | Tesla T4 Optimized | CUDA 12 | Google Colab Ready
- [x] cuBLAS matmul

### Phase 2: GGUF Integration 🚧 (In Progress)
- [x] Bootstrap refactor for T4-only
- [x] GGUF parser implementation
- [ ] Model loader for GGUF → Tensor

### Phase 3: Flash Attention 📅 (Planned)
- [ ] Custom FA2 CUDA kernels
- [ ] Long context optimization

### Phase 4: Unsloth Integration 📅 (Planned)
- [ ] NF4 quantization kernels
- [ ] Direct Unsloth model loading
- [ ] `model.save_pretrained_llcuda()` export

---

## 🔧 Troubleshooting

### GPU Not Compatible

```
❌ INCOMPATIBLE GPU DETECTED

Your GPU is not Tesla T4
Required: Tesla T4 (SM 7.5)

llcuda v2.0 requires Tesla T4 GPU.
Compatible environment: Google Colab
```

**Solution**: Use Google Colab with Tesla T4

### Binary Download Failed

```bash
# Download T4 binaries manually
wget https://github.com/waqasm86/llcuda/releases/download/v2.0.6/llcuda-binaries-cuda12-t4.tar.gz
mkdir -p ~/.cache/llcuda
tar -xzf llcuda-binaries-cuda12-t4.tar.gz -C ~/.cache/llcuda/
```

---

## 📚 Documentation

- **Colab Build Notebook**: [notebooks/build_llcuda_v2_t4_colab.ipynb](notebooks/build_llcuda_v2_t4_colab.ipynb)
- **GitHub Issues**: https://github.com/waqasm86/llcuda/issues

---

## 📄 License

MIT License - see [LICENSE](LICENSE) file for details.

---

## 🙏 Acknowledgments

- Built on [llama.cpp](https://github.com/ggerganov/llama.cpp)
- FlashAttention from [Dao et al.](https://github.com/Dao-AILab/flash-attention)
- Designed for [Unsloth](https://github.com/unslothai/unsloth) integration

---

## 🔗 Links

- **PyPI**: https://pypi.org/project/llcuda/
- **GitHub**: https://github.com/waqasm86/llcuda
- **Unsloth**: https://github.com/unslothai/unsloth

---

**Version**: 2.0.6
**Target GPU**: **Tesla T4 ONLY** (SM 7.5)
**Platform**: Google Colab
**License**: MIT