quicksilver-inference 0.2.2
High-performance GGUF model inference with quantized kernels
License: Apache-2.0
Requires Python: >=3.9
Keywords: llm, inference, gguf, quantization, transformer, ai

# Quicksilver

**High-Performance GGUF Inference Engine**

[![PyPI](https://img.shields.io/pypi/v/quicksilver-inference)](https://pypi.org/project/quicksilver-inference/)
[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)

Quicksilver is a standalone, production-ready library for GGUF quantized model inference. It achieves **84 tok/s** on CPU (95% faster than llama.cpp) through optimized C++ kernels with AVX2 SIMD and OpenMP parallelization.

## Install

```bash
pip install quicksilver-inference
```

## Performance

| Model | Size | Quicksilver | llama.cpp | Speedup |
|-------|------|-------------|-----------|---------|
| SmolLM2-135M | 101MB | **84 tok/s** | 43 tok/s | **+95%** |
| Qwen2.5-1.5B | 1.0GB | **12 tok/s** | ~12 tok/s | comparable |

*Benchmarked with OMP_NUM_THREADS=8 on Intel CPU*

## Backends

| Backend | Platform | Status |
|---------|----------|--------|
| **CPU** | All (AVX2/NEON) | ✅ Production |
| **CUDA** | NVIDIA GPUs | ✅ Ready |
| **Metal** | Apple Silicon | ✅ Ready |
| **CANN** | Huawei Ascend NPU | ✅ Ready |

## Features

- **Native GGUF Parsing** - Zero external dependencies for model loading
- **9 Quantization Types** - Q4_0, Q5_0, Q8_0, Q2_K, Q3_K, Q4_K, Q5_K, Q6_K, F16
- **AVX2 SIMD** - Vectorized Q4_K/Q6_K kernels with FMA intrinsics
- **OpenMP Parallelization** - Multi-threaded GEMV operations
- **Streaming Generation** - Real-time token-by-token output
- **Batch Inference** - Process multiple prompts efficiently
- **OpenAI-Compatible API** - Drop-in replacement for OpenAI API
- **Auto Backend Selection** - Automatically uses best available (CUDA > Metal > CPU)

## Installation

### CPU (All Platforms)

```bash
cd quicksilver
pip install -e .

# Build C++ kernels (requires C++ compiler + OpenMP)
cd csrc && python setup_quantized.py build_ext --inplace

# Optimal performance
export OMP_NUM_THREADS=8
```

### CUDA (NVIDIA GPUs)

```bash
# Requires: CUDA toolkit, nvcc compiler
cd quicksilver/csrc
python setup_cuda.py build_ext --inplace
```

### Metal (macOS)

```bash
# Requires: Xcode (not just Command Line Tools)
python quicksilver/backends/metal/compile_shaders.py
```

### CANN (Huawei Ascend NPU)

```bash
# Requires: CANN toolkit from https://www.hiascend.com/cann
# Install torch_npu first
pip install torch-npu

# Build CANN kernels
cd quicksilver/backends/cann
python setup_cann.py build_ext --inplace
```

## Quick Start

### Basic Inference

```python
from quicksilver.quantized_engine import QuantizedInferenceEngine

engine = QuantizedInferenceEngine("model.gguf")
tokens = engine.generate(prompt_tokens=[1, 2, 3], max_tokens=50)
```

### Streaming Generation

```python
from quicksilver.streaming import StreamingGenerator

generator = StreamingGenerator(engine, tokenizer)

for token in generator.stream(prompt="Hello!", max_tokens=50):
    print(token.token_text, end="", flush=True)
```

### Batch Processing

```python
from quicksilver.batch import BatchProcessor, BatchRequest

processor = BatchProcessor(engine, tokenizer)
requests = [
    BatchRequest(id="1", prompt="What is AI?"),
    BatchRequest(id="2", prompt="Explain quantum computing"),
]
results, metrics = processor.process_batch(requests)
```

### OpenAI-Compatible API Server

```bash
# Start server
python -m quicksilver.server --model model.gguf --port 8000

# Use with OpenAI client
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="quicksilver",
    messages=[{"role": "user", "content": "Hello!"}],
    stream=True  # Streaming supported!
)
```

### CLI

```bash
quicksilver generate -m model.gguf -p "Once upon a time"
quicksilver benchmark -m model.gguf -n 100
```

## Architecture

```
quicksilver/
├── core/           # GGUF parsing and quantization
│   ├── parser.py   # Native GGUF file parser
│   ├── tensor.py   # Quantized tensor operations
│   └── quantization.py  # Quantization type definitions
├── csrc/           # C++ kernels
│   └── fused_quantized.cpp  # Optimized transformer kernels
├── quantized_engine.py  # Main inference engine
└── cli.py          # Command-line interface
```

## Supported Quantization Types

| Type | Bits | Block Size | Status |
|------|------|------------|--------|
| Q4_0 | 4.0 | 32 | ✅ Native GEMV |
| Q5_0 | 5.0 | 32 | ✅ Native GEMV |
| Q8_0 | 8.0 | 32 | ✅ Native GEMV |
| Q2_K | 2.5 | 256 | ✅ Native GEMV |
| Q3_K | 3.4 | 256 | ✅ Native GEMV |
| Q4_K | 4.5 | 256 | ✅ AVX2 SIMD |
| Q5_K | 5.5 | 256 | ✅ Native GEMV |
| Q6_K | 6.5 | 256 | ✅ AVX2 SIMD |
| F16 | 16 | 1 | ✅ Native GEMV |

## Roadmap

- [x] CPU inference with quantized kernels
- [x] Beat llama.cpp performance (84 vs 43 tok/s = +95%)
- [x] Support 9 quantization types (covers 95%+ of GGUF models)
- [x] Metal GPU backend shaders
- [x] CUDA GPU backend kernels
- [x] CANN backend for Huawei Ascend NPUs
- [x] OpenAI-compatible API server
- [x] Streaming generation
- [x] Batch inference
- [ ] Continuous batching
- [ ] Speculative decoding

## Development

```bash
# Install dev dependencies
pip install -e ".[dev]"

# Run tests
pytest tests/

# Format code
black quicksilver/
ruff check quicksilver/
```

## License

Apache 2.0