sollol 0.9.66
Intelligent Load Balancer for Ollama Clusters - Adaptive Parallelism + VRAM-Aware GPU Routing + Ray/Dask/llama.cpp Distribution
License: MIT
Requires Python: >=3.8
Keywords: ai, llm, distributed, ollama, load-balancing, inference, llama-cpp

# SOLLOL - Orchestration Framework for Local LLM Clusters

<div align="center">

<sup>Maintained by [BenevolentJoker-JohnL](https://github.com/BenevolentJoker-JohnL)</sup>

[![PyPI version](https://img.shields.io/pypi/v/sollol.svg)](https://pypi.org/project/sollol/)
[![Development Status](https://img.shields.io/badge/status-alpha-orange.svg)](https://github.com/BenevolentJoker-JohnL/SOLLOL)
[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Tests](https://github.com/BenevolentJoker-JohnL/SOLLOL/actions/workflows/tests.yml/badge.svg)](https://github.com/BenevolentJoker-JohnL/SOLLOL/actions/workflows/tests.yml)
[![codecov](https://codecov.io/gh/BenevolentJoker-JohnL/SOLLOL/branch/main/graph/badge.svg)](https://codecov.io/gh/BenevolentJoker-JohnL/SOLLOL)
[![Ollama](https://img.shields.io/badge/Ollama-Compatible-green.svg)](https://ollama.ai/)
[![llama.cpp](https://img.shields.io/badge/llama.cpp-Integrated-orange.svg)](https://github.com/ggerganov/llama.cpp)

**Open-source orchestration framework that combines intelligent task routing with distributed model inference for local LLM clusters — designed for production scalability.**

[Quick Start](#-quick-start) • [**📚 Distributed Guide**](docs/DISTRIBUTED_OLLAMA_GUIDE.md) • [Features](#-why-sollol) • [Architecture](#-architecture) • [Documentation](#-documentation) • [Examples](#-integration-examples)

</div>

---

## ⚠️ Development Status

**SOLLOL is currently in alpha development.**

### ✅ Production-Stable Features:
- **Task distribution** across Ollama nodes (mature, tested)
- **Intelligent routing** with adaptive learning (stable)
- **Observability dashboard** (fully functional)
- **Auto-discovery** and failover (reliable)

### 🔬 Experimental Features:
- **Distributed inference** (llama.cpp RPC integration)
  - ⚠️ **Not production-ready** - experimental proof-of-concept only
  - Validated for basic functionality with 13B models (2-3 nodes)
  - Known issues: 5x slower than local, version-sensitive, manual setup
  - **Funding required for production optimization**

**See [EXPERIMENTAL_FEATURES.md](EXPERIMENTAL_FEATURES.md) for complete details, realistic expectations, and why we don't recommend this for production.**

**Our Recommendation:** Use SOLLOL for task distribution and intelligent routing (proven, stable, fast).

---

## 🎯 What is SOLLOL?

SOLLOL (Super Ollama Load balancer & Orchestration Layer) transforms your collection of Ollama nodes into an **intelligent AI cluster** with adaptive routing and automatic failover—all running on your own hardware.

**Why Now?** As local LLM deployment becomes mainstream (Ollama, LM Studio, Kobold), SOLLOL bridges the gap between one-machine demos and real distributed compute — letting independent engineers orchestrate AI clusters without cloud dependency.

### The Problem

You have multiple machines with GPUs running Ollama, but:
- ❌ Manual node selection for each request
- ❌ No way to run models larger than your biggest GPU
- ❌ Can't distribute multi-agent workloads efficiently
- ❌ No automatic failover or load balancing
- ❌ Zero visibility into cluster performance

### The SOLLOL Solution

**Production-Ready Features:**
- ✅ **Intelligent routing** that learns which nodes work best for each task
- ✅ **Parallel agent execution** for multi-agent frameworks
- ✅ **Auto-discovery** of Ollama nodes across your network
- ✅ **Built-in observability** with real-time metrics and dashboard
- ✅ **Automatic failover** and health monitoring

**Experimental Features (Research/Testing Only - Not for Production):**
- 🔬 **Distributed inference** via llama.cpp RPC
  - ⚠️ **Proof-of-concept only** - not optimized for production
  - See [EXPERIMENTAL_FEATURES.md](EXPERIMENTAL_FEATURES.md) for honest assessment and limitations

---

## ⚡ Quickstart (3 Commands)

```bash
# 1. Install SOLLOL
pip install sollol

# 2. Start the dashboard (optional but recommended)
python3 -m sollol.dashboard_service &

# 3. Run your first query
python3 -c "from sollol import OllamaPool; pool = OllamaPool.auto_configure(); print(pool.chat(model='llama3.2', messages=[{'role': 'user', 'content': 'Hello!'}])['message']['content'])"
```

**What just happened?**
- ✅ SOLLOL auto-discovered all Ollama nodes on your network
- ✅ Intelligently routed your request to the best available node
- ✅ Dashboard live at `http://localhost:8080` (shows routing decisions, metrics, logs)

**Expected output:**
```
Discovering Ollama nodes...
Found 3 nodes: 192.168.1.22:11434, 192.168.1.10:11434, localhost:11434
Selected node: 192.168.1.22:11434 (GPU, 12ms latency)
Hello! How can I help you today?
```

**Next steps:**
- Visit `http://localhost:8080` to see the dashboard
- **📚 [Read the Distributed Ollama Guide](docs/DISTRIBUTED_OLLAMA_GUIDE.md)** - Learn how to build distributed AI applications with proven patterns

---

## 📚 Building Distributed Applications

**New to distributed Ollama?** Read our comprehensive guide:

### **[Distributed Ollama Guide](docs/DISTRIBUTED_OLLAMA_GUIDE.md)**

Learn to build production-grade distributed AI applications with:
- **4 proven architecture patterns** (batch processing, multi-agent, code synthesis, distributed training)
- **Real performance data** from production applications (5.5x speedup on FlockParser)
- **Honest assessment** of what's proven vs experimental
- **Complete code examples** from real projects
- **Performance tuning guide** for your workload
- **Production best practices** and troubleshooting

**Quick preview:**
```python
from sollol import OllamaPool

# Auto-discover and distribute work across cluster
pool = OllamaPool.auto_configure()

# Batch process 10,000 embeddings with adaptive parallelism
embeddings = pool.embed_batch(
    model="mxbai-embed-large",
    inputs=texts,
    use_adaptive=True  # SOLLOL optimizes based on node speeds
)
# Automatic work stealing, retry logic, and real-time dashboard
```

---

## 🏗️ Production-Ready Applications

SOLLOL powers two production-ready applications that leverage its intelligent routing and task distribution capabilities:

### [FlockParser](https://github.com/BenevolentJoker-JohnL/FlockParser)
**Distributed PDF Processing & RAG System**
- ✅ Production-stable document parsing and embedding
- ✅ Distributed vector search across Ollama cluster
- ✅ Automatic load balancing for embedding generation
- ✅ Real-time monitoring via SOLLOL dashboard
- **Use Case**: Enterprise document processing, RAG pipelines, knowledge base systems

### [SynapticLlamas](https://github.com/BenevolentJoker-JohnL/SynapticLlamas)
**Multi-Agent Collaboration Framework**
- ✅ Production-stable parallel agent execution
- ✅ Collaborative workflow orchestration (research → critique → synthesis)
- ✅ Quality control with automated validation
- ✅ Distributed across multiple Ollama nodes via SOLLOL
- **Use Case**: Complex reasoning tasks, research synthesis, multi-perspective analysis

> **Note**: Both applications use SOLLOL's **task distribution** features (production-ready). The experimental model sharding feature is available in SynapticLlamas for research purposes only.

---

## 🔧 In Development - SOLLOL Integration

The following projects are actively being developed to integrate with SOLLOL:

### Hydra
**Multi-Model Distribution Workflows**
- **Status**: 🚧 Requires further development
- **Purpose**: Enable complex multi-model distribution workflows across SOLLOL clusters
- **Goal**: Orchestrate multiple models working together on distributed tasks
- Will leverage SOLLOL's intelligent routing and task distribution

### LlamaForge
**Distributed Training Infrastructure**
- **Status**: 🚧 Requires further development
- **Purpose**: Distributed training capabilities for local LLM clusters
- **Goal**: Enable collaborative model training across SOLLOL nodes
- Part of expanding SOLLOL beyond inference to training workloads

> **These projects are under active development and will expand SOLLOL's capabilities beyond inference orchestration.**

---
- Check [Full Quick Start](#-full-quick-start) for production setup
- Read [Examples](#-integration-examples) for multi-agent, batch, and distributed inference patterns

---

## 🚀 Full Quick Start

### Installation

```bash
pip install sollol
```

### Basic Usage

```python
from sollol import OllamaPool

# Auto-discover nodes and start routing
pool = OllamaPool.auto_configure()

# Make requests - SOLLOL routes intelligently
response = pool.chat(
    model="llama3.2",
    messages=[{"role": "user", "content": "Hello!"}]
)
```

### Enable Real-Time GPU Monitoring

For accurate VRAM-aware routing, install the GPU reporter on each node:

```bash
# On each Ollama node, run:
sollol install-gpu-reporter --redis-host <redis-server-ip>

# Example:
sollol install-gpu-reporter --redis-host 192.168.1.10
```

**What this does:**
- Installs vendor-agnostic GPU monitoring (NVIDIA/AMD/Intel via `gpustat`)
- Publishes real-time VRAM stats to Redis every 5 seconds
- SOLLOL uses this data for intelligent routing decisions
- See [GPU Monitoring Guide](GPU_MONITORING_GUIDE.md) for details

**Without GPU monitoring:** SOLLOL falls back to estimates which may be inaccurate.

---

## 📸 Screenshots

### Dashboard Overview
![SOLLOL Unified Dashboard](docs/screenshots/dashboard-overview.png)
*Real-time monitoring with P50/P95/P99 latency metrics, network nodes, RPC backends, and active applications*

### Ray & Dask Integration
![Ray and Dask Dashboards](docs/screenshots/dashboard-metrics.png)
*Embedded Ray and Dask dashboards for distributed task monitoring*

### Activity Monitoring
![Real-time Activity Logs](docs/screenshots/dashboard-activity.png)
*Live request/response activity streams from Ollama nodes and RPC backends*

### Applications & Traces
![Performance Analytics](docs/screenshots/dashboard-ray-dask.png)
*Applications, distributed traces, and Ollama activity logs with real-time request/response tracking*

---

## 🔥 Why SOLLOL?

### 1. **Two Distribution Modes in One System**

SOLLOL combines both task distribution and distributed inference:

#### 📊 Task Distribution (Horizontal Scaling)
Distribute **multiple requests** across your cluster in parallel:
```python
# Run 10 agents simultaneously across 5 nodes
pool = OllamaPool.auto_configure()
responses = await asyncio.gather(*[
    pool.chat(model="llama3.2", messages=[...])
    for _ in range(10)
])
# Parallel execution across available nodes
```

#### 🔬 Distributed Inference (Experimental - Research Only)

**⚠️ Proof-of-concept only - not production-ready. See [EXPERIMENTAL_FEATURES.md](EXPERIMENTAL_FEATURES.md) for details.**

**Our Recommendation:** Use task distribution (proven, stable, fast) instead.

---

### 2. **Intelligent, Not Just Balanced**

SOLLOL doesn't just distribute requests randomly—it **learns** and **optimizes**:

| Feature | Simple Load Balancer | SOLLOL |
|---------|---------------------|---------|
| **Routing** | Round-robin | Context-aware scoring |
| **Learning** | None | Adapts from performance history |
| **Resource Awareness** | None | GPU/CPU/memory-aware |
| **Task Optimization** | None | Routes by task type complexity |
| **Failover** | Manual | Automatic with health checks |
| **Priority** | FIFO | Priority queue with fairness |

**Example**: SOLLOL automatically routes:
- Heavy generation tasks → GPU nodes with 24GB VRAM
- Fast embeddings → CPU nodes or smaller GPUs
- Critical requests → Fastest, most reliable nodes
- Batch processing → Lower priority, distributed load

---

### 3. **Production-Ready from Day One**

```python
from sollol import SOLLOL, SOLLOLConfig

# Literally 3 lines to production
config = SOLLOLConfig.auto_discover()
sollol = SOLLOL(config)
sollol.start()  # ✅ Gateway running on :8000
```

**Out of the box**:
- Auto-discovery of Ollama nodes
- Health monitoring and failover
- Prometheus metrics
- Web dashboard
- Connection pooling
- Request hedging
- Priority queuing

---

### 4. **Unified Observability for Your Entire AI Network**

SOLLOL provides a **single pane of glass** to monitor every application and every node in your distributed AI network.

- ✅ **Centralized Dashboard**: One web interface shows all applications, nodes, and RPC backends.
- ✅ **Multi-App Tracking**: See which applications (e.g., SynapticLlamas, custom agents) are using the cluster in real-time.
- ✅ **Network-Wide Visibility**: The dashboard runs as a persistent service, discovering and monitoring all components even if no applications are running.
- ✅ **Zero-Config**: Applications automatically appear in the dashboard with no extra code required.

This moves beyond per-application monitoring to provide true, centralized observability for your entire infrastructure.

---

### 5. **Technical Differentiation**

#### SOLLOL vs. Simple Load Balancers

| Feature | nginx/HAProxy | SOLLOL |
|---------|--------------|---------|
| Routing | Round-robin/random | Context-aware, adapts from history |
| Resource awareness | None | GPU/CPU/memory-aware |
| Failover | Manual config | Automatic detection & recovery |
| Distributed Inference | ❌ | ✅ llama.cpp RPC integration |
| Task prioritization | ❌ | ✅ Priority queue |
| Observability | Basic | Rich metrics + dashboard |
| Setup | Complex config | Auto-discover |

#### SOLLOL vs. Kubernetes

| Feature | Kubernetes | SOLLOL |
|---------|-----------|---------|
| **Complexity** | High - requires cluster setup | Low - pip install |
| **AI-specific** | Generic container orchestration | Purpose-built for LLMs |
| **Intelligence** | None | Task-aware routing |
| **Distributed Inference** | Manual | Automatic via llama.cpp RPC |
| **Best for** | Large-scale production | AI-focused teams |

**Use both!** Deploy SOLLOL on Kubernetes for ultimate scalability.

---

## 🏗️ Architecture

### High-Level Overview

```
┌────────────────────────────────────────────────────────┐
│                  Your Application                       │
│         (SynapticLlamas, custom agents, etc.)          │
└──────────────────────┬─────────────────────────────────┘
                       │
                       ▼
┌────────────────────────────────────────────────────────┐
│                 SOLLOL Gateway (:8000)                  │
│  ┌──────────────────────────────────────────────────┐  │
│  │         Intelligent Routing Engine               │  │
│  │  • Analyzes: task type, complexity, resources    │  │
│  │  • Scores: all nodes based on context            │  │
│  │  • Learns: from performance history              │  │
│  │  • Routes: to optimal node                       │  │
│  └──────────────────────────────────────────────────┘  │
│  ┌──────────────────────────────────────────────────┐  │
│  │          Priority Queue + Failover               │  │
│  └──────────────────────────────────────────────────┘  │
└────────┬─────────────────────────┬─────────────────────┘
         │                         │
         ▼                         ▼
  ┌─────────────┐          ┌──────────────┐
  │ Task Mode   │          │  Shard Mode  │
  │ Ray Cluster │          │  llama.cpp   │
  └──────┬──────┘          └──────┬───────┘
         │                         │
         ▼                         ▼
┌────────────────────────────────────────────────────────┐
│              Your Heterogeneous Cluster                 │
│  GPU (24GB) │ GPU (16GB) │ CPU (64c) │ GPU (8GB) │...  │
└────────────────────────────────────────────────────────┘
```

### How Routing Works

```python
# 1. Request arrives
POST /api/chat {
  "model": "llama3.2",
  "messages": [{"role": "user", "content": "Complex analysis task..."}],
  "priority": 8
}

# 2. SOLLOL analyzes
task_type = "generation"       # Auto-detected
complexity = "high"             # Token count analysis
requires_gpu = True             # Based on task
estimated_duration = 3.2s       # From history

# 3. SOLLOL scores all nodes
Node A (GPU 24GB, load: 0.2, latency: 120ms) → Score: 185.3 ✓ WINNER
Node B (GPU 8GB,  load: 0.6, latency: 200ms) → Score: 92.1
Node C (CPU only, load: 0.1, latency: 80ms)  → Score: 41.2

# 4. Routes to Node A, monitors execution, learns for next time
```

**Scoring Algorithm**:
```
Score = 100.0 (baseline)
      × success_rate (0.0-1.0)
      ÷ (1 + latency_penalty)
      × gpu_bonus (1.5x if GPU available & needed)
      ÷ (1 + load_penalty)
      × priority_alignment
      × task_specialization
```

---

## 📦 Installation

### Quick Install (PyPI)
```bash
pip install sollol
```

### From Source
```bash
git clone https://github.com/BenevolentJoker-JohnL/SOLLOL.git
cd SOLLOL
pip install -e .
```

---

## ⚡ Quick Start

### 1. Synchronous API (No async/await needed!)

**New in v0.3.6:** SOLLOL now provides a synchronous API for easier integration with non-async applications.

```python
from sollol.sync_wrapper import OllamaPool
from sollol.priority_helpers import Priority

# Auto-discover and connect to all Ollama nodes
pool = OllamaPool.auto_configure()

# Make requests - SOLLOL routes intelligently
# No async/await needed!
response = pool.chat(
    model="llama3.2",
    messages=[{"role": "user", "content": "Hello!"}],
    priority=Priority.HIGH,  # Semantic priority levels
    timeout=60  # Request timeout in seconds
)

print(response['message']['content'])
print(f"Routed to: {response.get('_sollol_routing', {}).get('host', 'unknown')}")
```

**Key features of synchronous API:**
- ✅ No async/await syntax required
- ✅ Works with synchronous agent frameworks
- ✅ Same intelligent routing and features
- ✅ Runs async code in background thread automatically

---

### 2. Async API (Original)

For async applications, use the original async API:

```python
from sollol import OllamaPool

# Auto-discover and connect to all Ollama nodes
pool = await OllamaPool.auto_configure()

# Make requests - SOLLOL routes intelligently
response = await pool.chat(
    model="llama3.2",
    messages=[{"role": "user", "content": "Hello!"}]
)

print(response['message']['content'])
print(f"Routed to: {response['_sollol_routing']['host']}")
print(f"Task type: {response['_sollol_routing']['task_type']}")
```

---

### 3. Priority-Based Multi-Agent Execution

**New in v0.3.6:** Use semantic priority levels and role-based mapping.

```python
from sollol.sync_wrapper import OllamaPool
from sollol.priority_helpers import Priority, get_priority_for_role

pool = OllamaPool.auto_configure()

# Define agents with different priorities
agents = [
    {"name": "Researcher", "role": "researcher"},  # Priority 8
    {"name": "Editor", "role": "editor"},          # Priority 6
    {"name": "Summarizer", "role": "summarizer"},  # Priority 5
]

for agent in agents:
    priority = get_priority_for_role(agent["role"])

    response = pool.chat(
        model="llama3.2",
        messages=[{"role": "user", "content": f"Task for {agent['name']}"}],
        priority=priority
    )
    # User-facing agents get priority, background tasks wait
```

**Priority levels available:**
- `Priority.CRITICAL` (10) - Mission-critical
- `Priority.URGENT` (9) - Fast response needed
- `Priority.HIGH` (7) - Important tasks
- `Priority.NORMAL` (5) - Default
- `Priority.LOW` (3) - Background tasks
- `Priority.BATCH` (1) - Can wait

---

### 4. Distributed Inference (Experimental - Research/Testing Only)

**⚠️ EXPERIMENTAL: Proof-of-concept only - not optimized for production use.**

This feature demonstrates distributing model layer computation across multiple RPC backends. It has significant limitations including:
- 5x slower than local inference
- Version-sensitive (exact binary matching required)
- Coordinator still requires full model in RAM
- Manual setup and troubleshooting

**See [EXPERIMENTAL_FEATURES.md](EXPERIMENTAL_FEATURES.md) for honest assessment, realistic expectations, and known issues.**

**Recommended for production:** Use SOLLOL's stable task distribution features instead.

---

**⚠️ The following sections document experimental distributed inference setup. Use at your own risk for research/testing only.**

<details>
<summary><b>Click to expand experimental distributed inference documentation</b></summary>

#### Quick Start: Auto-Setup (Use at Your Own Risk)

```python
from sollol.sync_wrapper import HybridRouter, OllamaPool

# SOLLOL handles all setup automatically
router = HybridRouter(
    ollama_pool=OllamaPool.auto_configure(),
    enable_distributed=True,  # Enable distributed inference via llama.cpp RPC
    auto_setup_rpc=True,      # Auto-configure RPC backends
    num_rpc_backends=3        # Distribute across 3 machines
)

# Use large model that doesn't fit on one machine
response = router.route_request(
    model="llama3.1:70b",  # Automatically sharded across backends
    messages=[{"role": "user", "content": "Explain quantum computing"}]
)

print(response['message']['content'])
```

**What happens automatically:**
1. SOLLOL discovers available RPC backends on your network
2. Extracts the GGUF model from Ollama storage
3. Starts llama-server coordinator with optimal settings
4. Distributes model layers across backends
5. Routes your request to the coordinator

#### RPC Server Auto-Installation

**SOLLOL can automatically clone, build, and start llama.cpp RPC servers for you!**

**One-line installation:**

```python
from sollol.rpc_auto_setup import auto_setup_rpc_backends

# Automatically: clone → build → start RPC servers
backends = auto_setup_rpc_backends(num_backends=2)
# Output: [{'host': '127.0.0.1', 'port': 50052}, {'host': '127.0.0.1', 'port': 50053}]
```

**What this does:**
1. ✅ Scans network for existing RPC servers
2. ✅ If none found: clones llama.cpp to `~/llama.cpp`
3. ✅ Builds llama.cpp with RPC support (`cmake -DGGML_RPC=ON`)
4. ✅ Starts RPC servers on ports 50052-50053
5. ✅ Returns ready-to-use backend list

**CLI installation:**

```bash
# Full automated setup (clone + build + install systemd service)
python3 -m sollol.setup_llama_cpp --all

# Or step by step
python3 -m sollol.setup_llama_cpp --clone  # Clone llama.cpp
python3 -m sollol.setup_llama_cpp --build  # Build with RPC support
python3 -m sollol.setup_llama_cpp --start  # Start RPC server
```

#### Building llama.cpp with GPU Support 🚀

**For GPU-accelerated distributed inference**, rebuild llama.cpp with CUDA support. The GPU-enabled binaries work on ALL nodes (CPU-only nodes will just use CPU).

**Quick install script (automated CUDA + build):**

```bash
# Download and run installation script
curl -fsSL https://github.com/BenevolentJoker-JohnL/SOLLOL/raw/main/scripts/install_cuda_llama.sh | bash

# Or manual installation (see below)
```

**Manual Installation Steps:**

```bash
# 1. Install CUDA toolkit (Ubuntu/Debian)
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
sudo apt-get install -y cuda-toolkit-12-6

# 2. Set CUDA environment
export PATH=/usr/local/cuda-12.6/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-12.6/lib64:$LD_LIBRARY_PATH

# 3. Clone and build llama.cpp with CUDA
cd ~
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp

# Build RPC server with CUDA support
cmake -B build \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES="75;80;86;89;90" \
  -DBUILD_SHARED_LIBS=OFF \
  -DLLAMA_CURL=OFF \
  -DLLAMA_BUILD_TOOLS=ON \
  -DGGML_RPC=ON

cmake --build build --config Release --target rpc-server -j $(nproc)

# 4. Install binary
mkdir -p ~/.local/bin
cp build/bin/rpc-server ~/.local/bin/

# 5. Deploy and start RPC servers

# For GPU nodes (192.168.1.20, etc.):
# Copy binary to GPU node
scp ~/.local/bin/rpc-server <gpu-node>:~/.local/bin/

# On GPU node, start with CUDA support
ssh <gpu-node>
nohup ~/.local/bin/rpc-server --host 0.0.0.0 --port 50052 > /tmp/rpc-server.log 2>&1 &

# For CPU-only nodes:
# Build without CUDA (faster, smaller binary)
cd ~/llama.cpp
cmake -B build-cpu \
  -DGGML_CUDA=OFF \
  -DLLAMA_BUILD_TOOLS=ON \
  -DGGML_RPC=ON

cmake --build build-cpu --config Release --target rpc-server -j $(nproc)
cp build-cpu/bin/rpc-server ~/.local/bin/rpc-server-cpu

# Start CPU-only RPC server
nohup ~/.local/bin/rpc-server-cpu --host 0.0.0.0 --port 50052 > /tmp/rpc-server.log 2>&1 &
```

**Why build separate binaries?**
- ✅ **CUDA binary (689MB)**: Runs on GPU nodes, uses CUDA automatically
- ✅ **CPU-only binary (~200MB)**: Smaller, runs on coordinator/CPU-only nodes
- ✅ **Deployment**: CUDA binary requires NVIDIA drivers on target machine
- ✅ **Hybrid parallelization**: GPU nodes can use CPU + GPU workers

**Note**: The CUDA-enabled binary **will not run** on machines without NVIDIA drivers (like CPU-only coordinator). Build CPU-only version for coordinator nodes.

**For other GPU vendors:**
- **AMD GPUs**: Use `-DGGML_HIPBLAS=ON` instead of `-DGGML_CUDA=ON`
- **Intel GPUs**: Use `-DGGML_VULKAN=ON` for Intel/AMD/NVIDIA compatibility

**Enabling GPU Detection and Reporting:**

For SOLLOL to intelligently route requests to GPU nodes, you need to configure Redis and GPU registration:

📖 **See [GPU Detection Setup Guide](docs/GPU_DETECTION_SETUP.md)** for complete instructions on:
- Configuring Redis for network access
- Registering GPU nodes with Redis
- Automatic GPU capability detection
- Troubleshooting GPU detection issues

**Quick summary:**
1. Configure Redis to listen on network interface (not just localhost)
2. Run `scripts/register_gpu_node.py` on each GPU node
3. SOLLOL automatically detects GPU capabilities and routes accordingly

**Docker IP Resolution:**

SOLLOL automatically resolves Docker container IPs to accessible host IPs:

```python
# If Docker container reports IP 172.17.0.5:11434
# SOLLOL automatically resolves to:
# → 127.0.0.1:11434 (published port mapping)
# → host IP (if accessible)
# → Docker host gateway

from sollol import is_docker_ip, resolve_docker_ip

# Check if IP is Docker internal
is_docker = is_docker_ip("172.17.0.5")  # True

# Resolve Docker IP to accessible IP
accessible_ip = resolve_docker_ip("172.17.0.5", port=11434)
# Returns: "127.0.0.1" or host IP
```

**Network Discovery with Docker Support:**

```python
from sollol import OllamaPool

# Auto-discover nodes (automatically resolves Docker IPs)
pool = OllamaPool.auto_configure()

# Manual control
from sollol.discovery import discover_ollama_nodes
nodes = discover_ollama_nodes(auto_resolve_docker=True)
```

**Multi-Node Production Setup:**

For distributed clusters, use systemd services on each node:

```bash
# On each RPC node
sudo systemctl enable llama-rpc@50052.service
sudo systemctl start llama-rpc@50052.service
```

See [SOLLOL_RPC_SETUP.md](https://github.com/BenevolentJoker-JohnL/FlockParser/blob/main/SOLLOL_RPC_SETUP.md) for complete installation guide.

#### Architecture: Hybrid GPU+CPU Parallelization 🚀
**llama.cpp RPC backend only**

**NEW:** llama.cpp RPC backends support hybrid device parallelization - GPU nodes contribute BOTH their GPU (VRAM) AND CPU (RAM) as separate parallel workers!

> **Note:** This feature is specific to llama.cpp's `rpc-server` architecture and is NOT available with Ollama. Ollama nodes can only contribute one worker per instance.

**Example: 3 Physical Nodes → 4 Parallel Workers (+33% throughput!)**

```
Traditional Setup (3 workers):
CPU Node 1  → 1 worker (8GB RAM)
CPU Node 2  → 1 worker (8GB RAM)
GPU Node    → 1 worker (12GB VRAM)
Total: 3 parallel workers

Hybrid Setup (4 workers):
CPU Node 1  → 1 worker (8GB RAM)
CPU Node 2  → 1 worker (8GB RAM)
GPU Node    → 2 workers:
              ├─ CPU device (10GB RAM)  ← Both run in parallel!
              └─ GPU device (9.6GB VRAM) ←
Total: 4 parallel workers  (+33% more!)
```

**Fully Automatic** - SOLLOL detects and configures hybrid parallelization when starting llama.cpp RPC servers:
- Auto-detects GPU(s) and calculates safe VRAM allocations (80% with 20% reserve)
- Auto-detects system RAM and reserves 20% for OS stability
- Generates optimal `--device` and `--mem` flags for llama.cpp's `rpc-server` automatically
- No manual configuration needed!

**Why llama.cpp only?** Ollama doesn't support the `--device cpu,cuda:0` multi-device flag that enables this hybrid parallelization.

**Manual inspection** (optional) - See what SOLLOL would configure:
```bash
python scripts/setup_rpc_node.py
```

**Sample output (GPU node):**
```
✅ GPU(s) Found: 1
   GPU 0: cuda:0 - 9600 MB VRAM (safe allocation)
💾 CPU RAM: 10240 MB (safe allocation)
⚡ Total Parallel Workers: 2 (1 CPU worker + 1 GPU worker)

RPC-SERVER COMMAND:
rpc-server --host 0.0.0.0 --port 50052 --device cpu,cuda:0 --mem 10240,9600
```

**Key Benefits:**
- ✅ Maximizes hardware utilization (use ALL resources)
- ✅ No extra hardware needed (same 3 machines)
- ✅ Safe memory allocations (80% with 20% reserve)
- ✅ Automatic GPU detection (NVIDIA, AMD, Intel)

See [docs/HYBRID_RPC_PARALLELIZATION.md](docs/HYBRID_RPC_PARALLELIZATION.md) for complete guide.

#### Layer Distribution Example

```
┌────────────────────────────────────────────┐
│    Llama 3.1 70B Model (40GB total)        │
│     Hybrid Distributed Parallelization     │
└────────────────────────────────────────────┘
                    │
       ┌────────────┼────────────┬────────────┐
       │            │            │            │
       ▼            ▼            ▼            ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│  CPU Node 1  │ │  CPU Node 2  │ │  GPU Node    │ │  GPU Node    │
│  Layers 0-9  │ │ Layers 10-19 │ │ CPU Device   │ │ GPU Device   │
│   (8GB RAM)  │ │   (8GB RAM)  │ │ Layers 20-29 │ │ Layers 30-39 │
│ RPC Backend  │ │ RPC Backend  │ │ (10GB RAM)   │ │ (9.6GB VRAM) │
│              │ │              │ │ RPC Backend  │ │ RPC Backend  │
└──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘
       ▲            ▲                    ▲                ▲
       └────────────┴────────────────────┴────────────────┘
                               │
                    ┌──────────┴──────────┐
                    │ llama-server        │
                    │ Coordinator         │
                    │ (Port 18080)        │
                    └─────────────────────┘

4 parallel workers across 3 physical machines!
```

#### Manual Setup (Advanced)

For explicit control over RPC backends:

```python
from sollol.llama_cpp_coordinator import LlamaCppCoordinator
from sollol.rpc_registry import RPCBackendRegistry

# 1. Register RPC backends explicitly
registry = RPCBackendRegistry()
registry.add_backend("rpc_1", "grpc://192.168.1.22:50052")
registry.add_backend("rpc_2", "grpc://10.9.66.46:50052")
registry.add_backend("rpc_3", "grpc://10.9.66.47:50052")

# 2. Create coordinator
coordinator = LlamaCppCoordinator(
    coordinator_port=18080,
    rpc_backends=registry.get_all_backends(),
    context_size=4096,
    gpu_layers=-1  # Use all available GPU layers
)

# 3. Start and use
await coordinator.start(model_name="llama3.1:70b")
response = await coordinator.generate(
    prompt="Explain the theory of relativity",
    max_tokens=500
)
```

#### Performance Expectations

| Model Size | Single GPU | Sharded (3 nodes) | Trade-off |
|------------|-----------|-------------------|-----------|
| **13B** | ✅ 20 tok/s | ✅ 5 tok/s | -75% speed, works on 3×smaller GPUs |
| **70B** | ❌ OOM | ⚠️ 3-5 tok/s (est.) | Enables model that won't run otherwise |

**Trade-offs:**
- 🐌 **Startup**: 2-5 minutes (model distribution + loading)
- 🐌 **Inference**: ~4x slower than local (network overhead)
- ✅ **Capability**: Run models that won't fit on single GPU

**Learn More:**
- 📖 [Complete llama.cpp Guide](docs/llama_cpp_guide.md) - Setup, optimization, troubleshooting
- 💻 [Working Examples](examples/llama_cpp_distributed.py) - 5 complete examples including conversation, batch processing, error handling

#### ⚠️ Known Limitation: llama.cpp Coordinator Bottleneck

**The Problem:** llama.cpp's `--rpc` flag enables distributed *computation* but NOT distributed *storage*. The coordinator node must load the entire model into its own RAM before distributing layer computation to RPC workers.

**Impact:**
- ❌ 70B model (40GB GGUF) on coordinator with 8GB RAM → **Crashes**
- ❌ Even with 48GB total across 4 workers, coordinator still needs 40GB locally
- ✅ Works if coordinator node has sufficient RAM (e.g., 32GB+ for 70B models)

**Workaround:** Run the llama-server coordinator on your machine with the most RAM (e.g., your GPU node with 32GB). The RPC workers can still be CPU-only nodes with less RAM.

**Research Track (WIP):** SOLLOL includes experimental code (`src/sollol/distributed_pipeline.py`) exploring true distributed model weight sharding where NO single node needs the full model. This uses Ray-based pipeline parallelism inspired by prima.cpp's architecture. See module documentation for technical details and path forward.

#### 🔬 Future Work: Production-Ready Distributed Inference & Model Weight Sharding

**Current Status:** SOLLOL's distributed inference is **experimental only**:
- Basic functionality validated (13B models, 2-3 nodes)
- Not optimized for performance (5x slower than local)
- Requires manual setup and exact version matching
- **Not recommended for production use**

**What's Needed for Production:**
1. **Optimization of current RPC integration:**
   - Performance tuning (reduce 2-5min startup to <30s)
   - Automated version management
   - Better error handling and recovery
   - Comprehensive testing across model sizes

2. **True model weight sharding** (future):
   - NO single node needs full model in RAM
   - Requires significant R&D (see `distributed_pipeline.py`)

**What's Needed:**
- ✅ GGUF layer analysis (completed)
- ✅ Ray-based architecture design (completed)
- ⚠️ Quantization-aware GGUF writing (complex, blocked)
- ⚠️ Tensor streaming via Ray object store (unimplemented)
- ⚠️ Multi-node validation with 70B+ models (requires cluster access)

**Proposed Extension:**
- **Ollama ⇄ GGUF conversion** for tensor-level distribution across nodes
- **Weight-sharded model loading** with Ray actors (no single-node memory bottleneck)
- **Pipeline parallelism** with activation passing through Ray object store
- **Production validation** on real multi-node clusters with frontier models (70B-405B)

**Important Distinction:**
- **Current:** Distributed **inference** - layers spread across nodes, coordinator still needs full model
- **Future:** Distributed **model weights** - model parameters split across nodes, no single node needs full model

> 💡 **This feature is gated behind funding or partnership opportunities.**
>
> It addresses a genuine limitation in local LLM infrastructure: running frontier models on consumer hardware clusters without requiring any single machine to have enterprise-grade RAM (40GB+).
>
> **Why this matters:** Enables sovereign AI deployment at scale without cloud dependencies or specialized hardware.

**Status:** Research track with working foundation (570+ lines), documented blockers, and clear implementation path. See `src/sollol/distributed_pipeline.py` for technical details.

**Interested in funding or collaboration?** Open an issue or contact via GitHub.

</details>

---

### 5. Batch Processing API

**New in v0.7.0:** RESTful API for asynchronous batch job management.

Submit large-scale batch operations (thousands of embeddings, bulk inference) and track progress via job IDs:

```python
import requests

# Submit batch embedding job (up to 10,000 documents)
response = requests.post("http://localhost:11434/api/batch/embed", json={
    "model": "nomic-embed-text",
    "documents": ["Document 1", "Document 2", ...],  # Can be thousands
    "metadata": {"source": "knowledge_base"}  # Optional metadata
})

job_id = response.json()["job_id"]
print(f"Job submitted: {job_id}")

# Poll for job status
import time
while True:
    status = requests.get(f"http://localhost:11434/api/batch/jobs/{job_id}").json()

    progress = status["progress"]["percent"]
    print(f"Progress: {progress}%")

    if status["status"] == "completed":
        break
    time.sleep(1)

# Get results
results = requests.get(f"http://localhost:11434/api/batch/results/{job_id}").json()
embeddings = results["results"]  # List of embedding vectors
print(f"Processed {len(embeddings)} documents in {status['duration_seconds']}s")
```

**Available Batch Endpoints:**
- `POST /api/batch/embed` - Submit batch embedding job
- `GET /api/batch/jobs/{job_id}` - Get job status
- `GET /api/batch/results/{job_id}` - Get job results
- `GET /api/batch/jobs?limit=100` - List recent jobs
- `DELETE /api/batch/jobs/{job_id}` - Cancel job

**Use cases:**
- Embedding large document collections (thousands of documents)
- Bulk inference for batch predictions
- Background processing without blocking
- Long-running operations with progress tracking

---

### 6. SOLLOL Detection

**New in v0.3.6:** Detect if SOLLOL is running vs native Ollama.

```python
import requests

def is_sollol(url="http://localhost:11434"):
    """Check if SOLLOL is running at the given URL."""

    # Method 1: Check X-Powered-By header
    response = requests.get(url)
    if response.headers.get("X-Powered-By") == "SOLLOL":
        return True

    # Method 2: Check health endpoint
    response = requests.get(f"{url}/api/health")
    data = response.json()
    if data.get("service") == "SOLLOL":
        retur