isagellm-gateway 0.5.1.0
sageLLM Gateway - OpenAI/Anthropic compatible API Gateway (L6)
License: Apache-2.0
Requires Python: ==3.11.*
Keywords: sagellm, gateway, llm, openai, anthropic, inference

# sageLLM Gateway

## Protocol Compliance (Mandatory)

- MUST follow Protocol v0.1:
  https://github.com/intellistream/sagellm-docs/blob/main/docs/specs/protocol_v0.1.md
- Any globally shared definitions (fields, error codes, metrics, IDs, schemas) MUST be added to
  Protocol first.

[![CI](https://github.com/intellistream/sagellm-gateway/actions/workflows/ci.yml/badge.svg)](https://github.com/intellistream/sagellm-gateway/actions/workflows/ci.yml)
[![PyPI version](https://badge.fury.io/py/isagellm-gateway.svg)](https://badge.fury.io/py/isagellm-gateway)
[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![Pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white)](https://github.com/pre-commit/pre-commit)

**isagellm-gateway** - OpenAI/Anthropic Compatible API Gateway for sageLLM

## Overview

The `isagellm-gateway` package serves as the primary entry point for external API requests in the
sageLLM architecture. It provides an OpenAI-compatible interface and handles session management,
request routing, and integration with the sageLLM Control Plane.

**Key Features:**

- **OpenAI Compatibility**: Full support for `/v1/chat/completions` and `/v1/models`.
- **Session Management**: Built-in support for conversational sessions via `/sessions` endpoints.
- **Control Plane Integration**: Seamlessly connects with `isagellm-control-plane` for model
  management and routing.
- **Distributed Tracing**: Built-in OpenTelemetry support for observability across services
  (Jaeger/Zipkin compatible).
- **Protocol Compliance**: Strictly adheres to sageLLM Protocol v0.1.

## Dependencies

- **Python**: >= 3.10
- **isagellm-protocol**: >= 0.4.0.0
- **isagellm-control-plane**: Required for runtime operation.

## Installation

Install the gateway and the required control plane package:

```bash
pip install isagellm-gateway isagellm-control-plane
```

## Quick Start

### Start the Server

Start the gateway server on port 8080:

```bash
sagellm-gateway --port 8080
```

### Usage Examples

**1. Chat Completion (OpenAI API)**

```bash
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt2",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
```

**2. List Models**

```bash
curl http://localhost:8080/v1/models
```

**3. Health Check**

```bash
curl http://localhost:8080/health
```

## API Documentation

### Core Endpoints

- `POST /v1/chat/completions`: Generate chat completions (supports streaming).
- `POST /v1/embeddings`: Generate embeddings.
- `GET /v1/models`: List available models.
- `GET /health`: Health check and status.

### Session Management

- `POST /sessions`: Create a new session.
- `GET /sessions`: List all sessions.
- `GET /sessions/{session_id}`: Get session details.
- `DELETE /sessions/{session_id}`: Delete a session.

## Architecture

```
┌─────────────────────────────────────────┐
│         sageLLM Gateway                 │
│  • OpenAI-compatible API                │
│  • Session State Management             │
│  • Request Validation & Routing         │
└─────────────────────────────────────────┘
                   │
                   ▼
┌─────────────────────────────────────────┐
│         Control Plane                   │
│  (managed by isagellm-control-plane)    │
└─────────────────────────────────────────┘
```

## Development Guide

### Prerequisites

- Python 3.10+
- Git

### Setup

1. **Clone the repository:**

   ```bash
   git clone git@github.com:intellistream/sagellm-gateway.git
   cd sagellm-gateway
   ```

1. **Initialize development environment:**

   ```bash
   # Installs dependencies and hooks
   ./quickstart.sh

   # Note: You also need the control plane for local testing
   pip install isagellm-control-plane
   ```

### Testing

Run the test suite with `pytest`:

```bash
pytest tests/ -v
```

### Linting

Check code quality with `ruff`:

```bash
ruff check .
```

## Versioning

Current Version: **0.4.0.11**

For version history, see [CHANGELOG.md](CHANGELOG.md).