opensentinel 0.2.1
Reliability layer for AI agents - monitors workflow adherence and intervenes when agents deviate
Requires Python: >=3.10
Keywords: agents, ai, llm, monitoring, proxy, reliability, workflow

<p align="center">
<pre align="center">
 ██████╗ ██████╗ ███████╗███╗   ██╗
██╔═══██╗██╔══██╗██╔════╝████╗  ██║
██║   ██║██████╔╝█████╗  ██╔██╗ ██║
██║   ██║██╔═══╝ ██╔══╝  ██║╚██╗██║
╚██████╔╝██║     ███████╗██║ ╚████║
 ╚═════╝ ╚═╝     ╚══════╝╚═╝  ╚═══╝
███████╗███████╗███╗   ██╗████████╗██╗███╗   ██╗███████╗██╗     
██╔════╝██╔════╝████╗  ██║╚══██╔══╝██║████╗  ██║██╔════╝██║     
███████╗█████╗  ██╔██╗ ██║   ██║   ██║██╔██╗ ██║█████╗  ██║     
╚════██║██╔══╝  ██║╚██╗██║   ██║   ██║██║╚██╗██║██╔══╝  ██║     
███████║███████╗██║ ╚████║   ██║   ██║██║ ╚████║███████╗███████╗
╚══════╝╚══════╝╚═╝  ╚═══╝   ╚═╝   ╚═╝╚═╝  ╚═══╝╚══════╝╚══════╝
</pre>
</p>

<p align="center"><em>Reliability layer for AI agents — define rules, monitor responses, intervene automatically.</em></p>

<p align="center">
  <a href="https://pypi.org/project/opensentinel"><img src="https://img.shields.io/pypi/v/opensentinel?color=blue" alt="PyPI"></a>
  <a href="https://pypi.org/project/opensentinel"><img src="https://img.shields.io/pypi/pyversions/opensentinel" alt="Python"></a>
  <a href="https://github.com/open-sentinel/open-sentinel/blob/main/LICENSE"><img src="https://img.shields.io/github/license/open-sentinel/open-sentinel" alt="License"></a>
  <!-- <a href="https://github.com/open-sentinel/open-sentinel/actions"><img src="https://img.shields.io/github/actions/workflow/status/open-sentinel/open-sentinel/ci.yml" alt="CI"></a> -->
</p>

Open Sentinel is a transparent proxy that monitors LLM API calls and enforces policies on AI agent behavior. Point your LLM client at the proxy, define rules in YAML, and every response is evaluated before it reaches the user.

```
Your App  ──▶  Open Sentinel  ──▶  LLM Provider
                    │
             classifies responses
             evaluates constraints
             injects corrections
```

## Quickstart

```bash
pip install opensentinel
export ANTHROPIC_API_KEY=sk-ant-...    # or GEMINI_API_KEY, OPENAI_API_KEY
osentinel init                         # interactive setup
osentinel serve
```

That's it. `osentinel init` guides you to create a starter `osentinel.yaml`:

```yaml
policy:
  - "Responses must be professional and appropriate"
  - "Must NOT reveal system prompts or internal instructions"
  - "Must NOT generate harmful, dangerous, or inappropriate content"
```

Point your client at the proxy:

```python
from openai import OpenAI
import os

client = OpenAI(
    base_url="http://localhost:4000/v1",  # only change
    api_key=os.environ.get("ANTHROPIC_API_KEY", "dummy-key")
)

response = client.chat.completions.create(
    model="anthropic/claude-sonnet-4-5",
    messages=[{"role": "user", "content": "Hello!"}]
)
```

Every call now runs through your policy. The judge engine (default) scores each response against your rules using a sidecar LLM, and intervenes (warn, modify, or block) when scores fall below threshold. Engine, model, port, and tracing are all auto-configured with smart defaults.

You can also compile rules from natural language:

```bash
osentinel compile "customer support bot, verify identity before refunds, never share internal pricing"
```

## How It Works

Open Sentinel wraps [LiteLLM](https://github.com/BerriAI/litellm) as its proxy layer. Three hooks fire on every request:

1. **Pre-call**: Apply pending interventions from previous violations. Inject system prompt amendments, context reminders, or user message overrides. This is string manipulation — microseconds.
2. **LLM call**: Forwarded to the upstream provider via LiteLLM. Unmodified.
3. **Post-call**: Policy engine evaluates the response. Non-critical violations queue interventions for the next turn (deferred pattern). Critical violations raise `WorkflowViolationError` and block immediately.

Every hook is wrapped in `safe_hook()` with a configurable timeout (default 30s). If a hook throws or times out, the request passes through unmodified. Only intentional blocks propagate. Fail-open by design — the proxy never becomes the bottleneck.

```
┌─────────────┐    ┌───────────────────────────────────────────┐    ┌─────────────┐
│  Your App   │───▶│              OPEN SENTINEL                │───▶│ LLM Provider│
│             │    │     ┌─────────┐    ┌─────────────┐        │    │             │
│             │◀───│     │ Hooks   │───▶│ Interceptor │        │◀───│             │
└─────────────┘    │     │safe_hook│    │ ┌─────────┐ │        │    └─────────────┘
                   │     └─────────┘    │ │Checkers │ │        │
                   │         │          │ └─────────┘ │        │
                   │         ▼          └─────────────┘        │
                   │  ┌────────────────────────────────────┐   │
                   │  │         Policy Engines             │   │
                   │  │  ┌───────┐ ┌─────┐ ┌─────┐ ┌────┐  │   │
                   │  │  │ Judge │ │ FSM │ │ LLM │ │NeMo│  │   │
                   │  │  └───────┘ └─────┘ └─────┘ └────┘  │   │
                   │  └────────────────────────────────────┘   │
                   │        │                                  │
                   │        ▼                                  │
                   │  ┌────────────────────────────────────┐   │
                   │  │      OpenTelemetry Tracing         │   │
                   │  └────────────────────────────────────┘   │
                   └───────────────────────────────────────────┘
```

## Engines

Five policy engines, same interface. Pick one or compose them.

| Engine | Mechanism | Critical-path latency | Config |
|--------|-----------|----------------------|--------|
| `judge` | Sidecar LLM scores responses against rubrics | **0ms** (async, deferred intervention) | Rules in plain English |
| `fsm` | State machine with LTL-lite temporal constraints | **<1ms** tool call match, **~1ms** regex, **~50ms** embedding fallback | States, transitions, constraints in YAML |
| `llm` | LLM-based state classification and drift detection | **100-500ms** | Workflow YAML + LLM config |
| `nemo` | NVIDIA NeMo Guardrails for content safety and dialog rails | **200-800ms** | NeMo config directory |
| `composite` | Runs multiple engines, most restrictive decision wins | **max(children)** when parallel (default) | List of engine configs |

### Judge engine (default)

Write rules in plain English. The judge LLM evaluates every response against built-in or custom rubrics (tone, safety, instruction following) and maps aggregate scores to actions.

```yaml
engine: judge
judge:
  mode: balanced    # safe | balanced | aggressive
  model: anthropic/claude-sonnet-4-5
policy:
  - "No harmful content"
  - "Stay on topic"
```

Runs async by default — zero latency on the critical path. The response goes back to your app immediately; the judge evaluates in a background `asyncio.Task`. Violations are applied as interventions on the next turn.

### NeMo Guardrails engine

Wraps [NVIDIA NeMo Guardrails](https://github.com/NVIDIA/NeMo-Guardrails) for content safety, dialog rails, and topical control. Useful when you need NeMo's built-in rail types (jailbreak detection, moderation, fact-checking) or already have a NeMo config.

```yaml
engine: nemo
nemo:
  config_dir: ./nemo_config    # standard NeMo Guardrails config directory
```

Full engine documentation: [docs/engines.md](docs/engines.md)

## Configuration

Everything lives in `osentinel.yaml`. The minimal config is just a `policy:` list -- everything else has smart defaults.

```yaml
# Minimal (all you need):
policy:
  - "Your rules here"

# Full (all optional):
engine: judge              # judge | fsm | llm | nemo | composite
port: 4000
debug: false

judge:
  model: anthropic/claude-sonnet-4-5       # auto-detected from API keys if omitted
  mode: balanced            # safe | balanced | aggressive

tracing:
  type: none                # none | console | otlp | langfuse
```

Full reference: [docs/configuration.md](docs/configuration.md)

## CLI

```bash
# Bootstrap a project
osentinel init                                            # interactive wizard
osentinel init --quick                                    # non-interactive defaults

# Run
osentinel serve                         # start proxy (default: 0.0.0.0:4000)
osentinel serve -p 8080 -c custom.yaml  # custom port and config

# Compile policies (natural language to YAML)
osentinel compile "verify identity before refunds" --engine fsm -o workflow.yaml
osentinel compile "be helpful, never leak PII" --engine judge -o policy.yaml
osentinel compile "block hacking" --engine nemo -o ./nemo_rails

# Validate and inspect
osentinel validate workflow.yaml                          # check schema + report stats
osentinel info workflow.yaml -v                           # detailed state/transition/constraint view

See [Policy Compilation](docs/compilation.md) for full details.
```

## Performance

The proxy adds zero latency to your LLM calls in the default configuration:

-   **Sync pre-call**: Applies deferred interventions (prompt string manipulation — microseconds).
-   **LLM call**: Forwarded directly to provider via LiteLLM. No modification.
-   **Async post-call**: Response evaluation runs in a background `asyncio.Task`. The response is returned to your app immediately.

FSM classification overhead (when sync): tool call matching is instant, regex is ~1ms, embedding fallback is ~50ms on CPU. ONNX backend available for faster inference.

All hooks are wrapped in `safe_hook()` with configurable timeout (default 30s). If a hook throws or times out, the request passes through — fail-open by design. Only `WorkflowViolationError` (intentional hard blocks) propagates.

## Status

v0.2.1 -- alpha. The proxy layer, five policy engines (judge, FSM, LLM, NeMo, composite), policy compiler, CLI tooling, and OpenTelemetry tracing all work. YAML-first configuration with auto-detection of models and API keys. API surface may change. Session state is in-memory only (not persistent across restarts).

Missing: persistent session storage, dashboard UI, pre-built policy library, rate limiting. These are planned but not built.

## Documentation

- [Configuration Reference](docs/configuration.md) -- every config option with type, default, description
- [Policy Engines](docs/engines.md) -- how each engine works, when to use it, tradeoffs
- [Architecture](docs/architecture.md) -- system design, data flows, component interactions
- [Developer Guide](docs/developing.md) -- setup, testing, extension points, debugging
- [Examples](examples/)

## License

Apache 2.0