rethink-llmhub-runtime 2.3.0
A lightweight runtime for LLMHub that delegates to any-llm
License: MIT
Requires Python: >=3.10
Keywords: llm, ai, llmhub, runtime, any-llm

# llmhub_runtime

`llmhub_runtime` is a small Python library that lets you call LLMs **by role** using a simple YAML config (`llmhub.yaml`), while delegating all provider-specific logic to [`any-llm`](https://mozilla-ai.github.io/any-llm/).

It is designed to be:

- **Runtime-light** – minimal dependencies, no discovery logic.
- **Provider-agnostic** – supports any provider that `any-llm` supports.
- **Role-centric** – your application code never handles provider/model strings directly.

`llmhub_runtime` is intended for:

- Application backends (e.g. memory systems, agents, tools).
- The future `llmhub` CLI/Web tool, which will generate `llmhub.yaml` and then use this runtime internally.

## Installation

    pip install llmhub-runtime any-llm-sdk

(Exact package name to be confirmed when publishing.)

## Runtime Config: `llmhub.yaml`

`llmhub_runtime` reads a **generated** config file, typically named `llmhub.yaml`:

    project: memory
    env: dev

    providers:
      openai:
        env_key: OPENAI_API_KEY
      anthropic:
        env_key: ANTHROPIC_API_KEY

    roles:
      llm.preprocess:
        provider: openai
        model: gpt-4o-mini
        mode: chat
        params:
          temperature: 0.2
          max_tokens: 512

      llm.inference:
        provider: anthropic
        model: claude-3-5-sonnet-20241022
        mode: chat
        params:
          temperature: 0.7
          max_tokens: 2048

You typically do **not** edit this by hand; it is generated by higher-level tools (e.g. `llmhub` CLI/Web).

## Basic Usage

    from llmhub_runtime.hub import LLMHub

    hub = LLMHub(config_path="llmhub.yaml")

    response = hub.completion(
        role="llm.inference",
        messages=[{"role": "user", "content": "Hello"}],
    )

    print(response)

Embeddings:

    embedding = hub.embedding(
        role="llm.embedding",
        input="Hello world",
    )

To override parameters per call:

    response = hub.completion(
        role="llm.inference",
        messages=[...],
        params_override={"temperature": 0.1},
    )

## Architecture Overview

`llmhub_runtime` is intentionally small and has three main layers:

1. **Config layer**
   - `models.py` – Pydantic models for `RuntimeConfig`, `ProviderConfig`, `RoleConfig`, `ResolvedCall`.
   - `config_loader.py` – loads and validates `llmhub.yaml`.

2. **Resolution layer**
   - `resolver.py` – maps a logical `role` name to `{provider, model, mode, params}`, with optional fallback from `defaults`.

3. **Execution layer**
   - `hub.py` – exposes the `LLMHub` class:
     - Resolves roles.
     - Calls `any-llm` (`completion` / `embedding`) with the resolved settings.
     - Optional hooks for logging/metrics.

All domain-specific errors live in `errors.py`.

## Design Principles

- **No provider logic** – `llmhub_runtime` never talks to provider SDKs directly; it only calls `any-llm`.
- **No discovery or scoring** – it assumes `llmhub.yaml` already contains concrete provider/model choices.
- **Role-first** – application code only sees role names; you can swap models by editing/generating `llmhub.yaml` without changing app code.

## Roadmap

- Async APIs (`acompletion`, `aembedding`).
- Streaming interfaces.
- More modes (`image`, `audio`, `tool`).
- Tight integration with the `llmhub` CLI/Web for config generation.