Spaces:

codey-lab
/

Multi-LLM-API-Gateway

Running

App Files Files Community

Multi-LLM-API-Gateway / README.md

Alibrown

Update README.md

5523a2c verified 2 days ago

preview code

raw

history blame contribute delete

17.9 kB

metadata

title: Multi-LLM API Gateway
emoji: 🛡️
colorFrom: indigo
colorTo: red
sdk: docker
pinned: true
license: apache-2.0
short_description: LLM API Gateway with MCP interface

Multi-LLM API Gateway with MCP interface

— or Universal MCP Hub (Sandboxed)
— or universal AI Wrapper over SSE + Quart with some tools on a solid fundament

aka: a clean, secure starting point for your own projects. Pick the description that fits your use case. They're all correct.

A production-grade MCP server that actually thinks about security.
Built on PyFundaments — running on simpleCity and paranoidMode.

No key → no tool → no crash → no exposed secrets

Most MCP servers are prompts dressed up as servers. This one has a real architecture.

Why this exists

While building this, we kept stumbling over the same problem — the MCP ecosystem is full of servers with hardcoded keys, os.environ scattered everywhere, zero sandboxing. One misconfigured fork and your API keys are gone.

This is exactly the kind of negligence (and worse — outright fraud) that Wall of Shames documents: a community project exposing fake "AI tools" that exploit non-technical users — API wrappers dressed up as custom models, Telegram payment funnels, bought stars. If you build on open source, you should know this exists.

This hub was built as the antidote:

Structural sandboxing — app/* can never touch fundaments/ or .env. Not by convention. By design.
Guardian pattern — main.py is the only process that reads secrets. It injects validated services as a dict. app/* never sees the raw environment.
Graceful degradation — No key? Tool doesn't register. Server still starts. No crash, no error, no empty None floating around.
Single source of truth — All tool/provider/model config lives in app/.pyfun. Adding a provider = edit one file. No code changes.

Architecture

main.py (Guardian)
│
│  reads .env / HF Secrets
│  initializes fundaments/* conditionally
│  injects validated services as dict
│
└──► app/app.py (Orchestrator, sandboxed)
     │
     │  unpacks fundaments ONCE, at startup, never stores globally
     │  starts hypercorn (async ASGI)
     │  routes: GET / | POST /api | GET+POST /mcp
     │
     ├── app/mcp.py         ← FastMCP + SSE handler
     ├── app/tools.py       ← Tool registry (key-gated)
     ├── app/provider.py    ← LLM + Search execution + fallback chain
     ├── app/models.py      ← Model limits, costs, capabilities
     ├── app/config.py      ← .pyfun parser (single source of truth)
     └── app/db_sync.py     ← Internal SQLite IPC (app/* state only)
                              ≠ fundaments/postgresql.py (Guardian-only)

The sandbox is structural:

# app/app.py — fundaments are unpacked ONCE, NEVER stored globally
async def start_application(fundaments: Dict[str, Any]) -> None:
    config_service         = fundaments["config"]
    db_service             = fundaments["db"]          # None if not configured
    encryption_service     = fundaments["encryption"]  # None if keys missing
    access_control_service = fundaments["access_control"]
    ...
    # From here: app/* reads its own config from app/.pyfun only.
    # fundaments are never passed into other app/* modules.

app/app.py never calls os.environ. Never imports from fundaments/. Never reads .env.
This isn't documentation. It's enforced by the import structure.

Why Quart + hypercorn?

MCP over SSE needs a proper async HTTP stack. The choice here is deliberate:

Quart is async Flask — same API, same routing, but fully async/await native. This matters because FastMCP's SSE handler is async, and mixing sync Flask with async MCP would require thread hacks or asyncio.run() gymnastics. With Quart, the /mcp route hands off directly to mcp.handle_sse(request) — no bridging, no blocking.

hypercorn is an ASGI server (vs. waitress/gunicorn which are WSGI). WSGI servers handle one request per thread — fine for traditional web apps, wrong for SSE where a connection stays open for minutes. hypercorn handles SSE connections as long-lived async streams without tying up threads. It also runs natively on HuggingFace Spaces without extra config.

The /mcp route in app.py is also the natural interception point — auth checks, rate limiting, payload logging can all be added there before the request ever reaches FastMCP. That's not possible when FastMCP runs standalone.

Two Databases — One Architecture

This hub runs two completely separate databases with distinct responsibilities. This is not redundancy — it's a deliberate performance and security decision.

┌─────────────────────────────────────────────────────────────┐
│  Guardian Layer (fundaments/*)                              │
│                                                             │
│  postgresql.py   → Cloud DB (e.g. Neon, Supabase)          │
│                    asyncpg pool, SSL enforced               │
│                    Neon-specific quirks handled             │
│                    (statement_timeout stripped, keepalives) │
│                                                             │
│  user_handler.py → SQLite (users + sessions tables)        │
│                    PBKDF2-SHA256 password hashing           │
│                    Session validation incl. IP + UserAgent  │
│                    Account lockout after 5 failed attempts  │
│                    Path: SQLITE_PATH env var or app/        │
│                                                             │
└──────────────────────┬──────────────────────────────────────┘
                       │ inject as fundaments dict
                       ▼
┌─────────────────────────────────────────────────────────────┐
│  App Layer (app/*)                                          │
│                                                             │
│  db_sync.py  → SQLite (hub_state + tool_cache tables)      │
│                aiosqlite (async, non-blocking)              │
│                NEVER touches users/sessions tables          │
│                Relocated to /tmp/ on HF Spaces auto        │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Why two SQLite databases?

user_handler.py (Guardian) owns users and sessions — authentication state that must be isolated from the app layer. db_sync.py (app/*) owns hub_state and tool_cache — fast, async IPC between tools that doesn't need to leave the process, let alone hit a cloud endpoint.

A tool caching a previous LLM response or storing intermediate state between pipeline steps should never wait on a round-trip to Neon. Local SQLite is microseconds. Cloud PostgreSQL is 50-200ms per query. For tool-to-tool communication, that difference matters.

Table ownership — hard rule:

Table	Owner	Access
`users`	`fundaments/user_handler.py`	Guardian only
`sessions`	`fundaments/user_handler.py`	Guardian only
`hub_state`	`app/db_sync.py`	app/* only
`tool_cache`	`app/db_sync.py`	app/* only

db_sync.py uses the same SQLite path (SQLITE_PATH) as user_handler.py — same file, different tables, zero overlap. The db_query MCP tool exposes SELECT-only access to hub_state and tool_cache. It cannot reach users or sessions.

Cloud DB (postgresql.py):

Handles the heavy cases — persistent storage, workflow tool results that need to survive restarts, anything that benefits from a real relational DB. Neon-specific quirks are handled automatically: statement_timeout is stripped from the DSN (Neon doesn't support it), SSL is enforced at require minimum, keepalives are set, and terminated connections trigger an automatic pool restart.

If no DATABASE_URL is set, the entire cloud DB layer is skipped cleanly. The app runs without it.

Tools

Tools register themselves at startup — only if the required API key exists in the environment. No key, no tool. The server always starts.

ENV Secret	Tool	Notes
`ANTHROPIC_API_KEY`	`llm_complete`	Claude Haiku / Sonnet / Opus
`GEMINI_API_KEY`	`llm_complete`	Gemini 2.0 / 2.5 / 3.x Flash & Pro
`OPENROUTER_API_KEY`	`llm_complete`	100+ models via OpenRouter
`HF_TOKEN`	`llm_complete`	HuggingFace Inference API
`BRAVE_API_KEY`	`web_search`	Independent web index
`TAVILY_API_KEY`	`web_search`	AI-optimized search with synthesized answers
`DATABASE_URL`	`db_query`	Read-only SELECT — enforced at app level
(always)	`list_active_tools`	Shows key names only — never values
(always)	`health_check`	Status + uptime
(always)	`get_model_info`	Limits, costs, capabilities per model

Configured in .pyfun — not hardcoded:

[TOOL.code_review]
active           = "true"
description      = "Review code for bugs, security issues and improvements"
provider_type    = "llm"
default_provider = "anthropic"
timeout_sec      = "60"
system_prompt    = "You are an expert code reviewer. Analyze the given code for bugs, security issues, and improvements. Be specific and concise."
[TOOL.code_review_END]

Current built-in tools: llm_complete, code_review, summarize, translate, web_search, db_query
Future hooks (commented, ready): image_gen, code_exec, shellmaster, Discord, GitHub webhooks

LLM Fallback Chain

All LLM providers share one llm_complete tool. If a provider fails, the hub automatically walks the fallback chain defined in .pyfun:

anthropic → gemini → openrouter → huggingface

Fallbacks are configured per-provider, not hardcoded:

[LLM_PROVIDER.anthropic]
fallback_to = "gemini"
[LLM_PROVIDER.anthropic_END]

[LLM_PROVIDER.gemini]
fallback_to = "openrouter"
[LLM_PROVIDER.gemini_END]

Same pattern applies to search providers (brave → tavily).

Quick Start

HuggingFace Spaces (recommended)

Fork / duplicate this Space
Go to Settings → Variables and secrets
Add the API keys you have (any subset works)
Space starts automatically — only tools with valid keys register

That's it. No config editing. No code changes.

→ Live Demo Space (no LLM keys set!)

Local / Docker

git clone https://github.com/VolkanSah/Multi-LLM-API-Gateway
cd Multi-LLM-API-Gateway
cp example-mcp___.env .env
# fill in your keys
pip install -r requirements.txt
python main.py

Minimum required ENV vars (everything else is optional):

PYFUNDAMENTS_DEBUG=""
LOG_LEVEL="INFO"
LOG_TO_TMP=""
ENABLE_PUBLIC_LOGS="true"
HF_TOKEN=""
HUB_SPACE_URL=""
MCP_TRANSPORT="sse"

Connect an MCP Client

Claude Desktop / any SSE-compatible client

{
  "mcpServers": {
    "universal-mcp-hub": {
      "url": "https://YOUR_USERNAME-universal-mcp-hub.hf.space/sse"
    }
  }
}

Private Space (with HF token)

{
  "mcpServers": {
    "universal-mcp-hub": {
      "url": "https://YOUR_USERNAME-universal-mcp-hub.hf.space/sse",
      "headers": {
        "Authorization": "Bearer hf_..."
      }
    }
  }
}

Desktop Client

A full PySide6 desktop client is included in DESKTOP_CLIENT/hub.py — ideal for private or non-public Spaces where you don't want to expose the SSE endpoint.

pip install PySide6 httpx
# optional file handling:
pip install Pillow PyPDF2 pandas openpyxl
python DESKTOP_CLIENT/hub.py

Features:

Multi-chat with persistent history (~/.mcp_desktop.json)
Tool/Provider/Model selector loaded live from your Hub
File attachments: images, PDF, CSV, Excel, ZIP, source code
Connect tab with health check + auto-load
Settings: HF Token + Hub URL saved locally, never sent anywhere except your own Hub
Full request/response log with timestamps
Runs on Windows, Linux, macOS

→ Desktop Client docs

Configuration (.pyfun)

app/.pyfun is the single source of truth for all app behavior. Three tiers — use what you need:

LAZY:       [HUB] + one [LLM_PROVIDER.*]                    → works
NORMAL:     + [SEARCH_PROVIDER.*] + [MODELS.*]              → works better  
PRODUCTIVE: + [TOOLS] + [HUB_LIMITS] + [DB_SYNC]           → full power

Adding a new LLM provider requires two steps — .pyfun + one line in providers.py:

# 1. app/.pyfun — add provider block
[LLM_PROVIDER.mistral]
active        = "true"
base_url      = "https://api.mistral.ai/v1"
env_key       = "MISTRAL_API_KEY"
default_model = "mistral-large-latest"
models        = "mistral-large-latest, mistral-small-latest, codestral-latest"
fallback_to   = ""
[LLM_PROVIDER.mistral_END]

# 2. app/providers.py — uncomment the dummy + register it
_PROVIDER_CLASSES = {
    ...
    "mistral": MistralProvider,   # ← uncomment to activate
}

providers.py ships with ready-to-use commented dummy classes for OpenAI, Mistral, and xAI/Grok — each with the matching .pyfun block right above it. Most OpenAI-compatible APIs need zero changes to the class itself, just a different base_url and env_key. Search providers (Brave, Tavily) follow the same pattern and are next on the roadmap.

Model limits, costs, and capabilities are also configured here — get_model_info reads directly from .pyfun:

[MODEL.claude-sonnet-4-6]
provider           = "anthropic"
context_tokens     = "200000"
max_output_tokens  = "16000"
requests_per_min   = "50"
cost_input_per_1k  = "0.003"
cost_output_per_1k = "0.015"
capabilities       = "text, code, analysis, vision"
[MODEL.claude-sonnet-4-6_END]

Dependencies

# PyFundaments Core (always required)
asyncpg          — async PostgreSQL pool (Guardian/cloud DB)
python-dotenv    — .env loading
passlib          — PBKDF2 password hashing in user_handler.py
cryptography     — encryption layer in fundaments/

# MCP Hub
fastmcp          — MCP protocol + tool registration
httpx            — async HTTP for all provider API calls
quart            — async Flask (ASGI) — needed for SSE + hypercorn
hypercorn        — ASGI server — long-lived SSE connections, HF Spaces native
requests         — sync HTTP for tool workers

# Optional (uncomment in requirements.txt as needed)
# aiofiles       — async file ops (ML pipelines, file uploads)
# discord.py     — Discord bot integration (app/discord_api.py, planned)
# PyNaCl         — Discord signature verification
# psycopg2-binary — alternative PostgreSQL driver

The core stack is intentionally lean. asyncpg + quart + hypercorn + fastmcp + httpx covers the full MCP server. Everything else is opt-in.

Security Design

API keys live in HF Secrets / .env — never in .pyfun, never in code
list_active_tools returns key names only — never values
db_query is SELECT-only, enforced at application level (not just docs)
app/* has zero import access to fundaments/ internals
Direct execution of app/app.py is blocked by design — prints a warning and uses a null-fundaments dict
fundaments/ is initialized conditionally — missing services degrade gracefully, they don't crash

PyFundaments is not perfect. But it's more secure than most of what runs in production today.

→ Full Security Policy

Foundation

This hub is built on PyFundaments — a security-first Python boilerplate providing:

config_handler.py — env loading with validation
postgresql.py — async DB pool (Guardian-only)
encryption.py — key-based encryption layer
access_control.py — role/permission management
user_handler.py — user lifecycle management
security.py — unified security manager composing the above

None of these are accessible from app/*. They are injected as a validated dict by main.py.

→ PyFundaments Function Overview
→ Module Docs → Source of this REPO

History

ShellMaster (2023, MIT) was the precursor — browser-accessible shell for ChatGPT with session memory via /tmp/shellmaster_brain.log, built before MCP was even a concept. Universal MCP Hub is its natural evolution.

License

Dual-licensed:

Apache License 2.0
Ethical Security Operations License v1.1 (ESOL) — mandatory, non-severable

By using this software you agree to all ethical constraints defined in ESOL v1.1.

Architecture, security decisions, and PyFundaments by Volkan Kücükbudak.
Built with Claude (Anthropic) as a typing assistant for docs & the occasional bug.

crafted with passion — just wanted to understand how it works, don't actually need it, have a CLI 😄