kubeflow-mcp 0.1.0
Agentic MCP server for Kubeflow - High-level semantic AI training orchestration
Requires Python: >=3.11
Keywords: ai, fine-tuning, kubeflow, llm, mcp, model-context-protocol, training

# Kubeflow MCP Server

**AI model fine-tuning on Kubernetes via natural language.**

Train models by describing what you want, not by writing YAML.

> 📖 **New here?** Read the [OVERVIEW.md](OVERVIEW.md) to understand the vision and technology stack.

## Why?

| Without MCP | With MCP |
|-------------|----------|
| Learn TrainJob YAML specs | Just describe what you want |
| Debug kubectl + events + logs | "Why did my job fail?" |
| Manual PVC/Secret setup | Auto-setup infrastructure |
| Context-switch IDE ↔ terminal | Stay in conversation |

**Core principle**: 
Users should run training **right away** with just `model + dataset + config`. 
No custom code—built-in algorithms (TrainingHub-primary, TorchTune - not currently used) handle the training loop.

## Architecture

```mermaid
graph TB
    subgraph Clients["🤖 LLM = REASONING"]
        A[Claude]
        B[Cursor IDE]
        C[Ollama]
        D[Open WebUI]
        E[🦙 LlamaIndex]
        F[🦙 RamaLama]
    end

    A & B & C & D & E & F --> G[📡 MCP Protocol]

    G --> P[🔐 Policy: readonly - data-scientist - ml-engineer - platform-admin]

    HF[🤗 HuggingFace Hub] --> H

    subgraph MCPs[" "]
        direction LR
        H[⚡ Kubeflow MCP 35 Tools]
        I[☸️ Kubernetes MCP General K8s]
    end

    P --> H
    P --> I

    subgraph KFTools[" "]
        direction LR
        H1[🔍 Discovery]
        H2[📋 Planning]
        H3[🚀 Execution]
    end

    H --> H1
    H --> H2
    H --> H3

    subgraph K8STools[" "]
        direction LR
        I1[📦 Pods Resources]
        I2[⎈ Helm Namespaces]
        I3[📊 Events Nodes]
    end

    I --> I1
    I --> I2
    I --> I3

    H1 & H2 & H3 --> SDK[🐍 Kubeflow SDK TrainerClient Initializers]

    subgraph Trainers[" "]
        direction LR
        T1[🔧 BuiltinTrainer TorchTune]
        T2[🤗 TransformersTrainer HuggingFace TRL]
        T3[🎯 TrainingHubTrainer SFT OSFT]
        T4[📝 CustomTrainer User Scripts]
    end

    SDK --> T1
    SDK --> T2
    SDK --> T3
    SDK --> T4

    T1 & T2 & T3 & T4 --> J[🏃 Trainer V2 K8s Operator]

    J --> K[☸️ Kubernetes Cluster]
    I1 & I2 & I3 --> K

    subgraph Resources[" "]
        direction LR
        L[🎮 GPUs]
        M[💾 PVCs]
        N[🔐 Secrets]
        O[📊 Kueue]
        Q[📦 Pods Services]
    end

    K --> L
    K --> M
    K --> N
    K --> O
    K --> Q
```

## Quick Start

**Install from GitHub** (no clone needed):
```bash
pip install git+https://github.com/opendatahub-io/kubeflow-mcp.git
```

**Development setup** (clone repo):
```bash
git clone https://github.com/opendatahub-io/kubeflow-mcp.git
cd kubeflow-mcp
uv sync --group dev
```

| Use Case | Command |
|----------|---------|
| **Cursor / Claude** | `uv run kf-mcp setup` → restart IDE |
| **Terminal (Ollama)** | `uv sync --extra chat && uv run kf-mcp chat --mcp kubeflow-mcp` |
| **Web UI** | `uv sync --extra ui && uv run kf-mcp ui` |

> ⚠️ **Default: Read-only mode** for safety. Use `--policy data-scientist` for write access. See [policies](src/kubeflow_mcp/policies/README.md) for details.

Then ask:
```
> Fine-tune Qwen/Qwen2.5-7B-Instruct on tatsu-lab/alpaca using LoRA
> Why did my training job fail?
> Show cluster GPU resources
```

## Tools (35)

| Layer | Tools | Purpose |
|-------|-------|---------|
| **Discovery** | `list_training_jobs`, `get_cluster_resources`, `list_runtimes`... | Understand current state |
| **Planning** | `estimate_resources`, `check_prerequisites`, `validate_config`... | Make decisions |
| **Execution** | `fine_tune_model`, `suspend_job`, `resume_job`, `setup_storage`... | Apply changes |

Run `/mcp` in chat to see all available tools.

## Example Conversations

**Start training:**
```
User: "Fine-tune Qwen/Qwen2.5-0.5B on tatsu-lab/alpaca in kubeflow-mcp-test namespace"

Agent: → check_training_prerequisites() → "All ready"
       → fine_tune_model() → "Created job: ft-abc12345"
       → monitor_training() → "Training started, 0%"
```

**Debug failures:**
```
User: "Why did my job fail?"

Agent: → get_job_events() → "OOMKilled"
       → get_training_logs() → "CUDA out of memory at batch 42"
       
       "Your job ran out of GPU memory. Try reducing batch_size 
        from 8 to 4, or switch to QLoRA which uses ~50% less memory."
```

**Manage resources:**
```
User: "Pause training to free GPUs for urgent job"

Agent: → suspend_training_job() → "Job paused, GPUs released"
       
       "Done. Resume anytime with 'resume my training job'—
        it will continue from the last checkpoint."
```
## Documentation

| Topic | Location |
|-------|----------|
| **Agent Usage & MCP Config** | [`src/kubeflow_mcp/agents/README.md`](src/kubeflow_mcp/agents/README.md) |
| **Policy System (RBAC)** | [`src/kubeflow_mcp/policies/README.md`](src/kubeflow_mcp/policies/README.md) |
| **Training Adapters** | [`src/kubeflow_mcp/adapters/README.md`](src/kubeflow_mcp/adapters/README.md) |
| **OpenShift Deployment** | [`deploy/README.md`](deploy/README.md) |
| **Roadmap** | [`docs/ROADMAP.md`](docs/ROADMAP.md) |

## Policies

By default, the agent runs in **read-only mode** for safety:

| Policy | Access |
|--------|--------|
| `readonly` | Read-only (default) |
| `data-scientist` | Train + Monitor |
| `ml-engineer` | Full namespace |
| `project-admin` | Full + Runtimes |
| `platform-admin` | Unrestricted |

```bash
# Enable training capabilities
uv run kf-mcp chat --policy data-scientist --mcp kubeflow-mcp
```

> See [`src/kubeflow_mcp/policies/README.md`](src/kubeflow_mcp/policies/README.md) for custom policies.

## Multi-MCP: Deep K8s Debugging

For cluster-level issues (scheduling, RBAC, networking), pair with [kubernetes-mcp-server](https://github.com/manusa/kubernetes-mcp-server):

```bash
uv run kf-mcp setup --with-kubernetes
```

| Issue Type | kubeflow-mcp | kubernetes-mcp |
|------------|:------------:|:--------------:|
| Training OOM | ✓ | |
| Job progress | ✓ | |
| Pod scheduling | | ✓ |
| RBAC/Network | | ✓ |

## Debug with MCP Inspector

```bash
npx @modelcontextprotocol/inspector uv run kf-mcp server
```

## Requirements

- Python 3.11+
- Kubernetes cluster with [Kubeflow Training Operator](https://github.com/kubeflow/trainer)
- Valid kubeconfig (`~/.kube/config`)
- [Kubeflow SDK](https://github.com/opendatahub-io/kubeflow-sdk/tree/release-0.2) (v0.2+ with RHAI Features)

## Current Project Status and Implementation Plan

**✅ Implemented:**
- 35 MCP tools (discovery, planning, execution)
- Training job lifecycle (create, monitor, suspend, resume, delete)
- Auto-setup infrastructure (PVC, runtimes, HF credentials)
- RBAC-like policy system (5 personas)
- Multi-MCP support (kubeflow, kubernetes, github, etc.)
- Default read-only mode for safety
- Ollama agent with FunctionAgent + thinking mode
- OpenShift fsGroup auto-detection

**🔜 TODO (Can implement in future iterations):**
- Custom Trainer: LLM writes training code → TransformersTrainer (progression tracking, JIT checkpointing)
- Script adapter tool: Convert any training script to distributed-ready function for CustomTrainer/TransformersTrainer
- Kueue quota integration
- Training Hub resource estimator
- Katib HPO (hyperparameter optimization)
- KServe/VLLM model deployment
- Cost tracking & budget alerts

> See [`docs/ROADMAP.md`](docs/ROADMAP.md) for the full roadmap.