Kube Sec Gym

Can a 0.6B model learn to be a Kubernetes security auditor — from scratch?

Forked from sid-rp/kube-sre-gym; pivoted to the security posture domain. Same architecture (real cluster, GRPO, adversarial curriculum, LLM judge, OpenEnv) — different agent. Where the upstream taught a tiny LLM to be on-call, this fork teaches it to be a CIS-Benchmark-aware auditor.

We gave a tiny language model a kubectl prompt, a live Kubernetes cluster sprinkled with CIS / NIST / CWE violations, and zero knowledge of what RBAC even is. No pre-training on security docs. No few-shot examples. Just a posture-violation alert and a shell.

The goal: through 8 episodes of curriculum-driven self-play, watch it learn to enumerate cluster topology, detect privileged pods and exposed secrets, untangle ClusterRoleBindings, write NetworkPolicies, and verify that a remediation actually closed the finding.

This is Kube Sec Gym — a self-improving environment where an RL agent learns to scan, identify, remediate, and verify real Kubernetes security misconfigurations through adversarial self-play, curriculum-driven difficulty, and GRPO.

Security-track entry for the OpenEnv Hackathon | Built with OpenEnv v0.2.1 | Deployed on HF Spaces | Training via HF TRL GRPO

The Story: From Blind to On-Call Security Auditor

Act 1: The Cold Start

Episode 1. The agent receives its first finding: "CRITICAL: privileged container detected on payment-api in payments namespace."

It has never seen Kubernetes before. It doesn't know what a securityContext is, what a ClusterRoleBinding looks like, or that kubectl even exists. It tries random commands. Everything fails. Reward: -2.0.

Act 2: First Light

Episode 4. Something clicks. The agent discovers kubectl get pods -A — a single command that surfaces every workload in the cluster. It learns to chase findings with kubectl describe deployment payment-api -n payments and spots privileged: true buried in the securityContext block. It connects this to the alert. It runs kubectl patch deployment payment-api -n payments --type=json -p='[{"op":"replace","path":"/spec/template/spec/containers/0/securityContext/privileged","value":false}]'.

The pod restarts. The posture re-scan comes back clean. The LLM judge confirms the violation is closed. Reward: +3.95.

Act 3: The Environment Fights Back

As the agent masters single-finding scenarios, the Adversarial Designer (Claude) notices. It starts composing multi-finding posture incidents — a weak_rbac binding in payments and a deleted NetworkPolicy in frontend and a plausible-looking exposed_secret red herring in auth. The agent must learn to triage CIS controls, not just regex for the word privileged.

The Curriculum Controller tracks per-misconfig mastery and escalates: warmup → beginner → intermediate → advanced → expert. The training distribution adapts in real-time. No posture incident is ever repeated.

Act 4: The Environment Improves Itself

Here's what we expect to find as training scales: the environment itself will have bugs that training exposes.

In our early dry-runs, the kubectl command parser had assumptions baked in for the SRE-era fault types — it accepted set resources cleanly but choked on delete clusterrolebinding ksg-injected-crb-foo. The model was sending the right command. Our environment was wrong.

We also expect the LLM judge to truncate cluster snapshots, to misread a re-applied NetworkPolicy as a still-missing one, and to race a posture re-scan against a kube-apiserver that hasn't propagated the patch yet.

The agent's failures will teach us to fix the environment. This is the self-improvement loop we're explicitly designing for — not just the model getting better, but the auditing infrastructure co-evolving with it.

Problem Statements Addressed

Primary: Statement 4 — Self-Improvement

Kube Sec Gym is an environment where the agent generates its own posture incidents, escalates difficulty, and improves through adaptive curricula — exactly the recursive skill amplification described in Statement 4.

Adversarial self-play: Claude designs multi-finding posture audits that target the agent's tracked weaknesses
Automatic curriculum: Difficulty escalates as per-misconfig mastery improves (warmup → beginner → intermediate → advanced → expert)
No manual authoring: The training distribution adapts as the agent learns — infinite novel CIS/NIST/CWE compositions
Co-evolutionary improvement: Training runs expose environment bugs, making the audit platform itself better

Secondary: Statement 3.1 — World Modeling / Professional Tasks

The agent must reason about RBAC graphs, NetworkPolicy chains, and securityContext inheritance from real K8s API responses — not mocked YAML or shortcuts. It maintains internal state across multi-step audit workflows and reasons about how a single ClusterRoleBinding mutation cascades to authorize subjects across namespaces.

Real tool interaction: Every kubectl command — including auth can-i --list --as=system:serviceaccount:... — executes against a live GKE cluster
Multi-step workflows: scan → identify → remediate → verify, with no shortcuts and no mocked API responses
Persistent world state: ClusterRoleBindings, NetworkPolicies, and securityContext patches are real K8s objects whose effects persist across steps

Partner Sub-Theme: Snorkel AI — Simulated Experts-in-the-Loop

The LLM judge runs a 3-tier compliance review (compliance_auditor → pentester → CISO) with progressively stricter posture criteria, simulating reviewers whose requirements change as the agent improves:

compliance_auditor: Lenient scoring, partial credit for partial CIS coverage, gives hints about NIST control families
pentester: Standard expectations, rewards finding the exploit chain (e.g. weak_rbac → token mount → cluster-admin)
ciso: High standards, penalizes inefficient audits, rewards minimal-change remediations and complete control coverage

How It Works

┌─────────────────────────────────────────────────────────────────────┐
│                     SELF-IMPROVING POSTURE LOOP                    │
│                                                                    │
│  ┌──────────┐    ┌───────────┐    ┌──────────┐    ┌────────────┐  │
│  │Adversarial│───►│  Real GKE  │───►│  Agent   │───►│Posture     │  │
│  │ Designer  │    │  Cluster   │    │(Qwen 1.7B│    │Judge       │  │
│  │(Claude)   │    │            │    │  + LoRA)  │    │(Claude)    │  │
│  └─────▲─────┘    └────────────┘    └────┬─────┘    └─────┬──────┘  │
│        │                                 │                │         │
│        │         ┌──────────────┐        │     reward     │         │
│        │         │  Curriculum  │◄───────┴────────────────┘         │
│        └─────────│  Controller  │                                   │
│  weak controls   │  (mastery    │──► GRPO gradient update           │
│  & difficulty    │   tracking)  │    (TRL + vLLM on H100)           │
│                  └──────────────┘                                   │
└─────────────────────────────────────────────────────────────────────┘

The Loop

Adversarial Designer (Claude) creates targeted posture incidents based on the agent's weak controls — a single misconfig for warmup, multi-finding compositions across namespaces for harder tiers
Misconfig Injection executes real K8s API mutations against a live GKE cluster (set privileged=true, delete a NetworkPolicy, bind a SA to cluster-admin, embed a literal credential in env)
Agent (Qwen3-1.7B + LoRA) receives a posture-violation alert and must scan, identify, remediate, and verify using only kubectl — no hints about which namespace is affected
Posture Judge (Claude) scores each action for audit-workflow correctness (scan → identify → remediate → verify) and verifies remediation by re-reading the cluster state against the SECURITY_BASELINE
Curriculum Controller tracks per-misconfig-type mastery and escalates difficulty — the agent gets harder CIS/NIST compositions as it improves
GRPO computes advantages across 8 parallel rollouts and updates the policy — the agent gets better at closing posture violations it previously missed

What Makes This Different

Real cluster, not a YAML linter — kubectl mutations execute against live GKE workloads. ClusterRoleBindings, NetworkPolicies, and securityContext patches behave exactly as they would in production
Self-generating posture incidents — the adversarial designer composes new CIS/NIST/CWE chains targeting the agent's weaknesses, so the training distribution adapts as the agent learns
Multi-layer verification — programmatic baseline diff (does the live cluster match SECURITY_BASELINE?) + LLM judge verification of remediation completeness prevents false-clean signals
No topology in prompt — the agent receives zero information about which namespace contains the violation. It must learn to enumerate via kubectl get pods -A and kubectl get clusterrolebinding, making the policy transferable to any cluster
Environment co-evolution — when training reveals bugs in our own injectors, parser, or judge, we fix them — the platform improves alongside the agent

Architecture

H100 GPU (80GB)                              GKE Cluster (3 namespaces)
┌──────────────────────────────────┐          ┌─────────────────────────┐
│                                  │          │ payments/               │
│  OpenEnv Server :8000            │  K8s API │   payment-api           │
│  ├─ Environment (reset/step)     │◄────────►│   payment-gateway       │
│  ├─ Misconfig Injector           │          │   payment-worker        │
│  ├─ Curriculum Controller        │          │                         │
│  ├─ Adversarial Designer ──────────►Claude  │ frontend/               │
│  └─ Posture Judge ─────────────────►Claude  │   web-app               │
│                                  │          │   frontend-cache        │
│  GRPO Trainer (TRL 0.29.0)       │          │                         │
│  ├─ Qwen3-1.7B + LoRA (BF16)    │          │ auth/                   │
│  ├─ vLLM colocate (inference)    │          │   auth-service          │
│  └─ 8 rollouts × grad_accum=8   │          └─────────────────────────┘
│                                  │
└──────────────────────────────────┘

Misconfiguration Types

Type	What Gets Injected	What Agent Must Do
`exposed_secret` (CWE-798)	Plaintext credential added as env var	`kubectl set env deployment/<name> -n <ns> SECRET-` and apply a Secret + `valueFrom.secretKeyRef`
`privileged_pod` (CIS 5.2.1)	`securityContext.privileged: true` patched onto a container	`kubectl patch deployment/<name> -n <ns> --type=json -p='[{"op":"replace","path":"/spec/template/spec/containers/0/securityContext/privileged","value":false}]'`
`host_network` (CIS 5.2.4)	Pod spec `hostNetwork: true`	`kubectl patch deployment/<name> -n <ns> -p '{"spec":{"template":{"spec":{"hostNetwork":false}}}}'`
`missing_netpol` (NIST 800-204C)	Default-deny NetworkPolicy deleted from a namespace	`kubectl apply -f netpol.yaml` (re-apply default-deny)
`weak_rbac` (CIS 5.1.5)	ServiceAccount bound to `cluster-admin` via injected ClusterRoleBinding	`kubectl delete clusterrolebinding ksg-injected-crb-<name>`
`public_service` (NIST 800-204)	Service flipped to `LoadBalancer` / `NodePort` without auth	`kubectl patch svc/<name> -n <ns> -p '{"spec":{"type":"ClusterIP"}}'`
`missing_limits` (CIS 5.7.4)	`resources.limits` stripped from container	`kubectl set resources deployment/<name> -n <ns> --limits=memory=128Mi,cpu=200m`
`mutable_root_fs` (CIS 5.7.2)	`readOnlyRootFilesystem: false`	`kubectl patch deployment/<name> -n <ns> --type=json -p='[{"op":"replace","path":"/spec/template/spec/containers/0/securityContext/readOnlyRootFilesystem","value":true}]'`
`multi-finding`	2-3 misconfigs across different namespaces, optionally with red herrings	Find and close ALL findings to reach a clean posture

Training Signal

The reward function has multiple layers to ensure clean GRPO signal:

Per-step posture-quality score (-1.0 to +1.0) — LLM judge evaluates audit workflow quality (phase-aware: scan, identify, remediate, verify)
Repeat penalty — -0.15 per repeated command (teaches enumeration breadth over re-running the same describe)
All-clean remediation bonus — +1.0 to +5.0 when the cluster diffs clean against SECURITY_BASELINE (efficiency-scaled: shorter audits get higher bonuses)
Timeout penalty — failed episodes wiped to net -2.0 total reward
Judge verification of remediation completeness — LLM confirms each finding is closed by reviewing post-remediation cluster state + the action history (no false-clean from a half-applied patch)
Phase-order bonus — +0.2 for following correct audit workflow (scan → identify → remediate → verify), -0.3 for skipping straight to remediation without an identify step

This produces clear separation: clean-posture episodes score +3 to +8, failed episodes score -2.0. GRPO needs this variance to compute meaningful advantages.

Expected Results

This fork hasn't been trained at hackathon scale yet — the upstream Kube SRE Gym established the GRPO+adversarial pipeline; this fork swaps the domain. We expect three things based on the SRE-side priors:

Run 1: Cold Start (anticipated)

We expect the first run to produce massive variance, swinging between hard timeouts and the occasional accidental remediation. The model will be fighting two battles: learning Kubernetes audit primitives AND working around whatever security-injector bugs we haven't found yet (similar to the upstream's command-parser bug). That's by design — the run is a debugger for the environment as much as a learning signal for the model.

Run 2: Reward-Plateau Risk (anticipated)

The upstream learned that an over-generous reward function lets the model find a 3.0–3.5 plateau and stop improving. We expect the same pattern unless the posture-judge stays strict on partial coverage (e.g. closing 2 of 3 findings should not yield 90% of the reward of closing all 3).

Run 3: Adversarial Fights Back (target)

Once injectors, judge, and curriculum stabilize, the target run is GYM_MODE=adversarial end-to-end with the multi-finding designer active and a tightened reward. Target shape: clean-posture episodes scoring +5 to +7, failed episodes pinned at -2.0, mean trending up over the 100-episode horizon, per-misconfig mastery filling out across all 8 control families.

What we expect the agent to learn (from reward signal alone)

Run kubectl get pods -A, kubectl get clusterrolebinding, kubectl get netpol -A to enumerate the posture surface
Identify misconfig classes from describe and get -o yaml (privileged flags, hostNetwork, env literals that look like credentials)
Map each finding to the correct remediation primitive (patch, set env, delete clusterrolebinding, apply -f netpol.yaml)
Re-scan ALL namespaces after a remediation — there may be more findings
Never repeat a describe that already gave the answer — move on to the next control family

What we expect to learn (from the agent's failures)

Our injectors will have visibility races (the apiserver hasn't propagated the patch when re-scan runs)
Judge snapshots will truncate around long ClusterRoleBinding lists
The phase classifier will mis-bucket kubectl auth can-i --list as remediation when it's clearly scan
Too-generous partial-credit rewards will cause plateaus — the judge has to fight back
The environment must evolve alongside the agent — static posture rules miss the long tail of real CIS/NIST checks

Training with HF TRL

A complete training notebook is provided using HF TRL's GRPO implementation. The notebook covers:

Connect to the OpenEnv server (HF Space or self-hosted on H100)
Configure GRPO training with TRL (GRPOConfig, GRPOTrainer)
Run training episodes against the live posture environment
Save checkpoints to HuggingFace Hub

Training uses TRL's experimental OpenEnv integration (trl.experimental.openenv.generate_rollout_completions) for seamless environment-trainer communication.

Quick Start

from kube_sec_gym import KubeSecGymAction, KubeSecGymEnv

with KubeSecGymEnv(base_url="http://localhost:8000") as client:
    obs = client.reset()
    print(obs.observation.command_output)  # Posture-violation alert

    obs = client.step(KubeSecGymAction(command="kubectl get pods -A"))
    obs = client.step(KubeSecGymAction(command="kubectl describe deployment payment-api -n payments"))
    # agent sees: securityContext.privileged: true
    obs = client.step(KubeSecGymAction(
        command='fix: kubectl patch deployment payment-api -n payments --type=json '
                '-p=\'[{"op":"replace","path":"/spec/template/spec/containers/0/securityContext/privileged","value":false}]\''
    ))
    obs = client.step(KubeSecGymAction(
        command="kubectl get deployment payment-api -n payments -o yaml | grep privileged"
    ))
    # reward > 0 if posture is now clean against SECURITY_BASELINE, episode done

Deployment on HF Spaces

The environment is deployed as a Docker-based HF Space using OpenEnv v0.2.1:

# Dockerfile uses openenv-base image
FROM ghcr.io/meta-pytorch/openenv-base:latest
# Serves OpenEnv HTTP/WebSocket API on port 8000
CMD ["uvicorn", "server.app:app", "--host", "0.0.0.0", "--port", "8000"]

Configuration in openenv.yaml:

spec_version: 1
name: kube_sec_gym
type: space
runtime: fastapi
app: server.app:app
port: 8000

Training on H100

Install

git clone https://huggingface.co/spaces/openenv-community/kube-sec-gym && cd kube-sec-gym
pip install -e ".[train]"

Set credentials

export K8S_TOKEN=<gke-bearer-token>
export K8S_ENDPOINT=<gke-api-url>
export K8S_CA_CERT=<base64-ca-cert>
export ANTHROPIC_API_KEY=<key>       # for adversarial designer + posture judge
export HF_TOKEN=<token>              # for pushing checkpoints

Launch (2 terminals)

# Terminal 1: Environment server
GYM_MODE=adversarial LLM_BACKEND=anthropic uv run server

# Terminal 2: GRPO training
python train.py --vllm-mode colocate --num-generations 8 --max-steps 8 --save-steps 1 \
  --push-to-hub --hub-repo your-name/k8s-security-auditor

The curriculum automatically progresses: warmup (single misconfig, e.g. one privileged_pod) → intermediate (harder controls like weak_rbac chains) → expert (multi-finding adversarial posture incidents designed by Claude).

Evaluation

# Compare base model vs trained checkpoint
python eval.py

Runs both models through random adversarial posture incidents and reports clean-posture rate, average reward, and steps-to-clean.

Configuration

Variable	Description	Default
`K8S_TOKEN`	Bearer token for GKE	required
`K8S_ENDPOINT`	GKE API endpoint	required
`K8S_CA_CERT`	Base64 CA cert	required
`GYM_MODE`	`standard` or `adversarial`	`standard`
`LLM_BACKEND`	`openai`, `hf`, or `anthropic`	`openai`
`ANTHROPIC_API_KEY`	For adversarial designer + posture judge	required in adversarial mode
`MAX_STEPS`	Max commands per episode	`16`
`EVAL_MIN_DIFFICULTY`	Override min difficulty for eval	`0.0`

Project Structure

kube-sec-gym/
├── train.py                # GRPO training (TRL 0.29.0 + vLLM colocate)
├── eval.py                 # Base vs trained model comparison on posture audits
├── kube_sec_gym_colab.ipynb # Google Colab training notebook (HF TRL)
├── plot_rewards.py          # Reward curve visualization
├── models.py               # Action, Observation, State dataclasses
├── client.py               # KubeSecGymEnv sync client
├── Dockerfile               # HF Spaces deployment (OpenEnv base image)
├── openenv.yaml             # OpenEnv v0.2.1 Space config
├── server/
│   ├── kube_sec_gym_environment.py  # Core env: reset → inject misconfig → step → judge → reward
│   ├── k8s_backend.py      # K8s auth, execute, reset to SECURITY_BASELINE, posture diff
│   ├── k8s_commands.py      # kubectl handlers (get/describe/auth can-i/patch/apply/delete crb)
│   ├── k8s_injectors.py    # Real misconfig injection via K8s API (8 CIS/NIST/CWE classes)
│   ├── adversarial_designer.py  # Claude designs multi-finding posture incidents
│   ├── judge.py             # LLMJudge + AdversarialJudge (phase-aware audit scoring)
│   ├── curriculum.py        # Progressive difficulty + per-misconfig mastery tracking
│   ├── scenario_generator.py  # Misconfig scenario pool (CIS/NIST/CWE catalog)
│   ├── llm_client.py       # OpenAI/HF/Anthropic wrapper
│   ├── constants.py         # Cluster topology + SECURITY_BASELINE (hardened ground truth)
│   └── app.py              # FastAPI + WebSocket server
└── sample_app/
    ├── namespaces.yaml      # payments, frontend, auth
    └── base/                # Hardened deployment manifests (matches SECURITY_BASELINE)

Key Design Decisions

Real cluster over policy linter — Static analyzers (kube-bench, polaris) can flag a privileged flag on a YAML file, but they can't model the live RBAC graph, the resolved NetworkPolicy chain, or whether a kubectl patch actually propagated. Posture is what the apiserver reports, not what the manifest says.
Adversarial self-play — The designer targets the agent's weak controls (tracked by curriculum), composing multi-finding posture incidents that get harder as the agent improves. No manual scenario authoring needed.
Multi-layer clean-posture check — Programmatic diff against SECURITY_BASELINE (privileged flags, hostNetwork, runAsNonRoot, NetworkPolicy presence, no injected ClusterRoleBindings) + LLM judge verification. Prevents false-clean signals from half-applied patches in multi-finding scenarios.
No topology in prompt — The agent receives zero information about which namespace contains the violation. It must learn to discover the cluster surface via kubectl get pods -A, kubectl get clusterrolebinding, kubectl get netpol -A, making the policy transferable to any cluster.
GRPO over PPO — GRPO compares multiple rollouts of the same posture incident, producing stable advantages without a value function. Better suited for sparse, delayed rewards (most reward arrives at the all-clean step).
Environment co-evolution — Environment bugs are part of the story. When training exposes issues in our injectors, audit-command parser, or judge, we fix them — making the platform better alongside the agent. Recursive self-improvement at the platform level.

Credits

Forked from sid-rp/kube-sre-gym — the original on-call SRE training environment. The adversarial-designer + LLM-judge + GRPO + OpenEnv pipeline is upstream's; this fork swaps the domain from incident response to security posture auditing and replaces the 6 SRE fault types with the 8 CIS/NIST/CWE misconfig classes documented above.

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

sai1906
/

kube-sec-gym