OpenEnv-IncidentTriage

OpenEnv-IncidentTriage is a reinforcement-learning benchmark environment that simulates an on-call ML engineer responding to production incidents. An AI agent investigates synthetic logs, metrics, and alerts to diagnose issues, and then writes a diagnosis, fix plan, and postmortem — all within a specified step budget. It provides partial-credit rewards and features varied difficulties for evaluating agentic reasoning.

Architecture Diagram

+-------------------+       +---------------------+       +-----------------------+
|  Agent (LLM)      | <---> |  FastAPI Server     | <---> |  IncidentTriageEnv    |
|  (inference.py)   |       |  (app.py)           |       |  (environment.py)     |
+-------------------+       +---------------------+       |                       |
                            | - Rate Limiter      |       | - Step Budget         |
                            | - Secure Validation |       | - Tool Dispatch       |
                            | - State Management  |       | - Score Grader        |
                            +---------------------+       +-----------------------+
                                                                  |      |
                                        +-------------------------+      |
                                        |                                |
                                +-------v-------+                +-------v-------+
                                |  Tasks        |                |  Signals      |
                                | (task_easy,   |                | (JSON logs,   |
                                |  task_medium, |                |  metrics,     |
                                |  task_hard)   |                |  alerts)      |
                                +---------------+                +---------------+

Task Table

ID	Difficulty	Max Steps	Optimal Steps	Reward Threshold
task_easy	Easy	10	4	0.60
task_medium	Medium	20	8	0.55
task_hard	Hard	35	14	0.50

Action Space

Tool	Args	Description
read_log	{"signal_id": "str"}	Retrieve a server or application log file by signal ID.
read_metric	{"signal_id": "str"}	Retrieve a time-series metric by signal ID.
read_alert	{"signal_id": "str"}	Retrieve an alert notification by signal ID.
query_runbook	{"keyword": "str"}	Search all runbooks for a keyword.
write_diagnosis	{"text": "str"}	Submit your root-cause diagnosis.
write_fix	{"text": "str"}	Submit your fix plan with concrete actions.
write_postmortem	{"text": "str"}	Submit your postmortem report.
done	{}	End the episode and trigger final grading.

Observation Space

Field	Type	Description
task_id	string	The active task configuration.
step	integer	Number of steps taken (≥ 0).
budget_remaining	integer	Steps remaining in the episode (≥ 0).
available_signals	list[string]	List of signal IDs the agent can query.
last_tool_result	string	Output from the previous action.
incident_summary	string	High-level summary of the incident event.
done	boolean	Indicates if the episode has ended.

Reward Function

The reward function relies on the quality of the submitted diagnosis, fix and postmortem, and penalises inefficiency. Total reward formula bounded in [0.0, 1.0]:

## Weights:
## diagnostic_accuracy = 0.40
## fix_correctness = 0.25
## postmortem_quality = 0.20
## step_efficiency = 0.15

raw = sum(weight * getattr(breakdown, key) for key, weight in REWARD_WEIGHTS.items())
penalty = max(-0.30, -0.01 * max(0, steps_taken - optimal_steps))
total = max(0.0, min(1.0, raw + penalty))

• Diagnosis Accuracy: NLP-based rubric to identify appropriate root causes and ignore red herrings.
• Fix Correctness: Check against required remediation actions and specific resolution commands.
• Postmortem Quality: Verifies presence of key document sections (Timeline, Root Cause, Impact, Remediation, Prevention, and Lessons Learned).
• Step Efficiency: Rewards resolving incidents within the optimal step window, degrading linearly down to 0 if max_steps are reached. Negative penalty for excessive steps is capped at -0.30.

Setup Instructions

1. Install locally:

   pip install -r requirements.txt
   

2. Docker Build:

   docker build -t openenv-incident-triage .
   

3. Docker Run:

   docker run -p 7860:7860 openenv-incident-triage
   

   The backend API will now be available on port 7860.

Running Inference

The inference.py provides the baseline LLM execution path solving all three tasks sequentially. Output adheres precisely to the standardized hackathon [START]/[STEP]/[END] format.

Environment Variables:

Variable	Default Value	Description
API_BASE_URL	https://router.huggingface.co/v1	OpenAI API compatible endpoint.
MODEL_NAME	Qwen/Qwen2.5-72B-Instruct	Model identifier to use.
HF_TOKEN	None (Required)	Access token for Hugging Face inference.
ENV_BASE_URL	http://localhost:8000	The deployed OpenEnv FastAPI server URL.
TASK_NAME	task_easy	Task ID to run individually (though inference.py runs all).
MAX_STEPS	20	Override to set max steps in the inference wrapper.

Example Command:

export HF_TOKEN="hf_yourtokenhere"
export ENV_BASE_URL="http://localhost:7860"
python inference.py

Validation

To evaluate environment compliance against OpenEnv specifications, ensure your backend server is deployed and run:

validate-submission.sh <your-hf-space-url>

You can also run validation using the openenv CLI tool if installed:

openenv validate

Running Tests

Verify environment integrity and high coverage with:

pytest tests/ -v --cov=openenv_incident_triage --cov-report=term

Security Notes

OpenEnv-IncidentTriage employs strict runtime boundaries to execute safely:

• No Remote Code Execution: Rejects eval/exec strings and prevents interpolation of inputs into OS-level commands or paths.
• Input Sanitization: Detects and denies shell injection syntax (;&|()), path traversal (../), null bytes, and excessively large payloads.
• Secrets Scanning: Heuristically warns about accidental leakage of API/HF/AWS tokens in agent submissions.
• Pydantic Validation: All inputs/outputs strictly typed; rejecting unknown configuration settings (extra="forbid").
• Deployment Hardening: Runs via non-root user appuser with a read-only signals/ store.

Contributing

You can expand this environment by authoring new scenarios:

1. Generate signal JSON records mapping simulated time to anomaly indicators in signals/.
2. Subclass BaseTask in a new file (e.g., tasks/task_custom.py):
   • Define signal_ids matching your new signal file IDs.
   • Implement logic for grade_diagnosis, grade_fix, and get_ground_truth.
3. Register your class in tasks/__init__.py.
4. Run pytest tests/ to confirm your grading criteria executes securely.

TO RUN $env:HF_TOKEN="your_working_token"; $env:ENV_BASE_URL="http://localhost:7860"; .\venv\Scripts\activate; python inference.py