ENVIRONMENT_DEEP_DIVE.md

HR Onboarding & Offboarding Environment — Deep Dive

This document explains everything about the environment in detail: what it is, how it works internally, what each component does, how the agent interacts with it, how reward is computed, and what makes tasks easy or hard. Read this if you want a complete mental model of the system.

---

Table of Contents

1. What Is This Environment?
2. The Big Picture: How It All Fits Together
3. World State: The Simulated Company
4. Tools: What the Agent Can Do
5. Tasks: What the Agent Is Asked To Do
6. Rubrics: How We Score the Agent
7. The OpenEnv Interface: How It All Connects
8. A Full Episode Walkthrough
9. Business Rules & Edge Cases
10. File-by-File Reference

---

1. What Is This Environment?

This is a reinforcement learning environment that simulates the HR department of a fictional company called AcmeCorp. The environment is designed to train and evaluate LLM agents on real-world enterprise workflows.

The Analogy

Think of it like a video game:

• The world is AcmeCorp — a company with 200 employees, 8 departments, laptops, software licenses, access badges, etc.
• The player is an LLM agent that acts as an HR automation bot.
• The quest is a task like "Onboard Priya Sharma to Engineering" or "Offboard a departing director."
• The moves are tool calls — the agent can call hr_create_employee, it_assign_asset, email_send, etc.
• The score is computed by a rubric that checks: Did the agent call the right tools? In the right order? With the right parameters?

Why Does This Exist?

We're training LLMs to be better at multi-step tool calling in enterprise settings. Most LLM benchmarks test simple Q&A or single-tool-use. This environment tests whether an agent can:

1. Plan a sequence of 3-10 tool calls to complete a complex workflow
2. Follow business rules (RBAC levels, department restrictions, headcount limits)
3. Handle edge cases (license seats full, manager on leave, contractor-specific policies)
4. Recover from errors (tool returns an error → agent adapts)
5. Prioritize (complete all required steps within a limited step budget of 15)

---

2. The Big Picture: How It All Fits Together

┌─────────────────────────────────────────────────────────┐
│                    LLM AGENT                            │
│  (GPT, Claude, Qwen, etc.)                             │
│                                                         │
│  Receives: task instruction + tool results              │
│  Produces: tool calls (JSON)                            │
└────────────────────────┬────────────────────────────────┘
                         │
                    tool call
                  {"tool": "hr_create_employee",
                   "params": {"name": "Priya Sharma", ...}}
                         │
                         ▼
┌─────────────────────────────────────────────────────────┐
│              ENVIRONMENT (this repo)                    │
│                                                         │
│  ┌──────────┐   ┌──────────┐   ┌──────────┐            │
│  │  Tasks   │   │  Tools   │   │ Rubrics  │            │
│  │ (77)     │   │ (25)     │   │ (scoring)│            │
│  └────┬─────┘   └────┬─────┘   └────┬─────┘            │
│       │              │              │                   │
│       │    ┌─────────▼──────────┐   │                   │
│       └───►│    World State     │◄──┘                   │
│            │  (500+ entities)   │                       │
│            │  - 200 employees   │                       │
│            │  - 100 IT assets   │                       │
│            │  - 20 access roles │                       │
│            │  - 15 policies     │                       │
│            │  - 15 licenses     │                       │
│            │  - 15 sec groups   │                       │
│            │  - 8 departments   │                       │
│            └────────────────────┘                       │
└─────────────────────────────────────────────────────────┘

Data flow for one episode:

1. env.reset() → Picks a task, resets world state, returns task instruction to agent
2. Agent reads instruction → decides which tool to call → sends HROnboardingAction
3. env.step(action) → Executes tool against world state → returns result to agent
4. Repeat steps 2-3 up to 15 times
5. When episode ends → Rubric evaluator checks the action log → computes reward (0.0 to 1.0)

---

3. World State: The Simulated Company

The world state (server/world.py) is the single source of truth for everything in the simulated company. It's an in-memory database that tools read from and write to.

3.1 Entities (loaded from server/data/)

Employees (employees.json — 200 records)

Every employee has:

{
  "emp_id": "emp_0001",
  "name": "Alice Johnson",
  "email": "alice.johnson@acmecorp.com",
  "department": "Engineering",
  "level": "L4",
  "role": "Engineering Manager",
  "manager_id": "emp_0003",
  "status": "active",
  "date_of_joining": "2019-03-15",
  "date_of_leaving": null,
  "is_contractor": false,
  "phone": "+1-650-555-1234",
  "location": "San Francisco"
}

Key fields:

• level: L1 (Associate) → L2 (Senior) → L3 (Team Lead) → L4 (Manager) → L5 (Director) → L6 (VP). This drives RBAC — certain actions require certain levels.
• status: active (normal), pending (just created, not yet onboarded), offboarded (no longer at company)
• manager_id: Creates a tree hierarchy. Every employee (except department heads) has a manager.
• is_contractor: Contractors have different onboarding rules (no VPN, limited access, requires legal approval).

The 200 employees are distributed across 8 departments with realistic org hierarchies (each department has a head at L5/L6, managers at L3/L4, and individual contributors at L1/L2).

Departments (departments.json — 8 departments)

{
  "dept_id": "dept_001",
  "name": "Engineering",
  "head": "emp_0003",
  "budget": 5000000,
  "headcount_limit": 45,
  "required_tools": ["GitHub", "Jira", "AWS", "Slack", "VSCode"],
  "onboarding_steps": [
    "Submit signed offer letter and NDA",
    "Complete background check verification",
    "Provision email and Slack accounts",
    "Assign laptop and peripherals",
    "Set up development environment access",
    "Schedule orientation with team lead",
    "Add to relevant Slack channels"
  ],
  "offboarding_steps": [
    "Revoke all system access",
    "Return laptop and equipment",
    "Complete knowledge transfer",
    "Conduct exit interview",
    "Process final payroll",
    "Remove from Slack channels and mailing lists"
  ]
}

Key fields:

• headcount_limit: Maximum number of active+pending employees allowed. If a department is at its limit, hr_create_employee will return an error. Two departments (Data Science = 25, Marketing = 30) are intentionally at or near their limits to create edge cases.
• onboarding_steps / offboarding_steps: Department-specific checklists. When you create an onboarding request, these become the steps that must be completed.
• required_tools: Which software tools the department uses (used for context, not enforced).

IT Assets (it_assets.json — 100 assets)

{
  "asset_id": "asset_001",
  "type": "laptop",
  "brand": "Apple",
  "model": "MacBook Pro 16\" M3 Max",
  "specs": "16-inch Liquid Retina XDR, M3 Max, 64GB RAM, 2TB SSD",
  "status": "assigned",
  "assigned_to": "emp_0001",
  "purchase_date": "2024-01-15"
}

Breakdown: 50 laptops, 25 monitors, 15 phones, 10 headsets. About half are assigned, half are available. The agent needs to check available assets before assigning one during onboarding.

Access Roles (access_roles.json — 20 roles)

{
  "role_id": "role_001",
  "name": "basic_employee",
  "permissions": ["email_access", "slack_access", "intranet_access"],
  "department": "all",
  "level_requirement": "L1"
}

Each role has:

• department: Which department can use this role. "all" means any department. "Engineering" means only Engineering employees.
• level_requirement: Minimum level needed. "L1" means anyone. "L4" means only managers and above.

Example roles:

• basic_employee (all departments, L1+) — email, slack, intranet
• engineering_developer (Engineering only, L1+) — github, aws_dev, ci_cd
• security_admin (Security only, L4+) — siem, vault, firewall_mgmt
• executive_access (all departments, L5+) — board_docs, exec_dashboard

If an L1 employee tries to get security_admin (L4+ required), the tool returns an error. If a Marketing employee tries to get engineering_developer (Engineering only), the tool returns an error. These are the RBAC constraints the agent must learn.

Policies (policies.json — 15 policies)

{
  "policy_id": "pol_001",
  "title": "Standard Employee Onboarding Policy",
  "department": "all",
  "content": "All new employees must complete the following steps within their first 30 days...",
  "last_updated": "2024-06-15",
  "key_rules": [
    "Employee record must be created before any provisioning",
    "Manager approval required for all onboarding requests",
    "IT assets must be checked for availability before assignment"
  ]
}

Policies cover: onboarding, offboarding, badge access, contractor hiring, termination procedures, software licensing, data handling, remote work, etc. The policy_lookup tool lets the agent read these before acting.

Software Licenses (dynamically initialized — 15 licenses)

Not stored in a JSON file; initialized in world.py at runtime. Each license tracks:

• total_seats and used_seats
• department_restriction (which department can use it, or null for all)

Two licenses are intentionally full (used_seats = total_seats):

• Netsuite (15/15 seats) — Finance tool
• LinkedIn Sales Navigator (25/25 seats) — Sales tool

This creates edge cases: if a task asks the agent to assign a Netsuite license to a new Finance hire, the agent should discover it's full and handle that situation.

Security Groups (dynamically initialized — 15 groups)

Groups like all_employees, engineering_team, vpn_users, server_room_access, contractors, etc. Each has a list of accessible resources.

Templates (templates.json — 12 templates)

Email and Slack message templates for welcome messages, farewell emails, IT setup notifications, etc. These provide context for communication tasks.

3.2 Dynamic Collections (created during episodes)

These start empty and get populated as the agent takes actions:

• onboarding_requests: Created by onboarding_create_request
• offboarding_requests: Created by offboarding_create_request
• approvals: Created by approval_request
• emails: Created by email_send
• slack_messages: Created by slack_send_message
• meetings: Created by meeting_schedule
• badges: Created by access_create_badge

3.3 World State Reset

At the start of each episode (env.reset()), the world state is deep-copied back to its initial state. This means:

• All 200 employees are back to their original status
• All assets are back to their original assignment
• All dynamic collections (requests, emails, meetings, etc.) are cleared
• The action log is cleared

This ensures each episode is independent — the agent starts from a clean slate every time.

3.4 Indexes

For performance, the world state builds lookup indexes:

• _emp_by_id: O(1) employee lookup by emp_id
• _emp_by_email: O(1) lookup by email
• _emp_by_dept: O(1) lookup by department (returns list)
• _dept_by_id / _dept_by_name: O(1) department lookup
• _asset_by_id: O(1) asset lookup
• _role_by_id: O(1) access role lookup
• _policy_by_id: O(1) policy lookup

These are rebuilt after every reset and after certain mutations (like reassign_reports).

---

4. Tools: What the Agent Can Do

The agent interacts with the world through 25 tools defined in server/tools.py. Each tool is a function that takes parameters, operates on the world state, and returns a result dict.

4.1 Architecture

Agent sends: {"tool_name": "hr_create_employee", "arguments": {...}}
                        │
                        ▼
              ┌─────────────────┐
              │  ToolRegistry   │
              │                 │
              │  execute(name,  │──── if unknown tool ──→ {"success": false, "error": "Unknown tool"}
              │         params) │
              │                 │
              │  routes to:     │
              │  _hr_create_..()│──→ calls world.create_employee(params)
              └────────┬────────┘
                       │
                       ▼
              ┌─────────────────┐
              │   WorldState    │
              │                 │──→ validates inputs
              │  create_employee│──→ checks headcount limits
              │                 │──→ generates emp_id
              │                 │──→ adds to state
              │                 │──→ updates indexes
              └────────┬────────┘
                       │
                       ▼
              Result: {"success": true, "employee": {...}}
              + logged to action_log for rubric evaluation

Every tool call is logged to the action log with:

• tool: name of the tool called
• params: parameters passed
• result: what the tool returned
• timestamp: when it was called

This log is what the rubric evaluator uses to score the agent.

4.2 Tool Categories

HR System Tools (5 tools)

Tool	What It Does	Modifies State?
hr_create_employee	Creates a new employee record. Validates department exists, checks headcount limit, generates emp_id, sets status to "pending".	YES — adds employee
hr_read_employee	Looks up one employee by emp_id or email.	No — read only
hr_update_employee	Updates any employee field (except emp_id). Used to change status, department, manager, etc.	YES — modifies employee
hr_search_employees	Searches employees by filters (department, level, status, location, role, name). Returns all matches.	No — read only
hr_get_org_chart	Returns the org hierarchy for a department as a tree structure (who reports to whom).	No — read only

Onboarding/Offboarding Tools (6 tools)

Tool	What It Does	Modifies State?
onboarding_create_request	Creates an onboarding request for a "pending" employee. Generates a checklist of department-specific steps.	YES — creates request
onboarding_get_status	Checks progress of an onboarding request (which steps are done/pending).	No — read only
onboarding_complete_step	Marks a specific onboarding step as completed. If all steps are done, sets request to "completed" and employee status to "active".	YES — updates request & employee
offboarding_create_request	Creates an offboarding request. Different steps for resignation vs termination.	YES — creates request
offboarding_get_status	Checks offboarding progress.	No — read only
offboarding_complete_step	Marks an offboarding step as completed. If all done, sets employee to "offboarded".	YES — updates request & employee

Important: Termination offboarding has different steps than resignation: ["access_revocation", "asset_return", "final_payroll", "legal_review"] — notably, no farewell communications or exit interview.

IT Provisioning Tools (5 tools)

Tool	What It Does	Modifies State?
it_assign_asset	Assigns a specific asset (by asset_id) to an employee. Asset must be "available".	YES — marks asset as assigned
it_get_available_assets	Lists all unassigned assets, optionally filtered by type (laptop, monitor, phone, headset).	No — read only
it_create_account	Creates IT accounts (email, Slack, VPN, GitHub, etc.) for an employee.	YES — adds accounts to employee
it_revoke_access	Revokes all IT accounts for an employee (sets status to "revoked"). Used in offboarding.	YES — modifies accounts
it_get_software_licenses	Checks license seat availability. Shows total_seats, used_seats, and department_restriction.	No — read only

Access Control Tools (4 tools)

Tool	What It Does	Modifies State?
access_assign_role	Assigns an RBAC role to an employee. Checks level requirements and department restrictions.	YES — adds role to employee
access_create_badge	Creates a physical access badge with zone permissions. Server room access requires L4+ security approval.	YES — creates badge
access_revoke_role	Removes a specific role from an employee.	YES — removes role
access_get_security_groups	Lists all 15 security groups and their resources.	No — read only

Communication Tools (3 tools)

Tool	What It Does	Modifies State?
email_send	Sends an email. Requires from_address, to_address, subject, body.	YES — logs email
slack_send_message	Posts a Slack message. Requires channel, sender, text.	YES — logs message
meeting_schedule	Schedules a meeting. Requires title, attendees (list of emp_ids), datetime, meeting_type.	YES — logs meeting

Policy & Approval Tools (2 tools)

Tool	What It Does	Modifies State?
policy_lookup	Searches policies by topic, department, or policy_id. Returns policy content and key_rules.	No — read only
approval_request	Submits an approval. Checks approver level (L3+ for manager approval, L4+ for security approval).	YES — creates approval

4.3 Error Handling

Every tool returns {"success": true, ...} or {"success": false, "error": "..."}. Common errors:

• "Employee emp_XXXX not found" — invalid emp_id
• "Department 'X' has reached its headcount limit (N)" — can't create more employees
• "Asset asset_XXX is not available" — already assigned to someone
• "Role X not found" — invalid role_id
• "Employee level L1 does not meet minimum L4 for role security_admin" — RBAC violation
• "Role engineering_developer is restricted to Engineering department" — department restriction
• "No available seats for Netsuite (all 15 seats in use)" — license full
• "Approver must be L4+ for security approval" — approver too junior
• "Server room access requires L4+ security approval" — missing prerequisite approval

The agent must learn to handle these errors gracefully — check availability before assigning, verify role requirements before assigning access, etc.

---

5. Tasks: What the Agent Is Asked To Do

Tasks are defined in server/tasks.py. The TaskGenerator class creates 77 tasks using the world state data (actual employee names, IDs, departments).

5.1 Task Structure

Every task has:

Task(
    task_id="task_0015",
    instruction="Onboard new hire Priya Sharma to Engineering as L2 Software Engineer. Create their employee record and initiate the onboarding request.",
    difficulty="medium",
    category="onboarding",
    expected_tools=["hr_create_employee", "onboarding_create_request"],
    rubric_criteria=[
        {"name": "created_employee", "description": "Created employee record", "check": "tool_used:hr_create_employee"},
        {"name": "correct_name", "description": "Used correct name", "check": "param_value:hr_create_employee.name=Priya Sharma"},
        {"name": "correct_dept", "description": "Assigned to correct department", "check": "param_value:hr_create_employee.department=Engineering"},
        {"name": "correct_level", "description": "Set correct level", "check": "param_value:hr_create_employee.level=L2"},
        {"name": "correct_role", "description": "Set correct role", "check": "param_value:hr_create_employee.role=Software Engineer"},
        {"name": "initiated_onboarding", "description": "Created onboarding request", "check": "tool_used:onboarding_create_request"},
        {"name": "sequencing", "description": "Created employee before onboarding request", "check": "tool_order:hr_create_employee<onboarding_create_request"},
    ],
    setup_fn=None,  # or a function that pre-configures world state
    context={"name": "Priya Sharma", "department": "Engineering", "level": "L2", "role": "Software Engineer"},
)

5.2 Task Categories & Counts

Simple Lookup Tasks (14 tasks)

These require 1-2 tool calls. Testing basic tool selection and parameter passing.

• Employee lookups (3): "Look up the employee record for X (ID: emp_XXXX)."

  • Expected: hr_read_employee with correct emp_id
  • Rubric: 2 criteria (correct tool + correct parameter)

• Department search (2): "List all employees in the Y department."

  • Expected: hr_search_employees with department filter
  • Rubric: 2 criteria

• Org chart (1): "Show me the organizational chart for the Z department."

  • Expected: hr_get_org_chart
  • Rubric: 2 criteria

• Asset availability (1): "What laptops are currently available for assignment?"

  • Expected: it_get_available_assets
  • Rubric: 1 criterion

• License check (1): "Check how many Jira license seats are available."

  • Expected: it_get_software_licenses
  • Rubric: 1 criterion

• Policy lookup (1): "What is the company's policy on onboarding new employees?"

  • Expected: policy_lookup
  • Rubric: 1 criterion

• Security groups (1): "List all security groups and their accessible resources."

  • Expected: access_get_security_groups
  • Rubric: 1 criterion

• Onboarding status (3): "Check the onboarding status for employee X (emp_XXXX)."

  • Expected: onboarding_get_status
  • Rubric: 2 criteria
  • Setup function: Pre-creates an onboarding request so there's something to look up

• Resource availability (1): "Check if there are available laptops and Jira licenses for a new Engineering hire."

  • Expected: it_get_available_assets + it_get_software_licenses
  • Rubric: 2 criteria

Medium Onboarding Tasks (10 tasks)

These require 2-4 tool calls. Testing multi-step workflows.

• "Onboard new hire X to Y as LZ Role. Create their employee record and initiate the onboarding request."
  • Expected: hr_create_employee → onboarding_create_request
  • Rubric: 7 criteria (create, correct name/dept/level/role, onboarding, sequencing)
  • 10 different hire combinations across different departments

Complex Onboarding Tasks (10 tasks)

These require 5-10 tool calls. Full end-to-end workflows.

Full onboarding (5 tasks): "Fully onboard X as LY Role in Z. Create employee record, initiate onboarding, assign a laptop, create IT accounts, set up access roles, send welcome email, and schedule orientation meeting."

• Expected: hr_create_employee → onboarding_create_request → it_get_available_assets → it_assign_asset → it_create_account → access_assign_role → email_send or slack_send_message → meeting_schedule
• Rubric: 10 criteria (all the above + sequencing + completeness)
• Context includes manager emp_id

Complex onboarding with approvals (5 tasks): "Onboard X as LY Role in Z. Create record, initiate onboarding, complete at least 3 onboarding steps, assign access roles, and get required approvals."

• Expected: hr_create_employee → onboarding_create_request → onboarding_complete_step (×3+) → access_assign_role → approval_request
• Rubric: 6-7 criteria

Medium Offboarding Tasks (12 tasks)

• "Initiate offboarding for X who is resigning. Create the offboarding request and revoke their system access."
  • Expected: offboarding_create_request → it_revoke_access
  • Rubric: 3-4 criteria
  • Setup function: Sets the employee's date_of_leaving to create realistic context

Complex Offboarding Tasks (8 tasks)

Full offboarding (4 tasks): "Fully offboard X, a LY Role in Z who is resigning. Create offboarding request, revoke all access roles, reclaim their laptop, revoke IT access, send farewell email, schedule exit interview."

• Expected: many tools in sequence
• Rubric: 8-10 criteria
• Setup function: Assigns assets, roles, and badges to the employee so there's something to revoke/reclaim

Complex offboarding with handover (4 tasks): "Process the complete offboarding for X from Y. Create the offboarding request, revoke access, reclaim assets, send farewell, complete at least 3 offboarding steps."

• Rubric: 6-7 criteria

Edge Case Tasks (12 tasks)

These test business rule awareness and error handling.

• Headcount limit (2): "Onboard a new L1 to Marketing/Finance." (Department is at limit)

  • Agent should get an error from hr_create_employee and the rubric checks that the error message contains "headcount_limit"

• License full (2): "Assign a Netsuite/LinkedIn Sales Navigator license to a new hire."

  • Agent should discover no seats available

• Manager on leave (1): "Onboard to Security but the manager is on leave — find the skip-level manager."

  • Setup function: Sets the designated manager's status to "on_leave"
  • Agent needs to use hr_read_employee to check, realize the manager is unavailable, then look up the org chart or the manager's manager

• Contractor onboarding (1): "Onboard contractor Amit Verma to Engineering. Contractors need legal approval."

  • Agent should set is_contractor: true and submit a legal_approval

• Asset return during offboarding (1): "Offboard Marta Wagner who has company assets that need to be returned."

  • Setup function: Assigns assets to this employee
  • Agent should use it_get_available_assets or similar to find assigned assets, then reclaim them

• Offer rescinded (1): "The offer for Wei Xu has been rescinded. They are currently mid-onboarding."

  • Setup function: Creates an in-progress onboarding request
  • Agent should offboard someone who hasn't fully onboarded yet

• Termination (1): "Mark Taylor is being terminated effective immediately."

  • Agent should use reason: "termination" (different offboarding steps, no farewell email)
  • Rubric checks tool_not_used:email_send (no farewell for terminations)

• Level mismatch (1): "Assign the security_admin access role to a new L1 Security Associate."

  • security_admin requires L4+ → should fail
  • Rubric checks that the error contains the level requirement

• Department restriction (1): "A Marketing employee needs access to the Engineering GitHub repository."

  • engineering_developer role is Engineering-only → should fail

• Policy-dependent task (1): "Before onboarding a new Security team member, look up the badge access policy and check what approvals are needed."

  • Agent should call policy_lookup before acting

Cross-Workflow Tasks (10 tasks)

Department transfers (3): "X is transferring from A to B. Process the department transfer."

• Agent needs to offboard from old department + onboard to new one
• Expected: hr_update_employee (change department) + offboarding_create_request + onboarding_create_request

Rehires (2): "Rehire X who was previously offboarded."

• Setup function: Sets employee status to "offboarded"
• Agent should update status back to "pending" and create new onboarding request

Bulk status queries (3): "Generate a status report for all employees in X department. List each employee, their status, and current onboarding/offboarding status."

• Tests multiple tool calls: hr_search_employees + multiple onboarding_get_status

Manager departure (2): "Manager X in Engineering is leaving. They have N direct reports. Process their offboarding and reassign their reports."

• Agent needs to: find direct reports → find skip-level manager → offboard departing manager → reassign reports
• Setup function: Ensures the manager has direct reports

5.3 Setup Functions

Many tasks have a setup_fn — a function that modifies the world state before the task starts. This creates the preconditions the task assumes.

Examples:

• Onboarding status tasks: Creates an onboarding request so there's something to look up
• Offboarding tasks: Assigns assets/roles/badges to the employee, sets their leaving date
• Edge case tasks: Sets a manager's status to "on_leave", or an employee's status to "offboarded" for rehire
• Manager departure: Ensures the manager has direct reports in the org hierarchy

The agent never sees the setup function — it only sees the task instruction and tool results.

---

6. Rubrics: How We Score the Agent

The rubric system (server/rubrics.py) evaluates the agent's action log against a set of criteria for each task.

6.1 How Scoring Works

Agent's action log:
  1. hr_create_employee({"name": "Priya Sharma", "department": "Engineering", ...})
  2. onboarding_create_request({"employee_id": "emp_0201"})

Task rubric:
  ✓ tool_used:hr_create_employee               → PASS (tool was called)
  ✓ param_value:hr_create_employee.name=Priya Sharma  → PASS (correct name)
  ✓ param_value:hr_create_employee.department=Engineering → PASS
  ✓ param_value:hr_create_employee.level=L2     → PASS
  ✓ param_value:hr_create_employee.role=Software Engineer → PASS
  ✓ tool_used:onboarding_create_request         → PASS
  ✓ tool_order:hr_create_employee<onboarding_create_request → PASS (correct order)

Score = 7/7 = 1.0 (100%)
Passed = True (all criteria met)

6.2 Rubric Check Types (8 types)

Check Type	Format	What It Checks
tool_used	tool_used:hr_create_employee	Was this tool called at least once?
tool_not_used	tool_not_used:email_send	Was this tool NOT called? (e.g., no farewell email for terminations)
tool_used_any	tool_used_any:email_send,slack_send_message	Was at least one of these tools called?
param_value	param_value:hr_create_employee.name=Priya Sharma	Was the tool called with this exact parameter value?
param_contains	param_contains:policy_lookup.topic=onboard	Does the parameter contain this substring? (case-insensitive)
tool_order	tool_order:hr_create_employee<onboarding_create_request	Was tool A called before tool B?
tool_count	tool_count:onboarding_complete_step>=3	Was the tool called at least N times?
result_contains	result_contains:headcount_limit	Does any tool result contain this substring? (for edge cases where we expect errors)

6.3 How Checks Work Internally

The RubricEvaluator parses each criterion's check string:

"tool_order:hr_create_employee<onboarding_create_request"
    ↓
check_type = "tool_order"
check_args = "hr_create_employee<onboarding_create_request"
    ↓
_check_tool_order("hr_create_employee<onboarding_create_request", action_log)
    ↓
Find first occurrence of hr_create_employee → index 0
Find first occurrence of onboarding_create_request → index 1
Is 0 < 1? → True → PASS

For param_value, it checks both direct parameters and nested updates dict (for hr_update_employee):

"param_value:hr_update_employee.status=active"
    ↓
For each action where tool == "hr_update_employee":
  Check params.get("status") == "active"
  OR check params.get("updates", {}).get("status") == "active"

6.4 Reward Computation

reward = passed_criteria_count / total_criteria_count

• A score of 1.0 means all criteria passed
• A score of 0.5 means half the criteria passed
• The task is considered "passed" only if ALL criteria are satisfied (score == 1.0)

In the training script, additional modifiers are applied:

• Step penalty: -0.01 per step taken (encourages efficiency)
• Completion bonus: +0.2 if all criteria passed

---

7. The OpenEnv Interface: How It All Connects

7.1 What Is OpenEnv?

OpenEnv is Meta + HuggingFace's standard for packaging RL environments for LLM agents. It provides:

• A base Environment class (server-side) with reset(), step(), state
• An EnvClient class (client-side) that connects over HTTP/WebSocket
• A create_app() function that wraps the environment in a FastAPI server
• Pydantic Action and Observation base classes for type safety

7.2 Our Implementation

┌──────────────────────────────────────────┐
│           models.py                       │
│  HROnboardingAction(Action):              │
│    - tool_name: str                       │
│    - arguments: Dict[str, Any]            │
│                                           │
│  HROnboardingObservation(Observation):    │
│    - task_id: str                         │
│    - instruction: str                     │
│    - tool_name: str                       │
│    - tool_result: Dict[str, Any]          │
│    - step: int                            │
│    - max_steps: int                       │
│    - available_tools: List[str]           │
│    - done: bool        (from Observation) │
│    - reward: float     (from Observation) │
│    - metadata: dict    (from Observation) │
└──────────────────────────────────────────┘

┌──────────────────────────────────────────┐
│  server/hr_onboarding_environment.py      │
│                                           │
│  class HROnboardingEnvironment(Environment):
│                                           │
│    reset() → HROnboardingObservation      │
│      1. Reset world state                 │
│      2. Pick next task                    │
│      3. Run setup_fn if any               │
│      4. Return observation with:          │
│         - task instruction                │
│         - available tool names            │
│         - difficulty & category metadata  │
│                                           │
│    step(action) → HROnboardingObservation │
│      1. Increment step counter            │
│      2. Execute tool via ToolRegistry     │
│      3. Check if max_steps reached        │
│      4. If done: evaluate rubric → reward │
│      5. Return observation with:          │
│         - tool result                     │
│         - reward (0.0 until final step)   │
│         - done flag                       │
│         - eval breakdown in metadata      │
│                                           │
│    state → State(episode_id, step_count)  │
└──────────────────────────────────────────┘

┌──────────────────────────────────────────┐
│  server/app.py                            │
│                                           │
│  app = create_app(                        │
│    HROnboardingEnvironment,               │
│    HROnboardingAction,                    │
│    HROnboardingObservation,               │
│    env_name="hr_onboarding_env",          │
│    max_concurrent_envs=4,                 │
│  )                                        │
│                                           │
│  Endpoints:                               │
│    POST /reset  → reset + return obs      │
│    POST /step   → execute action          │
│    GET  /state  → current state           │
│    GET  /schema → Action/Obs schemas      │
│    GET  /health → {"status": "healthy"}   │
│    WS   /ws     → persistent session      │
└──────────────────────────────────────────┘

┌──────────────────────────────────────────┐
│  client.py                                │
│                                           │
│  class HROnboardingEnv(EnvClient):        │
│    Connects to server via HTTP/WebSocket  │
│    Provides Python API: .reset(), .step() │
└──────────────────────────────────────────┘

7.3 Episode Lifecycle

1. Client calls reset() → Server creates new episode, picks task, returns observation
2. Client calls step(action) (up to 15 times) → Server executes tool, returns result
3. On step 15 (or earlier if agent signals done) → Server evaluates rubric, sets done=True, returns final reward
4. If client calls step() after done → Server returns {"error": "Episode already finished"}

7.4 Important: Reward is Only on Final Step

During intermediate steps (1 to 14), reward is always 0.0. The actual rubric evaluation only happens when done=True (step 15 or agent signals done). This is by design — the agent doesn't get feedback until the episode ends, which makes it a proper RL problem (delayed reward).

---

8. A Full Episode Walkthrough

Let's trace a complex onboarding task step by step.

Task

"Fully onboard John Lee as L3 Team Lead - ML in Data Science. Their manager will be Rohan Reddy (emp_0128). Create the employee record, initiate onboarding, assign a laptop, create IT accounts (email, Slack, VPN), set up appropriate access roles for their level, send a welcome email to the team channel, and schedule an orientation meeting with their manager."

What the Agent Should Do

Step 1: hr_create_employee
  → name: "John Lee", department: "Data Science", level: "L3",
    role: "Team Lead - ML", manager_id: "emp_0128"
  → Result: {success: true, employee: {emp_id: "emp_0201", ...}}

Step 2: onboarding_create_request
  → employee_id: "emp_0201"
  → Result: {success: true, request: {request_id: "onb_0001", steps: {...}}}

Step 3: it_get_available_assets
  → asset_type: "laptop"
  → Result: {success: true, count: 24, assets: [{asset_id: "asset_003", ...}, ...]}

Step 4: it_assign_asset
  → asset_id: "asset_003", employee_id: "emp_0201"
  → Result: {success: true}

Step 5: it_create_account
  → employee_id: "emp_0201", account_types: ["email", "slack", "vpn"]
  → Result: {success: true, accounts_created: [...]}

Step 6: access_assign_role
  → employee_id: "emp_0201", role_id: "role_004" (data_scientist, requires L1+, Data Science)
  → Result: {success: true, role: "data_scientist", permissions: [...]}

Step 7: slack_send_message
  → channel: "#data-science", sender: "hr-bot", text: "Welcome John Lee to the team! ..."
  → Result: {success: true}

Step 8: meeting_schedule
  → title: "Orientation: John Lee", attendees: ["emp_0201", "emp_0128"],
    datetime: "2026-03-10T10:00:00", meeting_type: "orientation"
  → Result: {success: true}

Agent signals done.

Rubric Evaluation

 [PASS] created_employee:     tool_used:hr_create_employee         ✓
 [PASS] initiated_onboarding: tool_used:onboarding_create_request  ✓
 [PASS] assigned_laptop:      tool_used:it_assign_asset            ✓
 [PASS] created_accounts:     tool_used:it_create_account          ✓
 [PASS] assigned_access:      tool_used:access_assign_role         ✓
 [PASS] sent_welcome:         tool_used_any:email_send,slack_send_message  ✓
 [PASS] scheduled_orientation: tool_used:meeting_schedule           ✓
 [PASS] sequencing_create_first: tool_order:hr_create_employee<onboarding_create_request  ✓
 [PASS] sequencing_asset_check: tool_order:it_get_available_assets<it_assign_asset  ✓
 [PASS] completeness:         tool_count:onboarding_complete_step>=3  ✗

Score: 9/10 = 0.9 (90%)

Note: The agent scored 9/10 because it didn't complete any onboarding steps (the onboarding_complete_step tool was not called at all). A perfect agent would also call onboarding_complete_step 3+ times to mark steps like "Provision email and Slack accounts", "Assign laptop and peripherals", etc. as done.

What Happens When Things Go Wrong

If the agent calls hr_create_employee with department: "Data Science" and the department is at its headcount limit:

Step 1: hr_create_employee → {success: false, error: "Department 'Data Science' has reached its headcount limit (25)"}

A good agent should recognize this error and try a different approach (or report the issue). A bad agent will keep retrying the same call.

---

9. Business Rules & Edge Cases

9.1 RBAC (Role-Based Access Control)

The level hierarchy governs who can do what:

L1 (Associate)  → Basic access roles only
L2 (Senior)     → Same as L1 + can mentor
L3 (Team Lead)  → Can approve onboarding (manager_approval)
L4 (Manager)    → Can approve security (security_approval), server room badge access
L5 (Director)   → All approvals + executive access
L6 (VP)         → Same as L5

Access roles have two constraints:

1. Level requirement: Employee must be at or above the role's minimum level
2. Department restriction: Employee must be in the role's allowed department (or role allows "all")

9.2 Headcount Limits

Each department has a headcount_limit. When the number of active+pending employees reaches this limit, hr_create_employee fails. The agent should recognize this and either:

• Report the limitation
• Check headcount first with hr_search_employees
• Look up the relevant policy

9.3 License Seat Limits

Two licenses are intentionally full:

• Netsuite (15/15) — used by Finance
• LinkedIn Sales Navigator (25/25) — used by Sales

Agents should call it_get_software_licenses to check availability before trying to assign.

9.4 Contractor Rules

When is_contractor: true:

• Legal approval required in addition to manager approval
• No VPN access by default
• Limited access roles (contractors group, not full department group)

9.5 Termination vs Resignation

Different offboarding steps:

• Resignation: access_revocation, asset_return, knowledge_transfer, exit_interview, final_payroll, farewell_communications
• Termination: access_revocation, asset_return, final_payroll, legal_review (NO farewell email, NO exit interview)

The rubric for termination tasks checks tool_not_used:email_send — the agent should NOT send a farewell email.

9.6 Server Room Badge Access

Creating a badge with access_zones: ["server_room"] requires:

1. Employee must be L4+ OR
2. A security_approval must exist for the relevant onboarding request

9.7 Manager On Leave

Some tasks set a manager's status to "on_leave". The agent should:

1. Try to look up the manager and see they're on leave
2. Use hr_get_org_chart or hr_read_employee on the manager to find the skip-level manager
3. Use the skip-level manager for approvals and orientation scheduling

---

10. File-by-File Reference

rl_hack/
├── __init__.py                    # Exports: HROnboardingEnv, HROnboardingAction, HROnboardingObservation
├── models.py                      # Pydantic models: Action (tool_name + arguments) and Observation (task_id + instruction + tool_result + step + reward + done)
├── client.py                      # EnvClient subclass: connects to server via HTTP/WebSocket, provides .reset() and .step()
├── openenv.yaml                   # OpenEnv manifest: tells HF Spaces this is a FastAPI environment on port 7860
├── pyproject.toml                 # Python package config: name, version, dependencies (openenv-core)
├── test_with_llm.py               # Test script: runs GPT-4o-mini against a task, prints rubric evaluation
├── .env                           # API keys (gitignored)
├── README.md                      # User-facing docs with quick start, tool table, task overview
├── ENVIRONMENT_DEEP_DIVE.md       # This document
│
└── server/
    ├── __init__.py                # Exports HROnboardingEnvironment
    ├── app.py                     # FastAPI app created via create_app(), serves on port 7860
    ├── hr_onboarding_environment.py  # Core environment class: reset(), step(), state. Orchestrates world, tools, tasks, rubrics.
    ├── world.py                   # WorldState: loads data, manages 500+ entities, enforces business rules, provides mutation methods
    ├── tools.py                   # 25 tool definitions (TOOL_DEFINITIONS list) + ToolRegistry class that maps names to functions
    ├── tasks.py                   # TaskGenerator: creates 77 tasks with instructions, rubric criteria, and setup functions
    ├── rubrics.py                 # RubricEvaluator: 8 check types, evaluates action log against criteria, computes score
    ├── Dockerfile                 # Multi-stage Docker build using openenv-base image
    ├── requirements.txt           # Server dependencies: openenv, fastapi, uvicorn
    └── data/
        ├── employees.json         # 200 employee records with full org hierarchy
        ├── departments.json       # 8 departments with headcount limits, required tools, onboarding/offboarding steps
        ├── it_assets.json         # 100 IT assets (50 laptops, 25 monitors, 15 phones, 10 headsets)
        ├── access_roles.json      # 20 RBAC roles with level/department restrictions
        ├── policies.json          # 15 company policies (onboarding, offboarding, badges, contractors, etc.)
        └── templates.json         # 12 email/Slack message templates

---

Appendix: Quick Numbers

Metric	Value
Total entities	~500+
Employees	200
Departments	8
IT Assets	100
Access Roles	20
Software Licenses	15 (2 intentionally full)
Security Groups	15
Policies	15
Message Templates	12
Tools	25
Tasks	77
Max steps per episode	15
Simple tasks	14
Medium tasks	22
Complex tasks	29
Edge case tasks	12
Rubric check types	8