Repository: harbor-framework/harbor
Issue: Codex agent: Missing trajectory.json conversion (ATIF format)
Root cause: Unimplemented Feature Gap

# Codex agent: Missing trajectory.json conversion (ATIF format)

## Summary

The Codex agent successfully extracts token usage metrics but does **not** generate a `trajectory.json` file in ATIF (Agent Trajectory Interchange Format). This creates inconsistency with other agents (terminus-2, openhands, claude-code) that provide full trajectory logs for analysis and debugging.

## Current Behavior

The Codex agent:
- ✅ **Extracts metrics correctly** from `codex.txt` (input/output tokens, cache tokens)
- ✅ **Creates rollout JSONL files** with detailed session data in `sessions/YYYY/MM/DD/rollout-*.jsonl`
- ❌ **Does NOT convert rollout data to trajectory.json**
- ❌ **Does NOT follow ATIF format** for trajectory logging

## Root Cause

In `src/harbor/agents/installed/codex.py`:

```python
class Codex(BaseInstalledAgent):
    @property
    def _trajectory_path(self) -> Path:
        return EnvironmentPaths.agent_dir / "trajectory.json"  # ✅ Defined but never used
    
    def populate_context_post_run(self, context: AgentContext) -> None:
        # ✅ Extracts metrics from codex.txt
        codex_output_path = self.logs_dir / self._OUTPUT_FILENAME
        with open(codex_output_path) as f:
            lines = f.readlines()
        for line in reversed(lines):
            parsed = json.loads(line)
            if isinstance(parsed, dict) and "usage" in parsed:
                usage = parsed["usage"]
                context.n_input_tokens = usage.get("input_tokens")
                context.n_cache_tokens = usage.get("cached_input_tokens")
                context.n_output_tokens = usage.get("output_tokens")
                return  # ❌ Returns without creating trajectory.json
```

**Missing functionality:**
- No `_get_session_dir()` method to locate rollout files
- No `_convert_events_to_trajectory()` method to parse rollout JSONL
- No trajectory.json writing logic

## Comparison: Other Agents

### OpenHands Agent (Has trajectory conversion)

```python
def populate_context_post_run(self, context: AgentContext) -> None:
    session_dir = self._get_session_dir()
    events_dir = session_dir / "events"
    
    # Convert events to trajectory
    trajectory = self._convert_events_to_trajectory(events_dir)
    
    # Write trajectory.json
    trajectory_path = self.logs_dir / "trajectory.json"
    with open(trajectory_path, "w") as f:
        json.dump(trajectory.to_json_dict(), f, indent=2)
    
    # Populate metrics
    context.n_input_tokens = trajectory.final_metrics.total_prompt_tokens
    ...
```

### Terminus-2 Agent (Has trajectory conversion)

```python
def _dump_trajectory(self) -> None:
    trajectory = Trajectory(
        session_id=self._session_id,
        agent=Agent(name=self.name(), ...),
        steps=self._trajectory_steps,
        final_metrics=final_metrics,
    )
    
    trajectory_path = self.logs_dir / "trajectory.json"
    with open(trajectory_path, "w") as f:
        json_str = format_trajectory_json(trajectory.to_json_dict())
        f.write(json_str)
```

## Impact

- **No trajectory analysis**: Cannot debug agent behavior step-by-step
- **Inconsistent data format**: Codex outputs differ from other agents
- **Missing ATIF compliance**: Cannot use trajectory validation/analysis tools
- **Limited observability**: Only final metrics available, no intermediate states

## Available Data in Rollout Files

The rollout JSONL files contain all necessary data for trajectory conversion:

```bash
# Example rollout file location
/logs/agent/sessions/2025/11/29/rollout-2025-11-29T23-26-00-019ad1f0.jsonl
```

**Sample events:**
```json
{"timestamp":"2025-11-29T23:26:00.783Z","type":"session_meta","payload":{"id":"019ad1f0...","model_provider":"openai"}}
{"timestamp":"2025-11-29T23:26:00.790Z","type":"response_item","payload":{"type":"message","role":"user","content":[...]}}
{"timestamp":"2025-11-29T23:26:02.821Z","type":"event_msg","payload":{"type":"token_count","info":{"total_token_usage":{"input_tokens":3329,"output_tokens":34}}}}
{"timestamp":"2025-11-29T23:26:03.500Z","type":"tool_call","payload":{"name":"Write","input":{"file_path":"/workspace/test.txt"}}}
```

All data needed for ATIF conversion is present:
- Session metadata
- User/assistant messages
- Tool calls and results
- Token usage per API call
- Timestamps

## Reproduction

### Prerequisites
```bash
cd /path/to/harbor
uv sync --all-extras
export OPENAI_API_KEY=your_key_here
```

### Create minimal test task

```bash
mkdir -p /tmp/codex-test/{environment,tests}

cat > /tmp/codex-test/task.toml << 'TOML'
[task]
name = "codex-test"
timeout_sec = 300
cpus = 1
memory_mb = 2048
TOML

cat > /tmp/codex-test/instruction.md << 'INST'
Create a file at /workspace/hello.txt with "Hello Codex" and print it.
INST

cat > /tmp/codex-test/environment/Dockerfile << 'DOCKER'
FROM python:3.11-slim
WORKDIR /workspace
DOCKER

cat > /tmp/codex-test/tests/test.sh << 'TEST'
#!/bin/bash
[ -f /workspace/hello.txt ] && echo '{"reward": 1.0}' > /logs/verifier/reward.json || echo '{"reward": 0.0}' > /logs/verifier/reward.json
TEST
chmod +x /tmp/codex-test/tests/test.sh
```

### Run Codex

```bash
uv run harbor run \
  --path /tmp/codex-test \
  --agent codex \
  --model openai/gpt-5.1-codex \
  --jobs-dir /tmp/codex-results \
  --n-concurrent 1
```

### Verify Missing trajectory.json

```bash
TRIAL_DIR=$(find /tmp/codex-results -name "codex-test__*" -type d | head -1)

# Check metrics (THESE WORK)
cat "$TRIAL_DIR/result.json" | jq '.agent_result'
# Output:
# {
#   "n_input_tokens": 10261,
#   "n_cache_tokens": 3328,
#   "n_output_tokens": 99,
#   "cost_usd": null
# }

# Check trajectory.json (MISSING)
ls "$TRIAL_DIR/agent/trajectory.json"
# ls: cannot access: No such file or directory

# Verify rollout data exists
ls "$TRIAL_DIR/agent/sessions/2025/"*/*/*.jsonl
# Shows: rollout-2025-11-29T23-26-00-019ad1f0.jsonl

# Verify it has all necessary data
cat "$TRIAL_DIR/agent/sessions/2025/"*/*/*.jsonl | jq -r '.type' | sort -u
# Shows: session_meta, response_item, event_msg, tool_call, tool_result, turn_context
```

### Compare with OpenHands (Has trajectory.json)

```bash
uv run harbor run \
  --path /tmp/codex-test \
  --agent openhands \
  --model anthropic/claude-sonnet-4-5 \
  --jobs-dir /tmp/openhands-results \
  --n-concurrent 1

TRIAL_DIR=$(find /tmp/openhands-results -name "codex-test__*" -type d | head -1)

# OpenHands DOES create trajectory.json
ls "$TRIAL_DIR/agent/trajectory.json"
# -rw-rw-r-- 1 user user 12345 Nov 29 15:30 trajectory.json

cat "$TRIAL_DIR/agent/trajectory.json" | jq '.schema_version, .agent, .final_metrics'
# Shows ATIF-compliant trajectory with schema_version: "1.3"
```

## Proposed Solution

Implement trajectory conversion for Codex similar to OpenHands/Claude Code:

### 1. Add session directory locator

```python
def _get_session_dir(self) -> Path | None:
    """Locate the Codex session directory with rollout files."""
    sessions_dir = self.logs_dir / "sessions"
    if not sessions_dir.exists():
        return None
    
    # Codex uses date-based directory structure: sessions/YYYY/MM/DD/
    rollout_files = list(sessions_dir.glob("**/rollout-*.jsonl"))
    if not rollout_files:
        return None
    
    # Return the directory containing the most recent rollout file
    return rollout_files[0].parent
```

### 2. Add trajectory conversion logic

```python
def _convert_rollout_to_trajectory(self, session_dir: Path) -> Trajectory | None:
    """Convert Codex rollout JSONL to ATIF trajectory format."""
    rollout_files = list(session_dir.glob("rollout-*.jsonl"))
    if not rollout_files:
        return None
    
    # Parse rollout events
    events = []
    for rollout_file in rollout_files:
        with open(rollout_file) as f:
            for line in f:
                events.append(json.loads(line))
    
    # Extract session metadata
    session_meta = next((e for e in events if e.get("type") == "session_meta"), {})
    session_id = session_meta.get("payload", {}).get("id", "unknown")
    
    # Convert events to ATIF steps
    steps = []
    current_step = None
    
    for event in events:
        event_type = event.get("type")
        payload = event.get("payload", {})
        
        if event_type == "response_item" and payload.get("role") == "user":
            # Start new step with user message
            current_step = Step(
                step_number=len(steps),
                timestamp=event.get("timestamp"),
                action=Action(
                    type="message",
                    content=payload.get("content", [])
                ),
                tool_calls=[],
                observations=[]
            )
        
        elif event_type == "tool_call" and current_step:
            # Add tool call to current step
            current_step.tool_calls.append(
                ToolCall(
                    name=payload.get("name"),
                    input=payload.get("input")
                )
            )
        
        elif event_type == "tool_result" and current_step:
            # Add observation
            current_step.observations.append(
                Observation(
                    type="tool_result",
                    content=payload.get("output")
                )
            )
        
        elif event_type == "event_msg" and payload.get("type") == "token_count":
            # Extract metrics for step
            if current_step:
                usage = payload.get("info", {}).get("last_token_usage", {})
                current_step.metrics = Metrics(
                    prompt_tokens=usage.get("input_tokens"),
                    completion_tokens=usage.get("output_tokens"),
                    cached_tokens=usage.get("cached_input_tokens")
                )
                steps.append(current_step)
                current_step = None
    
    # Calculate final metrics
    total_input = sum(s.metrics.prompt_tokens for s in steps if s.metrics)
    total_output = sum(s.metrics.completion_tokens for s in steps if s.metrics)
    total_cached = sum(s.metrics.cached_tokens or 0 for s in steps if s.metrics)
    
    return Trajectory(
        session_id=session_id,
        agent=Agent(
            name=self.name(),
            version=self.version() or "unknown",
            model_name=self.model_name
        ),
        steps=steps,
        final_metrics=FinalMetrics(
            total_prompt_tokens=total_input,
            total_completion_tokens=total_output,
            total_cached_tokens=total_cached
        )
    )
```

### 3. Update populate_context_post_run

```python
def populate_context_post_run(self, context: AgentContext) -> None:
    # Get session directory
    session_dir = self._get_session_dir()
    if not session_dir:
        self.logger.debug("No Codex session directory found")
        # Fall back to existing codex.txt parsing for metrics
        self._extract_metrics_from_output(context)
        return
    
    # Convert to trajectory
    trajectory = self._convert_rollout_to_trajectory(session_dir)
    if not trajectory:
        self.logger.warning("Failed to convert Codex rollout to trajectory")
        self._extract_metrics_from_output(context)
        return
    
    # Write trajectory.json
    trajectory_path = self.logs_dir / "trajectory.json"
    try:
        with open(trajectory_path, "w") as f:
            json.dump(trajectory.to_json_dict(), f, indent=2)
        self.logger.debug(f"Trajectory written to {trajectory_path}")
    except Exception as e:
        self.logger.error(f"Failed to write trajectory: {e}")
    
    # Populate context from trajectory
    if trajectory.final_metrics:
        context.n_input_tokens = trajectory.final_metrics.total_prompt_tokens
        context.n_output_tokens = trajectory.final_metrics.total_completion_tokens
        context.n_cache_tokens = trajectory.final_metrics.total_cached_tokens

def _extract_metrics_from_output(self, context: AgentContext) -> None:
    """Fallback: extract metrics from codex.txt (current implementation)."""
    # ... existing code ...
```

## Benefits

- **Consistent format**: Codex trajectories match terminus-2, openhands, claude-code
- **Better debugging**: Step-by-step analysis of agent execution
- **ATIF compliance**: Can use trajectory validation and analysis tools
- **Full observability**: Track tool calls, observations, and metrics per step
- **No breaking changes**: Metrics extraction still works as fallback

## Environment
- Harbor version: latest (commit 8c737de)
- Codex CLI: 0.63.0
- Python: 3.13.7

## Related Files
- `src/harbor/agents/installed/codex.py:29-30` (_trajectory_path property - defined but unused)
- `src/harbor/agents/installed/codex.py:71-104` (populate_context_post_run - only extracts metrics)
- `src/harbor/agents/installed/openhands.py:274-315` (reference implementation for trajectory conversion)
- `src/harbor/agents/installed/claude_code.py:686-704` (reference implementation for trajectory conversion)
- `docs/rfcs/0001-trajectory-format.md` (ATIF specification)

## References
- Issue #178: Claude Code agent permission bug
- ATIF Format: `src/harbor/models/trajectories/trajectory.py`


Yeah that's because @alexgshaw removed the log temporarily to debug sth. @alexgshaw did you get a chance to revert?