StepEval

StepEval is an 8B language-model judge for evaluating generated procedures. Given a task goal, a reference procedure, and a candidate procedure, it identifies critical failures that prevent task completion or make the candidate instructions unusable.

StepEval is based on Qwen3-8B and trained using supervised fine-tuning (SFT) with TRL to reproduce GPT-5's critical-failure annotations. It complements text-similarity metrics by evaluating whether the candidate preserves the actions and dependencies needed to accomplish the goal.

Inputs

The judge receives three inputs through the evaluation prompt:

Input	Description
Goal	The task query or objective the procedure should accomplish.
L1	A reference procedure that achieves the goal.
L2	The candidate procedure being evaluated.

L1 provides a reference for essential actions and their dependencies. Alternative procedures are acceptable when they preserve the requirements needed to accomplish the goal. Differences in wording, harmless additions, and changes in ordering that do not affect the outcome are not sufficient reasons to mark a failure.

Critical failures

StepEval checks for issues such as:

• Contradictions with the goal or essential reference actions.
• Internal inconsistency, incoherence, or severe vagueness.
• Missing essential actions.
• Counterproductive extra actions or harmful repetition.

An issue must be severe enough to prevent task completion or make the instructions unusable. The complete instructions and examples are included in prompt.md.

Output format

The model is prompted to return a JSON object with these fields:

Field	Type	Description
reasoning	String	Explanation of the assessment and failure severity.
critical_failures	Array	Critical failures identified in the candidate procedure; empty when none are identified.
critical_failures[].failure	String	Description of the failure.
critical_failures[].L1_steps	Array of integers	Relevant one-based step numbers in the reference procedure.
critical_failures[].L2_steps	Array of integers	Relevant one-based step numbers in the candidate procedure.

For example, an illustrative response for a candidate that omits addressing an envelope is:

{
  "reasoning": "The candidate omits the destination address required by the goal.",
  "critical_failures": [
    {
      "failure": "The envelope is mailed without a recipient address.",
      "L1_steps": [3],
      "L2_steps": [4]
    }
  ]
}

The model produces text, so consumers should parse and validate the response before deriving a score. A malformed or incomplete response is an evaluation error, not a no_failure verdict.

Scoring

Derive the binary verdict from a successfully parsed response:

• no_failure: critical_failures is empty.
• has_failure: critical_failures contains one or more entries.

For a set of evaluated procedures, valid-plan rate (VPR) is the fraction judged to have no critical failures:

VPR = number of procedures with no_failure / number of evaluated procedures

Report unresolved evaluation errors separately and state how they affect the denominator. The verdict describes the judge's assessment; it is not a guarantee that the procedure will succeed in practice.

Quick start

Install the inference dependencies:

pip install "transformers>=4.57.6,<5" accelerate torch huggingface_hub

This example loads the full model and uses the same prompt template as the evaluation pipeline. Use a machine with sufficient memory for the model weights and generation.

import json
from pathlib import Path

from huggingface_hub import hf_hub_download
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "rahmanidashti/StepEval"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype="auto",
    device_map="auto",
)
model.eval()

prompt_path = hf_hub_download(repo_id=model_id, filename="prompt.md")
template = Path(prompt_path).read_text(encoding="utf-8")

goal = "Mail a letter in a sealed, addressed envelope."
reference_steps = [
    "Put the letter inside an envelope.",
    "Seal the envelope.",
    "Write the recipient's address on the envelope.",
    "Attach sufficient postage.",
    "Place the envelope in an outgoing mailbox.",
]
candidate_steps = [
    "Put the letter inside an envelope.",
    "Seal the envelope.",
    "Attach sufficient postage.",
    "Place the envelope in an outgoing mailbox.",
]


def numbered_steps(steps):
    return "\n".join(f"{i}. {step}" for i, step in enumerate(steps, start=1))


prompt = template.format(
    goal=goal,
    reference_steps=numbered_steps(reference_steps),
    steps=numbered_steps(candidate_steps),
)

inputs = tokenizer.apply_chat_template(
    [{"role": "user", "content": prompt}],
    tokenize=True,
    add_generation_prompt=True,
    enable_thinking=False,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

output_ids = model.generate(
    **inputs,
    max_new_tokens=2048,
    do_sample=False,
    temperature=None,
    top_p=None,
    top_k=None,
)
response = tokenizer.decode(
    output_ids[0, inputs["input_ids"].shape[1]:],
    skip_special_tokens=True,
).strip()

# Accept a JSON code fence if the model includes one.
if response.startswith("```") and response.endswith("```"):
    response = "\n".join(response.splitlines()[1:-1]).strip()

result = json.loads(response)
if not isinstance(result, dict):
    raise ValueError("Expected a JSON object from StepEval.")
if not isinstance(result.get("reasoning"), str):
    raise ValueError("Missing or invalid reasoning field.")
failures = result.get("critical_failures")
if not isinstance(failures, list):
    raise ValueError("Missing or invalid critical_failures field.")
for failure in failures:
    if not isinstance(failure, dict) or not isinstance(failure.get("failure"), str):
        raise ValueError("Invalid failure entry.")
    for field, steps in [("L1_steps", reference_steps), ("L2_steps", candidate_steps)]:
        indices = failure.get(field)
        if not isinstance(indices, list) or any(
            type(i) is not int or not 1 <= i <= len(steps) for i in indices
        ):
            raise ValueError(f"Invalid step references in {field}.")

verdict = "has_failure" if failures else "no_failure"
print(json.dumps(result, indent=2, ensure_ascii=False))
print("Verdict:", verdict)

The example uses non-thinking mode and greedy decoding explicitly. It overrides the sampling defaults stored in the checkpoint's generation configuration. If output is truncated before the JSON object is complete, increase the generation budget and retry the evaluation.

Training

StepEval uses supervised fine-tuning on GPT-5 judge annotations. Each example pairs a prompt containing the goal, reference procedure, and candidate procedure with the teacher's assessment, including its explanation and structured list of critical failures.

The model is intended for procedural evaluation, including the assessment of generated plans in StepBench. For next-step evaluation, construct the candidate procedure consistently with the evaluation protocol rather than treating an isolated step as a complete plan.

Framework versions

The checkpoint's existing training metadata reports:

• TRL: 0.24.0
• Transformers: 4.57.6
• PyTorch: 2.8.0
• Datasets: 4.5.0
• Tokenizers: 0.22.2

Limitations

StepEval depends on the quality and completeness of the supplied goal and reference procedure. It may miss failures or flag valid alternative procedures. Its assessments reflect the teacher annotations used for fine-tuning and should be interpreted as model judgements.

Related resources

• StepBench dataset
• Qwen3-8B
• Evaluation prompt
• TRL