xiongsir1/reposocratic-interviewer

RepoSocratic Interviewer

A 9B project-interview agent built with trajectory distillation and online GRPO.

RepoSocratic turns project descriptions, candidate participation statements, and optional code repositories into focused technical interview questions. It reads relevant code through a small tool interface, identifies useful discussion topics, and prepares questions about personal contributions, design decisions, failure handling, and verification.

Latest evaluation: 86.5% task completion; 98.2% of delivered questions rated Ready or Revise by AI reviewers (includes questions that require editing).

中文：面向 Agent 应用与算法岗位的项目面试官，通过 27B→9B 轨迹蒸馏、教师概率蒸馏及在线 GRPO，实现从自主阅读代码到有依据的技术提问。

What it does

• Read with purpose: use list_tree, search_literal, and read_lines to understand the project and verify relevant implementation details.
• Ask about real work: connect questions to the supplied role and project context, including contributions, trade-offs, troubleshooting, and validation.
• Handle different inputs: support repository-backed scenarios, text-only project descriptions, and scenarios where the candidate's contribution needs clarification.
• Produce structured interview plans: separate candidate-facing questions from the interviewer's internal evidence, intent, and conditional follow-ups.

Released adapters

Directory	Training stage	Configuration
k2-checkpoint-206/	Trajectory SFT + teacher-probability distillation	Qwen3.5-9B, LoRA r8 / alpha16; selected epoch-2 checkpoint
rl-73/	K2 followed by online GRPO	73 sampled groups, 292 trajectories, 67 optimizer updates; group size 4, fixed K2 reference

Both directories contain PEFT LoRA adapters for the BF16 Qwen3.5-9B base. Load the base model first, then the selected adapter. Repository interaction is provided by the interviewer harness and its three read-only tools.

Training pipeline

1. Trajectory dataset: construct 100 interview scenarios across 20 projects, producing 400 questions and 820 reviewed assistant targets. Scenarios cover Agent applications and algorithms, with repository, text-only, and uncertain-contribution inputs.
2. Teacher-probability distillation: score the reviewed targets with a local 27B teacher. Store Top-32 probabilities, target-token probabilities, residual probability mass, and EOS coverage for 369,032 target tokens. Optimize CE together with grouped KL.
3. Long-context training: train with BF16 LoRA, sequence parallelism, FSDP2, and activation checkpointing on two GPUs. The training qualification includes an input of 116,757 tokens. CE and CE+KD runs each complete three epochs; K2 is the selected epoch-2 distillation checkpoint.
4. Online GRPO: sample trajectories, score them, update the actor, and sample the next group using the updated weights. Record behavior probabilities, grammar masks, policy lineage, and recoverable checkpoints throughout the loop.

Evaluation

The benchmark contains 24 project families held out from the project's training material, with 2 scenarios and 2 generation seeds per family: 96 tasks per policy, 288 trajectories in total. The three policies use the same scenarios, repository snapshots, tools, and generation settings.

Question retention: Ready + Revise

The delivered-question retention rate counts questions rated Ready or Revise:

retention rate = (Ready + Revise) / (Ready + Revise + Reject)

Ready means directly usable under the rubric. Revise means the question is retained for revision and still needs editing; it does not mean those edits have already been made. Reject means it is not retained. This descriptive retention summary uses the existing review labels without changing them.

Policy	Completed tasks	Ready	Revise	Reject	Retained / delivered	Retention rate (includes Revise)
Qwen3.5-9B base	73 / 96	38	239	15	277 / 292	94.86%
SFT + distillation: K2	82 / 96	75	244	9	319 / 328	97.26%
SFT + GRPO: RL-73	83 / 96	81	245	6	326 / 332	98.19%

For the final adapter, 326 of 332 delivered questions are rated Ready or Revise, giving 98.2% retention among delivered questions. Task completion remains 86.5% (83/96).

Coverage and metric definitions

Each policy has 384 planned question slots (96 tasks × 4 questions). To show coverage as well as the composition of delivered questions, both full-denominator yields are retained below:

Metric, denominator = all 384 planned question slots	Base	K2	RL-73
Ready + Revise / planned slots (includes required editing)	72.14%	83.07%	84.90%
Ready / planned slots (usable as written)	9.90%	19.53%	21.09%

The 98.2% headline uses delivered questions as its denominator; uncompleted tasks are reflected in task completion and the full planned-slot yields.

Questions are assessed by AI reviewers on six dimensions: contribution fit, interview value, premise reliability, project specificity, clarity, and role/visibility. The main review covers all 952 delivered questions; a preselected six-family subset receives a second review covering 232 questions.

Inference settings

Use the Qwen3.5 chat template with thinking enabled and preserved, Hermes-style tool calls, and the interviewer harness's structured-plan output.

temperature = 1.0
top_p = 0.95
top_k = 20
enable_thinking = true
preserve_thinking = true
cross_request_prefix_cache = false

These are the settings used for the evaluation above. cross_request_prefix_cache refers to reuse across separate tool-call requests; it does not require disabling the ordinary KV cache within one generation.

Open release

This repository publishes the distilled and GRPO-trained adapter weights and their PEFT configurations. The adapters are released under the Apache-2.0 license recorded in this repository.