ObjRun GPQA Diamond - oracle correctness feedback
195/198 (98.484848%) with gold-answer correctness feedback during solving.
This repository identifies the ObjRun F evaluation system and publishes its retained experiment results. It does not contain model weights or claim a new pretrained checkpoint. The underlying API requested and served deepseek-flash on 2026-10-08; the provider alias is preserved rather than mapped to an unverified checkpoint name.
Evaluation setting
The agent could query the gold-answer checker, receive correctness feedback, and revise its answer during solving. This oracle-assisted multiple-choice setting has a material information advantage over ordinary GPQA evaluation. The reported 98.48% is a system success rate under this setting and must not be interpreted as standard no-oracle model accuracy, greedy pass@1, or a claim of superiority over standard leaderboard entries.
- Full GPQA Diamond: 198 tasks, 195 passed, all 198 final grades valid.
- ObjRun F: 5 checks per attempt, up to 4 branches, 600 seconds per task.
- API: DeepSeek
deepseek-flash; temperature 0, seed 42, reasoning effortnone, maximum 8,192 tokens per request. - Web search disabled.
- Campaign:
repair-2026-10-08T06-54-55-267Z. - Context only: the U arm without correctness feedback solved 163/198 (82.32%). Only the F result is submitted by this repository.
Evidence
Machine-readable report contains the settings, source revision, and provenance hashes. Per-task results contains the 198 task identifiers, boolean grades, candidate hashes, and trace hashes. These files were derived from the retained completed campaign without rerunning inference.
Question text, answer choices, gold answers, generated answers, and raw traces are not published, to respect GPQA's dataset restrictions. This result is self-reported and has no Hugging Face verified evaluation token. Appearance in Hub evaluation results does not constitute endorsement of this nonstandard setting by GPQA's authors.