Reverse Question Answering: Can an LLM Write a Question so Hard (or Bad) that it Can’t Answer? (2025)

Abstract: Question answering (QA)—giving correct answers to questions—is a popular task, but we test **reverse question answering (RQA)**: for an input answer, give a question with that answer. Past work tests QA and RQA separately, but we test them jointly, comparing their difficulty, aiding benchmark design, and checking reasoning consistency. We run 16 LLMs on QA and RQA with trivia questions/answers, revealing: 1) Versus RQA, LLMs are much less accurate in RQA for numerical answers, but slightly more accurate in RQA for textual answers; 2) LLMs often answer their own invalid questions from RQA accurately in QA, so RQA errors are not just from knowledge gaps; 3) RQA errors correlate with question difficulty and inversely correlate with answer frequencies in the Dolma corpus; and 4) LLMs struggle to give valid multi-hop questions. By finding question and answer types that lead to RQA errors, we suggest improvements for LLM reasoning.

## experimental setup we evaluate llm abilities in question answering (qa) and reverse question answering (rqa): qa succeeds if a matches a ˆsemantically. for example, given the input "what is the name of the polygon with three sides?" for q, an llm using qa should give an a ˆthat matches "triangle" for a. this typical qa setup tests deduction, since the model must reason to the correct answer of a based on the premises in q. this section describes the datasets ( §2.1), models ( §2.2), and metrics ( §2.3) used for rqa and qa. ## conclusion we test llm rqa and qa abilities. llms have notably low accuracy in numerical rqa which is not just due to knowledge gaps, as models can often answer their own invalid questions correctly. these weaknesses can be excised in future benchmarks to more holistically evaluate llm numerical abductive reasoning and math capabilities. to reduce inaccuracies in numerical rqa, often from generating overly-complex questions, we suggest calibrating models using difficulty scores, collecting user preferences that control for complexity bias, and adapting prior multi-hop qa methods-key steps for reliable llm reasoning in downstream tasks.