True Detective: A Deep Abductive Reasoning Benchmark Undoable for GPT -3 and Challenging for GPT -4 (2023)

Abstract: Large language models (LLMs) have demonstrated solid zero-shot reasoning capabilities, which is reflected in their performance on the current test tasks. This calls for a more challenging benchmark requiring highly advanced reasoning ability to be solved. In this paper, we introduce such a benchmark, consisting of 191 long-form (1200 words on average) mystery narratives constructed as detective puzzles. Puzzles are sourced from the “5 Minute Mystery” platform and include a multiple-choice question for evaluation. Only 47% of humans solve a puzzle successfully on average, while the best human solvers achieve over 80% success rate. We show that GPT-3 models barely outperform random on this benchmark (with 28% accuracy) while state-of-the-art GPT-4 solves only 38% of puzzles. This indicates that there is still a significant gap in the deep reasoning abilities of LLMs and humans and highlights the need for further research in this area. Our work introduces a challenging benchmark for future studies on reasoning in language models and contributes to a better understanding of the limits of LLMs’ abilities.

## introduction large language models (llms) have gained significant attention in recent years due to their impressive performance on a wide range of natural language processing tasks, including reasoning tasks @xcite . this calls for new, genuinely challenging benchmarks requiring llms to possess truly advanced reasoning capabilities to be solved. abductive reasoning is a type of inference aiming at finding the minimal and most justified explanation for the set of phenomena or observations. previous benchmarks on this topic, such as @xcite , consisted of short and straightforward common-sense observations and were solved by gpt models @xcite . however, the canonical example of abductive reasoning, a demanding process of a detective finding the best solution to a complex crime based on the clues and observations, was not explored as a foundation for the llm benchmark in the literature. motivated by the need for a new reasoning benchmark and inspired by the complexities and particularities of a detective enterprise, we present a novel abductive reasoning benchmark consisting of 191 detective puzzles/mysteries. mysteries are sourced from the "5 minute mystery" platform, where professional and aspiring authors wrote them. a puzzle is structured as @xmath0. over the last 15 years, puzzles were attempted by humans around 2000 times each with an average solve rate of 47% (only the first try for each human for each puzzle counts). however, top human solvers (top 10) achieve a success rate of over 80% solving more than 154 of 192 puzzles correctly. moreover, additional modifications such as chain-of-thought (cot) prompting @xcite ; @xcite that are meant to invoke emergent reasoning abilities in llms do not help for gpt-3. in this study, we also assess the performance of the current state-of-the-art gpt-3 and gpt-4 models on our newly proposed dataset. we show that these models, even equipped with the chain of thought prompts @xcite , are getting an accuracy rate of only 28%, barely better than random guessing (gpt-3.5), or scoring 38% (gpt-4), which is halfway between random guessing and average human baseline, and far behind top human solvers with their 80% solve rate. these results reveal a significant gap in the reasoning abilities of gpt models and humans. in our ablation study, we also supply models with golden cots. golden cots are narratives that represent the reasoning behind the correct answer for each story (written by the mystery authors). when we attach golden cots to the input prompt, the best-performing gpt-3.5 model only achieves a solve rate of 63%. this indicates llms' difficulty making even trivial inferences from the complex long-form story. gpt-4 models, however, get as good as the best human solvers when presented with our chain of thoughts (even though humans do not have access to the golden cots). our contributions in this paper are twofold: (1) a new challenging benchmark for evaluating llms for advanced abductive reasoning; (2) a showcase of gpt-3.5 and gpt-4 models failing to perform reasonably. natural language inference (nli) is another related domain, but nli tasks usually include much simpler and smaller inferences @xcite . @xcite introduced the swag dataset that offers a largescale natural language inference challenge where grounded knowledge is required to make an inference. this shares some commonality with our dataset, as some mysteries might require a share of grounded knowledge about the real world. unlike @xcite , we only offer a test set, but our stories are broader and more involved. on the other hand, @xcite introduced a question-answering benchmark that requires deeper text understanding based on the football match commentaries. their questions range from counting the number of goals to identifying the game-winner. while answers to many of these questions are not explicitly provided in the football commentary, our mysteries require solving the whole case specifically designed to be challenging even for humans. lastly, @xcite find that while eliciting "chain of thought" reasoning helps with stronger models, it can hurt when solving harder tasks with smaller models. we observe this behavior when comparing gpt-3.5 and gpt-4 on our benchmark. ## conclusion we presented a new benchmark in the form of detective puzzles to evaluate the abductive reasoning capabilities of large language models. results from state-of-the-art gpt-3.5 models across three prompting strategies showed poor performance close to random. gpt-4 managed to show comparably solid performance (when prompted with cot), but even this model is behind the average human solve rate on our benchmark. when provided with golden cots, which significantly simplifies the task, gpt-4 shows good performance, while gpt-3 is still unable to do a final inference well enough. overall, our benchmark offers insights into llms' limitations and provides a difficult challenge for future research on abductive reasoning in large lms.