D etect B ench: Can Large Language Model Detect and Piece Together Implicit Evidence? (2024)

Abstract: Detecting evidence within the context is a key step in the process of reasoning task. Evaluating and enhancing the capabilities of LLMs in evidence detection will strengthen context-based reasoning performance. This paper proposes a benchmark called DetectBench for verifying the ability to detect and piece together implicit evidence within a long context. DetectBench contains 3,928 multiple-choice questions, with an average of 994 tokens per question. Each question contains an average of 4.55 pieces of implicit evidence, and solving the problem typically requires 7.62 logical jumps to find the correct answer. To enhance the performance of LLMs in evidence detection, this paper proposes Detective Reasoning Prompt and Finetune. Experiments demonstrate that the existing LLMs’ abilities to detect evidence in long contexts are far inferior to humans. However, the Detective Reasoning Prompt effectively enhances the capability of powerful LLMs in evidence detection, while the Finetuning method shows significant effects in enhancing the performance of weaker LLMs. Moreover, when the abilities of LLMs in evidence detection are improved, their final reasoning performance is also enhanced accordingly.

## introduction the ability to perform reasoning over natural language is an important aspect of intelligence @xcite . tasks designed to assess inferential capabilities commonly consist of a context and a question, expecting the large language models (llms) to respond correctly @xcite . human annotators often conceal the evidence necessary for answering the question within the context. this raises a question: whether llms possess the capability to detect these pieces * corresponding authors. on a snowy winter night, the room was hot and sweaty, the windows were closed but only half the curtains were covered. i stood inside, looked out through the window and saw …… i look through the window and saw the streets and surroundings is covered in a blanket of snow. ## conclusion this paper introduces the detectbench to assess llms' abilities in evidence and multi-step commonsense reasoning within a long context. we also propose a novel type of prompt and finetuning method named detective reasoning to augment llm's performance in evidence detection and thereby augment performance in commonsense reasoning. the experiment results show that the abilities of evidence detection and reasoning performance are correlated. detective reasoning effectively enhances the capability of llms in evidence detection, thereby improving the llms' commonsense reasoning results in long text contexts.