Direct Evaluation of Chain-of-Thought in Multi-hop Reasoning with Knowledge Graphs (2024)

Abstract: Large language models (LLMs) have demonstrated strong reasoning abilities when prompted to generate chain-of-thought (CoT) explanations alongside answers. However, previous research on evaluating LLMs has solely focused on answer accuracy, neglecting the correctness of the generated CoT. In this paper, we delve deeper into the CoT reasoning capabilities of LLMs in multi-hop question answering by utilizing knowledge graphs (KGs). We propose a novel discriminative and generative CoT evaluation paradigm to assess LLMs’ knowledge of reasoning and the accuracy of the generated CoT. Through experiments conducted on 5 different families of LLMs across 2 multi-hop question-answering datasets, we find that LLMs possess sufficient knowledge to perform reasoning. However, there exists a significant disparity between answer accuracy and faithfulness of the CoT generated by LLMs, indicating that they often arrive at correct answers through incorrect reasoning.

## introduction while large language models (llms) have shown great potential as general-purpose task solvers, they tend to be unreliable reasoners @xcite . prior research suggests that llms demonstrate reasoning-like behaviors as the number of parameters increases @xcite . notably, chainof-thought (cot) prompting, where llms are explicitly instructed to decompose questions into a sequence of logical steps before generating answers, has achieved impressive performance in various reasoning tasks @xcite . however, as llms function as black-box models, the mechanism behind their reasoning processes remains largely unknown. ## preliminaries chain-of-thought (cot) reasoning chain-ofthought (cot) @xcite is a reasoning framework that prompts llms to generate a stepby-step reasoning process @xmath0, s 2 , . . . , s n } to a question q, where s i is a natural language sentence describing a step in the reasoning process. faithful cot a faithful cot should satisfy the following properties (creswell and shanahan, 2022): (i) there are no factual errors, (ii) the reasoning process is coherent (i.e., the conclusion of previous step s i-1 should be the prerequisite of the current step s i ), (iii) the reasoning process leads to the correct answers. examples of violations of these properties are shown in figure 1. knowledge graphs (kgs) knowledge graphs (kgs) are structured representations of knowledge that contain abundant facts in the form of triples @xmath1, r, e t ) | e h , e @xmath2, @xmath3, where e h and e t are head and tail entities, and r is the relation between them; e and r are the entity and relation sets respectively. a path in kgs is a sequence of triples: ## evaluating the cot reasoning of llms we propose a framework to evaluate the cot reasoning process of llms with the help of kgs. specifically, we propose two evaluation modules: discriminative evaluation and generative evaluation. the discriminative evaluation investigates whether llms possess enough knowledge for conducting faithful reasoning and the generative evaluation further analyzes whether llms can provide faithful reasoning process during cot generation. the overall framework is shown in figure 3 . ## experiment settings we use the proposed evaluations to understand the cot reasoning process of the state-of-the-art llms on complex question-answering (qa) tasks which requires performing multi-step reasoning to answer the questions. through analysis, we seek to answer the following research questions (rqs) ## conclusion we propose an evaluation framework to understand the cot reasoning capability of llms beyond the sole assessment of final answer accuracy. with the help of a kg and a careful prompting strategy, we can turn the unstructured cot into a structured format for automatic evaluation. our framework consists of two evaluation modules: (i) a discriminative module that isolates the effects of different reasoning errors to verify llms' knowledge about reasoning, and (ii) a generative module to assess the generated cot reasoning. while llms showcase remarkable capabilities in generating correct answers, our study emphasizes the need for more nuanced evaluations of their reasoning processes. addressing the gap between the final answer and reasoning accuracy remains a critical area for further exploration in enhancing the true reasoning capabilities and interpretability of llms.