Analysis of LLM ’s “Spurious” Correct Answers Using Evidence Information of Multi-hop QA Datasets (2024)

Abstract: Recent LLMs show an impressive accuracy on one of the hallmark tasks of language understanding, namely Question Answering (QA). However, it is not clear if the correct answers provided by LLMs are actually grounded on the correct knowledge related to the question. In this paper, we use multi-hop QA datasets to evaluate the accuracy of the knowledge LLMs use to answer questions, and show that as much as 31% of the correct answers by the LLMs are in fact spurious, i.e., the knowledge LLMs used to ground the answer is wrong while the answer is correct. We present an analysis of these spurious correct answers by GPT-4 using three datasets in two languages, while suggesting future pathways to correct the grounding information using existing external knowledge bases.

## introduction question answering (qa) is one of the hallmark tasks that evaluate language understanding capabilities of nlp systems. we are currently witnessing the flourishment of highly capable large language models (llms) that solve this complex task, requiring both knowledge and inference skills, with an impressive accuracy @xcite . on the other hand, it has been shown that llms can generate content that contradicts facts @xcite , and several verification results have been reported regarding the evaluation of llms' internal knowledge and whether llms can provide answers based on facts @xcite @xcite @xcite . at this point, it is not clear exactly to what extent such llms possess the knowledge needed to solve qa problems and how accurately they perform inference to leverage that knowledge. how often do llms rely on "hallucinated" knowledge during inference? can these hallucinations be remedied by structured knowledge bases (kbs) carefully crafted by humans? previous studies have reported that correct answers are often obtained despite errors in the reasoning path that llms output to solve qa @xcite @xcite . @xcite shows the specific error patterns by question type in such cases and the possibility of complementing errors with kbs using jemhopqa dataset foot_0 , which has evidence information in the form of triples, but their analysis is limited to one dataset in japanese. in this paper, we focus on investigating how such "spurious" correct answers by llms occur more deeply in other datasets and languages. we use three datasets from two languages -hot-potqa (yang et al., 2018) with r 4 c (inoue et al., ## conclusions in this paper, we presented the evaluation of gpt-4 on multi-hop qa in three datasets in english and japanese, focusing on how the answers are/are not grounded on the knowledge internal to the model. the results show that almost all of the incorrect answers are due to knowledge hallucination, and that even when the answer is correct, up to 31% of them (40% in numerical comparison questions) are in fact spurious. we also showed that the knowledge gpt-4 uses for grounding is complementary with external kbs, indicating a future direction of integrating them for solving multi-hop questions. our analysis is based on the assumption that the derivation triples generated by the llm are reasoning of the llm, but we hope to clarify whether this assumption is correct in the future.