Where am I ? Large Language Models Wandering between Semantics and Structures in Long Contexts (2024)

Abstract: As the utilization of Large Language Models (LLMs) becomes more widespread, there is a growing demand for their ability to handle more complex and longer external knowledge across various use cases. Most existing evaluations of the open-ended question answering (ODQA) task, which necessitates the use of external knowledge, focus solely on whether the model provides the correct answer. However, even when LLMs answer correctly, they often fail to provide an obvious source for their responses. Therefore, it is necessary to jointly evaluate and verify the correctness of the answers and the appropriateness of grounded evidence in complex external contexts. To address this issue, we examine the phenomenon of discrepancies in abilities across two distinct tasks—QA and evidence selection—when performed simultaneously, from the perspective of task alignment. To verify LLMs’ task alignment, we introduce a verification framework and resources considering both semantic relevancy and structural diversity of the given long context knowledge. Through extensive experiments and detailed analysis, we provide insights into the task misalignment between QA and evidence selection. Our code and resources will be available upon acceptance.

## introduction open-domain question answering (odqa) task aims to generate accurate answers for a wide range of contexts and queries @xcite . in the odqa task, large language models (llms) should be capable of leveraging fluent generative abilities and retrievalaugmented generation (rag)-based capabilities to effectively utilize more complicated and longer external knowledge from the real world. title: jingle bell rock [1] hall & oates released a version in 1983 as a non-album single which peaked at number 30 on the hot 100 recurrents chart in 2005 , it also reached number 6 on the billboard 's holiday airplay chart on december 13 , 2008 and number 24 on the hot holiday songs chart on december 10 , 2011 . helms' original version, on decca 9 … [2] … (…) [7] ``jingle bell rock'' is an american popular christmas song first released by bobby helms in 1957 ( after it was recorded in october @xcite . it has received frequent airplay in the united states during every christmas season since then . ``jingle bell rock'' was composed by joseph carleton beal @xcite and james ross boothe @xcite . beal was a massachusetts -born public relations professional and longtime resident of south ocean avenue in atlantic city , new jersey , and boothe was an american writer in the advertising business . one of the challenges in leveraging diverse realworld knowledge is that knowledge contexts may not contain only helpful information for generation @xcite . in other words, unrelated or fascinating but partially relevant information that may confuse the model can be spread across various locations within the knowledge context @xcite @xcite . this can lead llms to generate hallucinated content that contradicts reality or fail to clearly present the evidence source for their responses, as illustrated in figure 1 . in particular, this phenomenon becomes more severe as the knowledge context becomes longer @xcite . despite these challenges, conventional qa research primarily evaluates whether model outputs are included in the gold answer using metrics such as exact match (em) score, thus often not focusing on the correctness of the sources underlying the generated responses @xcite @xcite . recent studies on llms' evidence selection ability have explored various approaches, including modeling and prompting @xcite @xcite @xcite . however, these studies have typically treated qa and evidence source selection as separate tasks, relatively overlooking the discrepancy in the llms' task alignment capabilities. bridging this discrepancy is crucial for ensuring the explainability of llms' outputs and enhancing end-user reliability @xcite @xcite . therefore, when llms handle extensive external knowledge, it is essential to sufficiently verify not only the accuracy of responses but also the appropriateness of the corresponding evidence selected by models. to address these issues, we conduct an integrated and systematic study on the alignment between answer correctness and grounding appropriateness, which have traditionally been studied separately, in challenging long context settings. we explore task alignment by simultaneously performing two distinct tasks-qa and evidence selection, utilizing the same features. for systematic validation, we conduct an empirical study on llms' alignment capabilities, considering both the semantic relevancy and structural diversity of the long context knowledge they utilize. specifically, we construct semantically diverse surrounding input contexts by retrieving relevant and irrelevant information from multiple documents using the rag approach. additionally, we investigate the llms' behavior in detail by structurally diversifying the position of ground-truth evidence within complex contexts. from a task alignment perspective, we provide insights into the discrepancy between llms' capabilities in qa and evidence selection through extensive experiments and detailed analysis. furthermore, we conduct additional experiments and analyses based on context lengths and prompting strategies. our contributions are threefold: i) we introduce a framework and resource construction method for validating task misalignment occurring in realworld long context-based generation scenarios. ii) to achieve systematic validation, we devise the con-cepts of semantic relevancy and structural diversity, effectively investigating llms' task alignment capabilities and achieving enhanced explainability. iii) through extensive experiments and analysis, we provide various insights on the llms' behavior in qa and evidence selection tasks when generating based on long contexts. ## experimental setup in this section, we describe the process of constructing resources to verify the alignment between answer correctness and appropriateness of evidence selection in llms' responses within long-context scenarios. additionally, we explain the models and evaluation metrics adopted for the tasks. ## experiments and analysis in this section, we analyze the results of experimenting with llms by feeding challenging contexts that are semantically and structurally diversified in section 3.2. additionally, we provide various qualitative analyses (refer to appendix b). ## conclusion one of the challenges of utilizing long-context knowledge in real-world scenarios is that the knowledge context does not contain information only helpful for generation. consequently, llms often generate hallucinated contents or fail to adequately present the evidence source for their responses. therefore, we examine and interpret the phenomenon of discrepancies in llms' abilities when performing two different tasks-qa and evidence selection-simultaneously using the same input features, from the perspective of task alignment. to address this, we propose a task alignment verification framework and resource construction methodology based on the concepts of semantic relevance and structural diversity. through extensive experiments and analysis in settings that closely resemble real-world scenarios, we provide various insights to bridge the task alignment gap observed in llms. we hope our work serves as useful reference material for preventing hallucinations and enhancing the explainability of llms' generative behavior when given long contexts.