Context Matters: Pushing the Boundaries of Open-Ended Answer Generation with Graph-Structured Knowledge Context (2024)

Abstract: This paper introduces a novel framework that combines graph-driven context retrieval in conjunction to knowledge graphs based enhancement, honing the proficiency of LLMs, especially in domain specific community question answering platforms like AskUbuntu, Unix, and ServerFault. We conduct experiments on various LLMs with different parameter sizes to evaluate their ability to ground knowledge and determine factual accuracy in answers to open-ended questions. Our methodology GraphContextGen consistently outperforms dominant text-based retrieval systems, demonstrating its robustness and adaptability to a larger number of use cases. This advancement highlights the importance of pairing context rich data retrieval with LLMs, offering a renewed approach to knowledge sourcing and generation in AI systems. We also show that, due to rich contextual data retrieval, the crucial entities, along with the generated answer, remain factually coherent with the gold answer. We shall release the source code and datasets upon acceptance.

## introduction in artificial intelligence, large language models (llms) @xcite have revolutionized text understanding @xcite and generation @xcite . despite their impressive capabilities, llms struggle in low-resource settings @xcite , are constrained by knowledge cutoffs, and often produce hallucinations @xcite . additionally, managing the trade-off between quality and the vast number of parameters @xcite presents challenges, particularly for researchers with limited resources. to overcome these limitations, new methods such as grounding llms 1 and retrieval-augmented generation (rag) @xcite have been proposed. these approaches enable models to access external databases, enhancing their responses with current, detailed, and accurate information. a critical aspect of effective knowledge grounding is the retrieval mechanism @xcite . traditional text-based retrieval methods are evolving to handle more complex questions, moving beyond simple keyword matching. current techniques often struggle with determining optimal chunk sizes foot_0 for indexing and querying, leading to inconsistent results. graph-based retrieval systems offer a solution by capturing intricate relationships through structured data, providing deeper semantic understanding and more contextually relevant results @xcite . these systems adapt to evolving data, uncovering insights and forming connections among diverse entities. this technique is vital in various applications, including dialogue systems @xcite , opendomain question answering @xcite , and novelty-controlled paraphrasing @xcite . for example, stackoverflow's overflowai ## related work over the years several approaches such as feature based methods @xcite , cnn @xcite , rnn @xcite , attention mechanism @xcite have been proposed for answer selection and summarization. some recent research focuses on summarizing diverse content on stackoverflow @xcite , using answerbot-an answer summary generator @xcite , opiner-which summarizes api reviews @xcite , extracting key sentences to guide developers on stackoverflow @xcite , and multi-document summarization @xcite . in the realm of answer summarization, numerous studies @xcite harness graphical structures, leverage existing graph-based summarizers @xcite @xcite , and employ graph-centric measures. in the age of llms, some research has centered around controlled summary generation via effective keyword-based prompting @xcite . in this experiment, we select three domainspecific datasets from open-source cqa platforms: askubuntu, serverfault, and unix, all of which originate from a low-resource domain with minimal properly annotated data available on these topics. these datasets, considered from june 2023, includes questions (title and body), a list of answers, an accepted answer flag, tags for the questions, and the posting dates and times for both questions and answers. for each question, the accepted answer serves as the ground truth. due to the limited resources of the datasets and the high expenses associated with human involvement, we opt not to use human annotations. further we apply several filtering procedures, such as duplicate question removal, non-specific answer removal, and length constraints (token limit in llm) resulting in our dataset. adopting the temporal splitting approach inspired by @xcite , we consider training set from 2019-2020 and test dataset from 2021-2023. due to resource limitations, we randomly sample test dataset, the details of which are provided in ## methodology in this section, we explain our proposed framework graphcontextgen. the overall framework is shown in figure 1 . our proposed framework consists of three modules -(1) context retriever (2) kg-driven context enhancement (3) answer generation. before explaining every module, we describe the problem in detail below. ## human evaluation in the results of the automatic evaluation, it is essential to remember that low values of bertscore or factsumm might also correspond to lower-quality ground truth answers posted by humans. thus, there is a possibility that the model-generated answers are superior in quality compared to the ground truth answers. such points can be verified only by human judgment experiments presented in figure 3 and the appendix a.6. ## error analysis this section presents a systematic error analysis highlighting the error types and corresponding examples (see misaligned retrieval outcomes: this misalignment occurs when the retrieved content, accurate in its context, doesn't match the user's intended query, leading to off-target responses due to the generation's reliance on the retrieved data. on platforms like askubuntu, unix, and serverfault, overlapping themes, like a 'boot issues' query retrieving 'usb booting' content instead of 'system booting problems', exacerbates the issue. entity misalignment: this issue arises when the retrieval mechanism accurately finds data but incorrectly links it to an entity in the knowledge graph, causing responses to deviate from the user's context. for example, in unix, a term like 'read' might refer to a command or a configuration file, leading to misassociations if not accurately disambiguated. composite query conundrums: this problem occurs when a user's query involves multiple issues, and the retrieval system typically focuses primarily on one, neglecting the others ## complexity analysis as the dimension is fixed, the cosine similarity between two embeddings is o(1). for training set questions (say n @xmath0, it becomes n q × o(1) = o(n q ). by including the query in the graph, pagerank becomes o((n q +1)+(e q +e ′ )) = o(n q +1+e q +e ′ ) = o(n q +e q +e ′ ). assuming e ′ can be at max(n q ), it simplifies to o(n q +n q +e q ) = o(n q + e q ). therefore, the total complexity until pagerank calculation for each test instance is o(n q + e q ). the worst-case complexity could be o(n q + n 2 q ) = o(n 2 q ). for each dataset, n q and e q remain constant for all test instances. ## conclusion this study addresses a notable challenge in cqa platforms: the automatic generation of answers across popular platforms often lacks clear problem definitions, leading to issues with proper knowledge grounding and factually incoherent responses. we present graphcontextgen, which, to the best of our knowledge, is the first solution that uses graph retrieval combined with knowledge graph context for this challenge in domain-specific cqa platforms. our evaluations indicate that this model outperforms previous prominent approaches. notably, human evaluators determine that answers generated by graphcontextgen exhibit greater factual coherence and knowledge grounding. we further demonstrate that researchers with constrained gpu resources can adopt this solution with smaller parameter llms and achieve performance that are at par with larger models. 296 using knowledge graphs with llms to generate technical answers encounters several challenges. the effectiveness of this approach partially depends on the accuracy and completeness of the knowledge graphs, as any gaps or inaccuracies can mislead the llm's responses. llms tailored for summarization may sacrifice detail for conciseness, potentially overlooking critical nuances of technical topics. challenges in handling ambiguous contexts and the potential for biases introduced during knowledge graph construction and llm training further complicate accurate answer generation. moreover, the computational resources required for processing large-scale knowledge graphs pose scalability issues. ensuring that knowledge graphs remain up-to-date and that llms can adapt to new information without frequent retraining are ongoing concerns. finally, while llms are versatile, they may not achieve the level of specificity and accuracy provided by systems specialized in particular non niche domains.