Drilling Down into the Discourse Structure with LLM s for Long Document Question Answering (2023)

Abstract: We address the task of evidence retrieval for long document question answering, which involves locating relevant paragraphs within a document to answer a question. We aim to assess the applicability of large language models (LLMs) in the task of zero-shot long document evidence retrieval, owing to their unprecedented performance across various NLP tasks. However, currently the LLMs can consume limited context lengths as input, thus providing document chunks as inputs might overlook the global context while missing out on capturing the inter-segment dependencies. Moreover, directly feeding the large input sets can incur significant computational costs, particularly when processing the entire document (and potentially incurring monetary expenses with enterprise APIs like OpenAI’s GPT variants). To address these challenges, we propose a suite of techniques that exploit the discourse structure commonly found in documents. By utilizing this structure, we create a condensed representation of the document, enabling a more comprehensive understanding and analysis of relationships between different parts. We retain 99.6% of the best zero-shot approach’s performance, while processing only 26% of the total tokens used by the best approach in the information seeking evidence retrieval setup. We also show how our approach can be combined with *self-ask* reasoning agent to achieve best zero-shot performance in complex multi-hop question answering, just \approx 4 % short of zero-shot performance using gold evidence.

## introduction long document question answering (ldqa) is a complex task that involves locating relevant evidence from lengthy documents to provide accurate answers to specific questions @xcite . ldqa is challenging for the following reasons -a) long documents often exceed the maximum token limit of existing transformer-based pretrained language models (plms) @xcite @xcite , posing a challenge in directly processing their content to extract pertinent information @xcite . b) the information required to answer a question is often dispersed across different sections or paragraphs within the document which may require sophisticated reasoning process to identify and extract the relevant information @xcite . c) processing the entire document to find answers can be computationally expensive and inefficient @xcite . one popular approach for ldqa is the retrievethen-read method @xcite @xcite @xcite , where relevant paragraphs are retrieved from the document to provide the answer. a major drawback of existing works is reliance on supervised fine-tuning for the evidence selection phase, exhibiting poor generalization on out-of-distribution data @xcite . given the remarkable few-shot/zero-shot performance and enhanced generalization capabilities demonstrated by large language models (llms) across various natural language generation and understanding tasks @xcite @xcite @xcite , we investigate the potential of leveraging these llms for zero-shot evidence retrieval. notably, llms that have been instruction fine-tuned @xcite or trained using reinforcement learning with human feedback @xcite exhibit exceptional generalization performance even on unseen tasks @xcite @xcite . thus, we explore the feasibility of utilizing llms for zeroshot evidence retrieval. however, llms, which are based on transformer architecture @xcite , are limited by their context length and suffer from expensive inference times that increase quadratically with the number of tokens in the input. additionally, utilizing enterprise llm solutions such as openai's gpt-3.5-turbo, text-davinci-003, gpt-4, etc. 1 to process an entire long document without optimizations would incur significant monetary costs. this highlights the need for an llm-based evidence retrieval solution that can achieve faster and more cost-effective inference by selectively processing relevant portions of the document, without compromising downstream performance. to overcome these challenges, we harness the inherent discourse structure commonly present in long documents. this structure encompasses the organization of topics, semantic segments, and information flow, enabling effective information search and knowledge acquisition for question answering. @xcite @xcite @xcite . utilizing this valuable structure, we construct a condensed representation of the document by replacing the content within each section with a corresponding summary. this condensed representation is then fed to the llm, enabling efficient processing of tokens while allowing the model to comprehensively analyze the entire input context for identifying relevant sections. thereafter, the content within each relevant section is further processed by the llm for finegrained evidence retrieval. we call our proposed approach d 3 (drilling down into the discourse) due to the nature of the solution described above. our approach undergoes evaluation in two distinct settings: information seeking and multi-hop reasoning in question answering. in the information seeking experiments, our approach retains the best zero-shot state-of-the-art (sota) results, while only utilizing 26% of the tokens employed by the sota approach. additionally, we examine the robustness of our model across various document lengths and analyze the number of tokens required and latency for different zero-shot approaches. moreover, we explore the integration of our approach with other zero-shot techniques within an agent framework designed to break down intricate queries into a sequence of simpler followup queries. ## baselines: we consider the following zero-shot approaches for performance comparison: this model is offered in the sentence-transformers (reimers and gurevych, 2019) library. while the library provides many different model checkpoints, we chose cross-encoder-ms-marco-minilm-l-12-v2 as it yielded highest f 1 score for evidence retrieval. (4) paragraph: every paragraph in d is processed by the llm independently to assess its relevance to q through a boolean prompt. ## experiments and results we mainly assess the applicability of our method in two scenarios: (1) §5.1: information-seeking setting @xcite and (2) §5.1: multihop reasoning in question answering @xcite . thereafter, we conduct an extensive analysis of our approach's performance across various configurations, as discussed in §5.3.1, and examine its effectiveness in different document length categories, as outlined in §5.3.2. thereafter, we justify the need for the inclusion of global context modeling in §5.3.3, and highlight the significance of incorporating discourse information in §5.3.4. finally, §5.3.5 we compare our methods performance with the best performing zero-shot approach for different categories and identify scope for improvement. in our analyses, we will also include insights into the inference cost and latency associated with utilizing large language models (llms) for the evidence retrieval stage. this will encompass the monetary cost and the processing efficiency measured in terms of the number of tokens processed and the number of llm inferences. it is important to note that these measurements pertain exclusively to llms and do not encompass smaller fine-tuned models. unless otherwise specified, we would be using base approach for fine-grained retrieval and the overall approach would be called d 3 -base. due to the monetary cost associated with gpt-based llms, we experiment with 150 randomly sampled documents for all the experiments. ## conclusion we demonstrated a zero-shot approach for evidence retrieval which leverages the discourse structure of the document for information categorization which not only yielded competitive performance for information seeking and question answering with multihop reasoning setting, but also processed lowest number of tokens resulting in significant compute and cost savings. this approach demonstrates several desirable characteristics such as robustness in evidence retrieval performance, lower latency, etc. all across different document length ranges.