More Bang for your Context: Virtual Documents for Question Answering over Long Documents (2024)

Abstract: We deal with the problem of Question Answering (QA) over a long document, which poses a challenge for modern Large Language Models (LLMs). Although LLMs can handle increasingly longer context windows, they struggle to effectively utilize the long content. To address this issue, we introduce the concept of a virtual document (VDoc). A VDoc is created by selecting chunks from the original document that are most likely to contain the information needed to answer the user’s question, while ensuring they fit within the LLM’s context window. We hypothesize that providing a short and focused VDoc to the LLM is more effective than filling the entire context window with less relevant information. Our experiments confirm this hypothesis and demonstrate that using VDocs improves results on the QA task.

## introduction large language models (llms) such as openai's chatgpt 1 or anthropic's claude 2 , have demonstrated exceptional performance across a range of natural language processing (nlp) tasks and are capable of answering questions on various topics. recent advancements have extended their context window to encompass tens of thousands or even millions of tokens @xcite . this enables them to process long documents, or even books, and answer questions over them. despite this impressive progress, recent studies have shown that llms still struggle to utilize long contexts efficiently @xcite . llms' performance often deteriorates when confronted with a long document, hindering their ability to accurately answer questions on such documents. to address this challenge, we propose a method called virtual document @xcite . vdoc takes a 1 https://chat.openai.com/ 2 https://www.anthropic.com/news/claude-2-1 long document, a question, and a specified context length, and generates a shorter version of the document, containing the most relevant parts for the given length. vdoc operates in three steps: 1) document segmentation: break down the long document into smaller segments, 2) segments ranking: rank the segments based on their relevance to the question, and 3) vdoc reconstruction: reassemble the segments into a coherent virtual document that contains the most relevant information to the question, and still fits the given context length. the overall vdoc process is illustrated in figure 1 . we analyze experimentally, different approaches for each step. our results show that vdoc enhances question-answering performance on various datasets, including zeroscrolls @xcite , a benchmark designed for tasks requiring a long context. the paper's main contribution is the thorough analysis and experiments of the above three steps for qa over a long document. while the suggested solution is close to rag (retrieval augmented generation), it has some unique properties. for example, the ordering of the segments based on their sequence in the original document, that does not exist in rag. our code is available on git 3 2 related work many previous works focus on designing llms that can support long context. some works modify the transformer attention mechanism @xcite @xcite or make it more efficient @xcite @xcite . others have focused on modifying the positional embeddings @xcite . another line of research attempts to replace transformers, which have quadratic sequence length complexity, with new architectural designs like convolution and linear rnns, e.g., rwkv @xcite , s4 @xcite , hyena @xcite , or mamba @xcite . while those works were able to extend the context length of llms, subsequent research has found that long-context transformers exhibit a recency bias and do not effectively utilize long-range context @xcite @xcite . specifically, these models have been found to struggle with ignoring irrelevant information @xcite . moreover, benchmarks for long context reveal that there is still significant room for improvement compared to human performance, and for some tasks, models even struggle to surpass the naive baseline @xcite . ## method we deal with the problem of using an llm to answer questions over a long document. our method, termed vdoc, creates a shorter version of the document, which is then used as input to the llm, instead of the entire document, to generate an answer. creating the vdoc is done in three steps. i) document segmentation into manageable segments, ii) segments ranking by their potential to contain the answer to the user question, and iii) vdoc reconstruction. the detailed steps are described below. document segmentation. we try two methods. i) sliding-window on sentence boundaries using nltk foot_1 ii) context-aware (using html structure when available). we define a segment-size parameter that controls the max segment length. segments ranking. we experiment with three families of language models (lm). the first is encoder-only models, where we rank segments by the similarity of their representation to the representation of the user question. an example of such lm is minilm foot_2 . the second, is encoder-decoder or decoder-only models, where we rank segments by the inverse perplexity of the user question, given each of the segments. specifically, we give each segment as input to the model with the instruction: "generate a conversation between a user and an agent based on the given content:". we then run a forward loop and compute the cross-entropy loss for the question. the segments are ranked by @xmath0. an example is flan-t5 foot_3 . the third, is a sparse representation, elser foot_4 which expands and indexes documents into elas-ticsearch foot_5 index, with semantic terms that were learned to co-occur frequently with terms in each document. we note that among the three segment ranking methods, the first (comparing dense representations) and the last (sparse representation) can be pre-computed on the given set of documents. the second method on the other hand (using perplexity), requires running a forward loop of the model for each segment and question, and thus can not be pre-computed, because it depends on the question. thus the perplexity method can be quite slow for very long documents. ## conclusions and future work we described a method called vdoc, enabling question answering using llm over long documents. the method is based on creating a virtual document that contains relevant information to the user query, and adheres to the llm's contextwindow limitations. our method does not require any modifications or fine-tuning a llm. we compared different methods and showed experimentally, that it is better to aggressively shorten the document, even if the llm can handle a larger context-window size, both in terms of quality of the results, and in terms of better run times. you are given a scientific article and a question. answer the question as concisely as you can, using a single phrase or sentence if possible. if the question cannot be answered based on the information in the article,write "unanswerable". if the question is a yes/no question, answer "yes", "no", or "unanswerable".