Improving Retrieval Augmented Open-Domain Question-Answering with Vectorized Contexts (2024)

Abstract: In the era of large language models, applying techniques such as Retrieval Augmented Generation can better address Open-Domain Question-Answering problems. Due to constraints including model sizes and computing resources, the length of context is often limited, and it becomes challenging to empower the model to cover overlong contexts while answering questions from open domains. This paper proposes a general and convenient method to cover longer contexts in Open-Domain Question-Answering tasks. %It leverages a small encoder language model that effectively encodes contexts, and the encoding applies cross-attention with origin inputs.It leverages a small encoder and cross-attention mechanism and effectively encodes contexts. With our method, the original language models can cover several times longer contexts while keeping the computing requirements close to the baseline. Our experiments demonstrate that after fine-tuning, there is improved performance across two held-in datasets, four held-out datasets, and also in two In Context Learning settings. Our code will be released at https://github.com/Alibaba-NLP/Vec-RA-ODQA.

## introduction transformer-based @xcite architectures with pre-training on large corpus have become popular in recent natural language processing research @xcite @xcite . an increasing number of natural language processing (nlp) tasks need to process long contexts such as open-domain question answering (odqa) with retrieval augmented generation (rag) @xcite @xcite . however, the fine-tuning and inference stages in downstream tasks are still constrained by the input length, e.g., 2048 tokens for bloomz @xcite and llama-1 @xcite . with rag, the input can easily surpass the maximum length the model can handle and it becomes text form @xcite knowledge: alvin and the ch… q: who was the man behind the chipmunks? ## analysis in this section, we present the results of three analytical experiments. the first one shows the result of the other training strategy discussed in sec. 2.4. the second shows the evaluation results of optimizing objective 3. the third shows the effectiveness of our method in a more challenging setting. ## conclusion in this paper, we propose a method that incorporates a small encoder model for excessively long context encoding by applying cross-attention mechanism with the original task model. the method is simple and general for transformer-based language models. in our experiments, after fine-tuning on odqa dataset, we find improved performance across two held-in, four held-out and two icl settings, compared to a baseline that incorporates the reranking technique on training data, showing the effectiveness of our method in utilizing long contexts. we note that the intuitive explanations for the performance improvement are as follows: 1) the encoder model provides the ability to encode longer contexts; 2) the cross-attention mechanism is useful in selectively attending the correct parts of the inputs. regarding the efficiency, the need for gpu quantity remains unchanged and the run time remains competitive to the baseline.