Chain-of-Rewrite: Aligning Question and Documents for Open-Domain Question Answering (2024)

Abstract: Despite the advancements made with the retrieve-then-read pipeline on open-domain question answering task, current methods still face challenges stemming from term mismatch and limited interaction between information retrieval systems and large language models. To mitigate these issues, we propose the Chain-of-Rewrite method, which leverages the guidance and feedback gained from the analysis to provide faithful and consistent extensions for effective question answering. Through a two-step rewriting process comprising Semantic Analysis and Semantic Augmentation, the Chain-of-Rewrite method effectively bridges the gap between the user question and relevant documents. By incorporating feedback from the rewriting process, our method can self-correct the retrieval and reading process to further improve the performance. Experiments on four open-domain question answering datasets demonstrate the effectiveness of our system under zero-shot settings.

## introduction open-domain question answering (odqa) @xcite @xcite ) is a long-standing task aimed at answering a wide variety of user questions without providing specific background documents. recently, with the emergence of large language models (llms), odqa systems typically employ a retrieve-then-read paradigm @xcite @xcite , where an information retrieval system is used to identify relevant contexts from knowledge bases, and then an llm such as instructgpt @xcite and llama @xcite is used to read the context and generate comprehensive and accurate answers. due to its effectiveness and simplicity, the retrieve-then-read paradigm performs well ## proposed method our method is based on the retrieve-then-read pipeline @xcite @xcite , which first uses a retriever to retrieve relevant documents from external knowledge source, and then uses a reader to predict the what genre is the trip? ... one of the greatest television programs of all time at the 2013 british comedy awards. it was nominated for the 2011 bafta television award for best situation comedy ... … plot. the trip (season 1). in an effort to impress his gourmet girlfriend, mischa (margo stilley), actor steve coogan accepts a commission from "the observer"... ## conclusion we propose an unsupervised rewriting method, chain-of-rewrite, which utilizes pre-trained language models to effectively address the term mismatch and limited interaction problems. we design two modules, semantic analysis and semantic augmentation, to progressively bridge the mismatch between the user question and relevant documents while maintaining semantic consistency. during the two-step rewriting process, we incorporate automatic feedback into the retrieval and reading process to self-correct the potential errors. extensive experiments on four odqa datasets demonstrate the effectiveness of our system in performing question answering, passage retrieval and passage ranking tasks under zero-shot settings.