Chain-of-Skills: A Configurable Model for Open-Domain Question Answering (2023)

Abstract: The retrieval model is an indispensable component for real-world knowledge-intensive tasks, e.g., open-domain question answering (ODQA). As separate retrieval skills are annotated for different datasets, recent work focuses on customized methods, limiting the model transfer- ability and scalability. In this work, we propose a modular retriever where individual modules correspond to key skills that can be reused across datasets. Our approach supports flexible skill configurations based on the target domain to boost performance. To mitigate task interference, we design a novel modularization parameterization inspired by sparse Transformer. We demonstrate that our model can benefit from self-supervised pretraining on Wikipedia and fine-tuning using multiple ODQA datasets, both in a multi-task fashion. Our approach outperforms recent self-supervised retrievers in zero-shot evaluations and achieves state-of-the-art fine-tuned retrieval performance on NQ, HotpotQA and OTT-QA.

## introduction gathering supportive evidence from external knowledge sources is critical for knowledgeintensive tasks, such as open-domain question answering (odqa; @xcite and fact verification @xcite . since different odqa datasets focus on different informationseeking goals, this task typically is handled by customized retrieval models @xcite @xcite . however, this dataset-specific paradigm has limited model scalability and transferability. for example, augmented training with single-hop data hurts multi-hop retrieval @xcite . further, as new information needs constantly emerge, dataset-specific models are hard to reuse. in this work, we propose chain-of-skills (cos), a modular retriever based on transformer @xcite , where each module implements a reusable skill that can be used for different odqa datasets. here, we identify a set of such retrieval reasoning skills: single retrieval, expanded query retrieval, entity span proposal, entity linking and reranking ( §2). as shown in figure 1 , recent work has only explored certain skill configurations. we instead consider jointly learning all skills in a multi-task contrastive learning fashion. besides the benefit of solving multiple odqa datasets, our multi-skill formulation provides unexplored ways to chain skills for individual use cases. in other words, it allows flexible configuration search according to the target domain, which can potentially lead to better retrieval performance ( §4). for multi-task learning, one popular approach is to use a shared text encoder @xcite , i.e., sharing representations from transformer and only learning extra task-specific headers atop. however, this method suffers from undesirable task interference, i.e., negative transfer among retrieval skills. to address this, we propose a new modularization parameterization inspired by the recent mixture-ofexpert in sparse transformer @xcite , i.e., mixing specialized and shared representations. based on recent analyses on transformer @xcite , we design an attention-based alternative that is more effective in mitigating task interference ( §5). further, we develop a multi-task pretraining using self-supervision on wikipedia so that the pretrained cos can be directly used for retrieval without dataset-specific supervision. to validate the effectiveness of cos, we consider zero-shot and fine-tuning evaluations with regard to the model in-domain and cross-dataset generalization. six representative odqa datasets are used: natural questions (nq; @xcite , webquestions (webq; @xcite , squad @xcite , entityquestions @xcite , hotpotqa @xcite and ott-qa @xcite , where the last two are multi-hop datasets. experiments show that our multi-task pretrained retriever achieves superior zero-shot performance compared to recent state-of-the-art (sota) self-supervised dense retrievers and bm25 @xcite . when fine-tuned using multiple datasets jointly, cos can further benefit from high-quality supervision effectively, leading to new sota retrieval results across the board. further analyses show the benefits of our modularization parameterization for multi-task pretraining and finetuning, as well as flexible skill configuration via chain-of-skills inference. foot_0 2 background we consider five retrieval reasoning skills: single retrieval, expanded query retrieval, entity linking, entity span proposal and reranking. convention-ally, each dataset provides annotations on a different combination of skills (see table a1 ). hence, we can potentially obtain training signals for individual skills from multiple datasets. below we provide some background for these skills. single retrieval many odqa datasets (e.g., nq; @xcite concern simple/singlehop queries. using the original question as input (figure 2 bottom-left), single-retrieval gathers isolated supportive passages/tables from target sources in one shot @xcite . expanded query retrieval to answer complex multi-hop questions , it typically requires evidence chains of two or more separate passages (e.g., hot-potqa; @xcite or tables (e.g., ott-qa; @xcite . thus, follow-up rounds of retrieval are necessary after the initial single retrieval. the expanded query retrieval @xcite takes an expanded query as input, where the question is expanded with the previous-hop evidence (figure 2 bottom-center). the iterative retrieval process generally shares the same target source. entity span proposal since many questions concern entities, detecting those salient spans in the question or retrieved evidence is useful. the task is related to named entity recognition (ner), except requiring only binary predictions, i.e., whether a span corresponds to an entity. it is a prerequisite for generating entity-centric queries (context with target entities highlighted; figure 2 bottom-right) where targeted entity information can be gathered via downstream entity linking. entity linking mapping detected entities to the correct entries in a database is crucial for analyzing factoid questions. following @xcite , we consider an entity-retrieval approach, i.e., using the entity-centric query for retrieving its corresponding wikipedia entity description. rereanking previous work often uses a reranker to improve the evidence recall in the top-ranked candidates. typically, the question with a complete evidence chain is used together for reranking. ## approach in this work, we consider a holistic approach to gathering supportive evidence for odqa, i.e., the evidence set contains both singular tables/passages (from single retrieval) and connected evidence chains (via expanded query retrieval/entity linking). as shown in figure 2 , cos supports flexible skill configurations, e.g., expanded query retriever and the entity linker can build upon the single-retrieval results. as all retrieval skill tasks are based on contrastive learning, we start with the basics for our multi-task formulation. we then introduce our modularization parameterization for reducing task interference. lastly, we discuss ways to use selfsupervision for pretraining and inference strategies. ## question answering experiments here, we conduct end-to-end question-answering experiments on nq, ott-qa and hotpotqa, using retrieval results from cos. following the literature, we report exact match (em) accuracy and f1 score. for nq and ott-qa, we re-implement the fusion-in-encoder (fie) model @xcite because of its superior performance on nq. for nq, the model reads top-100 passages returned by cos, and for ott-qa, the model reads top-50 evidence chains, in order to be comparable with previous work. here, separate models are trained for each dataset independently. due to space constraints, we only present the results on ott-qa and leave the nq results to table a2 . the ott-qa results are summarized in finally, for hotpotqa, since the task requires the model to predict supporting sentences in addition to the answer span, we follow @xcite to train a separate reader model to learn answer prediction and supporting sentence prediction jointly. due to space constraints, we leave the full results to table a3 . overall, our method achieves competitive qa performance against the previous sota with improved exact match accuracy. ## related work dense retrievers are widely used in recent literature for odqa @xcite . while most previous work focuses on single retrieval @xcite , some efforts have also been made towards better handling of other query types. @xcite propose a joint model to handle both single retrieval and expanded query retrieval. @xcite train a dense model to learn salient phrase retrieval. @xcite build an entity linker to handle multi-hop retrieval. nevertheless, all those models are still customized for specific datasets, e.g., only a subset of query types are considered or separate models are used, making them un-reusable and computationally intensive. we address these problems by pinning down a set of functional skills that enable joint learning over multiple datasets. mixure-of-expert models have also become popular recently @xcite . methods like gated routing @xcite or stochastic routing of experts @xcite do not differentiate the knowledge learned by different experts. instead, our work builds expert modules that learn reusable skills which can be flexibly combined for different use cases. another line of work focus on unsupervised dense retrievers using self-supervised data constructed from the inverse-cloze-task @xcite , random croppings @xcite , truncation of passages with the same span @xcite , hyperlink-induced passages @xcite or synthetic qa pairs @xcite . other model architecture adjustments on transformer for retrieval are proposed @xcite . our work can be viewed as a synergy of both. our multi-task pretrained model can perform better zero-shot retrieval. our modular retriever can be further fine-tuned in a multi-task fashion to achieve better performance. ## conclusions in this work, we propose a modular model chain-of-skills (cos) that learns five reusable skills for odqa via multi-task learning. to reduce task interference, we design a new parameterization for skill modules. we also show that skills learned by cos can be flexibly chained together to better fit the target task. cos can directly perform superior zero-shot retrieval using multitask self-supervision on wikipedia. when finetuned on multiple datasets, cos achieves sota results across the board. for future work, we are interested in exploring scaling up our method and other scenarios, e.g., commonsense reasoning @xcite and biomedical retrieval @xcite . group at microsoft research for their helpful discussions and anonymous reviewers for their valuable suggestions on this paper.