Long-Tailed Question Answering in an Open World (2023)

Abstract: Real-world data often have an open long-tailed distribution, and building a unified QA model supporting various tasks is vital for practical QA applications. However, it is non-trivial to extend previous QA approaches since they either require access to seen tasks of adequate samples or do not explicitly model samples from unseen tasks. In this paper, we define Open Long-Tailed QA (OLTQA) as learning from long-tailed distributed data and optimizing performance over seen and unseen QA tasks. We propose an OLTQA model that encourages knowledge sharing between head, tail and unseen tasks, and explicitly mines knowledge from a large pre-trained language model (LM).Specifically, we organize our model through a pool of fine-grained components and dynamically combine these components for an input to facilitate knowledge sharing.A retrieve-then-rerank frame is further introduced to select in-context examples, which guild the LM to generate text that express knowledge for QA tasks. Moreover, a two-stage training approach is introduced to pre-train the framework by knowledge distillation (KD) from the LM and then jointly train the frame and a QA model through an adaptive mutual KD method. On a large-scale OLTQA dataset we curate from 43 existing QA datasets, our model consistently outperforms the state-of-the-art.

## introduction real-world data often have a long-tailed and openended distribution @xcite . as a cornerstone for ai applications @xcite , question answering (qa) is widely investigated to tackle various qa tasks involving diverse formats and domains @xcite . the frequency distribution of qa tasks in our daily life is long-tailed @xcite , with a few head tasks of adequate samples and many more tail tasks of limited samples, and we continuously encounter new tasks that are not seen during training in an open world. we formally study open long-tailed qa (oltqa) emerging in natural data settings. a practical qa system shall learn from long-tailed distributed data, i.e., a few head tasks and many tail tasks, and it is expected to perform well over a balanced test set which include head, tail, and unseen tasks. oltqa must handle not only few-shot learning for tail tasks in the closed world @xcite , but also zero-shot learning for unseen tasks in an open world @xcite with one unified model. a major challenge for oltqa is the lack of knowledge required for the language understanding and reasoning abilities of qa tasks, especially under such low resource conditions @xcite . therefore, it is important for an oltqa model to share knowledge between head, tail, and unseen qa tasks @xcite , and mine knowledge from external resources @xcite . however, it is non-trivial to directly extend previous methods to the oltqa setting. specifically, an effective implementation of knowledge sharing is the multi-task learning (mtl) approach @xcite , in which task-specific components are maintained to preserve learned knowledge @xcite . as we constantly encounter new tasks in practice, it is challenging to directly apply mtl methods since they do not explicitly model samples from unseen tasks. another challenge is the absence of samples from unseen tasks in the training process, which leads to poor prior knowledge about unseen tasks. fortunately, a large pre-trained language model (lm) embeds broad-coverage knowledge that can help a variety of tasks @xcite . one key ingredient in lm knowledge mining is to select demonstrative in-context examples, which guild the lm to generate text that express knowledge for downstream tasks @xcite . however, few studies have explored selecting in-context examples to directly optimize qa performance in the oltqa setting. in this study, we propose an oltqa model to address challenges mentioned above for the oltqa setting. specifically, to encourage knowledge sharing between head and tail tasks while acknowledging the emergence of unseen tasks, we organize our model at the instance-level and use a dynamic architecture for each input @xcite , i.e., a pool of fine-grained components are maintained and dynamically combined in each forward pass based on the input @xcite . this scheme tackles unseen tasks, since the learned knowledge is distributed into different model components @xcite . we further mine knowledge from a large pretrained lm. concretely, we employ a retrieve-thenrerank frame @xcite to select demonstrative in-context examples for a test instance, which guide the lm to decode the output @xcite . the lm outputs are viewed as hints for qa tasks @xcite and leveraged for improving qa performance. the retrieve-thenrerank frame consists of an efficient retriever and an effective re-ranker @xcite , which is optimized by a two-stage training approach. the first stage pre-trains the retrieve-then-rerank framework by knowledge distillation from a pre-trained lm @xcite . the second stage jointly train the above framework and an encoder-decoder qa model through adaptive mutual knowledge distillation @xcite to allow information exchange between each other. our key contributions are summarized as follows: an instance-level knowledge sharing mechanism is introduced, and a retrieve-then-rerank frame is employed to mine knowledge from a large pre-trained lm through a novel twostage knowledge distillation training process. long-tailed learning focuses on long-tail distributed data @xcite . recent approaches for long-tailed learning include rebalancing @xcite , information augmentation @xcite , and module improvement @xcite . in this study, we attempt to build a qa model from long-tail distributed data by knowledge sharing and knowledge mining. knowledge mining from external resources is essential for building robust qa models @xcite . wikipedia and knowledge bases are used to improve qa performance @xcite . large pre-trained lms store rich knowledge, which is used to solve various tasks via conditioned generation @xcite . recent approaches build prompt retrievers to select in-context examples from a training set to optimize lm generation performance @xcite . however, these approaches cannot directly optimize our oltqa model. in this study, we jointly train a retrieve-then-rerank framework and a qa model to enhance qa performance. knowledge distillation (kd) is often employed to learn a student model using the knowledge distilled from a teacher model by enforcing the agreement of outputs between the two models @xcite . mutual kd helps a group of models mutually generate knowledge to train each other @xcite . our oltqa model jointly trains the retrieve-then-rerank frame and the qa model through adaptive mutual kd, encouraging them to collaborate with each other @xcite .