Multi-source Meta Transfer for Low Resource Multiple-Choice Question Answering (2020)

Abstract: Multiple-choice question answering (MCQA) is one of the most challenging tasks in machine reading comprehension since it requires more advanced reading comprehension skills such as logical reasoning, summarization, and arithmetic operations. Unfortunately, most existing MCQA datasets are small in size, which increases the difficulty of model learning and generalization. To address this challenge, we propose a multi-source meta transfer (MMT) for low-resource MCQA. In this framework, we first extend meta learning by incorporating multiple training sources to learn a generalized feature representation across domains. To bridge the distribution gap between training sources and the target, we further introduce the meta transfer that can be integrated into the multi-source meta training. More importantly, the proposed MMT is independent of backbone language models. Extensive experiments demonstrate the superiority of MMT over state-of-the-arts, and continuous improvements can be achieved on different backbone networks on both supervised and unsupervised domain adaptation settings.

## introduction recently, there has been a growing interest in making machines to understand human languages, and a great progress has been made in machine reading comprehension (mrc). there are two main types of mrc task: 1) extractive/abstractive question answering (qa) such as squad @xcite and drop @xcite ; 2) multiplechoice qa (mcqa) such as multirc @xcite and dream @xcite . different from extractive/abstractive qa whose answers are usually limited to the text spans exist in the passage, the answers of mcqa may not appear in the text passage and may involve complex language inference. thus, mcqa usually requires more advanced reading comprehension abilities, including arithmetic operation, summarization, logic reasoning and commonsense reasoning @xcite , and etc. in addition, the size of most existing mcqa datasets is much smaller than that of the extractive/abstractive qa datasets. for instance, all the span-based qa datasets, except cq @xcite , contain more than 100k samples. in contrast, the data size of most existing mcqa datasets are far less than 100k (see the above two major challenges make mcqa much more difficult to optimize and generalize, especially for the low resource issue. in order to achieve better performance on downstream nlp tasks, it is inevitable to fine-tune the pre-trained deep language models @xcite @xcite @xcite with a large number of supervised target data for reducing the discrepancy between the training source and target data. due to the low resource nature, the performance of most existing mcqa methods is far from satisfactory. to alleviate such issue in mcqa, one straightforward solution is to merge all available data resources for training @xcite . however, the data heterogeneity of datasets (e.g., resource domains, answer types and varies diversity of choice size across different mcqa datasets.) hinders the practical use of this strategy. to better discover the hidden knowledge across multiple data sources, we propose a novel framework termed multi-source meta transfer (mmt). in this framework, we first propose a module named multi-source meta learning (mml) that extends traditional meta learning to multiple sources where a series of meta-tasks on different data resources is constructed to simulate lowresource target task. in this way, a more generalized representation could be obtained by considering multiple source datasets. on the top of it, the meta transfer learning (mtl) is integrated into multi-source meta training to further reduce the distribution gap between training sources and the target one. different from traditional meta learning that assumes tasks generated from the similar distribution/same dataset, mmt is able to discover the knowledge across different datasets and transfer it into the target task. more importantly, mmt is agnostic to the upstream framework, i.e., it can be seamlessly incorporated into any existing backbone language models to improve performance. figure 1 briefly illustrates both meta learning and the proposed mmt. ## methodology many existing mcqa tasks suffer from the lowresource issue, which requires a special training strategy to tackle it. recent advance of meta learning shows its advantages in solving the few-shot learning problem. typically, it can rely on only a very small number of training samples to train a model with good generalization ability @xcite . unfortunately, the existing meta learning algorithms are unable to be applied in our problem setting directly, since they are based on the assumption that the meta tasks are generated from the same data distribution @xcite . for example, one of the most popular benchmarks is the mini-imagenet dataset that was proposed by @xcite , and it consists of 100 sub-classes from imagenet dataset. all the meta tasks generated from the same training dataset have similar properties. in contrast, in our studied problem mcqa, data properties such as answer, question type, and commonsense are greatly vary across the mcqa datasets. specifically, the passages and questions come from different scenarios (such as exams, dialogues, and stories), and the answering choice contains more complex semantic information than the fixed categories in mini-imagenet. therefore, simply combining all the data resources into one and feeding it into existing meta learning algorithms is not an optimal solution (the experimental results in figure 5 also support this point). to address the data heterogeneity challenge and cater to the mcqa task, we extend the traditional meta learning method to multiple training sources scenarios, where we fully exploit multiple interdomain sources to learn more generalized representations. specifically, multi-source meta learning performs meta learning among multiple sources in sequence, thereby completing one iteration. however, multi-source meta learning alone cannot guarantee the desirable performance due to the data distribution gap between multiple sources and target data. therefore, transfer learning from multisources to target is required. here we introduce meta transfer learning into each meta learning iteration, which aims at reducing the discrepancy between the learned meta representation from multisource and target. ## conclusion in this work, we propose a novel method named multi-source meta transfer for multiple-choice question answering on low resource setting. our method considers multiple sources meta learning and target fine-tuning into a unified framework, which is able to learn a general representation from multiple sources and alleviate the discrepancy between source and target. we demonstrate the superiority of our methods on both supervised setting and unsupervised domain adaptation settings over the state-of-the-arts. in future work, we explore to extend this approach for other low resource tasks in nlp.