Cooperative Self-training of Machine Reading Comprehension (2022)

Abstract: Pretrained language models have significantly improved the performance of downstream language understanding tasks, including extractive question answering, by providing high-quality contextualized word embeddings. However, training question answering models still requires large amounts of annotated data for specific domains. In this work, we propose a cooperative self-training framework, RGX, for automatically generating more non-trivial question-answer pairs to improve model performance. RGX is built upon a masked answer extraction task with an interactive learning environment containing an answer entity Recognizer, a question Generator, and an answer eXtractor. Given a passage with a masked entity, the generator generates a question around the entity, and the extractor is trained to extract the masked entity with the generated question and raw texts. The framework allows the training of question generation and answering models on any text corpora without annotation. We further leverage a self-training technique to improve the performance of both question generation and answer extraction models. Experiment results show that RGX outperforms the state-of-the-art (SOTA) pretrained language models and transfer learning approaches on standard question-answering benchmarks, and yields the new SOTA performance under given model size and transfer learning settings.

## introduction recent studies have shown that language model pretraining provides high-quality text representations and significantly improves neural networks' performance on a variety of natural language processing (nlp) tasks @xcite . based on the popular transformer architecture @xcite , various language models have been proposed @xcite ; clark * * work is not related to the employment of the second and fourth authors at meta and amazon ## related work reinforcement learning and self-training have emerged recently for learning language generation in addition to maximum likelihood training. to optimize text generation models directly with non-differentiable objective functions, rennie et al. in the domain of question answering, a question generator can be used for joint answer prediction @xcite , and synthetic qa data are used for in-domain data augmentation @xcite @xcite and outof-domain adaptation. @xcite and @xcite introduced models for question answering under unsupervised/zero-shot settings. @xcite proposed generating synthetic question-answer pairs with an end-to-end model simultaneously. @xcite improved the question synthesis by training with difficult qa cases from the adversarialqa corpus @xcite and fine-grained answer synthesis by multi-model voting. we include more related studies in appendix a. in this work, we mainly compare our method with latest baselines, @xcite and @xcite that reported results on outof-domain adaptation. besides improved qa performance, our framework, rgx, differs from the previous work in the following aspects: (1) our method features reinforced finetuning of the qa synthesizer, (2) our framework supports and improves maximize mutual information inference in test time, and (3) our work did not use complicated data annotation, e.g. adversarialqa. ## rgx framework in this section, we first introduce (1) the qa synthesis pipeline, (2) cooperative self-training for both qa synthesis and question answering, and (3) an improved maximum mutual information inference strategy. the self-training pipeline of rgx is shown in figure 2 . ## experiments in this work, we train three modules for building the cooperative self-training environment rgx, i.e., the answer entity recognizer (aer), the question generator (qg), and the question-answering extractor (qae). we used a bert @xcite model for aer, a bart (lewis et al., 2019a) model for qg, and an electra (clark et al., 2020) model for aer and qae. to compare with the results reported in shakeri et al. (2020) and bartolo et al. (2021), we (1) pretrain question generation and answering models on the seed corpora, (2) generate synthetic qa data on the target domains, (3) finetune qa models with synthetic data, and (4) evaluate the finetuned qa model on humanlabeled evaluation sets. the source code and demo are publicly available foot_1 . ## conclusion we propose a cooperative self-training framework, rgx, consisting of an answer entity recognizer, a question generator, and an answer extractor, for question generation and answering. we also introduce in the framework an expectationmaximization method that measures the quality of generated questions for reinforced finetuning of the question generation models. experiments show that rgx significantly outperforms pretrained and self-trained model baselines while adapted to unseen domains, suggesting that rgx is a promising framework for making extractive question answering methods more scalable and less dependent on human annotation.