Clozer”:” Adaptable Data Augmentation for Cloze-style Reading Comprehension (2022)

Abstract: Task-adaptive pre-training (TAPT) alleviates the lack of labelled data and provides performance lift by adapting unlabelled data to downstream task. Unfortunately, existing adaptations mainly involve deterministic rules that cannot generalize well. Here, we propose Clozer, a sequence-tagging based cloze answer extraction method used in TAPT that is extendable for adaptation on any cloze-style machine reading comprehension (MRC) downstream tasks. We experiment on multiple-choice cloze-style MRC tasks, and show that Clozer performs significantly better compared to the oracle and state-of-the-art in escalating TAPT effectiveness in lifting model performance, and prove that Clozer is able to recognize the gold answers independently of any heuristics.

## introduction endowing machines with the proficiency to read, understand, and reason from unstructured text information is an ongoing aspiration in natural language processing. this aim raises a notable research focus: machine reading comprehension (mrc). given a question, the goal of mrc is to infer the correct answer based on important cues gathered through understanding relevant context passage. mrc tasks vary in structure, depending on their question construction (e.g., cloze-style) and answer type (e.g., multiple-choice) @xcite . various methods using large pre-trained language models (lms) have been proposed in mrc tasks. in recent years, adaptation methods such as task adaptive pre-training (tapt) have been widely adopted for mrc tasks @xcite @xcite . tapt uses in-domain unlabelled data of the downstream task to generate a synthetic pre-training dataset adapted to the downstream task through certain data augmentation methods, depending on the downstream task in use. for multiple-choice cloze-style mrc, data augmentation often involves two steps: 1) answer extraction or selection and 2) pseudo-answer * the authors contributed equally to this work. figure 1 : clozer extracts an answer for tapt generation (figure 1 ). both steps have been adopted in several studies with varying implementations @xcite @xcite . one notable work presents ta-mamc @xcite , which achieves state-of-the-art performance by adopting the tapt framework. however, this method relies heavily on the downstream task's heuristics in the answer selection step, which hinders its applicability to other multiple-choice cloze-style mrc tasks. in this paper, we take a step towards generalized synthetic pre-training dataset construction, to use tapt to solve multiple-choice cloze-style mrc. we propose clozer, a cloze answer extraction based on sequence tagging developed independently of pre-defined rules to improve the generalizability of the tapt method for the cloze-style mrc tasks. clozer learns the intrinsic pattern of the downstream task dataset and acts as an answer extractor for the unlabelled data (figure 1 ). to adapt to the downstream task, the extractions are grouped with several other options to form a triplet of {context passage, cloze question, multiple-choice options}, following the standard multiple-choice cloze-style mrc task format, as a synthetic sample for the second pre-training phase. we conduct our experiments on two downstream tasks. our experimental results show that employing clozer in tapt provides a substantial performance boost, while being generally applicable for both multiple-choice mrc tasks we experiment on. 1 60 our contributions are as follows: 1) to the best of our knowledge, we are the first to introduce an automatic generalizable cloze answer extraction method to support a generalized tapt method for multiple-choice cloze-style mrc tasks; 2) we show that clozer significantly outperforms all other baselines on two multiple-choice cloze-style mrc tasks without relying on any task-specific heuristics; and 3) we present further analysis to explain the effectiveness and efficiency of our clozer and provide insight on how to improve its generalizability. ## methodology our method follows the pipeline described in figure 2. we follow tapt's objective, in which a model learns on a small task-relevant set of data instead of doing another round of masked language modeling (mlm) for pre-training. utilizing clozer, we adapt a large unlabelled pre-training dataset based on the downstream task, which could be any multiple-choice cloze-style mrc task. we define the pre-training dataset @xmath0, s p i )} n @xmath1. p could be any unlabelled data of document and sentence pairs, e.g., headline-content of news, title-body of articles, and synopsis-narration of stories. through the task adaptation, we reconstruct p into a synthetic cloze-style mrc task, where the resulting task-adapted pre-training dataset is represented by we split the task adaptation into 1) gold answer extraction and 2) pseudo options generation, which are explained in §3.1 and §3.2 respectively. afterwards, we employ tapt using the task-adapted dataset t , the details of which are provided in §3.3. ## conclusion we have proposed clozer, an automatic generalizable cloze answer extraction method, to help in syn-4 63 thetic tapt dataset construction in multiple-choice cloze-style mrc tasks. performing tapt with gold answers extracted by our electra-based clozer produces stronger models than the baselines in terms of effectiveness (i.e., performance) and efficiency (i.e., the amount of data used in tapt). moreover, we also show that the quality of clozer's extracted answers is higher, despite its independence from the downstream task's heuristics