Self-Supervised Test-Time Learning for Reading Comprehension (2021)

Abstract: Recent work on unsupervised question answering has shown that models can be trained with procedurally generated question-answer pairs and can achieve performance competitive with supervised methods. In this work, we consider the task of unsupervised reading comprehension and present a method that performs “test-time learning” (TTL) on a given context (text passage), without requiring training on large-scale human-authored datasets containing context-question-answer triplets. This method operates directly on a single test context, uses self-supervision to train models on synthetically generated question-answer pairs, and then infers answers to unseen human-authored questions for this context. Our method achieves accuracies competitive with fully supervised methods and significantly outperforms current unsupervised methods. TTL methods with a smaller model are also competitive with the current state-of-the-art in unsupervised reading comprehension.

## introduction reading comprehension is the task in which systems attempt to answer questions about a passage of text. answers are typically found in the passage as text-spans or can be inferred through various forms of reasoning @xcite . the answer to the following question: "who is the president of the united states?" depends on the timeframe and context of the passage provided, and will be different for news articles written in 2001 vs. 2021. if the context is the script of the tv series "the west wing", the answer is "jed bartlet", and even in this fictional setting, it will later change to "matt santos". knowledge sources such as wikipedia get updated when new events occur (such as the outcome of elections), or new facts about the world are revealed (such as scientific discoveries), with contributors adding new information and removing information that is no longer valid @xcite . with such context-dependent answers and continual changes in knowledge, it is hard to justify training models over fixed corpora for tasks such as question answering (qa). we would like models to answer questions based on the given context and not to learn biases from datasets or historical news articles. moreover, supervised learning has been shown to perform poorly in qa tasks with adversarial examples @xcite , domain shift @xcite @xcite , and biased or imbalanced data @xcite . for example, qa systems trained on wikipedia fail to generalize to newer domains such as natural questions @xcite or biomedical data @xcite , and suffer a significant drop in accuracy. even small semantics-preserving changes to input sentences, such as the substitution of words by synonyms, have been shown to degrade performance in nlp tasks @xcite . continual changes in text corpora are inevitable, thus calling for the development of robust methods that can reliably perform inference without being subject to biases. supervised question answering faces challenges such as the need for large-scale (usually human-authored) training corpora to train models. such corpora typically require significant postprocessing and filtering to remove annotation artifacts @xcite . to address these challenges, some recent methods @xcite approach question answering as an unsupervised learning task. a significant advantage of this approach is that it can be extended to domains and languages for which collecting a large-sized human-authored training corpus is challenging. methods for unsupervised qa procedurally generate a large corpus of (context, question, answer) triples, and train large neural language models, such as bert @xcite . in this work, we focus on unsupervised reading comprehension (rc) under evolving contexts and present the "test-time learning" paradigm for this task. rc -the task of answering questions about a passage of text, acts as the perfect setting for robust question-answering systems that do not overfit to training data. while large-scale language models trained on large datasets may contain global information, the answer needs to be extracted from the given context. thus, our work seeks to learn unsupervised reading comprehension without access to human-authored training data but instead operates independently on each test context. this makes our method 'distribution-blind' where each new context is assumed to be a novel distribution. the test-time learning (ttl) framework enables smaller models to achieve improved performance with small procedurally generated question-answer pairs, and is summarized below: this framework has a simple assumption that every context comes from a distinct distribution. hence, parameters learned for the previous context might not be useful to generalize to other contexts. this assumption holds where the contexts evolve over time, and rote memorization of answers might lead to wrong predictions. as such, the above process is repeated for each new context c i . for question-answer generation, we use simple methods such as cloze-translation @xcite , template-based question-answer generation @xcite and question-answer semantic role labeling (qa-srl) @xcite . we use two neural transformer-based language models, bert-large @xcite and dis-tilbert @xcite , to study the efficacy of our framework with large and small transformer models. we evaluate our method on two reading comprehension datasets, squad @xcite and newsqa @xcite . we investigate test-time training under multiple learning settings: (1) single-context learning -the "standard" setting, (2) k-neighbor learning -by retrieving top-k multiple related contexts for each test context, (3) curriculum learning -progressively learning on question-types of increasing order of complexity, (4) online learning -sequentially finetuning models on each incoming test sample. our experimental findings are summarized below: • test-time learning methods are effective for the task of reading comprehension and surpass current state-of-the-art on two benchmarks: squad and newsqa. • online ttl trained over k-neighboring contexts of the test context is the best version with em/f1 gains of 7.3%/7.8% on squad 1.1 and 5.3%/6.9% on newsqa. • distilbert -which has less than 1 5 th of the number of model parameters of bert-large is competitive with current sota methods that use bert-large. ## test-time reading comprehension consider a reading comprehesion test dataset d @xmath0, q i , a i )} n @xmath1, human-authored questions q i and true answers a i . the qa model g(•) is parameterized by θ = (θ f , θ h ) where θ f are parameters for the feature extractor, and θ h for the answering head. the answer is predicted as a text-span, given by the start and stop positions [y start , y stop ]. contemporary unsupervised rc models @xcite ) are trained on a large dataset @xmath2, qi , âi )} n @xmath3, where the qa pairs are synthetically generated from the context. in our setting, we do not use such large training datasets, but instead directly operate on individual test contexts c @xmath4. given c i , m synthetic question-answer pairs {(q j i , âj i )} m @xmath5. the qa model parameters θ are trained over the synthetic data to predict the span of the answer [ŷ start , ŷstop ] by optimizing the loss ℓ ans : next, we describe the variants of test-time reading comprehension. single-context test-time rc. this is the standard formulation of test-time learning in this paper, with equation 1 optimizing over θ, i.e. for each context c i , the feature extractor θ f is re-initialized with pre-trained bert, and the answering head θ h is randomly initialized. ## experiments datasets. we evaluate our learning framework on two well-known reading comprehension datasets: squad 1.1 @xcite and newsqa @xcite . qa model. we focus on training two transformer-encoder based models, bert-large @xcite trained with wholeword masking and distilbert @xcite . bert-large is used by current state-of-the-art methods on unsupervised extractive qa tasks and has 345 million trainable parameters. on the other hand, distilbert is a knowledge-distilled transformer-encoder based model and only has 66 million parameters (∼ 5× smaller than bert-large), allowing us to study the efficacy of ttl with respect to model-size. metrics. we use the standard metrics for extractive qa -macro exact match, where the predicted answer span is directly matched with the groundtruth, and macro f1, which measures the overlap between the predicted and the ground-truth spans. for comparisons with existing unsupervised methods, since ttl operates directly on test instances, we report validation set performance only for squad 1.1, as the test set is hidden. training setup. for all test-time learning variants, we limit the maximum number of questions generated per context to 4000 and the maximum number of training steps to 1500. the number of training steps is linearly dependent on the selected batch @xmath6. for our k-neighbor ttl setup that uses context expansion, we limit the number of retrieved contexts to 500. in curriculum test-time rc, we ensure that all variants have an equal number (1000) of generated qapairs per-context. we evaluate multiple learning rates within the range 1e-5 to 5e-5. we use the adam (kingma and ba, 2014) optimizer and truncate the paragraphs to a maximum sequence length of 384. the number 384 was chosen by evaluating the 99 th percentile of the combined length of question and the contexts, to reduce training overhead and gpu memory size. long documents are split into multiple windows with a stride of 128. all baselines. as we generate our own data using qa-srl, we use the following strong baselines. first, we train bert-large with generated data from previous methods described in section 3 and our method (which contains additional qa-srl samples). second, we replicate the baselines using the low parameter-count model distilbert (66 million vs 345 million for bert-large). third, for a fair comparison to single-context and kneighbor test-time learning where we train models for each context independently, we propose a baseline where we train on all the test contexts together, referred to as "all test contexts". we also evaluate all ttl variants on two initializations of featureextractor parameters -1. "default" initialization of bert-large, i.e. θ f pre-trained on masked language modeling and next-sentence prediction tasks, and θ h randomly initialized for each context and trained from scratch, or 2. θ f and θ h further pre-trained on 100k synthetic qa pairs generated procedurally using our methods described in section 3 with contexts taken from the wikipedia corpus. ## conclusion in this work, we propose test-time learning (ttl) as a new framework for unsupervised extractive question answering (eqa). we present four variants of ttl with a simple but effective context expansion method. we utilize four question-answer pair generation methods for eqa and propose using qa-srl as an additional source of qa pairs, to supplement prior methods. we show ttl enables "understanding" of contexts at test-time, without human-authored annotations, and significantly improves eqa, including low parameter models. we envision ttl as a framework that can direct work in reading comprehension to be viewed as a problem of ever-evolving datasets instead of a static corpus. natural language itself undergoes continuous evolution @xcite @xcite via changes in preference for syntactical structures; creation of new words and phrases; and changing usage frequencies and semantics for existing words. ttl can potentially be applied to such scenarios with semantic drift or domain shift. further improvements w.r.t. selection of similar contexts for k-neighbor ttl could be explored by leveraging hard sample selection, hard negative mining, bootstrapping, and contrastive learning, along with improved currculum strategies.