Less Is More: Domain Adaptation with Lottery Ticket for Reading Comprehension (2021)

Abstract: In this paper, we propose a simple few-shot domain adaptation paradigm for reading comprehension. We first identify the lottery subnetwork structure within the Transformer-based source domain model via gradual magnitude pruning. Then, we only fine-tune the lottery subnetwork, a small fraction of the whole parameters, on the annotated target domain data for adaptation. To obtain more adaptable subnetworks, we introduce self-attention attribution to weigh parameters, beyond simply pruning the smallest magnitude parameters, which can be seen as combining structured pruning and unstructured magnitude pruning softly. Experimental results show that our method outperforms the full model fine-tuning adaptation on four out of five domains when only a small amount of annotated data available for adaptation. Moreover, introducing self-attention attribution reserves more parameters for important attention heads in the lottery subnetwork and improves the target domain model performance. Our further analyses reveal that, besides exploiting fewer parameters, the choice of subnetworks is critical to the effectiveness.

## introduction reading comprehension @xcite obtains great attention from both research and industry for its practical value. state-of-the-art systems based on pre-trained language models @xcite @xcite @xcite have achieved remarkable performance on the task. despite pretraining, they still rely on large amounts of annotated data @xcite @xcite to reach the desired task performance. manually collecting such high-quality datasets is costly and time-consuming, especially for cases that require specific domain knowledge. it hinders us from applying the datadriven solutions directly to scenarios or domains without sufficient annotation data. in this case, domain adaptation @xcite @xcite is used to obtain a reasonable target domain performance. unsupervised domain adaptation @xcite exploits the unlabeled context passages for adaptation. however, these methods have difficulties in adapting to the desiderata of questions and question-context reasonings in the target domain. in this paper, we focus on supervised domain adaptation for reading comprehension in the few-shot settings. we are devoted to transfer a model trained on a large amount of source domain data to the target domain with only limited annotated data. it is generally feasible to annotate a small amout of question answering pairs. typical reading comprehension models based on pre-trained language model contain at least hundreds of millions parameters, e.g., size of bertbase is 110m. previous works @xcite @xcite show that dense neural networks are over-parameterized and considerable parameters of a trained model can be pruned with marginal or even no loss in performance. meanwhile, "the lottery ticket hypothesis" @xcite argues that the initialization of over-parameterized neural networks contains sparse sub-network at initialization, which, when trained in isolation, rival the original network in task performance. on the other hand, our preliminary analysis (figure 2 ) using an effective attribution method @xcite shows that important attention heads are highly correlated across various domains. in view of the over-parameterized source domain model and our preliminary findings on attention head dynamics, we assume fine-tuning a small fraction of deliberately selected parameters is both more efficient and more effective for fewshot domain adaptation. specifically, we first prune the source domain model via magnitude pruning gradually. in addition, we introduce self-attention attribution @xcite to reserve more parameters for important heads. the corresponding connections of the survived parameters after pruning depict the exact sparse structure of the lottery network. then, we only fine-tune the lottery subnetwork, which consumes much less parameters, on the annotated target domain data for adaptation. the remaining parameters are frozen and will not be updated, but they also contribute to the predictions by participating in the forward computation. experimental results show that our method, exploiting small lottery subnetworks for few-shot domain adaptation, outperforms the full model finetuning on four out of five various domains with a range number of training examples. further analyses reveal several intriguing findings. first, introducing attention head importance yields better lottery subnetworks in highly sparse regimes in the source domain. and improves the performance regardless of the sparsity. secondly, the better source domain lottery subnetworks lead to the improved domain adaptation performance. finally, in addition to using fewer parameters, the choice of subnetwork structure is critical to effectiveness. ## method in this section, we describe our few-shot domain adaptation method for machine reading comprehension in detail. in the source domain, we have a model trained on a large-scale annotated dataset. we fine-tune bert-base @xcite , a representative transformer-based pre-trained language model with tremendous number of parameters, as our source domain model. in the target domain, only limited annotated data, 1k examples at most, can be used for domain adaptation. the mismatch between a small amount of data and a large number of parameters makes it challenging to adapt all source domain model parameters to the target domain. thus, we exploit a small fraction of deliberately selected parameters for domain adaptation by first identifying and then fine-tuning the lottery subnetwork. ## conclusions in this work, we propose alter, a simple and effective domain adaptation paradigm for few-shot reading comprehension. we exploit a small fraction of parameters of the over-parameterized source domain model to adapt to the target domain by first identifying and then fine-tuning the lottery subnetwork. we introduce self-attention attribution, an interpreting method for transformer, to identify better subnetworks and improve the target domain performance. further exploration on using several heuristic methods to reveal subnetwork structures find that subnetwork structures are critical to the effectiveness besides using fewer parameters.