On Continual Model Refinement in Out-of-Distribution Data Streams (2022)

Abstract: Real-world natural language processing (NLP) models need to be continually updated to fix the prediction errors in out-of-distribution (OOD) data streams while overcoming catastrophic forgetting. However, existing continual learning (CL) problem setups cannot cover such a realistic and complex scenario. In response to this, we propose a new CL problem formulation dubbed continual model refinement (CMR). Compared to prior CL settings, CMR is more practical and introduces unique challenges (boundary-agnostic and non-stationary distribution shift, diverse mixtures of multiple OOD data clusters, error-centric streams, etc.). We extend several existing CL approaches to the CMR setting and evaluate them extensively. For benchmarking and analysis, we propose a general sampling algorithm to obtain dynamic OOD data streams with controllable non-stationarity, as well as a suite of metrics measuring various aspects of online performance. Our experiments and detailed analysis reveal the promise and challenges of the CMR problem, supporting that studying CMR in dynamic OOD streams can benefit the longevity of deployed NLP models in production.

## introduction fine-tuning large pre-trained language models (lms) has become the de facto standard for training models of a variety of tasks in natural language processing (nlp). these success stories are usually in places where the training and testing data are drawn from the same distribution. however, in real-world scenarios, a deployed model (e.g., a question answering service) often encounters examples that are out of the training distribution (i.e., outof-distribution, ood). such distribution shift often leads to a high error rate. in practice, it is highly preferred to continually refine deployed models whenever new errors are reported and annotated, in order to reduce their further negative impacts. in spite of its importance, the challenge of continually refining a model over ood data streams has been underexplored. prior work in continual learning (cl) has primarily focused on taskincremental settings with boundary-aware data streams. these cl methods are usually evaluated on simple models and data (e.g., image classification with mnist) @xcite . it is not clear to what extent they can efficiently refine a model in boundary-agnostic streams for a complex language task (e.g., reading comprehension) with modern lms. in addition, there is no existing evaluation protocol for comprehensively comparing the collection of applicable methods for such a practical and complex problem. traditional cl paradigms mainly focus on incrementally learning a model from a data stream with a sequence of distinct tasks with explicit delineation, which is rather unrealistic in real-world nlp applications. to address these research questions, we propose a novel cl formulation named continual model refinement (cmr), which aims to efficiently update a model for error correction in an out-of-distribution data stream without catastrophically forgetting its acquired knowledge over time. in contrast to prior cl setups, cmr targets learning a model of a particular task (e.g., question answering) from its prediction errors in dynamic ood data streams. instead of assuming that the streams are drawn from a fixed unseen distribution, we study cmr under a more general and realistic scenario, where the underlying distribution of ood data streams is non-stationary across time steps without clear boundaries while being diverse at every time step. in this paper, we focus on studying whether existing methods can address cmr and how we should benchmark and analyze their performance. we first formulate the cmr problem with several ba-sic metrics covering multiple desiderata for a cmr method: the ability to instantly fix known errors, the retention of previously acquire knowledge from upstream/online data, and the generalization to unseen ood data (sec. 2). then, we propose a general method to create the dynamic data streams of the aforementioned characteristics and evaluation metrics to benchmark cmr methods, yielding a comprehensive evaluation protocol for cmr (sec. 3). we employ and extend several suitable methods from the cl literature to study the cmr problem, which is based on parameter regularization or memory replay (sec. 4). we have conducted a comprehensive analysis with extensive experimental results, which reveal many interesting, non-trivial findings (section 5). for example, we find that even though replay methods are generally better than regularization-based methods, ewc @xcite , a typical regularization method, achieves the best score in generalizing to unseen ood data. we also find that a simple variant of ranking criteria in conditional replay methods achieves more stable results. moreover, we find that different cmr methods have orthogonal improvements and our positive initial results suggest that integrating regularization terms for replay methods is a promising future direction to develop advanced cl methods to address cmr. ## continual model refinement in this section, we formally introduce the proposed continual learning setup, continual model refinement (cmr). we first define the notations and describe the learning objectives that are also illustrated in fig. 1 , then we design a few basic evaluation metrics for assessing cmr methods, and finally, we briefly discuss the unique challenges compared to other cl formulations. ## a comprehensive evaluation protocol we provide a comprehensive evaluation protocol for studying continual model refinement in ood streams. this section first briefly describes our selected task and datasets (sec. 3.1), then focuses on our proposed method to sample non-stationary ood data streams (sec. 3.2), and finally, illustrate how we use the basic metrics to benchmark various cmr methods in a comprehensive yet concise way. ## methods we first introduce our base lm and then illustrate several typical continual learning methods with our extensions to make them applicable to the cmr problem. we discuss other relevant yet not suitable methods in related work (sec. 6). ## evaluation & analysis we first present the setup in sec. 5.1, and report our main results in ## conclusion & future directions in this paper, we propose a novel continual learning formulation named continual model refinement (cmr). the cmr problem aims to efficiently fix prediction errors when learning in outof-distribution data streams without catastrophically forgetting the acquired knowledge. for studying such a realistic and complex problem, we presented a dedicated evaluation protocol with a general method to create non-stationary, diverse ood data streams for analysis. also, we design multiple evaluation metrics to deliver a comprehensive yet concise measurement of cmr methods. the proposed cmr problem with our comprehensive analysis opens up a range of new opportunities for studying continual learning problems that are closer to real-world applications for the nlp community and beyond. for example, based on our results and analysis about (q3) and (q6), we find that it is promising to study how we can integrate both regularization methods and replay methods for mitigating the forgetting issue while improving the generalization ability. the analysis about (q5) suggests that developing more stable ranking criteria is also important to conditional replay methods (e.g., our simple extension maxloss can outperform mir under specific settings). developing cmr methods of which the configurations can generalize to diverse types of streams is also an important challenge. we release our codebase and processed datasets for supporting the reproducibility of our experiments and future research.