Variance-reduced First-order Meta-learning for Natural Language Processing Tasks (2021)

Abstract: First-order meta-learning algorithms have been widely used in practice to learn initial model parameters that can be quickly adapted to new tasks due to their efficiency and effectiveness. However, existing studies find that meta-learner can overfit to some specific adaptation when we have heterogeneous tasks, leading to significantly degraded performance. In Natural Language Processing (NLP) applications, datasets are often diverse and each task has its unique characteristics. Therefore, to address the overfitting issue when applying first-order meta-learning to NLP applications, we propose to reduce the variance of the gradient estimator used in task adaptation. To this end, we develop a variance-reduced first-order meta-learning algorithm. The core of our algorithm is to introduce a novel variance reduction term to the gradient estimation when performing the task adaptation. Experiments on two NLP applications: few-shot text classification and multi-domain dialog state tracking demonstrate the superior performance of our proposed method.

## introduction meta-learning has recently emerged as a promising approach in solving many natural language processing tasks, such as few-shot text classification @xcite , low resource language understanding @xcite @xcite , and multi-domain dialogue systems @xcite . in particular, modelagnostic meta-learning (maml) @xcite , a widely-used meta-learning approach, trains an initial model that can be adapted to a new task with a small number of optimization steps and training data. however, maml requires the computation of second-order derivatives, which can be costly for reinforcement learning and nlp applications. therefore, numerous computationallyefficient maml variants @xcite @xcite @xcite have been proposed in recent years. first-order metalearning @xcite is a widely-used method in practice because it is easy to implement, eliminates computationally-intensive second-order derivatives in maml, and achieves state-of-the-art performance. although meta-learning including first-order meta-learning has shown promising performances in many applications @xcite , it still somewhat struggles to learn on diverse task distributions @xcite @xcite . for first-order metalearning, it consists of task adaptation and meta updates. task adaptation aims to obtain a taskspecific model for each task by performing several optimization steps based on the current meta model. then, the meta update aggregates the gradient information of task-specific models to obtain a new meta model. it has been observed in many previous works @xcite @xcite ) that local update methods, including first-order meta-learning, performing multiple optimization steps on local data can lead to overfitting to atypical local data. in the context of first-order meta-learning, due to the large variance of the gradient estimator, task adaptation will drive task-specific models to move away from each other, resulting in that the gradients used in meta update have diverse directions. furthermore, since the difference in gradient magnitudes will also be large, the task with a much larger gradient in magnitude will dominate the task adaptation. as a result, the meta update will overfit to this dominating task. similar issues have been studied in multi-task learning: @xcite showed that conflicting gradients, i.e., two gradients that have a negative cosine similarity, can lead to significantly degraded performance when the difference in gradient magnitudes is large. the above gradient variance issue, i.e., the large variance from the gradient estimator, is significant in nlp applications since many nlp datasets have diverse properties, and the tasks for meta-learning in nlp applications also have their unique characteristics. for example, the multiwoz dataset @xcite for dialog systems and the spider dataset @xcite for semantic parsing, both consist of complex and cross domain examples. to address the aforementioned gradient variance issue in nlp applications when applying first-order meta-learning approaches, we propose a variance-reduced first-order meta-learning (vfml) algorithm. the key idea of our algorithm is that we leverage a novel variance reduction term in the task adaptation steps to reduce the variance of the gradient estimator. we evaluate our proposed method on two nlp applications: few-shot text classification and domain adaptation in multi-domain dialog state tracking. we experiment on several benchmark datasets, finding that our method produces models that can achieve better performances than the baseline reptile @xcite . ## problem setup and preliminaries let @xmath0. suppose t i is drawn from t with probability p i , and we use p to denote the probability distribution over t . our goal is to find an initial model θ such that it will have a small loss on a new task t i after a few steps of updates. therefore, we want to solve the following problem ## method our proposed algorithm for meta-learning is illustrated in algorithm 1. in the following discussion, we use ∇l i,b i t to denote the mini-batch stochastic gradient for task i and b i t is the sample index set. the main idea of our method is to construct algorithm 1 variance-reduced first-order metalearning (vfml) algorithm input initialization θ 0 , initial variance reduction term v 0 , step size: η, τ , iteration numbers: t , k, parameters: β, γ 1: for @xmath1, 1, . . . , t -1 do for @xmath2: @xcite . v will be used in the task adaptation step (line 4 in algorithm 1) to reduce the variance of the gradient estimator. more specifically, we use the gradient and the variance reduction term v, and (1 -γ) is the weight for v. when γ = 1, it reduces to reptile. we initialize the variance reduction term v 0 by averaging the gradients from a set of tasks which are randomly sampled and computed using the initialization θ 0 . next, we briefly discuss the intuition of why our proposed method can reduce the variance of the gradient estimator. suppose in addition, the variance of the gradient estimator in vfml, i.e., e g i k -∇l(w i k ) if we have a large number of examples for each task, then σ 2 1 will be small, and the variance of the gradient estimator in reptile will be determined by o(σ 2 2 ). when we have very diverse task distributions, σ 2 2 will be large, which can lead to a significant degradation in performance. however, for vfml, the variance will be dominated by ## experiments we evaluate our proposed method on one simulation experiment and two nlp applications: text classification and dialog state tracking. ## conclusion we propose a novel first-order meta-learning method to reduce the variance of the gradient estimator used in task adaptation for nlp tasks. we show in both few-shot text classification and dst that our method can achieve better performance than existing methods. it is interesting to further study domain adaptation methods built upon our new algorithm.