Beyond Reptile: Meta-Learned Dot-Product Maximization between Gradients for Improved Single-Task Regularization (2021)

Abstract: Meta-learning algorithms such as MAML, Reptile, and FOMAML have led to improved performance of several neural models. The primary difference between standard gradient descent and these meta-learning approaches is that they contain as a small component the gradient for maximizing dot-product between gradients of batches, leading to improved generalization. Previous work has shown that aligned gradients are related to generalization, and have also used the Reptile algorithm in a single-task setting to improve generalization. Inspired by these approaches for a single task setting, this paper proposes to use the finite differences first-order algorithm to calculate this gradient from dot-product of gradients, allowing explicit control on the weightage of this component relative to standard gradients. We use this gradient as a regularization technique, leading to more aligned gradients between different batches. By using the finite differences approximation, our approach does not suffer from O(nˆ2) memory usage of naively calculating the Hessian and can be easily applied to large models with large batch sizes. Our approach achieves state-of-the-art performance on the Gigaword dataset, and shows performance improvements on several datasets such as SQuAD-v2.0, Quasar-T, NewsQA and all the SuperGLUE datasets, with a range of models such as BERT, RoBERTa and ELECTRA. Our method also outperforms previous approaches of Reptile and FOMAML when used as a regularization technique, in both single and multi-task settings. Our method is model agnostic, and introduces no extra trainable weights.

## introduction meta-learning algorithms such as maml @xcite , fomaml, and reptile @xcite , which modify gradient descent by effectively differentiating through it, have lead to performance improvements on several datasets such as miniimagenet @xcite , @xcite and java github corpus @xcite . when used in a single-task setting, the only significant difference between these algorithms and standard sgd is that the meta-gradient from these algorithms also contains as a component the gradient for maximizing the dot-product between gradients for examples/batches, as was theoretically proven in the reptile paper. the reptile algorithm has also been leveraged to improve single-task performance across a range of models and tasks, such as in @xcite . second-order methods for aligned gradients have been explored before in the context of continual learning, such as in @xcite , @xcite , @xcite . some recent work, such as @xcite , @xcite , and @xcite have also shown that aligned gradients are related to improved generalization and model performances. we conjecture that aligned gradients in single-task settings will also improve learning across examples, enabling better transfer from one example to another, similar to as often done in continual/multi-task approaches such as @xcite . however, a naive approach to directly maximize the dot-product of gradients requires a calculation of the hessian matrix, which scales as o(n 2 ) in memory usage where n is the number of model parameters. this approach also fails to work with gradient accumulation, leading to a hard limit on the batch size, reducing training accuracy. even though some recent works such as @xcite have tried to make this tractable using large distributed environments, the computation costs are extremely high for any reasonably large model. approaches like the hessian-vector product @xcite also do not work with gradient accumulation. inspired by the above approaches and to fix the aforementioned issues, we propose to explicitly calculate the gradient for maximizing the dot product between batches using a first-order approximation. we use this gradient as a regularizing term, and show that it results in improved performance across a range of models and tasks. our main contributions are-• use the first-order finite differences method @xcite to explicitly calculate the gradient from the dot-product of gradients. reptile and fomaml in single-task as well as multi-task setting. ## theoretical analysis in this section, we provide a theoretical analysis of our meta update of the m eta dotp rod algorithm. we generalize the taylor expansion approach as used in @xcite , and show how our approach maximizes inner product of gradients between different mini-batches. this approach is essentially the expectation of the finite differences method over stochastic mini-batch sampling for calculating the hessian-vector product, but we present it here for clarity. we consider two input batches b 0 , b 1 at the be-ginning of i th step. for @xmath0, 1} we define - for small α, we can ignore the terms involving o(α 2 ) in (3). this term becomes o(α) in (4), but it is still α = 1e -7 times smaller than g 1 h 2 and hence can be safely ignored. under the expectation of stochastic mini-batch sampling, e[g 1 h 2 ] = e[g 2 h 1 ], and the above equation (4) becomes - ## conclusion we propose to use finite-differences to calculate the gradient from the dot-product of gradients, and demonstrate its effectiveness as a regularization technique, leading to more aligned gradients between different batches. we leverage this approach to show performance improvements on several datasets such as squad-v2.0, quasar-t, and all the superglue datasets, and achieves state-of-the-art performance on gigaword. our method is effective over a range of models and model sizes, such as bert, roberta and electra. our method outperforms the reptile and the fomaml algorithm in single-task and few-shot multi-task settings, is first-order, is model-agnostic, and can be used with large models and large batches.