Keep Learning: Self-supervised Meta-learning for Learning from Inference (2021)

Abstract: A common approach in many machine learning algorithms involves self-supervised learning on large unlabeled data before fine-tuning on downstream tasks to further improve performance. A new approach for language modelling, called dynamic evaluation, further fine-tunes a trained model during inference using trivially-present ground-truth labels, giving a large improvement in performance. However, this approach does not easily extend to classification tasks, where ground-truth labels are absent during inference. We propose to solve this issue by utilizing self-training and back-propagating the loss from the model’s own class-balanced predictions (pseudo-labels), adapting the Reptile algorithm from meta-learning, combined with an inductive bias towards pre-trained weights to improve generalization. Our method improves the performance of standard backbones such as BERT, Electra, and ResNet-50 on a wide variety of tasks, such as question answering on SQuAD and NewsQA, benchmark task SuperGLUE, conversation response selection on Ubuntu Dialog corpus v2.0, as well as image classification on MNIST and ImageNet without any changes to the underlying models. Our proposed method outperforms previous approaches, enables self-supervised fine-tuning during inference of any classifier model to better adapt to target domains, can be easily adapted to any model, and is also effective in online and transfer-learning settings.

## introduction it is a common consensus that the performance of machine learning algorithms improves with increasing data. however, due to the difficulty of obtaining large quantities of labelled data, many models (particularly in natural language processing domain) such as bert @xcite , gpt @xcite and unilm @xcite rely on unsupervised pre-training on unlabelled data to learn useful features which are then fine-tuned for other downstream tasks. while this approach leads to large gains in performance, it leads to a mismatch between a network's pretraining and final fine-tuning. some approaches such as pseudo-labelling @xcite have proposed utilizing data-augmentation of unlabelled data with the model's own predictions to better pre-train a model. while these methods are limited to the training phase, @xcite proposed to continue training a language modeling model (which is the task of predicting the next token in a sequence of tokens) during the evaluation stage, achieving significant improvements as the model learns to better adapt to the inference data, without any modifications to the model architecture or any access to training data. for language modeling, the ground truth labels are the next input token, which are trivially accessible to the model to facilitate this learning. however, this method does not easily generalize to standard classification tasks due to the unavailability of labels during inference. this is the setting which we further explore in this paper, in which we are provided with a classification model already trained on training data, but with no access to the training data, and the aim is to further improve the performance of the model by utilizing self-training on the inference data. to solve the above problem, we propose a method to train any classifier model during inference, utilizing methods used in domain adaptation, noisy-label learning, and multi-task meta-learning. with ground truth labels being absent, we utilize the model's own predictions as the pseudo-labels for those samples and utilize class balanced self training (cbst) @xcite to filter samples based on the model's confidence while retaining class balance. however, naive online learning or re-training on the inference data is not optimal due to the noise in the labels biasing the network, as well as the small size of the inference set. we solve this issue by leveraging the reptile meta learning algorithm @xcite to improve generalization, supplemented with an explicit inductive bias towards the model's pre-trained weights. our experimental results and ablation studies show that our method improves the performance of standard backbones such as bert, electra @xcite and resnet @xcite on a wide variety of tasks, such as question answering on squad @xcite and newsqa @xcite , benchmark task superglue @xcite , and conversation re-ranking on ubuntu dialog corpus v2.0 @xcite for nlp, as well as object classification on mnist @xcite and imagenet @xcite without any changes to the underlying models, while outperforming previous approaches. our method can also be utilized for continual selfsupervised fine-tuning of classifiers on target domains, as well as in transfer-learning settings, without any model-level modifications. ## proposed method our proposed technique is the self-supervised training of a classifier model during inference, consisting of three parts -using confident predictions as pseudo-labels, utilizing the reptile algorithm to improve generalization, and an explicit inductive bias to minimize the effect of noisy labels. ## theoretical analysis in this section, we provide a theoretical analysis of the meta update of reptile + l 2 sp. we generalize the taylor expansion approach for reptile as used in @xcite to accommodate l 2 sp, and show how our approach maximizes the inner product of gradients between different mini-batches. we consider one set of k inner updates. for @xmath0, k], we define - in the following analysis, to keeps the analysis tractable, we assume both α and β are small and comparable, and ignore terms involving o(α 2 ), o(β 2 ) and o(αβ). using the first order taylor expansion of g i , we get - the following equations can be proved using simple induction on eq ( 1) and ( 2 ) - by summing up the displacements from all variable updates, the expectation of the meta-gradient from reptile + l 2 sp under mini-batch sampling is - when expanding the terms above with eq ( 3) and ( 4 ) and simplifying, we get - the first term in r.h.s. of eq ( 5) is the gradient which takes us to the minimum of the training problem. for the third term, note that - therefore the third term maximizes the dot product between the gradients of the batches for improved generalization, as in the original reptile algorithm. for the second and fourth terms, note that (θ 0 -w ) is the direction of the gradient of the l 2 sp, and hence can be interpreted similar to the first and third term, but with training gradients replaced by this l 2 gradient. hence, we have shown that the reptile algorithm maximizing product of gradients for improving generalization holds true in our extension as well. ## ablation studies we conduct extensive ablation studies to test the effectiveness of all parts of our approach. we perform these ablations on squad v2.0 with bertbase model. ## conclusion we propose a method for self-supervised learning for any classifier model during inference using the model's own predictions, adapting reptile algorithm from meta-learning and an inductive bias for maintaining generalization while improving performance. we demonstrate the effectiveness of our method on a wide range of tasks, including super-glue benchmark, question answering on squad v2.0 and newsqa, response selection on ubuntu dialog corpus v2.0, and image classification on imagenet and mnist. our approach consistently improves the performance of standard backbones such as bert, electra, and resnet. our method is effective for improving the performance of neural models without any changes to the underlying models, their training, or access to training data, requires minimum extra compute, and is also effective in online and transfer-learning settings. corpus p -value squad v2.0 electra (f1) 3e-5 ubuntu dialog v2.0 5e-5 newsqa (f1) 1e-9 imagenet 4e-7 as a further test of improved generalization, we split the squad dev set in two equal halves, performed our self-training on one half, and evaluated on the other half. scores in