Memorisation versus Generalisation in Pre-trained Language Models (2022)

Abstract: State-of-the-art pre-trained language models have been shown to memorise facts and perform well with limited amounts of training data. To gain a better understanding of how these models learn, we study their generalisation and memorisation capabilities in noisy and low-resource scenarios. We find that the training of these models is almost unaffected by label noise and that it is possible to reach near-optimal results even on extremely noisy datasets. However, our experiments also show that they mainly learn from high-frequency patterns and largely fail when tested on low-resource tasks such as few-shot learning and rare entity recognition. To mitigate such limitations, we propose an extension based on prototypical networks that improves performance in low-resource named entity recognition tasks.

## introduction with recent advances in pre-trained language models @xcite @xcite , the field of natural language processing has seen improvements in a wide range of tasks and applications. having acquired general-purpose knowledge from large amounts of unlabelled data, such methods have been shown to learn effectively with limited labelled data for downstream tasks @xcite and to generalise well to out-of-distribution examples @xcite . previous work has extensively studied what such models learn, e.g. the types of relational or linguistic knowledge @xcite @xcite . however, the process of how these models learn from downstream data and the qualitative nature of their learning dynamics remain unclear. better understanding of the learning processes in these widely-used models is needed in order to know in which scenarios they will fail and how to improve them towards more robust language representations. the fine-tuning process in pre-trained language models such as bert @xcite aims to strike a balance between generalisation and memorisation. for many applications it is important for the model to generalise-to learn the common patterns in the task while discarding irrelevant noise and outliers. however, rejecting everything that occurs infrequently is not a reliable learning strategy and in many low-resource scenarios memorisation can be crucial to performing well on a task @xcite . by constructing experiments that allow for full control over these parameters, we are able to study the learning dynamics of models in conditions of high label noise or low label frequency. to our knowledge, this is the first qualitative study of the learning behaviour of pre-trained transformer-based language models in conditions of extreme label scarcity and label noise. we find that models such as bert are particularly good at learning general-purpose patterns as generalisation and memorisation become separated into distinct phases during their fine-tuning. we also observe that the main learning phase is followed by a distinct performance plateau for several epochs before the model starts to memorise the noise. this makes the models more robust with regard to the number of training epochs and allows for noisy examples in the data to be identified based only on their training loss. however, we find that these excellent generalisation properties come at the cost of poor performance in few-shot scenarios with extreme class imbalances. our experiments show that bert is not able to learn from individual examples and may never predict a particular label until the number of training instances passes a critical threshold. for example, on the conll03 (sang and de @xcite dataset it requires 25 instances of a class to learn to predict it at all and 100 examples to predict it with some accuracy. to address this limitation, we propose a method based on prototypical networks @xcite that augments bert with a layer that classifies test examples by finding their closest class centroid. the method considerably outperforms bert in challenging training conditions with label imbalances, such as the wnut17 @xcite rare entities dataset. our contributions are the following: 1) we identify a second phase of learning where bert does not overfit to noisy datasets. 2) we present experimental evidence that bert is particularly robust to label noise and can reach near-optimal performance even with extremely strong label noise. 3) we study forgetting in bert and verify that it is dramatically less forgetful than some alternative methods. 4) we empirically observe that bert completely fails to recognise minority classes when the number of examples is limited and we propose a new model, protobert, which outperforms bert on few-shot versions of conll03 and jnlpba, as well as on the wnut17 dataset. ## previous work several studies have been conducted on neural models' ability to memorise and recall facts seen during their training. @xcite showed that pre-trained language models are surprisingly effective at recalling facts while @xcite demonstrated that lstm language models are able to consistently memorise single out-of-distribution (ood) examples during the very first phase of training and that it is possible to retrieve such examples at test time. @xcite found that regularising early phases of training is crucial to prevent the studied cnn residual models from memorising noisy examples later on. they also propose a regularisation procedure useful in this setting. similarly, @xcite analyse how early stopping and gradient descent affect model robustness to label noise. @xcite , on the other hand, study forgetting in visual models. they find that models consistently forget a significant portion of the training data and that this fraction of forgettable examples is mainly dependent on intrinsic properties of the training data rather than the specific model. in contrast, we show that a pretrained bert forgets examples at a dramatically lower rate compared to a bilstm and a non-pretrained variant. memorisation is closely related to generalisation: neural networks have been observed to learn simple patterns before noise @xcite and generalise despite being able to completely memorise random examples @xcite . @xcite also show that our current understanding of statistical learning theory cannot explain the superhuman generalisation performance of large neural models across many areas of study. @xcite show that pre-trained models generalise better on out-of-distribution data and are better able to detect such data compared to non-pretrained methods but that they still do not cleanly separate in-and out-of-distribution examples. @xcite find that pre-trained methods such as bert are sensitive to spelling noise and typos. in contrast to noise in the input, we focus on the models' learning dynamics in the presence of label noise and find that pre-trained methods are remarkably resilient to such cases. ## experimental setting we investigate the performance of pre-trained language models in specific adverse conditions. in order to evaluate generalisation abilities, we first create datasets with varying levels of label noise by randomly permuting some of the labels in the training data. this procedure allows us to pinpoint noisy examples and evaluate the performance on clean and noisy datapoints separately. then, in order to investigate memorisation we train the models on datasets that contain only a small number of examples for a particular class. this allows us to evaluate how well the models are able to learn from individual datapoints as opposed to high-frequency patterns. we make the code for the experiments available online. 1 datasets we focus on the task of named entity recognition (ner) and employ the conll03 (sang and de meulder, 2003), the jnlpba (collier and kim, 2004), and the wnut17 @xcite datasets. ner is commonly used for evaluating pre-trained language models on structured prediction and its natural class imbalance is well suited for our probing experiments. conll03 and jnlpba are standard datasets for ner and bio-ner respectively. the wnut17 dataset is motivated by the observation that state-of-the-art methods tend to memorise entities during training @xcite . the dataset focuses on identifying unusual or rare entities at test time that cannot be simply memorised by the model. we evaluate based on entity-level f 1 unless stated otherwise. language models we use bert-base @xcite as the main language model for our experiments, as bert is widely used in practice and other variations of pre-trained language models build on a similar architecture. the model is augmented with a classification feed-forward layer and fine-tuned using the cross-entropy loss with a learning rate of 10 -4 . adamw @xcite is used during training with weight decay of 0.01 and a linear warm-up rate of 10%. the test results are recorded using the model that produced the highest validation metrics. we compare bert's behaviour with that of other pre-trained transformers such as roberta @xcite and deberta @xcite fine-tuned with the same optimiser and hyperparameters as above. in order to also compare against non-transformer models, we report performance for a bi-lstm-crf @xcite model with combined character-level and wordlevel representations. the model is comprised of 10 layers, with 300-dimensional word representations and 50-dimensional character representations, for a total of approximately 30 million trainable parameters. in our experiments, the model is trained with the adam optimiser @xcite and a learning rate of 10 -4 for 100 epochs using a crf loss @xcite . ## generalisation in noisy settings we first investigate how bert learns general patterns from datasets that contain label noise. figure 1 shows how the model performance on the conll03 training and validation sets changes when faced with varying levels of noise, from 0% to 50%. based on the progression of performance scores, we can divide bert's learning process into roughly three distinct phases: a second phase of learning we find bert to exhibit a distinct second settling phase during which it does not over-fit. a resilience to label noise has been observed in other neural networks trained with gradient descent @xcite . however, we find this phase to be much more prolonged in bert compared to models pre-trained on other modalities such as a pre-trained resnet fine-tuned on cifar10, which immediately starts memorising noisy examples (see appendix a for a comparison). these results indicate that the precise point of early stopping is not as important when it comes to fine-tuning pre-trained language models. similar optimal performance is retained for a substantial period, therefore training for a fixed number of epochs can be sufficient. we illustrate bert's behaviour by evaluating the token-level classification accuracy of noisy examples in figure 2 . during the second phase, bert completely ignores the noisy tokens and correctly misclassifies them, performing "worse" than a random classifier. the step-like improvements during the third stage show that the model is unable to learn any patterns from the noise and improves by repeatedly optimising on the same examples, gradually memorising them. robustness to noise we also observe in figure 1 that bert is extremely robust to noise and overfitting in general. in the absence of noise, the model does not over-fit and maintains its development set performance, regardless of the length of training. even with a large proportion of noise, model performance comparable to training on the clean dataset can be achieved by stopping the training process somewhere in the second phase. 2 we also hypothesise that due to the robustness to noise shown in the second phase of training, a noise detector can be constructed based only on bert's training losses, without requiring any other information. we find that a simple detector that clusters the losses using k-means reliably achieves over 90% noise-detection f 1 score in all our experiments, further showing how the model is able to actively detect and reject single noisy examples (see appendix e for details about the noise detection process). impact of pre-training the above properties can mostly be attributed to bert's pre-training process-after large-scale optimisation as a language model, the network is primed for learning general patterns and better able to ignore individual noisy examples. we find that a randomly initialised model with the same architecture does not only achieve lower overall performance but crucially does not exhibit's bert's distinct second phase of 2 adding 30% noise to the conll03 dataset causes only a 0.9% decrease of validation performance in the second phase. other pre-trained transformers we also analyse the behaviour of other pre-trained transformers for comparison. specifically, studying roberta and deberta, we find the same training pattern that was observed in bert-all models show a clear division into the three phases described above. these models are also all very robust to label noise during the settling phase of training. notably, roberta is even more resilient to label noise compared to the other two analysed models, despite deberta outperforming it on public benchmarks @xcite . training and validation performance visualisations, such as those in figure 1 , can be found for both models in appendix i. ## forgetting of learned information evaluating only the final model does not always provide the full picture regarding datapoint memorisation, as individual datapoints can be learned and forgotten multiple times during the training process. following @xcite , we record a forgetting event for an example at epoch t if the model was able to classify it correctly at epoch t -1, but not at epoch t. similarly, we identify a learning event for an example at epoch t if the model was not able to classify it correctly at epoch t -1, but it is able to do so at epoch t. a first learning event thus happens at the first epoch when a model is able to classify an example correctly. we furthermore refer to examples with zero and more than zero forgetting events as unforgettable and forgettable examples, respectively, while the set of learned examples includes all examples with one or more learning events. in a particular dataset. 3 however, our experiments show that the same does not necessarily hold for pre-trained language models. specifically, there is a large discrepancy in the ratio between forgettable and learned examples for bert (∼3%) and a bi-lstm model (∼80%) on the conll03 dataset. we additionally analyse the distribution of first learning events throughout bert's training on conll03 with label noise between 0% and 50% (figure 3 ) and notice how bert learns the majority of learned examples during the first epochs of training. as the training progresses, we see that bert stops learning new examples entirely, regardless of the level of noise for the third and fourth epochs. finally, in the last epochs bert mostly memorises the noise in the data. 4 ## bert in low-resource scenarios in the previous sections, we have observed that bert learns examples and generalises very early in training. we will now examine if the same behaviour applies in low-resource scenarios where a minority class is only observed very few times. to this end, we remove from the conll03 training set all sentences containing tokens with the minority labels misc and loc except for a predetermined number of such sentences. we repeat the process for the jnlpba dataset with the dna and protein labels. we conduct similar experiments to the previous sections by studying how different numbers of sentences containing the target class affect bert's ability to learn and generalise. we report in figure 4 the training and validation classification f 1 score for the conll03 datasets from which all but few (5 to 95) sentences containing the loc label were removed. note that the reported performance in this experiment refers to the loc class only. in figure 5 we also report the distribution of first learning events for the loc class in the same setting. two phenomena can be observed: 1) reducing the number of sentences greatly reduces the model's ability to generalise (validation performance decreases yet training performance remains comparable); and 2) when fewer sentences are available, they tend to be learned in earlier epochs for the first time. corresponding experiments on the misc label can be found in appendix j. we also show the average entity-level f 1 score on tokens belonging to the minority label and the model performance for the full ner task (i.e. considering all classes) for the conll03 and jnlpba datasets in figures 6 and 7 respectively. for the conll03 dataset, we observe that bert needs at least 25 examples of a minority label in order to be able to start learning it. performance rapidly improves from there and plateaus at around 100 examples. for the jnlpba dataset, the minimum number of examples increases to almost 50 and the plateau occurs for a higher number of examples. on the challenging wnut17 dataset, bert achieves only 44% entity-level f 1 . this low performance is attributable to the absence of entity overlap between training set and test set, which increases the inter-class variability of the examples. ## protobert for few-shot learning in order to address bert's limitations in few-shot learning, we propose a new model, protobert that combines bert's pre-trained knowledge with the few-shot capabilities of prototypical networks @xcite for sequence labelling problems. the method builds an embedding space where the inputs are clustered on a per-class basis, allowing us to classify a token by finding its closest centroid and assigning it the corresponding class. the model can be seen in figure 8 . we first define a support set s, which we use as context for the classification and designate with s k all elements of s that have label k. we refer to the set of points that we want to classify as the query set q, with l(q i ) indicating the label of the i th element in q. we will also refer to f as the function computed by bert augmented with a linear layer, which produces an m dimensional output. the model then classifies a given input x as follows: for each class k, we compute the centroid of the class in the learned feature space as the mean of all the elements that belong to class k in the training data query set support set query embeddings support embeddings class centroids similarities output probabilities similarity per-class average support set s: then, we compute the distance from each input @xmath0: and collect them in a vector @xmath1. finally, we compute the probability of x belonging to class k as the model is trained by optimising the crossentropy loss between the above probability and the one-hot ground-truth label of x. crucially, s and q are not a fixed partition of the training set but change at each training step. following @xcite , we use euclidean distance as a choice for the function d. in order to take into account the extreme underrepresentation of some classes, we create the support by sampling s 1 elements from each minority class and s 2 elements from each non-minority class. a high ratio s 1 /s 2 gives priority to the minority classes, while a low ratio puts more emphasis on the other classes. we then similarly construct the query set with a fixed ratio n between the minority classes and the non-minority classes. for ner, rather than learning a common representation for the negative class "o", we only want the model to treat it as a fallback when no other similar class can be found. for this reason, we define the vector of distances v as follows: if no example of a certain class is available in the support set during the training, we assign a distance of 400, making it effectively impossible to mistakenly classify the input as the missing class during that particular batch. finally, we propose two ways to compute the class of a token at test time. the first method employs all examples from x to calculate the centroids needed at test time, which produces better results but is computationally expensive for larger datasets. the second method approximates the centroid c k using the moving average of the centroids produced at each training step: ## conclusion in this study, we investigated the learning process during fine-tuning of pre-trained language models, focusing on generalisation and memorisation. by formulating experiments that allow for full control over the label distribution in the training data, we study the learning dynamics of the models in conditions of high label noise and low label frequency. the experiments show that bert is capable of reaching near-optimal performance even when a large proportion of the training set labels has been corrupted. we find that this ability is due to the model's tendency to separate the training into three distinct phases: fitting, settling, and memorisation, which allows the model to ignore noisy examples in the earlier epochs. the pretrained models experience a prolonged settling phase when fine-tuned, during which their performance remains optimal, indicating that the precise area of early stopping is less crucial.