Investigating the effectiveness of Data Augmentation and Contrastive Learning for Named Entity Recognition (2025)

Abstract: Data Augmentation (DA) and Contrastive Learning (CL) are widely used in NLP, but their potential for NER has not yet been investigated in detail. Existing work is mostly limited to zero- and few-shot scenarios where improvements over the baseline are easy to obtain. In this paper, we address this research gap by presenting a systematic evaluation of DA for NER on small, medium-sized and large datasets with coarse and fine-grained labels. We report results for a) DA only, b) DA in combination with supervised contrastive learning, and c) DA with transfer learning. Our results show that DA on its own fails to improve results over the baseline and that supervised CL works better on larger datasets while transfer learning is beneficial if the target dataset is very small. Finally, we investigate how contrastive learning affects the learned representations, based on dimensionality reduction and visualisation techniques, and show that CL mostly helps to separate named entities from non-entities.

## introduction named entity recognition (ner) has been widely studied in nlp and has many applications in the computational social sciences and the digital humanities. many of these applications, however, require the adaptation to new languages or genres for which no or only small amounts of annotated data are available. a major disadvantage of supervised ner systems is their dependence on large and representative datasets for training @xcite . consequently, the scarcity of labelled data has become one of the major challenges impeding the performance of ner systems, especially in highly specialised domains. data augmentation (da) seems like a compelling solution to address this problem. by applying transformations to the data, new training instances can be generated, thus reducing the amount of manually annotated data needed to train the model @xcite . many studies have applied da to text classification tasks, summarisation, or question answering @xcite , with a focus on low-resource scenarios. we are not aware of any studies that report improved results for da over strong baselines, such as transformers, for medium to large data sizes. furthermore, there is a lack of research on da for token-level tasks such as ner, where the integration of da presents a unique challenge. several da techniques apply transformations directly to tokens, thus changing their contextual information. as a consequence, this process may inadvertently modify the associated entity labels, disrupting the correspondence between tokens and their intended nes @xcite . this challenge underscores the necessity of developing augmentation strategies that preserve the entity labels while enhancing the diversity and robustness of the training data for improved ner model performance. another promising approach to improve model performance is contrastive learning (cl), where the model learns to position representations of instances from the same class closer together in the embedding space while representations for data points that belong to different classes are pushed further apart. cl can be used on its own but can also be combined with da and transfer learning. in the paper, we address the question of which of the techniques described are effective in improving results for ner on small, medium-sized and large datasets. 1 our main contributions include: ## experimental settings the last section has shown that there is a severe lack of research regarding the effectiveness of da and cl for ner in scenarios where ample training data is available. we address this gap by providing a systematic investigation of both techniques in different settings and comparing their impact in isolation and in combination with transfer learning. ## analysis and visualisation to better understand the effect of cl, we visualise the learned representations before and after the cl step. as we cannot directly plot the 768-dimensional word embeddings produced by roberta on a two-dimensional graph, we apply dimensionality reduction techniques in order to obtain informative two-dimensional representations. a popular dimensionality reduction technique is principal component analysis (pca), which tries to reduce the dimensionality by choosing the linear combination of variables that explain the variance in the data @xcite . while pca is quite efficient, it can only be applied when all components are linear. a method that can perform non-linear dimensionality reduction is t-sne (van der @xcite . however, t-sne still has a high computational cost compared to pca, especially when dealing with large datasets of high dimensionality. to resolve this problem, we use a combination of pca and t-sne for dimensionality reduction. we first apply pca to reduce the dimensionality of the word embeddings from 768 to 50. then, t-sne is used to reduce the dimensions from 50 to two (see appendix b for more details). figure 1 shows the visualisations of the embeddings for the conll dataset, using cl only. results for ontonotes and fewnerd are similar, and can be found in appendix c. for all three datasets, the separation between non-entities and nes is greater than the separation of the representations for neighbouring ne classes. while a possible reason for this could simply be that the difference between nes and non-entities is greater and therefore easier to learn, a more likely reason is the distribution of nes and non-entities in the data where the latter significantly outnumber the former. in cl, this means that the model can minimise the loss by increasing the difference between the non-entities and nes even if this comes at the expense of decreasing the difference between two different ne classes. ## discussion while we found no evidence that the proposed data augmentations are effective, related work has shown that da can be beneficial in low-resource scenarios @xcite @xcite . we also observed consistent increases in results for cl for datasets with sizes of at least 5,000 sentences. our best results for a roberta-base model with cl on ontonotes (89.75% f1) are only slightly below the ones reported for much larger models (cf., 89.76% f1 for bart-large @xcite and 90.42% f1 for a t5-base model with da @xcite ). these results are promising, given the severe lack of methods for improving the performance on larger datasets, as da has only been successful when applied in low-resource and few-shot scenarios @xcite @xcite , and the same also applies to work on contrastve learning for ner. our experiments failed to show that cl works for smaller datasets. however, when combined with transfer learning, the results are improved. ctl works best when the fine-tuning data size is small, making it a good complement to cl without transfer learning. figure 2 summarises our results, showing which method might work best in different scenarios. while the improvements we obtained are small, they are still important given that increasing the performance of a model that is already performing quite well tends to be much harder than improving the performance of a poorly performing model. in addition, data augmentation and cl can be combined, as often done in related fields like computer vision @xcite . this might be a promising avenue for future work on developing cl methods that work well for smaller datasets. our experiments demonstrate that combining da and cl is possible (see mask + cl in addressing data imbalance in cl for ner in section 5, we showed that a major problem for applying cl to ner is the data imbalance as the majority of the token labels are non-entities. one approach to address this problem could include a modifcation of the cl loss function to account for the imbalance @xcite @xcite . assuming that equation ( 4 ) is used for the loss function, a modified loss function that includes weights is shown in equation ( 5 ), where 𝑤 𝑖 ∈ ℝ + is the weight for class 𝑖. there are many ways to set 𝑤 𝑖 , but one possibility is to set it to the ratio of the frequency of nonentities to the frequency of the class. this is shown in equation ( 6 ), where 𝑛 𝑖 is the frequency of class 𝑖 and 𝑛 𝑂 is the frequency of the non-entity class. this scales the loss function so that different classes can have different weights which might help encourage the model to differentiate between various types of nes. there are many alternatives for the loss and weight functions, and the functions proposed above might not be optimal. development and testing of a weighted loss function will be left to future work. ## conclusion we presented a systematic investigation of the effect of da, cl, and ctl for ner. our main results can be summarised as follows. first, while da has been shown to be effective in low-resource scenarios (specifically for pre-transformer-based taggers), we failed to demostrate an improvement in results in our experiments. cl, on the other hand, can effectively improve results over a strong roberta baseline when medium to large datasets are available for finetuning, but has a weaker performance on smaller datasets. for small dataset sizes, contrastive transfer learning is the most promising approach but requires the existence of suitable data for transfer learning. we hope that the insights from our experiments will foster more work on da and cl for ner especially for medium and large datasets. to address the problem of data imbalance for ner, where the majority of the labels are non-nes, we proposed a modification to the loss function, which we plan to explore in future work.