IITK at S em E val-2021 Task 10: Source-Free Unsupervised Domain Adaptation using Class Prototypes (2021)

Abstract: Recent progress in deep learning has primarily been fueled by the availability of large amounts of annotated data that is obtained from highly expensive manual annotating pro-cesses. To tackle this issue of availability of annotated data, a lot of research has been done on unsupervised domain adaptation that tries to generate systems for an unlabelled target domain data, given labeled source domain data. However, the availability of annotated or labelled source domain dataset can’t always be guaranteed because of data-privacy issues. This is especially the case with medical data, as it may contain sensitive information of the patients. Source-free domain adaptation (SFDA) aims to resolve this issue by us-ing models trained on the source data instead of using the original annotated source data. In this work, we try to build SFDA systems for semantic processing by specifically focusing on the negation detection subtask of the SemEval2021 Task 10. We propose two approaches -ProtoAUGandAdapt-ProtoAUGthat use the idea of self-entropy to choose reliable and high confidence samples, which are then used for data augmentation and subsequent training of the models. Our methods report an improvement of up to 7% in F1 score over the baseline for the Negation Detection subtask.

## introduction the availability of large scale datasets has been the main driving factor behind the success of supervised deep learning in the recent times. however, the process of data annotation is very expensive and time consuming, being one of the major challenges in extending deep learning techniques for new tasks. one possible way to solve this problem is to train a machine learning model using an annotated source dataset to assist the annotation process over some unlabelled target dataset. however, there may be differences in the source and target domain distributions, which may lead to inaccuracies. thus the challenge is to update the weights of the source classifier to generalize it well on the target domain. this aligns with the well studied problem of unsupervised domain adaptation (uda) @xcite @xcite a common denominator across many popular uda methods is their dependence on large amounts of labelled source domain data @xcite . however, many a times it is not possible to release the source domain dataset because of privacy concerns. this problem becomes particularly relevant when working with clinical natural language processing (nlp) datasets because they contain highly sensitive information which cannot be freely distributed. to tackle these data sharing constraints, the framework of source-free domain adaptation (sfda) is gaining interest @xcite . in sfda, instead of sharing the source domain data, only a model that has been trained on the source domain data is shared. this model is then used for solving the original task for the unlabelled target domain. semeval 2021 task 10 (laparra et al., 2021) asks participants to develop sfda models for two subtasks. the first subtask involves negation detection, where we are required to determine whether or not a clinical entity (diseases, symptoms, etc.) mentioned in a sentence is negated in the given context. the second subtask is of time expression recognition, where the objective is to detect and label all time expressions mentioned in a given document. in this work we have focused on the negation subtask. for solving this subtask our strategy is to make use of high-confidence prototypes from the target domain to reinforce the target-specific features of the source model. we propose a simple augmentation technique that makes use of these high-confidence prototypes to generate labelled artificial datapoints. these augmented samples are then used to perform supervised fine-tuning of our source model. using our methods, we were able to obtain upto a 7% improvement in f1 score over the baseline. the code for models and experiments is made available via github. 1 ## background in source free domain adaptation (sfda) problem for semantic analysis, the goal is to create accurate systems for un-annotated target domain data. for these tasks, we are provided with un-annotated target domain data, and a model trained on the annotated source domain data for a similar task. the shared task is further divided into 2 subtasks -negation detection and time expression recognition. in this work we focus only on the first subtask. negation detection: the task is to classify clinical event mentions for whether they are negated by their context. this is essentially a "span-incontext" classification problem, where both the entity to be classified and its surrounding context are to considered. for example, the sentence -"has @xmath0, and the model's task is to correctly identify this entity as negated. pretrained models: for negation detection, a given pre-trained classification model has been finetuned on the 10,259 instances in the sharp seed dataset @xcite of de-identified clinical notes from mayo clinic of which 902 instances are negated. practice data: the development data for the negation task is a subset of the i2b2 2010 challenge (uzuner et al., 2011) on concepts, assertions, and relations in clinical text. the practice dataset is further divided into train and dev splits. the train split contains 2886 unlabeled sentences while the dev split is composed of 5545 labeled sentences. test data: a part of the mimic iii corpus v1.4 @xcite ,is used as the test set for the negation detection subtask. the processed test data contains around 600k instances. ## system overview in this paper we offer a novel perspective on the problem of domain adaptation for the negation detection task in clinical nlp. the proposed approaches attempt to utilize some of the aspects of both self-learning and semi-supervised learning, as explained next. class prototypes: if there was any access to the labeled target data then the most intuitive approach would have been the fine-tuning. but for unlabeled case, in order to fine-tune the pre-trained network s, it would become necessary to generate a labeled set of data from the given unlabeled target domain data. one way to approach this would be through a concept from self-learning, i.e., by finding the most reliable samples from the target data over which the model s is sufficiently confident and using these predictions as the corresponding ground truth. in order to find reliable target samples, self-entropy h can be used to quantify the prediction uncertainty: here, s refers to the network (pre-trained classifier), k are total number of classes, and p s k (x) is the probability of the instance x belonging to the class k. the samples with smaller self-entropy indicate that the classifier is more confident over them, and these are referred to as prototypes. the relationship between the self-entropy value and certainty in the prediction can be clearly observed in figure 1. due to very high confidence of the model and accuracy over these prototypes, we can safely consider their predicted labels as their actual true labels. here, one crucial hyperparameter to consider is the self-entropy threshold below which the residing target samples will be identified as prototypes. the key issue faced while determining the value of this hyperparameter is the disparities and highly imbalanced data distribution between the classes. especially, in the negation task, the presence of the negated sentences in the practice data itself is in a very small proportion than that of non-negated sentences making negated a minority and non-negated a majority class. analysis of the practice data (figure 2 ) further showed that the lowest self-entropy achieved by the negated class is far higher than that of non-negated class. for now, the self-entropy threshold value is defined by the 50th percentile (median) self-entropy value of the minority class i.e. negated class. the rationale for this is provided in the analysis section (section 5). unfortunately, the fine-tuning of the pre-trained network s with the prototypes identified from the target samples, didn't enable the trained network to generalise its performance over unseen target data as it made it highly likely to overfit on the prototypes without any early intervention. with this approach, we were able to get improvement of about 1% in f1 score over the baseline but that too with the highly unstable and unreproducible training results. only using the reliable samples became a major issue as the trained model seemed to be unaware of the samples over which the model was less confident before. augmentation: to address the issue of lack of generalisation of the model towards the unseen and not-so-reliable target data, we propose the use of augmentation, inspired by the mean teacher model proposed by @xcite . the basic idea is to regularize the network by introducing noise to the prototypes in a way that can keep the label same and thereby creating new data points with the known labels. mean teacher model uses similar strategy for the labeled data points, instead here we apply that to the identified prototypes from the target data. this way, the pre-trained network can be subjected to a further constructive training by introducing a set of new labelled samples, which could help with the generalisation for the trained model. another use of the augmentation here is to address the issue of highly skewed and imbalanced distribution of data between the classes. here, the augmentation is utilized not just to generate new samples to regularize the network but also to make the data distribution balanced across the classes by adding new samples in accordance with the preferred proportions. although there could be many possible ways to augment the samples so that their labels can be maintained, for this specific task, since we have a highlighted concept term in each sentence, we have used the augmentation by replacing that concept term. the new sentences are generated from their parent prototype sentences by replacing the concept term with a concept term from any randomly selected sentence of the same class in the data set. here we have assumed that most of the concept terms will represent medical condition and thus nouns. even when the grammar and the sentence structure is preserved, the result of the augmentation might still not be perfect due to the possible ambiguity in the concept term selection or incorrect grammar in the original sentence itself. for example, consider the sentence "...<@xmath1.". here it is not clear if the entity "cortisol" is negated or non-negated. however, it should be noted that such ambiguities are relatively less frequent which in turn implies that the augmentation will suffice its purpose for the majority of the time. protoaug and adapt-protoaug: combining the concepts discussed above, here we propose our two approaches protoaug and adapt-protoaug, both of which share the same set of concepts of class prototypes and augmentation as explained before. for both protoaug and adapt-protoaug, the common underlying procedure is as follows: first the prototypes are identified from the target domain data and then the augmentation follows with the intent of regularizing the network and to create a more balanced distribution of samples across the classes. now both the prototypes and their augmented samples with their respective labels are used with cross entropy loss to update the weights of the feature extractor module f of the pre-trained network s, with classifier module c of the pretrained network being frozen. the fundamental difference between protoaug and adapt-protoaug is about the adaptive nature of adapt-protoaug in recognizing the prototypes from the original target domain dataset after every epoch, which protoaug does only at the beginning of the training. adapt-protoaug makes incremental changes to the percentile score (initially 50) for the self-entropy threshold. the intuition behind using this strategy is that as the training proceeds, model will become more confident on the training samples and the entropy values for all the samples will significantly decrease. a possible drawback of using a fixed percentile criteria for the threshold at every epoch would likely exclude some reliable samples even when they achieve objectively quite lower values of self-entropy. to avoid such scenarios, apart from repeating the same process of prototype identification followed by augmentation after every epoch, we also propose to increase the percentile score for determining the self-entropy threshold in an uniform manner throughout the training with some fixed upper bound (70). this is also explained further in the analysis (section 5). ## experimental setup model: the pre-trained model used in subtask-1 is a roberta @xcite based sequence classification model foot_0 provided by organizers after training on source domain data inaccessible to us. it has two modules -a feature extractor and a classifier that operates over the output @xcite token from the feature extractor. data: in the practice phase, train split of the practice dataset was used to further train the pre-trained model whereas the dev split was used as a validation set for the hyper-parameter tuning. in the evaluation phase, due to our computational constraints, we were only able to utilize a randomly selected subset of 25k samples from around 600k sentences of the test dataset, for retraining of the pre-trained model. for evaluation the organisers use an annotated subset of the test dataset. during the evaluation phase this subset was kept hidden from the participants. hyper-parameters setting: for protoaug, selfentropy percentile threshold is set to 50% whereas as for adapt-protoaug it is uniformly increased after every epoch from being 50% at the first to being 70% at the final epoch. using augmentation, the final number of samples per class is set to be x times the number of prototypes belonging to the majority (non-negated) class. in our experiments, we choose x to be 4. for both the approaches, the model training is performed for 10 epochs. during the test phase, we reuse the hyper-parameters obtained from the practice phase. further details about hyper-parameter selection can be found in appendix a. ## results ## conclusion in this work, we carefully explored the problem of source-free domain adaptation for the negation detection subtask. we studied the importance of the confidence that a model places on its prediction and analyzed its formulation in terms of the samples' self-entropy scores. further, using those insights, we proposed two simple and intuitive approaches, namely protoaug and adapt-protoaug for the negation detection subtask and got an improvement of 7% on the test set with respect to the baseline model.