BLCUFIGHT at S em E val-2021 Task 10: Novel Unsupervised Frameworks For Source-Free Domain Adaptation (2021)

Abstract: Domain adaptation assumes that samples from source and target domains are freely accessible during a training phase. However, such assumption is rarely plausible in the real-world and may causes data-privacy issues, especially when the label of the source domain can be a sensitive attribute as an identifier. SemEval-2021 task 10 focuses on these issues. We participate in the task and propose novel frameworks based on self-training method. In our systems, two different frameworks are designed to solve text classification and sequence labeling. These approaches are tested to be effective which ranks the third among all system in subtask A, and ranks the first among all system in subtask B.

## introduction deep neural networks have achieved remarkable success in a variety of applications across different fields while with huge expense of laborious largescale training data annotation. to avoid expensive data labeling, domain adaptation(da) methods was proposed to fully utilize previously labeled datasets and unlabeled data on hand in a transductive manner, which obtained promising results in sentiment analysis, part-of-speech tagging, machine translation, etc. @xcite @xcite unsupervised domain adaptation(uda) aims to reduce the domain shift between labeled and unlabeled target domains. early works @xcite learnt domain-invariant features to link the target domain to the source domain. along with the growing popularity of deep learning, plenty of works benefited from its powerful representation learning ability for domain adaptation. those methods typically minimized the distribution discrepancy between two domains @xcite , or deployed adversarial training @xcite @xcite . however, a crucial requirement in the methodology of these methods is that all samples from both domains are freely available during the training process, which is inefficient in data transmission and may violate the data privacy policy. for example, it is not allowed to share tweet texts according to twitter policies, though tweet ids can be shared. the situation is even more common in clinical nlp, where patient health information must be protected, and annotations over health text, when released at all, often require the signing of complex data use agreements. semeval 2021 task 10 focuses on the problem of source-free domain adaptation for semantic processing. subtask a of task 10 is negation detection which aims to classify clinical event mentions (e.g., diseases, symptoms, procedures) for whether they are being negated by their context. traditional systems, such as one of the first algorithms negex @xcite was based on rules. subsequently, syntax-based methods were developed @xcite . in recent years, some researchers explored new generation of transfer learning models (bert) to solve this task @xcite , outperforming the previous state-of-the-art systems by a significant margin. subtask b of task 10 is time expression recognition which aims to find time expressions in text. it is a sequence labeling task as @xcite described in their work. a few of works combined traditional machine learning with rules achieved good performances @xcite . some studies got character-level contextual embeddings @xcite and applied to this task, yielding major performance improvements over the previous state-of-the-art. in this paper, we propose two different unsupervised frameworks for each subtask in source-free setting. for negation detection task, we design a framework which obtain pseudo labels with high confidence by using reliable pseudo labels as prototypes. for time expression recognition, we design an unsupervised teacher-student framework with mean teacher. ## system description for subtask a, we used a pseudo-labeling training method. to reduce the uncertainty from the pseudo labels, we only chose those with high confidence to fine-tune the model. finally, we ensembled 5 models to make the model have better robustness and results. for subtask b, we started by data preprocessing. then, we enlarged the training set with pseudo-labeled sentences, which were predicted on the test set by teacher model. in addition, mean teacher helps to generate better pseudo labels. finally, we used the ensemble model to make predictions and add manual expressions. each module will be introduced in detail in the following sections. ## results the performance of our system and the task baselines for both subtasks are shown in following tables. figures 3 and 4 show the confusion matrix of the classification results of baseline model and our model on the test dataset. this corresponds to baseline and sfda+ensemble in table2, respectively. compared with baseline, we predicted more true positive samples. however, the false prediction of negative samples as positive has increased. as a result, the recall and f1 score of our model have been improved, but the precision has decreased a little. error analysis. for subtask a, we conducted statistics and analysis on the classification results of baseline model and our best model. there are 474 sentences that both baseline model and our best model predict correctly. the entity being detected usually follow the word that express negative in this part, there are some long-distance keywords or parallel phrases. e.g. "... no tobacco, etoh, @xmath2. for these, we consider to add some handcraft rules to improve the results of the model. for subtask b, we conducted a manual error analysis. for the raw text "during the harvest season", both our model and baseline model incorrectly labeled "harvest" with "season-of-year" instead of "harvest season", "harvest" is just the activity, though if it instead said "harvest season", we would annotate that whole thing as a "season-of-year". for the raw text "february (27,661)", baseline model incorrectly label "27" with "day-of-month" while our model didn't, which proves the effectiveness of our architecture. in addition, we list a detailed description of the recall of subtask b in ## conclusion we introduce two different frameworks which are both based on self-training method for text classification and sequence labeling in semeval 2021 task 10, in order to address the problems of source-free, labeled training data scarcity. in subtask a, we used a metric learning method, combining pseudo labeling with prototype network and achieve good results. in subtask b, we employed teacher-student framework, and then we propose to use high-confidence soft labels to further improve the self-training. our system takes third place in subtask a and first place in subtask b. in future, we would like to introduce adversarial training and more data augmentation approaches in our model to further facilitate source-free domain adaptation.