M ed AI at S em E val-2021 Task 10: Negation-aware Pre-training for Source-free Negation Detection Domain Adaptation (2021)

Abstract: Due to the increasing concerns for data privacy, source-free unsupervised domain adaptation attracts more and more research attention, where only a trained source model is assumed to be available, while the labeled source data remain private. To get promising adaptation results, we need to find effective ways to transfer knowledge learned in source domain and leverage useful domain specific information from target domain at the same time. This paper describes our winning contribution to SemEval 2021 Task 10: Source-Free Domain Adaptation for Semantic Processing. Our key idea is to leverage the model trained on source domain data to generate pseudo labels for target domain samples. Besides, we propose Negation-aware Pre-training (NAP) to incorporate negation knowledge into model. Our method win the 1st place with F1-score of 0.822 on the official blind test set of Negation Detection Track.

## introduction the negation detection track of semeval 2021 task 10: source-free domain adaptation for semantic processing provides a new setting for unsupervised domain adaptation task which ask participants to conduct negation detection in target domain only with model trained on source domain (namely source model) and unlabeled target domain data. negation detection is a span-in-context classification problem, where the model will jointly consider both the target mention to be classified and its surrounding context. for example, in sentence has @xmath0, the target span diarrhea is negated by its surrounding context no. this task is important for physicians to extract key information from clinical text. the test dataset used is based on mimic-iii version 1.4 @xcite , which is a large, freely-available english database comprising de-identified health-related data. we approach this task as a problem of learning with pseudo labels. our main interests include 1) negation knowledge infusion through pre-training on target domain and 2) high-quality pseudo label generation. we divide the task into two stages: negation-aware pre-training (nap) stage and pseudo label training stage. in the nap stage, token-level and sentence-level negation semantic are embedded into model. in the pseudo label training stage, confidence threshold search and mean self-entropy are used to select target domain samples with highly confident pseudo labels. ## related work traditional negation detection method are mostly rule-based. these methods @xcite @xcite used regular expression algorithm, dependency parsing and grammatical parsing to perform negation cue detection and scope resolution. recent years, deep learning has been applied to negation detection task. in @xcite , convolutional neural network was used to recognize negation scope in the sentence. @xcite and @xcite leveraged recurrent neural network variants to perform negation scope resolution and achieved better performance with bilstm, which further indicates the potential in deep learning-based methods. joint model to detect negation cues and targets simultaneously had been studied by @xcite . more recently, popular transformer-based model @xcite had also been used to perform negation detection. due to data privacy and data transmission problem, several source-free unsupervised domain adaptation methods @xcite @xcite have been proposed for image classification task. these methods mostly focus on generating high-quality pseudo labels for target domain samples before or during training phase and do not involve self-supervised pre-training. in the natural language processing filed, pretraining is popular. we train language models on huge corpora and fine-tune the pre-trained architectures @xcite @xcite in downstream tasks, achieving state-of-the-art results on most nlp tasks. prior studies @xcite @xcite has further shown the potential of domain-adaptive and task-adaptive pre-training. ## method we approach the task of source-free domain adaptation for negation detection as a problem of learning with pseudo labels. to generate high-quality pseudo labels, we use mean self-entropy as metric to search appropriate probability threshold, which is inspired by @xcite . besides, in order to learn more negation semantic knowledge from target domain, we propose negation-aware pre-training to incorporate negation knowledge by self-supervised training. ## source model and data source domain data the source domain data is from sharp seed dataset which consists of deidentified clinical notes from mayo clinic. in the sharp data, clinical events are marked with a boolean polarity indicator, with values of either asserted or negated. there are 10259 samples provided including 902 negated instances for source model training. source model since the data privacy limitation, sharp seed dataset cannot be distributed. thus organizers provide a "span-in-context" negation detection model trained on sharp seed dataset as the source model. the source model is robertabased and could achieve promising result on the sharp seed dataset. ## results the result is evaluated using the standard precision, recall and f1 scores as used in most published work. we achieve the best performance on the official blind test dataset. ## conclusion in this paper, we model source-free domain adaptation as learning with pseudo label. we leverage mean self-entropy of dataset to search appropriate probability threshold for high-quality pseudo label generation. besides, we propose negation-aware pre-training to integrate different types of negation knowledge to improve the generalization of negation representation. the result shows that the additional negation-aware pre-training is helpful for negation detection task. in the future, we will work towards more robust pseudo label generation method and effective pre-training task introducing more knowledge such as part-of-speech tag and dependency relation.