S em E val-2021 Task 10: Source-Free Domain Adaptation for Semantic Processing (2021)

Abstract: This paper presents the Source-Free Domain Adaptation shared task held within SemEval-2021. The aim of the task was to explore adaptation of machine-learning models in the face of data sharing constraints. Specifically, we consider the scenario where annotations exist for a domain but cannot be shared. Instead, participants are provided with models trained on that (source) data. Participants also receive some labeled data from a new (development) domain on which to explore domain adaptation algorithms. Participants are then tested on data representing a new (target) domain. We explored this scenario with two different semantic tasks: negation detection (a text classification task) and time expression recognition (a sequence tagging task).

## introduction data sharing restrictions are common in nlp datasets. for example, twitter policies do not allow sharing of tweet text, though tweet ids may be shared. the situation is even more common in clinical nlp, where patient health information must be protected, and annotations over health text, when released at all, often require the signing of complex data use agreements. the source-free domain adaptation shared task presents a new framework that asks participants to develop semantic annotation systems in the face of data sharing constraints. a participant's goal is to develop an accurate system for a target domain when annotations exist for a related domain but cannot be distributed. instead of annotated training data, participants are given a model trained on the annotations. then, given unlabeled target domain data, they are asked to make predictions. this is a challenging setting, and much previous work on domain adaptation does not apply, as it assumes access to source data @xcite @xcite , or assumes that labeled target domain data is available @xcite @xcite . two different semantic tasks in english were created to explore this framework: negation detection and time expression recognition. these represent two common types of classification tasks: negation detection is typically formulated as predicting an attribute of a word or span given its context, and time expression recognition is typically formulated as a named entity tagging problem. both of these tasks have previously been run as shared tasks, and had at least two different domains of data available, and we had access to experienced annotators for both tasks, allowing us to annotate data in a new domain. negation detection is the task of identifying negation cues in text. this task has been widely studied by previous work @xcite @xcite including the development of a variety of datasets @xcite . however, there are still large performance losses in the cross-domain setting @xcite . for negation detection, we provided a "spanin-context" classification model, fine-tuned on instances of the sharp seed dataset of mayo clinic clinical notes, which the organizers have access to but cannot currently be distributed. (models were approved to be distributed, as the data is deidentified.) in the sharp data, clinical events are marked with a boolean polarity indicator, with values of either asserted or negated. as development for time expression recognition, we provided a sequence tagging model, fine-tuned on deidentified clinical notes from the mayo clinic, which were available to the task organizers, but are difficult to gain access to due to the complex data use agreements necessary. (models were approved to be distributed, as the data is deidentified.) the development data was the annotated news portion of the semeval 2018 task 6 data whose source text is from the freely available timebank. for evaluation, we used a set of annotated documents extracted from food security warning systems. the main impact of this task is to drive the nlp community to address the serious challenges of data sharing constraints by designing new domain adaptation algorithms that allow source data and target data to remain separate, rather than assuming they can be shared freely with each other. ## data and resources in this section, we describe both negation detection and time expression recognition tasks, the models fine-tuned on a difficult-to-obtain set of annotated data, the development data representing a new domain on which participants can explore their approaches for domain adaptation, and the test data representing another new domain on which the systems developed by participants are evaluated. details of the different data sets can be found in tables 1 and 2 . ## evaluation metrics negation detection was evaluated using the precision/recall/f 1 of the negated class, as used in most published work. time expression recognition was evaluated using the standard precision/recall/f 1 previously used for the entityfinding portion of semeval 2018 task 6. in both cases, the metrics are defined as: dev-tuned models pre-trained on the source data (i.e., source-trained) and then fine-tuned on the labeled dev data. all baselines were built on roberta @xcite using the huggingface transformers library. 5 ## participating systems since our goal was to see a set of experiments as varied as possible, we did not impose any constraint on the approaches participants could submit, including the use of any of the unlabeled or labeled data provided. the task had 9 participants that submitted 20 unique runs in total, as shown in ## evaluation results tables 5 and 6 shows the performance of the systems described in section 5 on negation detection and time expression recognition. for comparison, the tables also include the performance of the baselines described in section 4. ## future directions self-learning and data augmentation were the most frequently used techniques. some systems, including the best performing ones, incorporated some kind of manual supervision in the form of active-learning, hand-crafted heuristics or semiautomatically building a training set. this suggests that future work on source-free domain adaptation will focus on acquiring data instances for the target domain either automatically or manually, and use such data to continue fine-tuning the sourcedomain model. any new approaches will have to address some fundamental challenges. errors in the generation of pseudo-labels propagate in successive self-learning iterations degrading the performance. continual fine-tuning on data from a new domain can lead to catastrophic forgetting, especially if the data is restricted to certain instances like those drawn from high-confident predictions of the source model. manually supervised approaches, such as active learning, do not necessarily solve these problems due to the complexity of some annotation schemes, like in time expressions recognition, and the reduced number of labels that this methods can yield. some of the experiments carried out during this task have approached these issues and should be taken as an starting point for future research. ## conclusion in this paper, we have described the source-free domain adaptation shared task held within semeval-2021. in this task, participants were asked to adapt a given model to a target domain when the access to both labeled and unlabeled source data is restricted. in contrast to previous tasks on domain adaptation, participants were only provided with a trained model and the target unlabeled data. systems were evaluated on two tasks, negation detection and time expression recognition, that are paradigmatic examples of two common types of machine-learning problems in natural language processing: text classification and sequence labeling. the scripts and the code of the baselines, along with the development and test data, can be obtained from the task's github repository. 6 the trained models are available in the huggingface model hub for both negation detection 7 and time expression recognition. 8 the codalab 9 leader-board of the of the post-evaluation phase will continue to accept submissions indefinitely.