The U niversity of A rizona at S em E val-2021 Task 10: Applying Self-training, Active Learning and Data Augmentation to Source-free Domain Adaptation (2021)

Abstract: This paper describes our systems for negation detection and time expression recognition in SemEval 2021 Task 10, Source-Free Domain Adaptation for Semantic Processing. We show that self-training, active learning and data augmentation techniques can improve the generalization ability of the model on the unlabeled target domain data without accessing source domain data. We also perform detailed ablation studies and error analyses for our time expression recognition systems to identify the source of the performance improvement and give constructive feedback on the temporal normalization annotation guidelines.

## introduction unsupervised domain adaptation (uda) is a task that generalizes knowledge acquired from a model trained on labeled data in one domain @xcite to unlabeled data in a different domain (target domain). conventional uda algorithms usually require access to both source-domain and target-domain data @xcite @xcite . however, sharing source-domain data is often not practical for clinical texts due to their highly sensitive personal information and complex data use agreement procedures @xcite . to overcome this difficulty, @xcite propose a new task of source free domain adaptation (sfda) where only models trained on source-domain data are shared, which allows the possibility of using the information from the source-domain while reducing private information leakage. the biggest challenge of this task is to transfer task-related information embedded in the trained models. our team participated in both subtasks of sem-eval 2021 task 10 @xcite , source free domain adaptation for semantic processing: negation detection and time expression recognition. for both tasks, participants were given a roberta model @xcite fine-tuned on the sourcedomain, and asked to make predictions in the targetdomain. the goal of the negation detection task is to predict whether an event in a sentence is negated by its context. this is a binary sentence classification task. for example, given the event diarrhea and the sentence has no diarrhea and no new lumps or masses, the goal is to predict that diarrhea is negated by its context. the goal of time expression recognition sub-task @xcite is to recognize time expressions in the target domain. this is a named entity recognition (ner) task. the number of entity types (inside-outside-beginning format) is 65. entity types in this task are formally defined time entity types from the semantically compositional annotation of time expressions (scate) @xcite ) annotation schema. for example, in 2021-02-19, 2021 will be labeled as year, 02 will be labeled as month-of-year and 19 will be labeled as day-of-month. we investigate self-training, active learning, and data augmentation techniques on negation detection and time expression recognition under the sfda setting. our contributions are: ## system description the source-domain models for both subtasks are roberta-base models with linear classification output layers, implemented via the huggingface transformers library @xcite , using robertaforsequenceclassification for negation, and robertafortokenclassification for time. the input to the models is a sequence tokenized by byte-pair encoding (bpe). following the conventions of the roberta model input format, two special @xmath0, respectively. in the negation detection task, targeted events are denoted with two special @xmath1. for example, the sentence has no diarrhea and no new lumps or masses with event diarrhea will be converted @xmath2.
. the model output for negation detection is whether the target event is negated and the model output for time expression detection is the labels for each input tokens. ## data all data used is in english. both subtasks had training, development and test data, each representing different domains. as participants, we did not have access to the training set. the training sets are used by organizers to fine-tune the pretrained roberta-base models to obtain the sourcedomain models. we used the source-domain models and development sets to develop source-free domain adaptation systems during the practice phase, and tested our systems during test phase. we summarize the data in ## experiments the organizers provided two baseline models for each task: the source-domain model, and the source-domain model fine-tuned on the development set. the official evaluation metric is the f1 score. precision and recall scores are also reported. ## time expressions ablation study effect of fine-tuning on dev data from the baseline models' performances (row 1 vs row 2 in ## time expressions annotation analysis though gold annotations led to large performance improvements, the annotation for this task is challenging for untrained people. through reading the scate annotation guidelines and posting questions on the share task google group, our team annotated 160 sentences of which 48 sentences were in the labeled portion of the test set. we annotated 13008 tokens in total (including padding tokens) and our overall accuracy on the gold 48 sentences is 0.991 for all categories and 0.785 excluding the category o. we report detailed performance for each of the entity types in we found several annotation patterns where our team consistently disagreed with the gold annotations. our errors can be broadly attributed to two reasons: misinterpretation/underspecification of annotation guidelines, and ambiguity of the phrases. errors from misinterpretations/underspecification of annotation schema : we annotated the token seasonal(ly) (e.g., seasonal progress, seasonal rainfall) as calendar-interval instead of season-of-year as we thought season-of-year is applied to seasons that are explicitly specified (such as summer). we considered seasonal similar to weekly, both referring to an interval unspecified. however, season-of-year could be applied to very broad categories such as dry seasons and rainfall seasons including seasons that are not specified. also, seasonal unlike weekly/monthly/yearly only refers to one season of a year instead of every season of a year. due to the ubiquity of this token in the dataset, this error affects our overall performance. correcting the annotation of this particular token leads to +.042 f1 score improvement (row 4 vs row 3 in errors from ambiguity within phrases some phrases allow multiple interpretations that lead to different ways of annotations. for instance, con-fusion between 'period' and 'calendar interval' occurred frequently (e.g., we annotated weeks in recent weeks as 'period' rather than 'calendar-interval'). although "/" between seasons is commonly annotated as 'i-season-of-year' in the gold annotations, we found different roles it might play in specific contexts. for example, if "/" is used between two terms that refer to the same season, then it should be annotated as 'i-season-of-year'; if it is used between two non-adjacent seasons, it should be annotated as 'union'; and if it is used between two adjacent seasons, then it could be annotated as 'between' or 'union'. thus, the correct annotation requires a surprising amount of external knowledge about ethiopian season terms. in fact, there are still cases that remained uncertain: for example, xaran refers to seasonal rains from april through september and xagaa refers to the second dry season (july to september) and when the two tokens joined by "/" it is difficult to interpret the meaning of "/". we also found the conjunction and causes ambiguity. for example, and in rains in may and august could be considered as an operator over months (i.e., rains in union(may, august)) or an operator over rains (i.e., union(rains in may, rains in august)). the former understanding requires annotating and whereas the latter does not despite the fact that the two interpretations are essentially semantically equivalent. lastly, we also found the particle the is difficult to annotate. for instance, the in the month depending on the context may be annotated as this or last and sometimes the context may not be clear enough to tell the differences. our annotation analysis leads to several suggestions for the annotation schema and the documentation. our errors in the first category indicate some potential helpful updates can be made such as including more examples in categories (e.g., 'season-of-year'), explicit documentation of whether the certain category is closed or open as well as the specific manner to deal with multi-word phrases or even circumpositions. the second category of errors, however, might involve the refinement of the annotation schema. for example, maybe 'between' and 'union' can be unified together, and 'period' can be merged into 'calendar interval' or confined to an explicit set of circumstances. ## conclusion our overall rank (by f1 score) for negation detection task was 2 nd and for time expression recogni-tion was 6 th . our results suggest that simple self-training can be used in sentence-level sfda tasks to improve a trained model's performance on a new domain. as for token-level tasks, our analysis shows that both active learning and data augmentation can bring significant performance improvements, but the premise is that the data in active learning can be correctly annotated. our analysis and feedback could also be used to improve the scate annotation guidelines/schema in future work.