A Comparison of Strategies for Source-Free Domain Adaptation (2022)

Abstract: Data sharing restrictions are common in NLP, especially in the clinical domain, but there is limited research on adapting models to new domains without access to the original training data, a setting known as source-free domain adaptation. We take algorithms that traditionally assume access to the source-domain training data—active learning, self-training, and data augmentation—and adapt them for source free domain adaptation. Then we systematically compare these different strategies across multiple tasks and domains. We find that active learning yields consistent gains across all SemEval 2021 Task 10 tasks and domains, but though the shared task saw successful self-trained and data augmented models, our systematic comparison finds these strategies to be unreliable for source-free domain adaptation.

## introduction deep neural networks achieve high performance in many tasks, but typically require annotated training data for each new domain. domain adaptation algorithms aim to take models trained on one domain (the "source domain") and transfer the model's knowledge to another domain (the "target domain"). they typically try to do this without a huge amount of annotated data in the target domain. domain adaptation can be easy if the source and target domain have similar distributions, but domains often differ substantially @xcite . while there has been much progress in domain adaptation methods @xcite and even in unsupervised domain adaptation where there are no target-domain labels @xcite , most methods assume access to the labeled source data. yet this assumption is often not satisfied, especially in the clinical domain due to privacy concerns @xcite . semeval 2021 task 10 @xcite , on source-free domain adaptation, called attention to this challenging but more realistic scenario where labeled source data are not accessible, only the model trained on the source domain data can be shared foot_0 , and little or no labeled target data are available. participants explored methods including selftraining, active learning, and data augmentation @xcite but it is hard to make fair comparisons between algorithms since different teams varied in their base implementations. we therefore conducted experiments to provide a systematic comparison of algorithms for sourcefree domain adaptation. our contributions are: 1. the first systematic comparison of self-training, active learning, and data augmentation for source-free domain adaptation, carried out across multiple tasks and domains. 2. we identify a formulation of source-free active learning that consistently improves performance of the source-domain model, and sometimes even outperforms fine-tuning on a large set of labeled target domain data. 3. we perform an error analysis across tasks and domains and show that the selected formulation of active learning corrects several types of errors that self-training does not. our code is publicly available. 2 2 related work ## data we base our experiments off of the data and sourcedomain models from the tasks of semeval 2021 task 10: negation detection and time expression recognition. we select these tasks because: 1. they represent real-world data-sharing problems: the negation source-domain data "cannot currently be distributed" and the time expression source-domain data is "difficult to gain access to due to the complex data use agreements" @xcite . only the task organizers had access to the data and permission to distribute models trained on the (de-identified) data. 2. the annotation schemes are complex enough that the problem cannot be easily solved by manually annotating the target domain. @xcite found that annotations from annotators given only the time annotation guidelines yielded no gains to models, while annotations from heavily trained annotators did yield gains. 3. these two tasks suffer a large performance loss under domain shift: the source-trained model is 15+ points of f1 lower on the target test set than on the source test set @xcite . the popular amazon reviews sentiment analysis dataset @xcite violates the points above: labeled source and target data are easily available, the annotation scheme is easy (it is artificially balanced and removes reviews with neutral labels, as others have noted @xcite ), and the source domain model performs well on the target domain (within 0-4 points of f1). we nonetheless include some experiments on this dataset in appendix a.3. we find that with simple data preprocessing and source-domain hyperparameter tuning, the source-domain model alone outperforms all domain adaptation models from @xcite and @xcite . semeval 2021 task 10 negation detection is a "span-in-context" classification task. the goal is to predict whether an event (denoted by two special @xmath0. for example, given the sentence: has @xmath1. the source-domain negation detection model was trained on mayo clinic clinical notes. the target domains are partners healthcare clinical notes from the i2b2 2010 challenge and beth israel icu progress notes from the mimic iii corpus. semeval 2021 task 10 time expression recognition is a sequence-tagging task. the goal is to identify the time entities in the document and label them with scate types @xcite . for example, given the sentence: the patient underwent appendicitis surgery on august 29, 2018, the goal is to label august as month-of-year, 29 as day-of-month, and 2018 as year. the sourcedomain time expression recognition model was trained on the mayo clinic clinical notes of sem-eval 2018 task 6 @xcite . the target domains are news articles (also from semeval 2018 task 6) and reports from food security warning systems including the un world food programme and the famine early warning systems network. each task has a model trained from a source domain and a test set for each of two target domains. for each target domain, we split the data into 20% as a development set and 80% as a test set. detailed data information is shown in we use only the english roberta-base models @xcite (approx. 125m parameters) that the task organizers fine-tuned on the source domain data sets via the huggingface transform-ers library v3.5.1 @xcite . target development data we use the development data for fine-tuning the model. for active learning, to simulate manual annotation, we finetune on a small number of automatically selected labeled examples. for self-training, no labels are used; we fine-tune on predictions (pseudo-labels) generated by the model on the development data. for oracle experiments, we fine-tune the model on all labeled examples in the development set. target test data we evaluate on the test data. no fine-tuning is performed. models always treat this data as unlabeled foot_2 . its labels are used only during evaluation. we use the same evaluation metrics as in semeval 2021 task 10: precision, recall, and f1 score. ## research questions we aim for a systematic analysis of three strategies with many different implementations in semeval 2021 task 10: self-training, active learning, and data augmentation. our research questions are: 1. how much can we gain from having human intervention (active learning) and not just the model alone (self-training)? 2. for active learning, given a fixed annotation budget, is it better to do several iterations of selecting examples for annotation and retraining the model, or to select and retrain just once? 3. for self training, given a fixed confidence threshold, is it better to do several iterations of generating pseudo-labels and retraining the model, or to generate and train only once? 4. in each iteration of active learning or selftraining, should we use the training data from the previous iteration or start anew? 5. in each iteration of active learning or selftraining, should we continue training the model from the previous iteration or the model from the source-domain? 6. do active learning and self-training improve with data augmentation or work better alone? ## method we design source-free variants of self-training, active learning, and data augmentation that incorporate the following parameters, allowing us to investigate the questions above. ## experiments the input to the source-domain models for both tasks is a sentence. the output for the negation detection model is a sentence label (negated or not negated). the output for the time expression model is one label per token (its time entity type). for both tasks, we use the conventional roberta input format, surrounding the sentence with the special @xmath2. the negation detection data is already split into sentences. for the time recognition data, we split it into sentences using the english sentencizer from spacy v2.3.2 @xcite . when we fine-tune the source-domain model on the target domain, we keep the same training hyperparameters as were used when the shared task organizers trained the models on the source domains. in source-free domain adaptation, there is no (or very little) labeled development data available, so it is not possible to tune hyperparameters. all hyperparameters are given in appendix a.1. all experiments are run on a single nvidia p100 gpu. the total approximate gpu hours are 70 hours. in self-training, we set the threshold τ to 0.95, and experiment with running just a single iteration and with running 30 iterations with the different s d and s m strategies. the threshold and the number of iterations are adapted from @xcite . training may run for fewer iterations when the stopping conditions are met. in active learning, we set our annotation budget to 96 sentences, and experiment with spending these 96 sentences at once and in 8 iterations with the different s d and s m strategies. for all experiments, we run one version with data augmentation (with @xmath3. for each source and target domain pair, we compare our adapted model with the following models. ## discussion tables 2 and 3 show the results of our experiments. we are interested less in the best model for a particular configuration, but rather in which configurations are successful across multiple tasks and domains. this is because in source-free domain adaptation, there is typically no (or very little) labeled target domain data available for hyperparameter tuning. therefore, what we need is a universal strategy that does not require careful tuning. for source-free active learning, we find that even small amounts of annotated data are useful, and that smart data selection (e.g., using uncertainty scores) is usually helpful. the active learning keep-data models (rows 6, 8, 11, and 13 in tables 2 and 3) have higher f1s than the baseline source domain models across all tasks and domains (0.054 f1 higher on average). active learning keepdata models also outperform passive learning models (that randomly select data) in 14 out of 16 cases, and are at least as good as, and typically much better than, the self-training models (rows 15-24 in tables 2 and 3). the resetmodel+resetdata models always have the worst f1s of the active learning models (rows 7 and 12 in tables 2 and 3). several active learning models achieve higher f1s than the "oracle" model that fine-tuned on the full labeled development set (row 8, 10, 11, 13, 14 in for source-free self-training, we find that iteratively updating both model and data is slightly above baseline, and that it is better to start from the source-domain model than from roberta without fine-tuning. the keepmodel+keepdata (without data augmentation) is slightly above the sourcedomain model across all tasks and domains (0.013 f1 higher on average). every other configuration, even if they outperform keepmodel+keepdata in one task or domain, is below the source-domain baseline in another. all self-trained models without data augmentation (which start from the sourcedomain model) do at least outperform self-distilled models (which start from the roberta model without fine-tuning; row 3 in tables 2 and 3). the small gains from the only self-training configuration that consistently outperformed the sourcedomain model suggest that self-training may not be worthwhile for source-free domain adaptation. data augmentation helped in some cases (e.g., self-training time expression recognition on news), and hurt in others (e.g., self-training time expression recognition on food security). data augmentation sometimes led to ill-behaving models: on the negation mimic-iii dataset, data augmentation made the self-trained model predict all examples as not negated resulting in 0.000 f1 (rows 21 -24 in we thus make the following suggestions for source-free domain adaptation: ## error analysis we performed an error analysis to try to determine if different adaptation strategies resulted in different types of errors being corrected (as compared to the source domain model). for negation detection we sampled and categorized around 200 errors of the source-domain model for each target domain. when the model failed to predict a negation, we manually categorized the error by the negation cue (no, free, absent, etc.). when the model predicted a negation it should not have, we manually categorized the error into "wrong cue" (there was a negation cue in the sentence but it did not apply to the target event) or "short sentence" (especially on the i2b2 domain, the model liked to predict all short sentences as negated). for time expression recognition, we categorized all errors of the source-domain model by entity type (inside-outside-beginning format) for each target domain. for both tasks, we then calculated how many of these source-domain model errors the best adapted models continued to make. heatmaps of these analyses are plotted in appendix a.2. across all tasks and domains, we see that the best self-trained models correct errors roughly evenly across sourcedomain error categories, while the best active learning models correct different errors, more like the oracle (target-fine-tuned) model. for example, the oracle model and active learning adapted models correct many more "wrong cue" errors in the negation i2b2 domain, more denies and none errors in the negation mimic iii domain, more b-period and b-month-of-year entities in the time news domain, and more b-season-of-year, i-season-of-year, and b-this entities in the time food domain. some error types appear to be only learnable with substantially more data. only the oracle model is able to correct errors with the non and afebrile negation cues in the i2b2 domain and with the hold negation cue in mimic-iii domain. this suggests that the source-domain model may be very confident in some types of wrong examples causing them not to be selected in active learning and generating poor pseudo-labels in self-training. ## conclusion in this paper, we present a detailed comparison of the use of active learning, self-training and data augmentation to adapt a source-domain model on a target domain when the source-domain training data is unavailable. we identify a specific formula-tion of source-free active learning that consistently improves performance of the source-domain model. we believe our work highlights the interesting challenges of source-free domain adaptation, and its systematic comparison provides a solid base for future research in this area. pages 400-410, vancouver, canada. association for computational linguistics.