Self-Adapter at S em E val-2021 Task 10: Entropy-based Pseudo-Labeler for Source-free Domain Adaptation (2021)

Abstract: Source-free domain adaptation is an emerging line of work in deep learning research since it is closely related to the real-world environment. We study the domain adaption in the sequence labeling problem where the model trained on the source domain data is given. We propose two methods: Self-Adapter and Selective Classifier Training. Self-Adapter is a training method that uses sentence-level pseudo-labels filtered by the self-entropy threshold to provide supervision to the whole model. Selective Classifier Training uses token-level pseudo-labels and supervises only the classification layer of the model. The proposed methods are evaluated on data provided by SemEval-2021 task 10 and Self-Adapter achieves 2nd rank performance.

## introduction domain adaptation (da) is the task of applying an algorithm trained on a source domain data to a different target domain data with limited/undefined labels. da has gotten significant attention as an alternative of fine-tuning approach @xcite @xcite , especially in situations rich supervision is not possible @xcite . da is an important way of overcoming the data shortage of deep learning since it enables the utilization of knowledge from other labeled data. source-free da is then proposed to cope with such data sharing in the general setting of da, the data distribution in the source domain and the target domain are related but different @xcite , and annotated samples from the source domain are available during the training process. however, many of the data resources are not allowed to be shared in real-life environments as there are increasing concerns for privacy issues. for example, twitter has a regulation that prevents sharing tweet text. the policy is even more rigorous in the financial/clinical domain under the privacy protection issue. unlike conventional da, one can not get access to the source domain data in source-free da but is provided a model trained on the source domain data. about source-free da in computer vision, several approaches have been proposed; @xcite assumes the target domain data is a transformation from the source domain data along natural axes such as brightness and contrast; @xcite proposes universal da that is trained via two-stage learning of procurement and deployment; @xcite progressively updates the target model with pseudo-labels which are selected under self-entropy criterion. as for natural language processing (nlp), the application of source-free da is slightly more complicated since sentences are usually considered as having discrete representations. in this context, semeval-2021 task 10 has proposed a challenge that is related to source-free domain adaptation for semantic processing. in this paper, we propose self-adapter for the time expression recognition sub-task in semeval-2021 task 10. following @xcite , we employ pseudo-labels from the target domain to further supervise the model trained on the source domain data, while the entropy-based evaluation of reliable pseudo-labels is adopted in consideration of the discrete text data. in addition, we adopt sloughing trick to prevent over-fitting. to demonstrate the efficacy of the self-adapter, we evaluate the proposed method on the dataset by @xcite . we also compare the proposed method with several variations and another method we come up with, named selective classifier training (sct). in the end, the self-adapter has achieved 2 nd rank in the official evaluation period getting 0.811 f 1 which is 1.7 percentage points higher than the roberta-based sequence tagging model pre-trained only on source data. 2 systems description our proposed methods have three operations in common: (1) generating pseudo-labels, (2) filtering out pairs of reliable samples and pseudo-labels based on models' self-confidence, and (3) doing supervised learning using the pseudo-labels. we concentrate on sorting out 'reliable' pseudo-labels since training with incorrect labels harms the performance of the model. self-entropy is usually treated as an indicator of self-confidence @xcite . we adopt normalized self-entropy as the evaluation metric for pseudo-labels: specifically, we propose two adaptation methods to efficiently fit the model trained on a source domain to a target domain: self-adapter and sct. ## experiments we evaluate our two models: self-adapter, sct and their variations. ## conclusion in this paper, we propose novel training methods self-adapter and selective classifier training that improve model performance on the target domain only by leveraging the roberta-based model pretrained on source data. both models rely on selflearning with highly credible pseudo-labels that are filtered based on self-entropy, differ only in the range of trainable parts. also, we propose sloughing trick to prevent over-confidence of the model by softening the network output. our work is highly applicable in the real world since we have achieved remarkable improvement in performance using only a few test data which is not annotated at all, without any manual supervision.