YNU - HPCC at S em E val-2021 Task 10: Using a Transformer-based Source-Free Domain Adaptation Model for Semantic Processing (2021)

Abstract: Data sharing restrictions are common in NLP datasets. The purpose of this task is to develop a model trained in a source domain to make predictions for a target domain with related domain data. To address the issue, the organizers provided the models that fine-tuned a large number of source domain data on pre-trained models and the dev data for participants. But the source domain data was not distributed. This paper describes the provided model to the NER (Name entity recognition) task and the ways to develop the model. As a little data provided, pre-trained models are suitable to solve the cross-domain tasks. The models fine-tuned by large number of another domain could be effective in new domain because the task had no change.

## introduction data sharing constraints are common in natural language processing (nlp) datasets. for example, twitter policies prohibit the sharing of tweet text, although tweet ids may be shared. in clinical nlp, the situation is even more prevalent because information on patients' health must be protected. obtaining annotations about health texts often requires the signing of complex data usage agreements. during the competition, the organizers provided models which fine-tuned on the annotated source domain data, while the source domain data could not be distributed. the organizers also provided labeled data as dev data and unlabeled target domain data as test data. there were two sub-tasks for semeval task 10: 1) to classify clinical event mentions (e.g., diseases, symptoms, procedures, etc.) for whether they are being negated by their context. this is a text-classification task @xcite . 2) to find time expressions @xcite ) in text, this is a sequence-tagging task. semeval task 10 was different from traditional nlp tasks, where training and testing were in the same domain. predictions could be out of control due to the different target domain. the domain of the dev data is related to the source and target domains, therefore the model can be developed. in semeval task 10, it is necessary to develop an existing model along with training the model with labeled data or unlabeled data. furthermore, the model can be developed by fine-tuning it in different ways. for certain reasons, only the results for the second subtask were submitted, that is, time expression recognition, which can be considered as a name entity recognition (ner) task. the participants were asked to find time expressions in the text through the task. this is a sequence-tagging task that uses fine-grained time expression annotations that are a component of semeval 2018 task 6 @xcite . to deal with this task, deep learning models, such as long short-term memory (lstm) @xcite , bidirectional lstm with conditional random field (bilstm-crf) @xcite , and bidirectional encoder representation from transformers (bert) @xcite have been developed to challenge the ner task. owing to the lack of a training set, pre-trained models @xcite were considered. the ner task can become difficult as the number of classifications increase. in this task, 33 types of entities need to be recognized, and phrases and words may be one of the types. for the base model, b-prefix and i-prefix were used to discriminate the beginning of classification and ensure the accuracy of the tokens; therefore, 65 types of entities were used in the model. in this paper, an ensemble model is proposed for time expression recognition with a hard voting strategy. each input sample was first tokenized using the matched model tokenizer. base models, such as bert @xcite and its variants, including roberta @xcite , distilbert @xcite and albert @xcite , were used to learn hidden representations for each token. then, a fully connected dense layer with a sof tmax function was used for classification. by using a hard voting strategy, the predictions from different base models were merged with the final result. experimental results show that the proposed ensemble models achieve a competitive result with the official baseline model, which ranked fifth in sub-task 2. the remainder of this paper is organized as follows. section 2 describes the overall structure of the proposed ensemble model. the comparative experimental results and discussion are presented in section 3. finally, conclusions are presented in section 4. ## ensemble model for source-free domain adaptation the official baseline model is a fine-tuned roberta model: clulab/roberta-timex-semeval. it uses roberta @xcite for token classification architecture, which is pre-trained using no next-sentence prediction (nsp) and dynamic masking. the roberta model achieved better performance than the bert model because it was pretrained by larger volume data, more steps, larger batches, and larger vocabulary than the bert model. in addition, the provided baseline model was fine-tuned with approximately 25,000 expressions in de-identified clinical notes as well as the development dataset @xcite . four other models were also implemented using a hard voting strategy and a hard voting result for a single submission. ## conclusions the semeval-2021 task 10 framework requests participants to develop semantic annotation systems in the face of data sharing constraints. in this study, we fine-tuned the official baseline model and then combined it with four other pre-trained models with a hard voting strategy for time expression recognition. experimental results showed that the proposed model outperformed the previously released baseline model, achieved a competitive result with the newly released official baseline model, and finally ranked fifth on the leaderboard. future work will attempt to improve the performance of cross-domain ner tasks.