PTST - U o M at S em E val-2021 Task 10: Parsimonious Transfer for Sequence Tagging (2021)

Abstract: This paper describes PTST, a source-free unsupervised domain adaptation technique for sequence tagging, and its application to the SemEval-2021 Task 10 on time expression recognition. PTST is an extension of the cross-lingual parsimonious parser transfer framework, which uses high-probability predictions of the source model as a supervision signal in self-training. We extend the framework to a sequence prediction setting, and demonstrate its applicability to unsupervised domain adaptation. PTST achieves F1 score of 79.6% on the official test set, with the precision of 90.1%, the highest out of 14 submissions.

## introduction semeval-2021 task 10 presents source-free unsupervised domain adaptation (sfuda) for semantic processing. 2 the goal of unsupervised domain adaptation is to transfer a model from a source domain to another, different domain -called target domain -using only unlabelled data in the target domain. source-free unsupervised domain adaptation additionally assumes no access to source domain data: only the source model pre-trained on the source domain data is available. this situation may occur when the source domain data contains protected information that cannot be shared, or even if it can, requires signing a complex data use agreement. while there are numerous works on sfuda outside nlp @xcite @xcite , sfuda research for nlp is severely lacking in spite of its importance in, for example, clinical nlp @xcite . there are two tasks involved in semeval-2021 task 10: negation detection and time expression recognition. we participate only in the latter. our approach is an extension of the parsimonious parser transer framework (ppt; @xcite ). ppt allows cross-lingual transfer of dependency parsers in a source-free manner, requiring only unlabelled data in the target side. it leverages the output distribution of the source model to build a chart containing high probability trees for each sentence in the target data. we extend this work by (1) formulating ppt for chain structures and evaluating it on a semantic sequence tagging task; and (2) demonstrating its effectiveness in a domain adaptation setting. we call our method parsimonious transfer for sequence tagging (ptst). we find ptst effective for improving the precision of the system in the target domain. it ranks 7th out of 14 submissions in the official leaderboard in terms of f 1 score, but 1st in precision, with a gap of 3 points from the second best. drawing on the model calibration literature, we provide a way to combat the problem of model overconfidence which is key to make ptst outperform a simple transfer of a source model to the target domain. however, we also find that ptst struggles in improving recall. in conclusion, our results suggest that ptst can be used for sfuda, but further work is required to improve the precision-recall trade-off in the target domain. ## background in the semeval-2021 task 10 time expression recognition task, the input is a single sentence, and the output is a sequence of tags indicating the time entity type of a word (if any). there are 32 time entities in total (e.g., year, hour-of-day), and the tags are coded in bio format. fig. 1 shows an example input sentence and its output. the task organisers provided a pre-trained source model for the task. this source model is roberta-base @xcite that is pre-trained on more than 25k time expressions in english clinical notes from mayo clinic in semeval-2018 task 6. the model is distributed online via huggingface models, foot_0 which can be obtained with huggingface transformers library. 4 the organisers also released trial data for the practice phase containing 99 annotated english articles from the news domain. the official test data released by the organisers in the evaluation phase contains 47 articles that are in a different domain from the source and development data. the time expression recognition task is formalised as a sequence tagging task. the literature on sequence tagging in nlp is massive @xcite @xcite @xcite . one closely related task is named-entity recognition (ner) whose goal is to detect mentions of named entities such as a person or organisation in an input sentence. @xcite introduced a now widely adopted neural architecture for this task, where input word embeddings are encoded with a bidirectional lstm @xcite before they are passed through per-word softmax layers. in recent times, it is common to replace the lstm with a transformer @xcite . with the advancements of large pretrained language models, the standard is to use a model such as bert @xcite as the encoder and fine-tune the model on labelled data. the source model of the time expression recognition task provided by the organisers was also trained in this manner. in the context of unsupervised domain adaptation, a popular approach is domain adversarial training. introduced by @xcite , it leverages multi-task learning in which the model is optimised not only on the main task objective but also a domain prediction objective. the learning signal for the latter is passed through a gradient reversal layer, ensuring that the learnt parameters are predictive for the main task, but general across domains. with pre-trained language models, @xcite proposed to continue pre-training on the unlabelled target domain data, prior to finally finetuning on the labelled source domain data. unfortunately, these approaches require access to the source domain data. there is relatively little work on sfuda in nlp, however, some works on source-free cross-lingual transfer exist. @xcite employ teacherstudent learning for source-free cross-lingual ner. a teacher model trained on the source side predicts soft labels on the unlabelled target side data, and a student model is trained on those soft labels. their method outperforms a simple direct transfer method where the source model is directly applied on the target side. more recently, a method for source-free cross-lingual transfer of dependency parsers was introduced by @xcite . the key idea is to build a chart of high probability trees based on arc marginal probabilities for each unlabelled sentence on the target side, and treat all those trees as a weak supervision signal for training. their method outperforms direct transfer as well as a variety of recent cross-lingual transfer methods that are not source-free. that said, the effectiveness of their method on (a) semantic (sequence labelling) tasks and (b) in a domain adaptation setting is unexplored, which is what we aim to address in this work. ## system description we first describe our sequence tagging model (section 3.1), before we present parsimonious transfer for sequence tagging (ptst) in section 3.2. ## experimental setup we use the pre-trained source model and data provided for semeval-2021 task 10. we only use the model and data for the time expression recognition task (timex hereinafter) as we only participate in that task. we use the practice data as the development set to tune the hyperparameters of our model with random search. 5 we set the threshold σ = 0.95 following the setup of @xcite . we do not use any data sets other than those provided by the task organisers for timex. we train ptst on the unlabelled test data for 5 epochs. as described in section 3.1, we initialise the neural network for the emission scoring with the source roberta model provided by the task organisers. we enforce the bio constraints by initialising the transition matrix φ with -∞ for entries corresponding to illegal transitions, and zero otherwise. 6 we use the linear crf implementation provided in the torch-struct library @xcite . to avoid out-of-memory error, we discard sentences longer than 30 tokens. additionally, we find that it is useful to freeze the embedding and the first few layers of the roberta encoder. thus, in our main result, we freeze the embedding layer and the first 6 (out of 12) layers of the encoder. an analysis is provided in section 5. ## results and discussion ## conclusions we present ptst, our submission to the time expression recognition task of semeval-2021 task 10. we describe our sequence tagging model as a crf over chain structures, parameterised by a neural network. our domain adaptation approach leverages the output distribution of the source model to build a chart of high probability tag sequences for every sentence in the unlabelled target domain data. ptst ranks 7th in terms of f 1 score in the official leaderboard, but achieves the highest precision out of 14 submissions. we provide analyses on the importance of temperature scaling to mitigate model overconfidence and the patterns of errors.