Learning from Unlabelled Data for Clinical Semantic Textual Similarity (2020)

Abstract: Domain pretraining followed by task fine-tuning has become the standard paradigm for NLP tasks, but requires in-domain labelled data for task fine-tuning. To overcome this, we propose to utilise domain unlabelled data by assigning pseudo labels from a general model. We evaluate the approach on two clinical STS datasets, and achieve r= 0.80 on N2C2-STS. Further investigation reveals that if the data distribution of unlabelled sentence pairs is closer to the test data, we can obtain better performance. By leveraging a large general-purpose STS dataset and small-scale in-domain training data, we obtain further improvements to r= 0.90, a new SOTA.

## introduction semantic textual similarity (sts) measures the degree of semantic equivalence between two text snippets, based on a graded numerical value, with applications including question answering @xcite , duplicate detection @xcite , and entity linking @xcite . modern pretrained language models have achieved impressive results for general sts @xcite . however in low-resource domains without in-domain labelled data, results are generally lower @xcite . in the clinical domain in particular, annotation requires medical experts @xcite , meaning that labelled datasets are generally small, hampering clinical sts. we address the question of how to apply pretrained language models to such domain-specific tasks where there is little or no labelled data, focusing specifically on the task of clinical sts. employing a general sts model generally yields poor results over technical domains due to covariate shift. to bridge this gap, a standard approach is to pretrain the lm on in-domain text, such as clin-icalbert @xcite using mimic-iii @xcite . however, existing research has tended to estimate effectiveness under the fine-tuning setting, rather than via inference tasks @xcite . in this paper, we first evaluate domain pretraining approaches for clinical sts, with no labelled data. based on the assumption that general sts models trained on large-scale sts datasets will perform reasonably well on clinical sentence pairs (section 4), we then experiment with learning from the pseudo-labelled data (section 5). experimental results show both domain pretraining and pseudo-labelled data fine-tuning improve clinical sts, and the combination of the two achieves the best performance of @xmath0.80 on n2c2-sts (section 6.3). further analysis shows that the score distribution and volume of pseudolabelled pairs influence the performance of finetuning. we also find that training for more iterations leads to minor improvements. the paper makes three major contributions: (1) we propose a simple pseudo-training method, and show it to perform well on clinical sts; (2) we evaluate several existing approaches to clinical sts in a zero-shot setting, and benchmark against our method; and (3) we achieve state-of-the-art results of @xmath1.90 for n2c2-sts. ## related work the general approach to domain-specific task modelling is: (1) pretrain a language model (lm) on a large volume of open-domain text @xcite ; and (2) fine-tune on domainspecific text and task-specific labelled data @xcite . for this approach, however, domain-specific labelled data is required, an assumption that we seek to relax. for sts, in the absence of labelled data, the simplest approach is to calculate the cosine similarity between the cls-vectors of two sentences or averaged last-layer embeddings, but this tends to perform poorly, even worse than averaged glove @xcite embeddings. sbert @xcite proposed to use a siamese structure based on bert to learn sentence representations, where they fine-tuned the model over general nli data, and continued to fine-tune on general sts data (sts-b) @xcite . in this work, we experiment with this approach specifically in the clinical context. ## datasets and tasks we select two available clinical sts benchmark datasets for evaluation: medsts @xcite and n2c2-sts @xcite . the latter annotated 412 instances as new test bed, and updated train partition by labelling extra 574 instances and merging the former train and test cases (see for example, s1: discussed goals, risks, alternatives, advanced directives, and the necessity of other members of the surgical team participating in the procedure with the patient. s2: discussed risks, goals, alternatives, advance directives, and the necessity of other members of the healthcare team participating in the procedure with the patient and his mother. label: 4, as the two sentences are mostly equivalent and differ only in unimportant details (in bold). pearson's correlation (r) and spearman's correlation (ρ) between the predicted and gold standard scores are used as evaluation metrics. ## observations in modern nlp, large amounts of high-quality training data are a key element in building successful systems @xcite . this is also the case with sts, where additional training data has been shown to improve accuracy @xcite . however, domain shifts inevitably lead to performance drops @xcite . therefore, we ask: rq1 can large-scale generaldomain labelled sts data be transferred to train effect of larger general sts corpus. we source general-domain labelled data from: (1) semeval-sts shared tasks 2012-2017 @xcite @xcite @xcite ; and sick-r @xcite . this results in a total of 28,518 labelled sentence pairs, which we refer to as "sts-g". we adapt a bert encoder connected to a linear regression layer to fine-tune a general-domain sts model using sts-g, where the cls-vector is used to represent the sentence pair (cls-bert). we compare this with a model trained only on sts-b. we evaluate both models on sts-b dev (same setup as section 6.1). for clinical sts, we employ a hierarchical convolution (hconv) model based on bert (updating parameters of the last four layers), where the model is first fine-tuned with sts-b, then n2c2-sts is augmented by back-translation @xcite . the model architecture and hyperparameter settings are the same as the original paper, such that we merely replace sts-b with sts-g, and observe that more training data improves clincial sts. as shown in discussion. though general-domain data lacks clinical information, the model clearly benefits from the extra out-of-domain training data (answering rq1). this inspires us to rethink the clinical sts task as a combination of domain-specific text understanding and domain-invariant task learning, leading to the question: can the two aspects be learned separately? that is, can task learning take place via large volumes of general-domain labelled data, and domain-specific characteristics be learned from silver-standard labelled domain data, such as low-quality clinical sentence pairs labelled by a general sts model? ## method next, we investigate the use of pseudo-labelled clinical data based on the general sts model. ## experiments we first evaluate existing approaches for clinical sts in the zero-shot setting, and compare with our method. then we analyse the impact of the volume of sampled instances and data distribution on the fine-tuning quality. we experiment with the number of iterations in section 6.5. ## conclusion in this paper, we have proposed a simple method of pseudo-labelling in-domain data and iterative training, to improve clinical sts. evaluation over two clinical sts datasets demonstrates the effectiveness of the approach, and domain pretraining is shown to achieve further improvements. further investigation indicated that keeping the distribution of pseudo-labelled instances close to that of the in-domain data improves performance. we also observed modest improvements through more iterations of iterative training. our work provides an alternative approach to employing domain-specific unlabelled data to support clinical sts. as future work, we plan to explore the application of our method to other model structures such as sbert.