Evaluating the Utility of Model Configurations and Data Augmentation on Clinical Semantic Textual Similarity (2020)

Abstract: In this paper, we apply pre-trained language models to the Semantic Textual Similarity (STS) task, with a specific focus on the clinical domain. In low-resource setting of clinical STS, these large models tend to be impractical and prone to overfitting. Building on BERT, we study the impact of a number of model design choices, namely different fine-tuning and pooling strategies. We observe that the impact of domain-specific fine-tuning on clinical STS is much less than that in the general domain, likely due to the concept richness of the domain. Based on this, we propose two data augmentation techniques. Experimental results on N2C2-STS 1 demonstrate substantial improvements, validating the utility of the proposed methods.

## introduction semantic textual similarity (sts) is a language understanding task, involving assessing the degree of semantic equivalence between two pieces of text based on a graded numerical score @xcite . it has application in tasks such as information retrieval @xcite , question answering @xcite , and summarization @xcite . in this paper, we focus on sts in the clinical domain, in the context of a recent task within the framework of n2c2 (the national nlp clinical challenges) 1 , which makes use of the extended medsts data set @xcite , referring to n2c2-sts, with limited annotated sentences pairs (1.6k) that are rich in domain terms. neural sts models typically consist of encoders to generate text representations, and a regression layer to measure the similarity score (he et al., 2015; mueller and thyagarajan, 2016; he and lin, 1 https://portal.dbmi.hms.harvard.edu/ projects/n2c2-2019-t1/ 2016; @xcite . these architectures require a large amount of training data, an unrealistic requirement in low resource settings. recently, pre-trained language models (lms) such as gpt-2 @xcite and bert @xcite have been shown to benefit from pre-training over large corpora followed by fine tuning over specific tasks. however, for small-scale datasets, only limited finetuning can be done. for example, gpt-2 achieved strong results across four large natural language inference (nli) datasets, but was less successful over the small-scale rte corpus @xcite , performing below a multi-task bilstm model. similarly, while the large-scale pre-training of bert has led to impressive improvements on a range of tasks, only very modest improvements have been achieved on sts tasks such as sts-b @xcite and mrpc @xcite (with 5.7k and 3.6k training instances, resp.). compared to general-domain sts benchmarks, labeled clinical sts data is more scarce, which tends to cause overfitting during fine-tuning. moreover, further model scaling is a challenge due to gpu/tpu memory limitations and longer training time @xcite . this motivates us to search for model configurations which strike a balance between model flexibility and overfitting. in this paper, we study the impact of a number of model design choices. first, following @xcite , we study the impact of various pooling methods on sts, and find that convolution filters coupled with max and mean pooling outperform a number of alternative approaches. this can largely be attributed to their improved model expressiveness and ability to capture local interactions @xcite . next, we consider different parameter fine-tuning strategies, with varying degrees of flexibility, ranging from keeping all parameters frozen during training to allowing all pa-rameters to be updated. this allows us to identify the optimal model flexibility without over-tuning, thereby further improving model performance. finally, inspired by recent studies, including sentence ordering prediction @xcite and data-augmented question answering @xcite , we focus on data augmentation methods to expand the modest amount of training data. we first consider segment reordering (sr), in permuting segments that are delimited by commas or semicolons. our second method increases linguistic diversity with back translation (bt). extensive experiments on n2c2-sts reveal the effectiveness of data augmentation on clinical sts, particularly when combined with the best parameter fine-tuning and pooling strategies identified in section 3, achieving an absolute gain in performance. ## sts model configurations in this section, we study the impact of a number of model design choices on bert for sts, using a 12-layer base model initialized with pretrained weights. ## data augmentation the accuracy of an sts model unsurprisingly depends on the amount of labeled data. this is reflected in figure 2 , where models trained with more data outperform those with fewer training instances. in this section, we propose two data augmentation methods, namely segment reordering (sr) and back translation (bt), to address the data sparsity issue in clinical sts. ## conclusions in this paper, we have presented an empirical study of the impact of a number of model design choices on a bert-based approach to clinical sts. we have demonstrated that the proposed hierarchical convolution mechanism outperforms a number of alternative conventional pooling methods. also, we have investigated parameter fine-tuning strategies with varying degrees of flexibility, and identified the optimal number of trainable transformer blocks, thereby preventing over-tuning. lastly, we have verified the utility of two data augmentation methods on clinical sts. it may be interesting to see the impact of leveraging target languages other than chinese in bt, which we leave for future work.