C linical T 5: A Generative Language Model for Clinical Text (2022)

Abstract: In the past few years, large pre-trained language models (PLMs) have been widely adopted in different areas and have made fundamental improvements over a variety of downstream tasks in natural language processing (NLP). Meanwhile, domain-specific variants of PLMs are being proposed to address the needs of domains that demonstrate a specific pattern of writing and vocabulary, e.g., BioBERT for the biomedical domain and ClinicalBERT for the clinical domain. Recently, generative language models like BART and T5 are gaining popularity with their competitive performance on text generation as well as on tasks cast as generative problems. However, in the clinical domain, such domain-specific generative variants are still underexplored. To address this need, our work introduces a T5-based text-to-text transformer model pre-trained on clinical text, i.e., ClinicalT5. We evaluate the proposed model both intrinsically and extrinsically over a diverse set of tasks across multiple datasets, and show that ClinicalT5 dramatically outperforms T5 in the domain-specific tasks and compares favorably with its close baselines.

## introduction in the past few years, large pre-trained language models (plms), such as bert @xcite , roberta @xcite , @xcite , bart @xcite , t5 @xcite , etc., have achieved great success over a variety of downstream tasks in natural language processing (nlp). these plms mainly depend on self-supervised pre-training on large amounts of general-domain textual data, e.g., wikipedia, news articles, web crawl corpus, etc., and are widely adopted in downstream applications. despite the superior performance of these plms on general-domain text, their performance over domain-specific text is relatively poor (ma et al., 1 we will release the models upon decision of the paper. 2019). to bridge this gap, researchers propose to build domain-specific plms through fine-tuning or pre-training from scratch over domain corpora. for example, in the biomedical and clinical domains, various domain-specific plms have been explored and released, including biobert @xcite , scibert @xcite , bluebert @xcite , clinicalbert @xcite , bioclinicalbert foot_0 @xcite , umls-bert @xcite , diseasebert @xcite , scifive @xcite , and biobart @xcite . domain-specific language models have been extensively explored in different kinds of nlp-related downstream applications, ranging from entity linking @xcite to document classification @xcite . generally, a typical and popular usage of the aforementioned plms is to leverage them to encode domain text, the learned representations of which are then fed into some task-specific structures for label prediction. taking a complicated real-world task as an example, @xcite predicts patients' risk of readmission within 30 days after discharge using clinical notes in the electronic health records (ehrs). essentially, they encode discharge summaries of patients with clinicalbert, and put the learned embeddings of the [cls] token to a linear layer on top for prediction, leading to better performance than traditional models. moreover, @xcite constructs a document-level multi-view graph out of each clinical note and predicts patients' 30-day readmission risk with a graph-based model, and they use bioclinicalbert @xcite as the encoder within the graph model. recently, generative language models, e.g., bart @xcite and t5 @xcite , have attracted attention since they are naturally effective for natural language generation tasks, such as document summarization @xcite , question answering @xcite , data augmentation @xcite , etc. meanwhile, a novel paradigm of leveraging generative language models has gained popularity, where researchers cast non-generation tasks as generative problems, e.g., to directly generate textual labels to incorporate their semantics, and report promising results @xcite . however, such approaches are still underexplored in certain domains due to lack of domain-specific generative language models, i.e., most of the aforementioned domain-specific plms are notably domain-adapted bert-style models. in the biomedical domain, two generative language models scifive @xcite and biobart @xcite have been released, but in the clinical domain, the situation is worse and no such generative models exist to our knowledge. though the two domains are relatively close, clinical text poses unique challenges compared to general and non-clinical biomedical text due to its specific linguistic characteristics @xcite . previous studies list some of the linguistic features of clinical text, e.g., heavy use of professional technical terminology, abbreviations and acronyms, passive verbs, omission of subjects and verbs, etc., and these features make clinical text divergent from standard language @xcite . aiming to fulfill this gap, we adapt t5 @xcite to the clinical domain by training a domain-specific variant using clinical text, i.e., clinicalt5. we demonstrate the capabilities of the model by conducting both intrinsic and extrinsic evaluations. for intrinsic evaluation, we aim to evaluate its capability to capture the similarity and relatedness of the unified medical language system (umls) concept pairs, where we measure the correlation coefficient between the similarity scores of the encoded representations for the concept pairs and those judged by human experts. for extrinsic evaluation, we evaluate the proposed model along with baselines over a diverse set of benchmark datasets, ranging from document classification (dc), named entity recognition (ner), to natural language inference (nli). furthermore, we also evaluate on three more complicated real-world tasks of clinical importance, i.e., patients' 30-day readmission risk, 30-day and 1-year mortality risk. we show that clinicalt5 dramatically outperforms t5 and compares favorably with its close baselines across all of these tasks. ## experiments in this section, we evaluate clinicalt5 both intrinsically and extrinsically, along with the following generative baselines (for both general and domainspecific texts): bart @xcite , bio-bart @xcite , t5 @xcite , scifive @xcite , to demonstrate the capabilities of clinicalt5 across different applications. ## conclusion in this study, we explore and propose clinicalt5, a t5-based text-to-text transformer model for clinical text. we evaluate the proposed model both intrinsically and extrinsically, and the results show that clinicalt5 compares favorably with its close baselines. we also test upon more complicated patient outcomes prediction tasks, the results of which indicates its potential for these real-world downstream tasks in the clinical domain.