M e DAL : Medical Abbreviation Disambiguation Dataset for Natural Language Understanding Pretraining (2020)

Abstract: One of the biggest challenges that prohibit the use of many current NLP methods in clinical settings is the availability of public datasets. In this work, we present MeDAL, a large medical text dataset curated for abbreviation disambiguation, designed for natural language understanding pre-training in the medical domain. We pre-trained several models of common architectures on this dataset and empirically showed that such pre-training leads to improved performance and convergence speed when fine-tuning on downstream medical tasks.

## introduction recent work in mining medical texts focus on building deep learning models for different medical tasks, such as mortality prediction @xcite and diagnosis prediction @xcite . however, because of the private nature of medical records, there are few large-scale, publicly available medical text datasets that are suitable for pretraining models, and real-world, private datasets are often small-scale and imbalanced. as a result, one of the biggest challenge in building deep learningbased nlp systems for biomedical corpora is the availability of public datasets @xcite . to tackle this problem, we present medical dataset for abbreviation disambiguation for natural language understanding (medal) 1 , a large dataset of medical texts curated for the task of medical abbreviation disambiguation, which can be used for pre-training natural language understanding models. figure 1 shows an example of sample in the dataset, where the true meaning of the abbreviation 'dhf' is inferred from its context, and figure 2 shows the pretraining framework. although this dataset can be used for building abbreviationexpansion systems, its main purpose is to enable 1 https://github.com/brucewen120/medal second, we believe that understanding natural language in a knowledge-rich domain such as medicine requires understanding of domain knowledge at some level, similar to how humans can understand medical text only after receiving medical training. the abbreviation disambiguation task enables models to use domain knowledge to understand the global and local context, as well as the possible meanings of the abbreviation in the medical domain. medical abbreviation disambiguation has long been studied @xcite @xcite @xcite and our work builds upon many of them. in particular, our data generation process is inspired by the reverse substitution technique @xcite . our work differs from them in mainly two aspects. first, instead of trying to improve performance on abbreviation disambiguation itself, we propose to use it as a pre-training task for transfer learning on other clinical tasks. second, existing datasets for medical abbreviation disambiguation, for instance casi @xcite , are small compared to datasets used for general language model pre-training, and as noted by @xcite some are erroneous. thus, we chose to construct a new dataset large enough for effective pre-training. our main contributions are: a) we present a large dataset for pre-training on the task of medical abbreviation disambiguation. b) we provide empirical evidence of the benefit of abbreviation pre-training for a wide range of deep learning architectures. ## models the models are first pre-trained on the medal dataset, then pre-trained weights are used to initialize models for training on the downstream tasks. we compared this training strategy with training respective models from scratch to validate the benefit of pre-training. lstm bilstm is used as a baseline model. specifically, the bilstm consists of three layers with hidden size of 512. pre-trained fasttext model is used for word embeddings @xcite . ## results models' performance on the pre-training task, abbreviation disambiguation, is shown in figure 5 . as the goal is not to optimize performance on this the benefit of pre-training is more significant on diagnosis prediction, shown in figure 6 . both lstm and lstm + self attention perform considerably better if they pre-trained. in fact, the two models' performance increase by more than 70% relatively. while for electra the gain is not as significant, pre-training leads to faster convergence during fine-tuning. on the two downstream tasks, experiment results show that pre-training improves electra's performance even when the model is already fully pre-trained on non-medical texts and is among the state-of-the-art, and bring the other models' performance close to electra's. this shows that pre-training on the medal dataset can generally improves models capabilities of understanding language in medical domain. the complete results can be found in appendix c. ## conclusion and discussion in this work, we present medal, a large dataset on abbreviation disambiguation, designed for pretraining natural language understanding models in the medical domain. we pre-trained a variety of models using common architectures and empirically showed that such pre-training leads to improvement in performance as well as faster convergence when fine-tuning on two downstream clinical tasks.