Intermediate Domain Finetuning for Weakly Supervised Domain-adaptive Clinical NER (2023)

Abstract: Accurate human-annotated data for real-worlduse cases can be scarce and expensive to obtain. In the clinical domain, obtaining such data is evenmore difficult due to privacy concerns which notonly restrict open access to quality data but also require that the annotation be done by domain experts. In this paper, we propose a novel framework - InterDAPT - that leverages Intermediate Domain Finetuning to allow language models to adapt to narrow domains with small, noisy datasets. By making use of peripherally-related, unlabeled datasets,this framework circumvents domain-specific datascarcity issues. Our results show that this weaklysupervised framework provides performance improvements in downstream clinical named entityrecognition tasks.

## introduction domain adaptation is a vast topic that is central to the field of biomedical nlp largely due to the limited number of high-quality, task and domainspecific biomedical and clinical corpora available. in such situations, it would be advantageous if we could leverage any available poorly-annotated, unlabeled datasets to boost model performance as opposed to entirely relying on one small, noisy dataset. in this work, we propose an ensemble training strategy combining multiple machine learning training methods to mitigate the issues impacting downstream named entity recognition (ner) performance caused by incorrect and missing labels in highly domain-specific datasets. biomedical applications of nlp often require high-quality annotation of task-specific datasets which can be very expensive. in addition, such data may be protected by privacy concerns which would make human annotation harder to perform. another issue that is commonly observed with hu-man annotations is that there may be variability in labels from different annotators. for example, ner labels in the beginning-inside-outside (bio) format only allow annotators to assign one entity label to each span. however, depending on the sentence structure and the definitions of each entity category, the same span might be considered to be a candidate for two or more entity labels. in such situations, different annotators might assign labels differently regardless of the standardized guidelines provided to all annotators. such issues may be difficult to identify and expensive to rectify. due to these issues, leveraging weak supervision techniques which use unlabeled data to buttress the impact of noisy labels would be of interest. our proposed training framework uses noisy, machine-labeled data to help create a transferable middle layer model which, when combined with dapt (domain adaptive pre-training) @xcite would allow a pre-trained model to better adapt to noisy clinical data. our generalized training framework which we refer to as the intermediate finetuning for weakly supervised sub-domain adaptation (interdapt) framework is described in figure 1 . in the first phase, we perform continual domain adaptive pre-training (dapt) @xcite on bioclinicalbert @xcite using generalized patient data. in parallel, we label orthopedics-related operative notes using radiology ner models provided by john snow @xcite . this machine-annotated dataset is used to train the intermediate model which is then used to finetune downstream ner tasks. we apply this approach on two different datasets from orthopedics-related clinical notes -spine and hip. our preliminary results indicate that our proposed framework achieves similar or better performance for the same dataset with noisy labels as can be achieved with clean labels. ## disambiguating domain and task as there is little domain variability in most of the task-specific corpora that are commonly used for benchmarking in nlp research, many works tend to conflate tasks with domains. in this paper, we seek to disambiguate these terms as the goal of interdapt is to improve the performance of entity recognition models in the absence of highquality data in some areas by leveraging relatively large datasets in other areas with the assumption that there is some overlap of implicit information among these datasets by virtue of their domains being related. hereafter, we refer to nlp tasks such as named entity recognition as tasks. we break down categorizations of domains further into metadomains, tangential domains, and sub-domains. we define the following terms: as the main motivation for this work is to improve model performance in highly specific clinical domains with inconsistent or incorrect labeling, we evaluate interdapt's performance on the named entity recognition (ner) task. we keep the task stable for the purposes of our experiments in order to assess the impact of the different types of domain-specific datasets while ruling out possible performance variations brought on by variations in the type of task. figure 2: generalized intermediate finetuning for weakly supervised sub-domain adaptation (interdapt) framework 4 method 4.1 intermediate finetuning for weakly supervised sub-domain adaptation (interdapt) the intermediate finetuning for weakly supervised sub-domain adaptation (interdapt) framework is a training paradigm that allows pre-trained language models to adapt to small, weakly labeled domain-specific datasets without requiring further human annotation to correct or add labels. inter-dapt is comprised of three training components -dapt for clinical domain adaptation on metadomain dataset, intermediate domain finetuning using weak supervision to allow the model to learn task-specific information on tangential-domain, and finally downstream finetuning to a variety of target sub-domains. ## experimental setup in order to evaluate the efficacy of interdapt we run our preliminary experiments on datasets of clinical notes described below. as there is significant variability in the baseline performance of target tasks due to variation in the nature of the task and factors such as the number and types of entities, we compare the performance of interdapt against direct finetuning from bioclinicalbert to the target sub-domain where the test set is cleaned and human-annotated. we report results on datasets for which data cleaned and vetted by domain experts is available at this time. ## results results on the spine dataset show that the inter-dapt framework is effective at improving domain adaptation especially as we approach an intermediate task size of 5,000,000. interestingly, noisy labels seem to provide slightly better baseline results than baseline results with a clean dataset. notably, we observe a strong initial improvement in results for noisy labels with interdapt and further improvements are correlated with increasing intermediate task size. our preliminary experiments with the hip dataset (noisy) notes provide an f1 score improvement of 10 points from 70.91 on the baseline bch-bert model to 80.23 on a model configuration of bch-bert + interner-bert-1m. this increase in performance could indicate that intermediate domain finetuning is much better at distinguishing between ambiguous entities. during our error analysis, we found that the baseline model without interdapt had relatively more issues with distinguishing between entities that were very similar presentations. however, further experiments with cleaned datasets are required to confirm this hypothesis. ## future work while our experiments show that interdapt is an effective framework that helps ameliorate issues stemming from noisy labeling, more exper in continuing work, we aim to expand to more sub-domains as we continue to obtain high-quality human-annotated data to compare the relative performance of interdapt against.