Leveraging Task Transferability to Meta-learning for Clinical Section Classification with Limited Data (2022)

Abstract: Identifying sections is one of the critical components of understanding medical information from unstructured clinical notes and developing assistive technologies for clinical note-writing tasks. Most state-of-the-art text classification systems require thousands of in-domain text data to achieve high performance. However, collecting in-domain and recent clinical note data with section labels is challenging given the high level of privacy and sensitivity. The present paper proposes an algorithmic way to improve the task transferability of meta-learning-based text classification in order to address the issue of low-resource target data. Specifically, we explore how to make the best use of the source dataset and propose a unique task transferability measure named Normalized Negative Conditional Entropy (NNCE). Leveraging the NNCE, we develop strategies for selecting clinical categories and sections from source task data to boost cross-domain meta-learning accuracy. Experimental results show that our task selection strategies improve section classification accuracy significantly compared to meta-learning algorithms.

## introduction an important part of electronic health records (ehrs) is the digitized clinical notes that contain the medical and treatment histories of patients. the section of clinical notes can be defined as a text segment that clusters consecutive sentences with relevant content of one dimension of a patient's health encounter @xcite . clinical note sections, labeled with either headings or subheadings, make the notes well organized and offer improved clinical information extraction @xcite . however, many clinical notes contain narratives that are in an unstructured free-text format, (e.g., history of present illnesses described in paragraph form), which makes it challenging to retrieve and utilize this information. in the united states, physicians generally spend an excessive amount of time interfacing with ehrs and computerized physician order entry (cpoe) workflows in their aftercare work, resulting in burnout, low job satisfaction, and system-wise inefficiencies @xcite . an automated section classifier can play a key role in mitigating this problem. in some cases, section classification serves as an end task of automatic report segmentation. for example, according to an internal survey we conducted with amazon care providers, we found evidence that classifying sentences related to the history of present illness from medical encounters can greatly assist providers with their documentation. for computer-assisted report generation, understanding clinical notes from an unstructured format is an important data pre-processing @xcite . there are some challenges for clinical note section classification in practice. first, it is difficult to collect and access a large amount of in-domain data. second, section types and medical contents within a section substantially vary depending on care providers, which makes it hard to utilize opensource datasets. even though some sections exist in multiple different sources, their contents vary across clinical categories. for example, the diagnosis section for nutrition specialty and rehabilitation service specialty vary in types of content. recently developed neural network language technologies capture rich contextual information in sentences. among them, bidirectional encoder representations from transformers (bert) achieved significant improvements in multiple natural language processing (nlp) tasks, establishing strong baselines in low-resource scenarios @xcite . however, there remains room for performance improvement because bert uses source data -data outside of in-domain or targetdomain data -in an unsupervised training fashion only. another approach for low-resource indomain nlp tasks is multi-task learning (mtl). the mtl adopts shared text encoding layers across all tasks while the top layers are task-specific for each dataset @xcite . the target task with limited data benefits from the knowledge learned from source tasks. instead of mtl, which minimizes the loss of the source tasks, @xcite proposed a model-agnostic meta-learning algorithm that finds optimal model parameters for better adaptation capability to new tasks. in classification tasks, @xcite proposed reptile, an optimization-based meta-learning algorithm for section classification, and achieved comparable accuracy on well-established benchmarks on low resourced image datasets. in the present paper, we adopted these methods as strong baselines in our experiments and computed the relative performance improvement of our method. task transferability denotes how easy it is to transfer the representation learned from one task to another task @xcite . it helps discover the relationship between two types of tasks and provides supporting evidence for developing transfer learning strategies. task transferability becomes more useful in realistic situations where the assumption of the metalearning, which is that data of the target task can be drawn from the distribution of the source tasks, does not hold. one common example is that there are 'outlier tasks' in the training (source) tasks, which are dissimilar from the testing (target) ones @xcite . for this problem, good selection of relevant source tasks can benefit knowledge transfer to unseen tasks @xcite @xcite . in clinical section classification, we suppose how close a source task is toward the target task is determined by its specialty and the section types included. however, few studies of task transferability estimation have discussed the function of each label. thus we propose an information-theoretic metric for task transferability, namely normalized negative conditional entropy (nnce). the nnce score is calculated by the classifier of a source task and target data samples without training on the target task, thus saving expensive computation for model optimization. we hypothesize that this score correlates with how well the source data labels (sec-tions) distinguish the target labels. leveraging the nnce, we explore strategies of source task selection to improve the performance of meta-learning. the goal is to make the best use of available data from various clinical specialties for any target tasks. specifically, we explore two strategies: 1) category selection -we select a subset of clinical categories that are relevant to the target task; 2) section selection -for a clinical category, we filter out the samples of certain section types which are not relevant to the target task and merge similar sections by assigning the same label. the category selection is informed directly by the best nnce scores. for section selection, however, there are too many combinations, and it is timeconsuming to train models for every possible task and find optimal ones. to handle that, we apply a backward selection method for heuristic search. the experiment results show that our task selection strategies improve the meta-transfer learning of section classification in low-resource scenarios. our work has the following contributions: ## related work in this section, we briefly discuss several areas in machine learning that are related to our work. ## dataset we conduct experiments on the medical information mart for intensive care iii (mimic-iii) database @xcite , a large open-access dataset of de-identified patient records. we collected data from 9 different clinical categories of mimic-iii and randomly picked 200 clinical notes for each. there are nearly 1,000 section labels of these categories, and most of them contain very few sentence instances. to handle the sparsity, we only keep the section types of each category satisfying the following conditions: ## conclusion and future work in this paper, we explored the clinical section classification with limited in-domain data. we applied a meta-learning algorithm utilizing multiple out-ofdomain clinical datasets, improving the classification accuracy and adaptation speed. we proposed a normalized negative conditional entropy measure to estimate the task transferability and leverage it to select the clinical categories and sections related to the target task that best improves knowledge transfer. in addition, we examined a backward selection method to reduce the computational complexity of section selection. our study suggests that both category selection and section selection outperform the baseline meta-learning approach, and combining two strategies results in better performance than adopting each of them independently. future work will look to develop a joint optimization of category selection and section selection. we also plan to apply our approach to other styles of text data. for example, section classification on spoken utterances of doctor-patient conversations is an exciting extension of the present work, which we plan to explore @xcite . finally, we will continue to apply the proposed method to other text processing applications, e.g., medical information retrieval @xcite .