Parameter Efficient Transfer Learning for Suicide Attempt and Ideation Detection (2022)

Abstract: Pre-trained language models (LMs) have been deployed as the state-of-the-art natural language processing (NLP) approaches for multiple clinical applications. Model generalisability is important in clinical domain due to the low available resources. In this study, we evaluated transfer learning techniques for an important clinical application: detecting suicide attempt (SA) and suicide ideation (SI) in electronic health records (EHRs). Using the annotation guideline provided by the authors of ScAN, we annotated two EHR datasets from different hospitals. We then fine-tuned ScANER, a publicly available SA and SI detection model, to evaluate five different parameter efficient transfer learning techniques, such as adapter-based learning and soft-prompt tuning, on the two datasets. Without any fine-tuning, ScANER achieve macro F1-scores of 0.85 and 0.87 for SA and SI evidence detection across the two datasets. We observed that by fine-tuning less than ~2% of ScANER’s parameters, we were able to further improve the macro F1-score for SA-SI evidence detection by 3% and 5% for the two EHR datasets. Our results show that parameter-efficient transfer learning methods can help improve the performance of publicly available clinical models on new hospital datasets with few annotations.

## introduction in the past decade, 90% of the us hospitals have adopted a certified electronic health record (ehr) system @xcite . this has led to an enormous availability of ehrs with rich information about patients' health @xcite . with the advancement of natural language processing (nlp), there has been a significant improvement in the development of clinical models and systems to extract clinically relevant information from the ehrs for further downstream tasks @xcite . recent years have seen clinical datasets being publicly released for different nlp tasks such as named entity recognition, relation extraction, text de-identification and disease classification @xcite @xcite . medical information mart for intensive care -iii (mimic) @xcite has enabled a large and continually growing set of deidentified ehr notes from an intensive care unit for developing other publicly available datasets such as emrqa @xcite , @xcite and adverse drug reaction (adr) extraction @xcite . this increase in availability of the clinically annotated datasets has led to the improvement in performance of different nlp models. while this improvement is great, a key question is whether these improvements generalize to new datasets of the same task or not. this question is quite difficult to answer because it requires annotating multiple datasets or new datasets with the same guidelines when it is already difficult to annotate a single dataset @xcite . in this study, we evaluate different parameter efficient transfer learning techniques on the task of an important clinical application, namely suicide attempt (sa) and suicide ideation (si) detection from ehrs. recently, a sa-si detection dataset (scan) @xcite was publicly released in an effort to extract suicidal information from patients' ehrs. scan was released along with the annotation guidelines used by the experts and the baseline model to detect the suicidal evidences from ehr notes (scaner). we followed the annotation guideline to annotate two new datasets: ehr notes from school of medicine at university of pittsburgh (hereby referred as scan_up) and ehr notes from the us veterans health administration (scan_va). we used scan and scaner as our base dataset and model for creating the two new datasets and evaluating different transfer learning techniques. in order to evaluate the transfer learning performance of scaner, we kept the size of scan_up and scan_va relatively smaller than scan for further fine-tuning. these fine-tuned models could eventually help clinical professionals in making patient-aware clinical judgements for further treatments. pre-trained language models have significantly grown in size since the inception of bert @xcite model. bert was introduced with 110 million parameters but recent lms such as generative pre-trained transformer (gpt-3) @xcite and open pretrained transformer (opt) @xcite have ∼ 175 billion parameters. given their unprecedented performance gains over different downstream tasks, the researchers in the clinical community have also adopted these models. but all hospitals or medical organizations do not have the resources to adapt these billion parameter models in their ecosystem. hence it is important to evaluate parameterefficient transfer learning techniques that keep most of the model parameters frozen during fine-tuning on a newer dataset for the same task. we decided to try five different techniques: fine-tuning the classification layer, bitfit @xcite , adding adapter modules @xcite , softprompt fine-tuning @xcite and tuning the last four layers @xcite . most of these techniques require fine-tuning of less than 2% of scaner's parameters except tuning the last four layers which requires tuning of ∼ 23% parameters. in this study, we found that scaner @xmath0. we were able to further improve the sa-si evidence detection by 3% for scan_up and 5% for scan_va by fine-tuning less than ∼ 2% of scaner's parameters. both scan_up and scan_va contain less than 8% annotations when compared to the original scan dataset. this shows that parameter-efficient transfer learning methods can help in improving the performance of publicly available clinical models on new hospital datasets with few annotations. ## dataset in order to evaluate different transfer learning techniques, we focused heavily on choosing a task that has a publicly available dataset along with the annotation guidelines and the baseline model. the annotation guidelines are very important because they would help us in keeping the annotation decisions across different datasets uniform. hence, we chose the task of detecting suicide attempt and ideations events in ehrs because of the availability of scan dataset @xcite . the annotations guidelines for creating scan are publicly available along with their proposed baseline model (scaner). ## results and discussion for evidence retrieval, even without fine-tuning the original scaner model is able to achieve a macro-f1 score of 0.87 and 0.85 for scan_up and scan_va datasets respectively, refer ## conclusion in this paper, we evaluated different parameter efficient transfer learning techniques on the task of suicide attempt (sa) and suicide ideation (si) events detection in the ehr notes. according to the publicly available annotation guidelines of @xcite ) dataset, we created two new datasets: scan_up and scan_va. we tested the baseline model scaner on these two datasets and achieved macro f1-scores of 0.87 and 0.85 for sa-si evidence detection. we were able to further improve the performance of scaner by at least 3% after fine-tuning only 2% of scaner's parameters. we show that parameter efficient transfer learning can help improve the performance of publicly available clinical models on new hospital datasets with few annotations. we would recommend the use of adapter modules for further transfer learning of clinical models as they consistently performed well for sa-si detection while tuning only 2% of the parameters. if the computational resources are still a constraint, we would recommend using softprompt tuning as they only tune 0.02% of the parameters while achieving a performance quite close to adapter module tuning.