Transfer Learning for Low-Resource Clinical Named Entity Recognition (2023)

Abstract: We propose a transfer learning method that adapts a high-resource English clinical NER model to low-resource languages and domains using only small amounts of in-domain annotated data. Our approach involves translating in-domain datasets to English, fine-tuning the English model on the translated data, and then transferring it to the target language/domain. Experiments on Spanish, French, and conversational clinical text datasets show accuracy gains over models trained on target data alone. Our method achieves state-of-the-art performance and can enable clinical NLP in more languages and modalities with limited resources.

## introduction clinical text such as physician notes, discharge summaries, and patient encounter transcripts contains a wealth of information critical for healthcare. however, much of the data within these texts remain locked away due to challenges in automatically processing clinical narratives. the field of clinical natural language processing (nlp) aims to develop methods and tools to unlock this data but has lagged behind general domain nlp in applying many recent neural and machine learning advances. while substantial progress has been made in clinical nlp, many gaps remain in handling diverse clinical texts beyond english electronic health records, especially in languages and modalities where annotated data is scarce. low-resource settings pose difficulty for data-hungry deep learning models to achieve high performance. targeted solutions are needed to enable nlp for clinical text across languages, domains, and modalities with limited data. in this work, we propose a transfer learning method for low-resource clinical named entity recognition (ner) that relies on translating indomain datasets to a high-resource language rather than requiring large amounts of direct annotation in the target language or domain. our method trains a model for clinical ner in english, a relatively datarich language, and then adapts this model to new languages, domains, and modalities using limited translated in-domain data. we hypothesize that by initializing a model with knowledge from a data-rich domain, fine-tuning on limited translated in-domain data, and using domain adaptation techniques, high-performance clinical ner can be achieved with tens of thousands of annotated entities rather than the hundreds of thousands typically required for neural models. experiments on spanish, french, and conversational (e.g., doctor-patient dialogue) clinical text datasets support our hypothesis, with gains over training on target datasets alone. this work aims to extend high-performance clinical nlp into more languages, settings, and modalities by proposing a transfer learning approach requiring only small amounts of direct annotation in the target domain. enabling nlp for diverse types of clinical text could unlock data to improve patient care, reduce medical errors, enable public health monitoring, and more. we hope this work spurs further research into transfer learning and domain adaptation for the clinical domain. ## related work recent approaches for low-resource named entity recognition (ner) include using cross-lingual word embeddings @xcite , bilingual lexicon induction @xcite , and model transfer between high-and low-resource languages @xcite . transfer learning, where a model trained on a highresource domain is fine-tuned on a low-resource target domain, has shown promise for clinical nlp @xcite but typically requires larger target datasets than we assume in this work. translating datasets to a high-resource language is an intuitive approach but has not been extensively explored for transfer learning. @xcite translate a german dataset to english to augment training data for english ner, showing small gains over english training data alone. @xcite translated spanish datasets to english to improve an english ner model, then transferred back to spanish with limited success. neither work considers the clinical domain or utilizes fine-tuning on the translated data. domain adaptation techniques like weight freezing @xcite , parameter shuffling @xcite , and dropout @xcite have improved transferability between domains in computer vision and nlp. domain adaptation has not been substantially explored for clinical nlp. work on processing conversational or dialogue text with nlp has focused on domains like customer service @xcite , tutoring systems @xcite , and captioning @xcite . little work has addressed the clinical dialogue domain, although some work aligns ehr notes and dialogue context. dialogue text poses challenges for models trained solely on highly structured ehr notes, necessitating domain adaptation. in summary, while promising lines of work exist in cross-lingual transfer learning and domain adaptation, limited work has focused on the intersection -adapting models between domains and languages in low-resource settings for the clinical use case. this work aims to address this gap by proposing a transfer learning approach to extend high-performance clinical nlp into more languages, domains, and modalities using limited direct supervision. ## proposed method we propose a transfer learning method that adapts a high-resource english clinical ner model to low-resource languages and domains using only small amounts of in-domain annotated data. our approach involves: this work aims to extend the capabilities of stateof-the-art clinical nlp models to low-resource languages, domains, and modalities where annotated data is scarce by proposing a novel transfer learning approach requiring only small amounts of direct in-domain annotation. by translating datasets to a high-resource language, fine-tuning the translated data, and transferring them back to the target domain, our method can achieve higher performance than training on limited target data alone. ## training we trained our english clinical ner model on 950,000 anonymized ehr notes provided by anthropic, pbc. the notes span multiple years and institutions, covering patient encounters, progress notes, discharge summaries, and other clinical texts. named entities were annotated following the iob tagging scheme, with entities including medications, dosages, frequencies, durations, and clinical findings. the model was implemented in pytorch and trained for 10 epochs with the following hyperparameters: the english model achieved 94.3% precision, 92.5% recall, and 93.4% f1 score on a held-out test set of 50,000 notes. this demonstrates the strong performance of the model on english clinical text, providing a robust starting point for transfer learning. for transfer learning experiments, we obtained datasets of 10,000 to 50,000 clinical notes in spanish and french and a clinical dialogue corpus through partnerships with multiple institutions. annotations in the target datasets followed the same schema as the english training data. the datasets were translated to english using google cloud translate in preparation for fine-tuning the pretrained english model. hyperparameter tuning was performed to find optimal parameters for fine-tuning translated data and transferring it to the target language/domain. the following hyperparameters were used for finetuning, with dropout and weight freezing employed to prevent overfitting to the translated data: the fine-tuned models were evaluated on heldout test portions of the untranslated target datasets to assess performance after transferring back to the original language/domain. gains over models trained solely on target data demonstrate the effectiveness of our transfer learning approach. ## results we evaluated our transferred models on held-out test sets in each target dataset and compared performance to models trained solely on the target data. ## conclusion we proposed a transfer learning method that adapts a high-resource english clinical ner model to low-resource languages and domains using only small amounts of in-domain annotated data. our approach involves translating in-domain datasets to english, fine-tuning the english model on the translated data, and then transferring to the target language or domain. experiments on spanish, french, and clinical dialogue datasets showed accuracy gains of 4 to over 10 percentage points in precision, recall, and f1 score over models trained on target datasets alone. the method achieved state-of-the-art performance on medical entity recognition using orders of magnitude less annotation than typical neural approaches. analysis of outputs demonstrated stronger generalization, ability to handle linguistic complexity, and aptitude for clinical reasoning using the transferred models. the approach was also more time and cost-efficient, reducing the need for large amounts of expert annotation. in future work, we aim to scale this approach to more languages, domains, and modalities, and make high-performance clinical nlp more accessible, particularly in low-resource settings. we plan to: by advancing transfer learning and domain adaptation for the clinical domain, we can unlock more data, gain deeper insights, and develop ai systems that adapt to diverse real-world settings -leading to benefits for healthcare worldwide, especially for underserved populations. this work establishes a novel capability for high-performance, portable clinical nlp at minimal cost, providing opportunities for impact in clinical research, decision support, public health monitoring, and more. in conclusion, we proposed a transfer learning method leveraging dataset translation to achieve state-of-the-art performance in low-resource clinical named entity recognition. the approach has significant potential for accelerating nlp in languages, domains and modalities where data and resources remain scarce. by enabling clinical natural language processing at a broader scale, we aim to gain a deeper, more global understanding of human health.