Few-Shot Cross-lingual Transfer for Coarse-grained De-identification of Code-Mixed Clinical Texts (2022)

Abstract: Despite the advances in digital healthcare systems offering curated structured knowledge, much of the critical information still lies in large volumes of unlabeled and unstructured clinical texts. These texts, which often contain protected health information (PHI), are exposed to information extraction tools for downstream applications, risking patient identification. Existing works in de-identification rely on using large-scale annotated corpora in English, which often are not suitable in real-world multilingual settings. Pre-trained language models (LM) have shown great potential for cross-lingual transfer in low-resource settings. In this work, we empirically show the few-shot cross-lingual transfer property of LMs for named entity recognition (NER) and apply it to solve a low-resource and real-world challenge of code-mixed (Spanish-Catalan) clinical notes de-identification in the stroke domain. We annotate a gold evaluation dataset to assess few-shot setting performance where we only use a few hundred labeled examples for training. Our model improves the zero-shot F1-score from 73.7% to 91.2% on the gold evaluation set when adapting Multilingual BERT (mBERT) (CITATION) from the MEDDOCAN (CITATION) corpus with our few-shot cross-lingual target corpus. When generalized to an out-of-sample test set, the best model achieves a human-evaluation F1-score of 97.2%.

## introduction with growing interest and innovations in datadriven digital technologies, privacy has become an important legal topic for the technology to be regulations-compliant. in europe, the general data protection regulation (gdpr) @xcite requires data owners to have a legal basis for processing personally identifiable information (pii), which also includes the explicit consent of the subjects. in cases where explicit consent is not possible, anonymization is often seen as a resorted-to figure 1 : the process of text de-identification involves the removal of a predefined set of direct identifiers in text @xcite . for clinical notes, this set is often the phi categories (or types) defined by the health insurance portability and accountability act (hipaa) @xcite . the example here shows a deidentified excerpt of a patient note from the spanish-catalan stroke dataset used in this study (translated into english here for readability). in this work, we focus on the task of removing phi from clinical texts, also called de-identification (fig. 1 ). we address a real-world challenge where the target texts are code-mixed (spanish-catalan) and domain-constrained (stroke). to avoid high annotation costs, we consider a more realistic setting where we annotate a gold evaluation corpus and a few hundred examples for training. our approach is motivated by strong performance of pre-trained lms in few-shot cross-lingual transfer for ner with high sample efficiency (see fig. 2 ) in comparison to supervised or unsupervised approaches. our contributions are summarized as follows: ## problem statement we approach the de-identification problem as an ner task. given an input sentence x with n words: @xmath0:n , we feed it to an encoder we feed h into the ner classifier which is a linear classification layer with the softmax activation function to predict the phi label of x: for pre-trained lms, this setting corresponds to ner fine-tuning @xcite . when we jointly fine-tune on more than one ner dataset, we refer to it as multi-task learning. definition 1 (few-shot ner). given an entity label set p, we define the task of few-shot ner as having access to @xmath1, where k is a small number (e.g., in @xcite ) and m is orders of magnitude larger (e.g., ≥ 1000). definition 2 (few-shot ner transfer). given an ner dataset in a source domain (or language), we define the task of few-shot cross-domain (or cross-lingual) ner transfer as adapting a model trained on the source domain (or language) to a target domain (or language) with access to a fewshot corpus (definition 1). this setting is different from prior studies in ner transfer including few-shot @xcite , unsupervised @xcite , and semisupervised ner @xcite . ## data and annotation our dataset consists of stroke patient records collected at instiut guttmann. foot_0 following the protocol in gao et al. (2021) for constructing manually annotated distantly supervised relation extraction test sets, we train mbert on the meddocan corpus, using coarsegrained phi categories {date, age, location, name, contact, profession, id} with the bio scheme @xcite , for evaluation and few-shot training data selection. we use the trained model to make predictions on the dataset and observe that the model predicts phi on only ## experiments and results we conduct our experiments with the t2ner framework @xcite . 3 for the baseline, we consider zero-shot performance on the evaluation set of the mbert encoder fine-tuned on the meddocan training set consisting of 16,299 samples. we then fine-tune it on the few-shot target corpus as outlined in fig. 3 . following the multi-task learning @xcite approach in t2ner, we jointly fine-tune mbert on the med-docan and few-shot target corpora. since the few-shot corpus is much smaller, the multi-task learning helps the model transfer. it further acts as a regularization approach by sharing parameters between the datasets. to improve performance on the target data, we further fine-tune with the few-shot target corpus after the first step of fine-tuning to have improved target performance; for the model to be an expert in target @xcite . all the models are trained for 3 epochs with a learning rate of 3e-5 and linear warm-up of 10%. for few-shot fine-tuning only, the model is trained for 25 epochs. to test the model's generalizability, we tokenize the 100 out-of-sample notes into sentences and make predictions with our best model. the resulting annotated sentences are reconstructed into patient notes, which are manually evaluated by two reviewers (one external and one annotator) for occurrences of true and false positives and negatives. the model achieves precision, recall, and f1-scores of 95.1%, 99.3%, and 97.1% respectively on the out-of-sample notes, highlighting the effectiveness of our approach. ## conclusion we address the task of clinical notes deidentification in a low-resource scenario. by investigating the few-shot cross-lingual transfer property of mbert, we propose a strategy that significantly boosts zero-shot performance while keeping the number of required annotated samples low. our results highlight the effectiveness of the proposed strategy for the task with a potential for future applications in other low-resource scenarios. description observation sentences annotated by annotator (a) 4400 sentences annotated by annotator (b) 4343 sentences annotated and revised by (a, b) 4314 agreements 3924 disagreements 390 token-level cohen's kappa score 0.898 development corpus 3924 w entity mentions 1493 w/o entity mentions 2431 few-shot target corpus 384 w entity mentions 369 w/o entity mentions 15 table a.2: dataset annotation and final statistics.