Cross-Language Transfer of High-Quality Annotations: Combining Neural Machine Translation with Cross-Linguistic Span Alignment to Apply NER to Clinical Texts in a Low-Resource Language (2022)

Abstract: In this work, cross-linguistic span prediction based on contextualized word embedding models is used together with neural machine translation (NMT) to transfer and apply the state-of-the-art models in natural language processing (NLP) to a low-resource language clinical corpus. Two directions are evaluated: (a) English models can be applied to translated texts to subsequently transfer the predicted annotations to the source language and (b) existing high-quality annotations can be transferred beyond translation and then used to train NLP models in the target language. Effectiveness and loss of transmission is evaluated using the German Berlin-Tübingen-Oncology Corpus (BRONCO) dataset with transferred external data from NCBI disease, SemEval-2013 drug-drug interaction (DDI) and i2b2/VA 2010 data. The use of English models for translated clinical texts has always involved attempts to take full advantage of the benefits associated with them (large pre-trained biomedical word embeddings). To improve advances in this area, we provide a general-purpose pipeline to transfer any annotated BRAT or CoNLL format to various target languages. For the entity class medication, good results were obtained with 0.806 F1 -score after re-alignment. Limited success occurred in the diagnosis and treatment class with results just below 0.5 F1 -score due to differences in annotation guidelines.

## introduction clinical texts contain many important buried information that can be accessed through natural language processing (nlp). systematic analysis of this vast amount of data can improve clinical care and aid in decision making. there are many other applications already in use, such as cohort selection, pharmacovigilance, and quality reporting @xcite . clinical text is often available as unstructured texts: retrospective analysis therefore involves an enormous amount of work @xcite . by using nlp, biomedical concepts can be extracted and processed using named entity recognition (ner), allowing large amounts of text on specific topics of interest to be retrospectively analyzed. while biomedical text is intended for publications, clinical text is written by and aimed at health care professionals. they are written under time pressure and are heterogeneous in terms of abbreviations, omission of words, and medical jargon to keep information density high @xcite . compared to english texts, the processing of non-english clinical texts by nlp is far from what is actually possible by the current state-of-the-art @xcite . this is due to the fact that in the u.s., health insurance portability and accountability (hipaa) clearly regulates which 18 different identifiers of protected health information (phi) must be removed in order for a document to be considered anonymized, creating many facilitators for de-identification of clinical texts @xcite . based on these rules, large clinical datasets such as medical information mart for intensive care iii (mimic-iii) @xcite and shared tasks with high-quality annotations have been published, resulting in research and tools for processing english clinical texts being widely developed. with regard to the availability of nlp tools for other languages, there are major differences, for example in the processing of german clinical texts: anonymization is left to individual institutions, data protection officers, and ethics committees, which means that there are no uniform regulations. the state-of-the-art for german texts lags behind and, despite great efforts @xcite , continues to be limited to rule-based systems @xcite or is often based on in-house data @xcite , which means that neither the data nor the trained models can be shared @xcite . freely available large anonymized datasets with high-quality annotated german clinical texts are therefore non-existent. in order to bridge this gap, this work provides a general-purpose pipeline to transfer annotated datasets in brat or conll format to various target languages foot_0 . approaches based on neural machine translation (nmt) have recently been applied to ner tasks @xcite @xcite . improved translation quality through advances in neural machine translation @xcite have reached a level that allows the transfer of predictions or annotated data in combination with word alignments @xcite to other languages. in this work, the berlin-tübingen-oncology corpus (bronco) @xcite is used and treated as a zero-resource dataset, for which english models and external biomedical and clinical datasets are used instead. the aim is to evaluate whether low-resource languages can benefit from the available english resources. the methodology of this work can be applied to other clinical datasets and languages, as word alignment with contextualized embeddings through multilingual bert @xcite covers 104 languages. accordingly, multilingual models are available for translation, e.g., the mbart @xcite many-to-many model covers 50 languages. ## data the bronco corpus @xcite is the first small, fully anonymized dataset for german clinical texts, that can be accessed via a data usage agreement form request. the dataset contains 200 discharge reports of hepatocellular carcinoma and melanoma, with 50 reports retained by the authors as independent test data. due to strict data protection regulations and to make de-anonymization more difficult, the discharge summaries were shuffled into sentences so that the clinical context is only preserved at sentence level. it includes three annotated entity classes: diagnosis, medication and treatment (see for the 3 entity classes in bronco, 3 existing english external datasets are used. in order to use external data, the underlying documents and annotation guidelines should match if possible. ## methods the experiments are divided into two parts. first, the german clinical dataset is treated as a zeroresource problem. this means that none of the annotated data is used to develop recognition models for the three entity classes diagnosis, medication and treatment. instead, three existing english highquality annotated datasets as described in section 2 are used to train on the entity classes. inferences are either made based on the translation and are then retroactively aligned to the german text, or models are fine-tuned on the translated form of the english datasets and are directly applied to the german clinical texts. the second part focuses on the extent to which english pre-trained biomedical language models can be adapted for use in another language. for this purpose, the german dataset is translated and aligned in order to fine-tune large english pretrained biomedical transformer-based models. the inference is then re-aligned to the german language. this is compared to non-biomedical german and cross-lingual transformer-based language models. in this way, the loss due to translation and subsequent alignment can be determined and weighed against the benefits of large biomedical language models that would not otherwise be available. based on current benchmarks @xcite , the selection for translation models fell on the directional wmt 19 en ↔ de model from facebook ai research (fair) as well as the multilingual wmt 21 model that covers 7 different languages. since careful review of the translation quality of some clinical texts did not reveal any relevant deficiencies, the more resource-friendly wmt 19 model was chosen. for the span alignment of the annotations, simalign (jalili sabet et al., 2020) is used without fine-tuning a parallel corpus. the work of jalili sabet et al. (2020) has shown that word alignments via contextualized embeddings from multilingual language models achieve good results. here, the itermax algorithm is used with contextualized word embeddings from multilingual bert @xcite . itermax aligns two parallel sentences at token level with cosine-similarity, where for each token the parallel vectors co-represent the context of the token within its sentence. since for many sentences no mutual argmaxes are available, the suggestion mentioned by the authors to perform this process iteratively is followed. this also allows for token of the source language to be mapped to multiple token in the target language. this seems reasonable for clinical entities. for example, urinary tract obstruction is merged to only one token harnwegsblockade in the german language (see figure 1 ). for fine-tuning language models, all experiments use the hyperparameters as described in ## discussion in the zero-resource setting, there is an advantage in the backward-pass approach over the forwardpass models. good results could only be achieved for the medication class, but this is not necessarily due to translation and word alignment, but to the nature of the data. for the diagnosis and treatment class, there is no equivalent english dataset that fully matches the annotation guidelines of the german clinical text. the medication class seems unproblematic in that medication terms are more easily aligned, one-to-many token constellations due to translation are rare, and medications are often represented similarly in both languages. nevertheless, the underlying sentence structure is fundamentally different between english and german, which means that the transfer of the results can be considered successful. further limitations are discussed in section 6. these assumptions are also supported by the finetuning results, which show that although translation and alignment result in a loss, it is still competitive compared to the initial baseline. only in the comparison with multilingual and german transformer architectures the disadvantage becomes clear. provided that annotations are available, a general advantage of english biomedical models over non- ## conclusion and future work the results of this work show that english language models can in principle be applied to other languages in clinical contexts. translated training data can serve as a good basis and approach for languages where there are otherwise no resources. in a zero-resource scenario, the approach is limited to the extent that it works for data where the documents and annotation guidelines match across languages. cross-linguistic differences in the available standards that annotators work with also play a limiting role here. bronco corpus is based on german icd-10 and german ops standards, which is also reflected in the annotation guidelines, making it difficult to apply external data. transfer in the clinical setting was evaluated with only one language pair (en ↔ de). success with other language pairs depends not only on the annotation standard, but also on the similarity of the languages (grammar and morphology). transfer can only succeed if the quality of translation and word alignment is sufficient, which can be expected between indo-european languages, but can be much more difficult when transferring between language families. practical applications on other low-resource languages is left for future work. it would be interesting to see the effect of adding a few annotated samples to the external data. in this context, zeroshot and few-shot approaches would be a useful addition as a comparator. for comparison, it would also be helpful to have a non-alignment baseline that is fine-tuned to english data and directly infers german test data.