Multilingual Clinical NER : Translation or Cross-lingual Transfer? (2023)

Abstract: Natural language tasks like Named Entity Recognition (NER) in the clinical domain on non-English texts can be very time-consuming and expensive due to the lack of annotated data. Cross-lingual transfer (CLT) is a way to circumvent this issue thanks to the ability of multilingual large language models to be fine-tuned on a specific task in one language and to provide high accuracy for the same task in another language. However, other methods leveraging translation models can be used to perform NER without annotated data in the target language, by either translating the training set or test set. This paper compares cross-lingual transfer with these two alternative methods, to perform clinical NER in French and in German without any training data in those languages. To this end, we release MedNERF a medical NER test set extracted from French drug prescriptions and annotated with the same guidelines as an English dataset. Through extensive experiments on this dataset and on a German medical dataset (Frei and Kramer, 2021), we show that translation-based methods can achieve similar performance to CLT but require more care in their design. And while they can take advantage of monolingual clinical language models, those do not guarantee better results than large general-purpose multilingual models, whether with cross-lingual transfer or translation.

## introduction in recent years, pre-trained language models based on the transformer architecture @xcite have demonstrated high performance on many natural language tasks such as named entity recognition (ner), natural language inference or question-answering @xcite . these models, which are generally pre-trained on general domain data, can be finetuned on downstream tasks to achieve state-of-theart results. such models can also be adapted to a * equal contribution. specific domain such as the legal @xcite or the biomedical fields @xcite and can then outperform the general-domain models on domain-specific tasks. extracting medical entities from unstructured texts has become an essential tool to structure medical reports, and pre-trained language models have been naturally used to perform this task @xcite . these models use training data for fine-tuning that come from biomedical ner datasets like ncbi-disease @xcite or n2c2 @xcite . however most of these datasets are only available in english and consequently the majority of such medical ner algorithms are developed for the english language, while medical reports or drug prescriptions are rather written in the country's language. in the clinical domain, non-english datasets to fine-tune a model are even rarer than in the general domain. even large domain-specific unlabeled corpora are mostly found in english. for example, the biomedical scientific literature is mostly written and available in english. moreover, gathering medical texts and annotating them using expert knowledge is very expensive for low-resource languages and more generally for any non-english language. this restricts the development of medical ner models for non-english languages. fortunately, cross-lingual transfer (clt) can work around the absence of training data in the target language, by making use of language models pre-trained on multilingual data. clt consists in applying on a specific task a multilingual large language model (mllm) fine-tuned in another language for the same task. for example, models like mbert @xcite and xlm-r @xcite can be fine-tuned on a generaldomain ner task in english and provide competitive results when evaluated in other languages on this task @xcite . before mllms, cross-lingual adaptation was generally tackled using translation methods @xcite . the lack of annotated data can be overcome by translating an english training set into the target language and by projecting the annotations with an alignment algorithm. another approach consists in translating a test set into english to apply an english ner on it and in projecting the predicted labels in the original language. recently, translation models based on the transformer architectures @xcite have demonstrated huge improvements over other machine translation algorithms. such models can therefore be leveraged to perform crosslingual learning. however, comparing clt with translation-based methods has attracted little attention and only comparisons in the general domain @xcite has been done. our proposed contribution is three-fold: (1) we perform extensive experiments with clt and translation-based methods on a medical ner task in french and in german without any training data in those languages, (2) we release mednerf, a french clinical ner dataset based on drug prescriptions, which serves as a test set for our experiments in french, (3) we demonstrate that clt and translation provide comparable results for clinical ner, that the choice of the technique depends on the model's size and that using domain-specific models does not necessarily improve the results over using multilingual models. ## related work multilingual large language models (mllms). mllms are the logical multilingual extension of large language models, trained on multilingual corpora. there exist several versions of mllms among which the most popular are mbert @xcite , which is bert multilingual version pre-trained on wikipedia in a hundred different languages instead on only english wikipedia (and bookcorpus), and xlm-r @xcite which is a multilingual transformer encoder using the same pre-training principles as roberta @xcite , trained in 100 languages on a larger and more diverse corpus than wikipedia. xlm-r, mbert and its distilled version dis-tilmbert @xcite are not trained on any parallel data and show nevertheless strong clt abilities, for example on ner tasks, even between languages which use different sets of characters @xcite . ## method and models we now describe the different methods we compare to perform ner without any annotations in the target language: cross-lingual transfer and two translation-based methods, where either the train set or the test set is translated. all of them only use the english annotations from the n2c2 dataset (track 2, adverse drug events and medication extraction) @xcite , which is an english dataset of medical entities extracted from ehr. ## evaluation data while the model is trained on the english dataset n2c2 or a translation of it, it must be evaluated on french and german data, annotated similarly to n2c2 to assess its cross-lingual adaptation abilities. ## pre-selecting translation and alignment the translation-based methods require a translation and an alignment models. we present in this section how we fine-tuned translation and alignment algorithms and how we chose which algorithms to use in our experiments. the choice of these algorithms can be seen as an hyper-parameter and for fair comparison with clt, the selection should not be based on downstream cross-lingual abilities as this would mean cross-lingual supervision. ## results having selected the best translation and alignment model for each language based on intrinsic evaluation, translate-train and translate-test approaches can be compared to cross-lingual transfer (clt). the impact of the translation and alignment models can also be analyzed, as well as using the translation fine-tuning data to improve directly the models used in clt. since translate-train can leverage monolingual domain-specific model, we evaluate the outcome of such a strategy. five different random seeds were used for each model and results presented in this section show the average performance along with the standard deviation (more implementation details in appendix b). ## conclusion this paper shows that cross-lingual transfer with general-domain mllms is efficient for a domainspecific task like clinical ner, giving comparable results with translating the training set. but clt has the advantage of working off-the-shelf, while translation-based methods require choosing the translation and alignment models carefully. selecting these models based on intrinsic domainspecific values, like fine-tuning scores on clinical parallel data, or using a domain-specific language model does not provide significantly better downstream results in the target language. the selection of the alignment model was shown to be particularly crucial, and results of translate-train could probably be improved by post-processing the alignment. clt also has a margin of progression as realigning the representations of mllms can increase the results dramatically in some cases. it is also worth noting that training on translated data provide better results than translating at inference time. the translate-test approach should then be used only when large multilingual models cannot be used. while training on translated data allows to leverage domain-specific monolingual language models, those latter models can give better results over multilingual models like xlm-r only if pre-trained with sufficient data. pre-training a mllm with only clinical data is a good lead for further improvements in clinical cross-lingual transfer. while the results show that using a domain-specific monolingual model in translate-train or translate-test is not on par with general-purpose multilingual models, they also show that the french clinical model drbert provides the best results for translate-train when it uses the english biomedical model pubmedbert as initialization. we finally advocate for the release of more non-english clinical datasets annotated with similar guidelines as english (or other) ones. even a relatively small dataset like mednerf or the gernermed test set are crucial to evaluate crosslingual adaptation in the clinical domain.