Learning Domain-Specialised Representations for Cross-Lingual Biomedical Entity Linking (2021)

Abstract: Injecting external domain-specific knowledge (e.g., UMLS) into pretrained language models (LMs) advances their capability to handle specialised in-domain tasks such as biomedical entity linking (BEL). However, such abundant expert knowledge is available only for a handful of languages (e.g., English). In this work, by proposing a novel cross-lingual biomedical entity linking task (XL-BEL) and establishing a new XL-BEL benchmark spanning 10 typologically diverse languages, we first investigate the ability of standard knowledge-agnostic as well as knowledge-enhanced monolingual and multilingual LMs beyond the standard monolingual English BEL task. The scores indicate large gaps to English performance. We then address the challenge of transferring domain-specific knowledge in resource-rich languages to resource-poor ones. To this end, we propose and evaluate a series of cross-lingual transfer methods for the XL-BEL task, and demonstrate that general-domain bitext helps propagate the available English knowledge to languages with little to no in-domain data. Remarkably, we show that our proposed domain-specific transfer methods yield consistent gains across all target languages, sometimes up to 20 Precision@1 points, without any in-domain knowledge in the target language, and without any in-domain parallel data.

## introduction recent work has demonstrated that it is possible to combine the strength of 1) transformer-based encoders such as bert @xcite , pretrained on large general-domain data with 2) external linguistic and world knowledge @xcite @xcite . such expert human-curated knowledge is crucial for nlp applications in specialised domains such as biomedicine. there, @xcite recently proposed self-alignment pretraining (sap), a technique to fine-tune bert on phraselevel synonyms extracted from the unified medical language system (umls; @xcite ). 1 their sapbert model currently holds state-of-theart (sota) across all major english biomedical entity linking (bel) datasets. however, this approach is not widely applicable to other languages: abundant external resources are available only for a few languages, hindering the development of domainspecific nlp models in all other languages. simultaneously, exciting breakthroughs in crosslingual transfer for language understanding tasks have been achieved @xcite . however, it remains unclear whether such transfer techniques can be used to improve domain-specific nlp applications and mitigate the gap between knowledge-enhanced models in resource-rich versus resource-poor languages. in this paper, we thus investigate the current performance gaps in the bel task beyond english, and propose several cross-lingual transfer techniques to improve domain-specialised representations and bel in resource-lean languages. in particular, we first present a novel crosslingual bel (xl-bel) task and its corresponding evaluation benchmark in 10 typologically diverse languages, which aims to map biomedical names/mentions in any language to the controlled umls vocabulary. after empirically highlighting the deficiencies of multilingual encoders (e.g, mbert and xlmr; @xcite ) on xl-bel, we propose and evaluate a multilingual extension of the sap technique. our main results suggest that expert knowledge can be transferred from english to resource-leaner languages, yielding huge gains over vanilla mbert and xlmr, and english-only sapbert. we also show that leveraging general-domain word and phrase translations offers substantial gains in the xl-bel task. ## methodology background and related work. learning biomedical entity representations is at the core of bionlp, benefiting, e.g., relational knowledge discovery @xcite and literature search @xcite . in the current era of contextualised representations based on transformer architectures @xcite , biomedical text encoders are pretrained via masked language modelling (mlm) on diverse biomedical texts such as pubmed articles @xcite , clinical notes @xcite , and even online health forum posts @xcite . however, it has been empirically verified that naively applying mlmpretrained models as entity encoders does not perform well in tasks such as biomedical entity linking @xcite . recently, @xcite proposed sap (self-alignment pretraning), a fine-tuning method that leverages synonymy sets extracted from umls to improve bert's ability to act as a biomedical entity encoder. their sapbert model currently achieves sota scores on all major english bel benchmarks. in what follows, we first outline the sap procedure, and then discuss the extension of the method to include multilingual umls synonyms ( §2.1), and then introduce another sap extension which combines domain-specific synonyms with generaldomain translation data ( §2.2). ## the xl-bel task and evaluation data a general cross-lingual entity linking (el) task @xcite aims to map a mention of an entity in free text of any language to a controlled english vocabulary, typically obtained from a knowledge graph (kg). in this work, we propose xl-bel, a cross-lingual biomedical el task. instead of grounding entity mentions to english-specific ontologies, we use umls as a language-agnostic kg: the xl-bel task requires a model to associate a mention in any language to a (language-agnostic) cui in umls. xl-bel thus serves as an ideal evaluation benchmark for biomedical entity representations: it challenges the capability of both 1) representing domain entities and also 2) associating entity names in different languages. ## experiments and results umls data. we rely on the umls (2020aa) as our sap fine-tuning data, leveraging synonyms in all available languages. the full multilingual fine-tuning data comprises ≈15m biomedical entity names associated with ≈4.2m individual cuis. as expected, english is dominant (69.6% of all 15m names), followed by spanish (10.7%) and french (2.2%). the full stats are in app. §a.3. translation data. we use (a) "muse" word translations @xcite , and (b) the parallel wikipedia article titles (phrase-level translations; referred to as "wt"). we also list results when using "muse" and "wt" combined ("wt+ muse"). training and evaluation details. our sap finetuning largely follows liu et al. (2021); we refer to the original work and the appendix for further tech-@xmath0 ## conclusion we have introduced a novel cross-lingual biomedical entity task (xl-bel), establishing a widecoverage and reliable evaluation benchmark for cross-lingual entity representations in the biomedical domain in 10 languages, and have evaluated current sota biomedical entity representations on xl-bel. we have also presented an effective transfer learning scheme that leverages general-domain translations to improve the cross-lingual ability of domain-specialised representation models. we hope that our work will inspire more research on multilingual and domain-specialised representation learning in the future. yifan peng, shankai yan, and zhiyong lu. 2019. transfer learning in biomedical natural language processing: an evaluation of bert and elmo on ten benchmarking datasets. in proceedings of the 18th bionlp workshop and shared task, pages 58-65, florence, italy. association for computational linguistics. nils reimers and iryna gurevych. 2020. making monolingual sentence embeddings multilingual using knowledge distillation. in proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), pages 4512-4525, online. association for computational linguistics. mujeen sung, hwisang jeon, jinhyuk lee, and jaewoo kang. 2020. biomedical entity representations with synonym marginalization. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 3641-3650, online. association for computational linguistics. chen-tse tsai and dan roth. 2016. cross-lingual wikification using multilingual embeddings. in proceedings of the 2016 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 589-598, san diego, california. association for computational linguistics. elena tutubalina, artur kadurin, and zulfat miftahutdinov. 2020. fair evaluation in concept normalization: a large-scale comparative analysis for bertbased models. in proceedings of the 28th international conference on computational linguistics, pages 6710-6716, barcelona, spain (online). international committee on computational linguistics. a appendix a a.1 xl-bel: full statistics