KB io XLM : A Knowledge-anchored Biomedical Multilingual Pretrained Language Model (2023)

Abstract: Most biomedical pretrained language models are monolingual and cannot handle the growing cross-lingual requirements. The scarcity of non-English domain corpora, not to mention parallel data, poses a significant hurdle in training multilingual biomedical models. Since knowledge forms the core of domain-specific corpora and can be translated into various languages accurately, we propose a model called KBioXLM, which transforms the multilingual pretrained model XLM-R into the biomedical domain using a knowledge-anchored approach. We achieve a biomedical multilingual corpus by incorporating three granularity knowledge alignments (entity, fact, and passage levels) into monolingual corpora. Then we design three corresponding training tasks (entity masking, relation masking, and passage relation prediction) and continue training on top of the XLM-R model to enhance its domain cross-lingual ability. To validate the effectiveness of our model, we translate the English benchmarks of multiple tasks into Chinese. Experimental results demonstrate that our model significantly outperforms monolingual and multilingual pretrained models in cross-lingual zero-shot and few-shot scenarios, achieving improvements of up to 10+ points.

## introduction in recent years, biomedical pretrained language models (plms) @xcite @xcite have made remarkable progress in various natural language processing (nlp) tasks. while the biomedical annotation data @xcite @xcite @xcite are predominantly in english. therefore, non-english biomedical natural language process- * corresponding author. wikipedia aligned en:42030 laudanum contains approximately 10% opium poppy, equivalent to 1% morphine… cn:6640907 鸦片酊通过将罂粟的提取物溶解 在酒精中制备，含有约10%重量 的鸦片粉末… umlstrans entity knowledge alignment: 鸦片酊 罂粟花 吗啡 [cls] laudanum contains approximately 10% opium poppy, equivalent to 1% morphine… entities: @xmath0; @xmath1; opium @xmath2: fact knowledge alignment: unlike in general domains, there is a scarcity of non-english biomedical corpora and even fewer parallel corpora in the biomedical domain, which presents a significant challenge for training mul-tilingual biomedical models. in general domains, back translation (bt) @xcite is commonly used for data augmentation. however, our experiments (refer to "xlm-r+bt" listed in to compensate for the lack of cross-lingual evaluation datasets, we translate and proofread four english biomedical datasets into chinese, involving three different tasks: named entity recognition (ner), relation extraction (re), and document classification (dc). the experimental results demon-strate that our model consistently outperforms both monolingual and other multilingual pretrained models in cross-lingual zero-shot and few-shot scenarios. on average, our model achieves an impressive improvement of approximately 10 points than general multilingual models. meanwhile, our method maintains a comparable monolingual ability by comparing common benchmarks in both chinese and english biomedical domains. to summarize, our contributions can be outlined as follows: ## method this section presents the proposed knowledgeanchored multi-lingual biomedical plm called kbioxlm. considering the scarcity of biomedical parallel corpora, we utilize language-agnostic knowledge to facilitate text alignment at three levels of granularity: entity, fact, and passage. subsequently, we train kbioxlm by incorporating these knowledge-anchored aligned data on the foundation of the multilingual xlm-r model. figure 2 provides an overview of the training details. ## biomedical dataset construction after @xmath3, she @xmath4. ## conclusion this paper proposes kbioxlm, a model that transforms the general multi-lingual plm into the biomedical domain using a knowledge-anchored approach. we first obtain biomedical multilingual corpora by incorporating three levels of knowledge alignment (entity, fact, and passage) into the monolingual corpus. then we design three train-ing tasks, namely entity masking, relation masking, and passage relation prediction, to enhance the model's cross-lingual ability. kbioxlm achieves sota performance in cross-lingual zero-shot and few-shot scenarios. in the future, we will explore biomedical plms in more languages and also venture into multilingual plms for other domains.