Can Language Models be Biomedical Knowledge Bases? (2021)

Abstract: Pre-trained language models (LMs) have become ubiquitous in solving various natural language processing (NLP) tasks. There has been increasing interest in what knowledge these LMs contain and how we can extract that knowledge, treating LMs as knowledge bases (KBs). While there has been much work on probing LMs in the general domain, there has been little attention to whether these powerful LMs can be used as domain-specific KBs. To this end, we create the BioLAMA benchmark, which is comprised of 49K biomedical factual knowledge triples for probing biomedical LMs. We find that biomedical LMs with recently proposed probing methods can achieve up to 18.51% Acc@5 on retrieving biomedical knowledge. Although this seems promising given the task difficulty, our detailed analyses reveal that most predictions are highly correlated with prompt templates without any subjects, hence producing similar results on each relation and hindering their capabilities to be used as domain-specific KBs. We hope that BioLAMA can serve as a challenging benchmark for biomedical factual probing.

## introduction recent success in natural language processing can be largely attributed to powerful pre-trained language models (lms) that learn contextualized representations of words from large amounts of unstructured corpora @xcite . there have been recent works in probing how much knowledge these lms contain in their parameters @xcite and how to effectively extract such knowledge. @xcite @xcite . while factual probing of lms has attracted much attention from researchers, a more practical application would be to leverage the power of domainspecific lms @xcite as domain knowledge bases (kbs). unlike recent works that probe general domain knowledge, we ask whether it is also possible to retrieve expert knowledge from lms. specifically, we tune our focus on factual knowledge probing for the biomedical domain as shown in figure 1 . to inspect the potential utility of lms as biomedical kbs, we create and release the biomedical language model analysis (biolama) probe. biolama consists of 49k biomedical factual triples whose relations have been manually curated from three different knowledge sources: the comparative toxicogenomics database (ctd), the unified medical language system (umls), and wikidata. while our biomedical factual triples are inherently more difficult to probe (see initial probing results on biolama show that the best performing lm achieves up to 7.28% acc@1 and 18.51% acc@5, and outperforms an information extraction (ie) baseline @xcite . although this result seems promising, we find that their output distributions are largely biased to a small number of entities in each relation. along this line, we use two metrics, prompt bias @xcite and synonym variance, to investigate the behavior of lms as kbs. our analysis shows that while lms seem to be more aware of synonyms than the ie baseline, they output highly biased predictions given the prompt template of each relation. our result calls for better lms and probing methods that can retrieve rich but still useful biomedical entities. ## biolama in this section, we detail the construction of bi-olama including the data curation process and pre-processing steps. statistics and examples of biolama are shown in ## conclusion in this work, we explore the possibility of using lms as biomedical kbs. to this end, we release bi-olama as a probing benchmark to measure how much biomedical knowledge can be extracted from lms. while biomedical lms can extract useful facts to some extent, our analysis shows that this is largely due to their predictions being biased towards certain prompts. in future work, we plan to overcome the underlying challenges in bio-lama and improve the probing accuracy of lms. automatically generated prompts. in proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), pages 4222-4235, online. association for computational linguistics. mujeen sung, hwisang jeon, jinhyuk lee, and jaewoo kang. 2020. biomedical entity representations with synonym marginalization. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 3641-3650, online. association for computational linguistics. houcemeddine turki, thomas shafee, mohamed ali hadj taieb, mohamed ben aouicha, denny vrandečić, diptanshu das, and helmi hamdi. 2019. wikidata: a large-scale collaborative ontological medical database. journal of biomedical informatics, 99:103292. andra waagmeester, gregory stupp, sebastian burgstaller-muehlbacher, benjamin m good, malachi griffith, obi l griffith, kristina hanspers, henning hermjakob, toby s hudson, kevin hybiske, et al. 2020. science forum: wikidata as a knowledge graph for the life sciences. elife, 9:e52614. zexuan zhong, dan friedman, and danqi chen. 2021. factual probing is [mask]: learning vs. learning to recall. in proceedings of the 2021 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 5017-5033, online. association for computational linguistics.