S ci C oncept M iner: A system for large-scale scientific concept discovery (2021)

Abstract: Scientific knowledge is evolving at an unprecedented rate of speed, with new concepts constantly being introduced from millions of academic articles published every month. In this paper, we introduce a self-supervised end-to-end system, SciConceptMiner, for the automatic capture of emerging scientific concepts from both independent knowledge sources (semi-structured data) and academic publications (unstructured documents). First, we adopt a BERT-based sequence labeling model to predict candidate concept phrases with self-supervision data. Then, we incorporate rich Web content for synonym detection and concept selection via a web search API. This two-stage approach achieves highly accurate (94.7%) concept identification with more than 740K scientific concepts. These concepts are deployed in the Microsoft Academic production system and are the backbone for its semantic search capability.

## introduction scientific knowledge has been expanded at an exponential rate over the past decades and the fastgrowing volume of academic literature accentuates a pressing need for automated capture of finegrained emerging concepts. statistical topic models @xcite , such as latent dirichlet allocation (lda) @xcite , have been wellrecognized for automatically extracting the topic structure of large document collections for past decades. however, it has two main limitations to prevent it from being widely applied in a modern large-scale document collection. first, it is the scalability issue on the number of topics an lda can model. the latest development @xcite can process 131m documents with 28b tokens efficiently, however, it only extracts 1,722 topics. with the fast-growing body 1 https://academic.microsoft.com/ of scholarly communications, a comprehensive manually controlled vocabulary like medical subject headings(mesh) @xcite ) contains tens of thousands of subjects (concepts) mostly in the bio-med domain; and an automated scientific knowledge exploration system such as microsoft academic graph (mag) @xcite has hundreds of thousands of topics across all academic disciplines. a topic modeling system that is scalable not only to the size of documents but also to the number of topics is imperative. second, the result of an lda model is a list of frequency-based terms that form a topic. it requires manual efforts to annotate such lists to generate a human-readable theme or topic name. an automatic process of identifying topic themes with authoritative names and meaningful descriptions is desired to reduce costly human interventions. in this paper, we introduce a self-supervised end-to-end system, sciconceptminer, for automatically discovering scientific concepts from both semi-structured independent knowledge sources and unstructured academic documents. it first obtains a list of concept candidates, either from external knowledge repositories such as wikipedia @xcite and unified medical language system (umls) @xcite , or directly mining concepts from a collection of academic documents. such concept lists are large and noisy. they are in the scale of millions and dominated by invalid or duplicate terms. we then send these candidates as queries to a search engine api and leverage rich web content to identify legitimate concepts, cluster synonyms, and discard improper terms. the search api is also used to retrieve highquality concept descriptions. one example is shown in figure 1 . 2 four out of five trending topics (network embedding, triplet loss, network representation learning, and zero shot learning) under embedding are extracted by our automatic concept extractor model trained on cs corpus. it demonstrates that our designed model can effectively capture the emerging trending topics from the latest scientific articles. the sciconceptminer has been deployed to identify concepts from millions of scholarly communications in microsoft academic graph (mag) @xcite @xcite . the mag with the full list of 740k scientific concepts can be freely accessed via the microsoft academic foot_1 search website and mag data set foot_2 . ## system description as shown in figure 2 , the sciconceptminer system has two stages: the first is the concept candidates discovery from various data sources; the second is synonym detection and concept clustering via a web search api. in the concept candidates discovery stage, we first integrate the semi-structured independent knowledge sources, wikipedia and umls, into the system. such an existing concept list in the system with associated documents enables us to train a concept extractor learning model with selfsupervision. we design a bert-based sequence labeling model to make a binary prediction on whether a word or phrase in a sentence is a scientific concept or not. this proposed model is trained on self-supervised data generated from existing concepts (from wikipedia and umls) tagged to a collection of academic documents. we do the concept inference with the trained model to generate concept candidates for the next stage. concept candidates, as the input to the second stage, are either from external knowledge sources or inferred from academic documents. both sources have high noisy ratios with different natures. the independent source such as wikipedia has high-quality entities (well-defined names and descriptions, rare duplication, and rich links and relationships with each other) but type noisy (many other types of entities than academic concepts). the umls candidates and the inferred candidates from an unstructured corpus have more irrelevant phrases and concept synonyms. with the help of a search engine api to retrieve top n documents by using concept candidates as queries, we analyze the returning web pages and associated url domain information collectively. this process would iden-tify around 3-5% of candidates from the first stage as proper scientific concepts with consistently high accuracy (94-95% based on sample results) across all data sources, with over 740k concepts in total. ## conclusion in this work, we demonstrated a large-scale scientific concept discovery production system, scicon-ceptminer, for automatically capturing academic concepts from both semi-structured data and unstructured documents. the system has two parts: the first is the concept candidate identification, and the second is synonym detection and concept selection. we used a bert-based sequence labeling model to learn concept phrases with selfsupervision and leverage a web search api to cluster synonyms and identify valid concepts. sciconceptminer has discovered more than 740k scientific concepts across all research domains from wikipedia, umls, and scholarly articles with high accuracy (94.7%). these concepts are integrated to build the microsoft academic graph, which publishes one of the largest crossdomain scientific taxonomy. it enables easy exploration of scientific knowledge as well as facilitates many downstream applications like information retrieval, question answering, and recommendations.