K en M e SH : Knowledge-enhanced End-to-end Biomedical Text Labelling (2022)

Abstract: Currently, Medical Subject Headings (MeSH) are manually assigned to every biomedical article published and subsequently recorded in the PubMed database to facilitate retrieving relevant information. With the rapid growth of the PubMed database, large-scale biomedical document indexing becomes increasingly important. MeSH indexing is a challenging task for machine learning, as it needs to assign multiple labels to each article from an extremely large hierachically organized collection. To address this challenge, we propose KenMeSH, an end-to-end model that combines new text features and a dynamic knowledge-enhanced mask attention that integrates document features with MeSH label hierarchy and journal correlation features to index MeSH terms. Experimental results show the proposed method achieves state-of-the-art performance on a number of measures.

## introduction the pubmed foot_0 database is a resource that provides access to the medline bibliographic database of references and abstracts together with the full text articles of some of these citations which are available in the pubmed central foot_1 (pmc) repository. medline foot_2 contains more than 28 million references (as of feb. 2021) to journal articles in the biomedical, health, and related disciplines. journal articles in medline are indexed according to medical subject headings (mesh) 4 , an hierarchically organized vocabulary that has been developed and maintained by the national library of medicine (nlm) 5 . currently, there are 29,369 main mesh headings, and each medline citation has 13 mesh indices, on average. mesh terms are distinctive features of medline and can be used in many applications in biomedical text mining and information retrieval @xcite @xcite , being recognized as important tools for research (e.g., knowledge discovery and hypothesis generation). currently, mesh indexing is done by human annotators who examine full articles and assign mesh terms to each article according to rules set by nlm 6 . human annotation is time consuming and costly -the average cost of annotating one article in medline is about $9.40 @xcite . nearly 1 million citations were added to medline in 2020 (approximately 2,600 on a daily basis) 7 . the rate of articles being added to the medline database is constantly increasing, so there is a huge financial and time-consuming cost for the status quo. therefore, it is imperative to develop an automatic annotation system that can assist mesh indexing of large-scale biomedical articles efficiently and accurately. automatic mesh indexing can be regarded as an extreme multi-label text classification (xmc) problem, where each article can be labeled with multiple mesh terms. compared with standard multilabel problems, xmc finds relevant labels from an enormous set of candidate labels. the challenge of large-scale mesh indexing comes from both the label and article sides. currently, there are more than 29,000 distinct mesh terms, and new mesh terms are updated to the vocabulary every year. the frequency of different mesh terms appearing in documents are quite imbalanced. for instance, the most frequent mesh term, 'humans', appears in more than 8 million citations; 'pandanaceae', on the other hand, appears in only 31 documents @xcite . in addition, the mesh terms that have been assigned to each article varies greatly, ranging from more than 30 to fewer than 5. furthermore, semantic features of the biomedical literature are complicated to capture, as they contain many domain-specific concepts, phrases, and abbreviations. the aforementioned difficulties make the task more complicated to generate an effective and efficient prediction model for mesh indexing. in this work, inspired by the rapid development of deep learning, we propose a novel neural architecture called kenmesh (knowledge-enhanced mesh labelling) which is suitable for handling xmc problems where the labels are arrayed hierarchically and could capture useful information as a directed graph. our method uses a dynamic knowledge-enhanced mask attention mechanism and incorporates document features together with label features to index biomedical articles. our major contributions are: ## proposed model mesh indexing can be regarded as a multi-label text classification problem in which, given a set of biomedical documents @xmath0, x 2 , ..., x n } and a set of mesh labels @xmath1, y 2 , ..., y l }, multi-label classification learns the function f : @xmath2, 1] y using the training set @xmath3, y i ), @xmath4, ..., n, where n is the number of documents in the set. figure 1 illustrates our overall architecture. our model is composed of a multi-channel document representation module, a label features learning module, a dynamic semantic mask attention module, and a classifier. ## results and ablation studies we evaluate our proposed model with five state-of-the-art models: mti, deepmesh, fullmesh, bertmesh and hgcn4mesh. among these, mti, deepmesh, bertmesh, and hgcn4mesh are trained with abstracts and titles only; fullmesh (full) and bertmesh (full) are trained with full pmc articles. our proposed model is trained on titles and abstracts, and is tested using 20,000 of the latest articles. we mainly focus on mif, which is the main evaluation metric in mesh indexing task. we compare our model against previous related systems on micro-average measure and examplebases measure in impacts on multi-channel settings as shown in ## conclusion we propose a novel end-to-end model integrating document features and label hierarchical features for mesh indexing. we use a novel dynamic knowledge-enhanced mask attention mechanism to handle the large universe of candidate mesh terms and employ gcn in extracting label correlations. experimental results demonstrate that our proposed model significantly outperforms the baseline models and provides especially large improvements on infrequent mesh labels. in the future, we believe two important research directions will lead to further improvements. first, we plan to explore full text articles, which contain more information, to see whether our model takes advantage of the full text to improve the performance of large-scale mesh indexing. second, we are interested in integrating knowledge from the unified medical language system (umls) @xcite , a comprehensive ontology of biomedical concepts, in our model.