Annotating P ub M ed Abstracts with M e SH Headings using Graph Neural Network (2023)

Abstract: The number of scientific publications in the biomedical domain is continuously increasing with time. An efficient system for indexing these publications is required to make the information accessible according to the user’s information needs. Task 10a of the BioASQ challenge aims to classify PubMed articles according to the MeSH ontology so that new publications can be grouped with similar preexisting publications in the field without the assistance of time-consuming and costly annotations by human annotators. In this work, we use Graph Neural Network (GNN) in the link prediction setting to exploit potential graph-structured information present in the dataset which could otherwise be neglected by transformer-based models. Additionally, we provide error analysis and a plausible reason for the substandard performance achieved by GNN.

## introduction many scientific publications are available on the internet, and the number of publications is continuously increasing with time. the digital library pubmed 2 currently consists of 33 million citations and is based on the medical database medline. the articles available on pubmed are indexed with concepts that come from the medical subject headings (mesh) ontology. the human and financial effort needed to keep up with the rapid pace of development is steadily increasing @xcite . there was a 5% increase in the number of citations in 2018 for medline. moreover, these citations are manually indexed with mesh headings, which cost $9.4 per citation on average. a large-scale biomedical semantic indexing task (10a) in the bioasq 3 challenge is designed to 1 github repository 2 pubmed website 3 bioasq website help develop systems that can automatically index pubmed publications using headings from the mesh ontology 4 . the fact that a publication can be assigned more than one mesh heading makes it a multi-label classification task. additionally, there are approximately 30k mesh headings which makes it an extreme multi-label classification task. gnn has been shown to achieve unprecedented performance on the benchmarks of link prediction and recommender systems @xcite . a considerable amount of real-world datasets contains latent graph-structured information that could be effectively exploited to improve performance by modeling the task as a graph-related task. the models proposed in previous versions of the bioasq challenge do not formulate the problem as gnn modeling, which can curtail the performance gain due to the omission of crucial graph-structured information present in the dataset. task 10a of the bioasq challenge is to assign mesh headings to pubmed articles based on the title and abstract of each article. in this work, we work on the following points to solve the problem. ## literature review the methods used for the task in the previous versions of the bioasq challenge can be classified into three categories @xcite . the first category named binary relevance consists of models such as metalabeler @xcite which uses linear svms in a one vs all multi-label classification setting to select the most probable mesh headings. the second category consists of models like deepmesh @xcite and mesh now @xcite which rely on the widely used information retrieval technique named learning to rank in order to obtain the most relevant mesh headings. the final category is based on deep learning e.g. meshprobenet @xcite and attentionmesh @xcite which uses rnn and attention mechanism to get the most probable mesh headings. most teams in the 2021 version of the bioasq challenge relied on contextualized language models, such as bert @xcite . the top performing model bertmesh @xcite al. (2020) also uses bert as a foundational model. ## methodolgy our proposed gnn model is implemented in the link prediction setting consisting of two modules, namely, document embedding module and link prediction module. ## dataset the dataset provided by the organizers of the 2022 version of the bioasq challenge for task 10a is composed of articles obtained from pubmed. the training dataset consists of 16,218,838 articles and 29,681 distinct mesh headings. mesh headings are the concepts that are part of the mesh ontology, which makes it easy to index and search medical and health-related information. each article is assigned 12.68 mesh headings on average. each human-annotated mesh heading has a unique id assigned to it, which needs to be predicted for each article. an example of the mesh heading named adult is shown in figure 1 where the heading is described by "scope note". the test set provided by the challenge organizers for the first week containing 9659 samples is used for testing. mesh headings are categorized into 16 categories, which are further divided into subcategories. each subcategory has a hierarchical depth of up to 13 where headings are organized from general to specific foot_1 . one important property of the mesh hierarchy is that it is a graph instead of a tree. in a tree, each node can have only one parent, which does not hold true in the case of the mesh hierarchy. figure 2 shows that the mesh heading named glucocorticoids has 2 parent nodes, namely adrenal cortex hormones and hormones. ## experimental setup taking into consideration the large size of the dataset, 70k articles are randomly sampled from the original training set to be used as the training set. additionally, the validation set of 10k samples is sampled from the original training set. the random sampling of a small subset of articles could lead to a training dataset that has a considerably different distribution than the original dataset, resulting in non-generalizable results. we tried to mitigate that by sampling the training articles using the mesh ontology, which is described further in appendix a. however, there was no improvement observed over the random sampling. therefore, we report results on randomly sampled training data to keep the method intelligible. ## results ## error analysis the results obtained using the focal loss indicate that the number of false positive predictions can be improved using methods that give more importance to hard negatives. the negative edges which are difficult to discern from the positive edges are called hard negatives. therefore, we assumed that the creation of hard negative samples improves the fp results and used dynamic random sampling @xcite and @xcite to add hard negatives during the training process instead of randomly sampling negative edges. for dynamic random sampling, we start adding the hard negatives after second epoch. for each article, 5 random negatives and up to 10 hard negatives are added. negative edges for which the dot product is too high (fp) are ignored in order to avoid the hardest negatives. to this end, negative edges which have dot product between 0.6 and 0.95 are considered hard negatives. the 2-dimensional representation of the embeddings obtained at the output layer of gnn model is shown in figure 3 . it can be observed that the hard negatives are closer to article embeddings in vector space as compared to the embeddings of remaining headings. to empirically verify our observation, we calculated the cosine similarity between the mean of article embeddings and the mean of hard negatives, which turns out to be -0.14. similarly, a cosine similarity of -0.75 was obtained between the mean of article embeddings and heading embeddings. although the model correctly selects the hard negatives as indicated by cosine similarity, the results obtained on the evaluation metrics do not surpass the results obtained using focal loss only. the second approach mixup uses a linear interpolation of the positive and negative samples to create hard negatives. the following equation is used to linearly interpolate article embedding e p and heading embedding e n to create hard negative e h . we set λ equal to 0.9 for the experimentation. this approach also yields no improvements over the results obtained using focal loss only. ## gnn limitations although gnn has improved performance on many tasks which benefit from graph-structured data, the architecture of gnn has some inherent limitations. one of the problems that neural networks has is that the performance is decreased as the number of layers is increased. the vanishing gradient coerces us to limit the number of layers, resulting in a shallow network that is not able to generalize. in addition to the vanishing gradient problem, the gnn model is limited to a small number of layers due to over-smoothing. @xcite have shown that the convolution operation of gnn is the source of its predictive power, but is also the cause of its limitation. they proved that the convolution operation of gnn is a kind of laplacian smoothing, which helps to learn new embedding from the neighboring nodes. however, the repeated application of laplacian smoothing results in the features of all nodes being identical, which deteriorates the predictive power of the model. as the number of layers increased, the nodes in the graph increasingly have similar neighbors to update their embeddings, resulting in identical nodes. the architecture of gnn has another limitation, named over-squashing. gnn is less effective on tasks that benefit from long-distance interactions. equation 1 shows a node update using neighborhood aggregation for a particular layer. it can be seen that as the number of layers increases, the receptive field also grows exponentially. therefore, the need for the model to encode information from long-distance neighbors creates a bottleneck because the model tries to cram too much information into a single vector. @xcite has shown that the information from exponentially growing k-hop neighbors for a k-layer gnn can not be crammed into a single vector representation, which results in low performance for tasks that require long-distance information. figure 4 illustrates the bottleneck while updating a node's feature representation based on its 3-hop neighbors. . additionally, the degree distribution of our bipartite graph follows a power law and is potentially scale-free graph @xcite . this also forces us to cram a lot of information into high-degree nodes. over-smoothing and negative sampling does not seem to be the main cause of low performance in our case. the potential reason for the superior performance of transformer-based models than gnn is the mitigation of the over-squashing problem. bertmesh avoids over-squashing by making a unique representation for each label using multilabel attention instead of making a single vector representation as described in the paper. this allows the model to avoid over-squashing, which leads to improved performance. ## conclusion taking into consideration, the need for an efficient system to automatically classify mesh headings, we implemented gnn in the link prediction setting. the use of advanced negative sampling strategies did not yield improved results. we highlighted the limitations of gnn and hypothesized that gnn is not able to generalize due to the over-squashing.