Biomedical Document Classification with Literature Graph Representations of Bibliographies and Entities (2023)

Abstract: This paper proposes a new document classification method that incorporates the representations of a literature graph created from bibliographic and entity information. Recently, document classification performance has been significantly improved with large pre-trained language models; however, there still remain documents that are difficult to classify. External information, such as bibliographic information, citation links, descriptions of entities, and medical taxonomies, has been considered one of the keys to dealing with such documents in document classification. Although several document classification methods using external information have been proposed, they only consider limited relationships, e.g., word co-occurrence and citation relationships. However, there are multiple types of external information. To overcome the limitation of the conventional use of external information, we propose a document classification model that simultaneously considers bibliographic and entity information to deeply model the relationships among documents using the representations of the literature graph. The experimental results show that our proposed method outperforms existing methods on two document classification datasets in the biomedical domain with the help of the literature graph.

## introduction document classification has improved significantly with large language models, such as bidirectional encoder representations from transformers (bert) @xcite . however, these methods use only text information and ignore a lot of information behind the text, such as bibliographic information (e.g., authors and publishing journals), citation information, and information about entities that appear in the target text, which can be considered as one of the keys to better classification. with the help of neural models, several document classification methods have been proposed that use both text information and external information. @xcite classified documents using representations of a text graph. in the text graph, nodes correspond to papers and words, and edges are connected between word nodes and their paper nodes and between word nodes with high co-occurrence frequency. the text graph allows paper nodes to be connected through the common word nodes and makes document classification take into account the relationships among papers. bert-gcn @xcite further adds text information to the node representations of the text graph using bert. @xcite proposed a pretrained language model linkbert that considers the relationship between documents. the inputs of the model are texts from documents in a citation relationship in pre-training. linkbert achieved higher performance than existing bert models. although these studies indicate the effectiveness of external information other than text information, they only consider limited information, such as the co-occurrence of words and citation relationships, and do not simultaneously consider multiple types of external information. this paper proposes a document classification model that incorporates multiple types of external information into the target text information. specifically, we first build the literature graph using bibliographic information, including authors and publishing journals, and entity information, including descriptions of entities and their taxonomic information. then, we create representation vectors of the nodes that consider various relationships among different types of nodes in the literature graph so that the vectors can contain multiple types of external information. finally, we build a document classification model that receives both the representation vectors and the target text information as input. the contributions of this paper are as follows: ## proposed method this section proposes a novel document classification model that incorporates a representation of a literature graph with bibliographic and entity information. figure 1 shows an overview of the proposed method. we first explain the definition and representation learning of the literature graph in sections 3.1 and 3.2, respectively. we then introduce a document classification model that uses both vector representations of the literature graph containing bibliographic and entity information, and the target text information is described in section 3.3. ## experimental settings the quality of the representation vectors obtained by representation learning of the literature graph is evaluated by link prediction, which predicts the target nodes that are related to the node. we also evaluated document classification using the representation of the literature graph to confirm the effectiveness of the literature graph in document classification. we used biolinkbert-base (yasunaga et al., 2022) in both experiments. ## analysis to check the effects of the representation vectors of the literature graph and the entity graph alone, we conducted two experiments: document classification with masking the entities in the target text, which eliminates the impact of entity information in the text, and document classification with no text information, which eliminates the impact of paper and entity information in the text. the overview of these experiments is shown in figure 3 . the results of each experiment are shown in ## conclusions in this study, for the purpose of document classification that can use multiple types of information at the same time, we proposed a document classification model that creates a representation vector from a literature graph that contains a lot of information, such as bibliographic and entity information, considering various relationships between documents, and incorporates the representation vector and textual information about the documents. experimental results on two datasets, ohsumed and hoc, show that both models improve performance with the information from the literature graph, and the models show state-of-the-art performance on both datasets. we also found that the performance degraded when we simultaneously used multiple information of paper and entity, showing that the incorporation of different types of information is not simple. in the future, we will investigate representation learning of large-scale literature graphs using methods such as gcn to obtain better representations. we will also explore methods for simultaneously learning representations of literature graphs and document classification.