Beyond Text: Incorporating Metadata and Label Structure for Multi-Label Document Classification using Heterogeneous Graphs (2021)

Abstract: Multi-label document classification, associating one document instance with a set of relevant labels, is attracting more and more research attention. Existing methods explore the incorporation of information beyond text, such as document metadata or label structure. These approaches however either simply utilize the semantic information of metadata or employ the predefined parent-child label hierarchy, ignoring the heterogeneous graphical structures of metadata and labels, which we believe are crucial for accurate multi-label document classification. Therefore, in this paper, we propose a novel neural network based approach for multi-label document classification, in which two heterogeneous graphs are constructed and learned using heterogeneous graph transformers. One is metadata heterogeneous graph, which models various types of metadata and their topological relations. The other is label heterogeneous graph, which is constructed based on both the labels’ hierarchy and their statistical dependencies. Experimental results on two benchmark datasets show the proposed approach outperforms several state-of-the-art baselines.

## introduction with the rapid growth of scientific documents, it is difficult to track related literature manually. for example, there are more than 200,000 publications related to covid-19 by april 2021. therefore, it is crucial to automatically assign publications with their corresponding categories. multi-label document classification (mldc), associating one document instance with a set of most relevant labels, is attracting more and more research attention. to tackle the problem, early work focused on learning semantic representations of the input text using some text encoders. for example, liu et al. (2017) proposed the xml-cnn model using a convolutional neural network and @xcite proposed the attentionxml model using a recurrent neural network as text semantic encoder. recently, @xcite proposed the x-transformer model, a deep transformer model fine-tuned for mldc. different from the aforementioned approaches, there have been attempts exploring information beyond text for mldc. on the one hand, the information associated with labels such as label semantics and the relationships between labels are employed. for example, @xcite generated labelspecific document representation using the label semantic information. @xcite improved the classification performance by constructing a hierarchical label tree. to model label dependency, mldc is cast as a seq2seq task @xcite . on the other hand, the document metadata is incorporated. for example, to employ the metadata information, the representation of the document and its metadata are learned in the same embedding space @xcite . the label hierarchy is also applied to regularize the output probability of each child label by its parents. however, existing methods have two limitations: to tackle the above limitations, in this paper, we propose a novel neural network based approach for multi-label document classification using two heterogeneous graphs. specifically, a metadata heterogeneous graph with four node types and five edge types is constructed to capture the metadata information and a label heterogeneous graph with two edge types is constructed to capture both the label hierarchy and labels' statistical dependencies. both graphs are learned using the heterogeneous graph transformer. moreover, to fully utilize the label information, the label-specific document representation is learned. the main contributions of this paper are listed as follows: ## related work based on the information utilized for model learning, approaches for multi-label document classification can be categorized into two types: using textual information only or additionally incorporating external information. for approaches solely based on textual information, early attempts @xcite employed bag-of-words representations. recently, deep learning based approaches were proposed to learn better text representations. for example, @xcite proposed to use a convolutional neural network for text encoding. @xcite proposed a neural network approach with the attention mechanism to capture the most relevant part of the input text for each label. @xcite employed a pre-trained transformer @xcite to capture textual information for text classification. however, such methods ignore the information beyond text, which we believe is crucial for accurate multi-label document classification. other approaches attempted to incorporate external information, which can be further classified into two categories, using the document metadata and using label information. for approaches utilizing metadata, @xcite proposed a neural network approach for sentiment analysis which incorporates user and product meta information by a vector space model. @xcite employed categorical metadata signals as additional features to train a deep neural network classifier. @xcite developed an approach called match, which pre-trained the embeddings of text and metadata in the same space, and used the transformer to capture the relationship between them. nevertheless, these methods ignored the heterogeneous structure of a document's metadata. approaches using label information have considered label semantic information, label hierarchy and label dependency. to incorporate label semantic information, @xcite proposed an interactive mechanism that incorporates word-level matching signals into the text classification task. lsan @xcite leveraged label semantic information to determine the semantic connection between each label and the document. pappas and henderson (2019) proposed the gile model, which is a joint input label embedding model for neural text classification. to incorporate label hierarchy, gopal and yang (2015) employed a recursive regularization to encourage the similarity between the child classifiers and their parent classifier. peng et al. (2018) further extended this regularization to a text semantic encoder based on graph convolutional neural networks. zhang et al. (2018) constructed a label dependency graph to model the label embedding in the label space based on the graph prior. yang et al. (2018) converted multilabel document classification into a seq2seq task, and predicted label sequences with label dependence. zhang et al. ( 2021 ) proposed the match model and employed different ways to regularize the parameters and output probability of each child label by its parents. however, these methods cannot jointly consider label semantic information, label hierarchy, and label dependency. our approach is partly inspired by match @xcite , but with the following differences: (1) the heterogeneous structure of the document's metadata is modeled in the proposed approach, which is ignored in @xcite ; and (2) the proposed approach incorporates the implicit statistical dependencies between labels, which are not considered in @xcite . ## conclusions we propose a novel neural network approach for multi-label document classification, in which two heterogeneous graphs are incorporated and learned using heterogeneous graph transformers. one is the metadata heterogeneous graph, which models various type of metadata and their topological relations. the other is the label heterogeneous graph, which is constructed based on the labels' hierarchy and statistical dependency. experiments on two datasets show the superior performance of the proposed approach over existing approaches. in addition, results of the ablation experiments show the effectiveness of incorporating both the metadata heterogeneous graph and the label heterogeneous graph for multi-label document classification.