Content- and Topology-Aware Representation Learning for Scientific Multi-Literature (2023)

Abstract: Representation learning forms an essential building block in the development of natural language processing architectures. To date, mainstream approaches focus on learning textual information at the sentence- or document-level, unfortunately, overlooking the inter-document connections. This omission decreases the potency of downstream applications, particularly in multi-document settings. To address this issue, embeddings equipped with latent semantic and rich relatedness information are needed. In this paper, we propose SMRC ^{2} , which extends representation learning to the multi-document level. Our model jointly learns latent semantic information from content and rich relatedness information from topological networks. Unlike previous studies, our work takes multi-document as input and integrates both semantic and relatedness information using a shared space via language model and graph structure. Our extensive experiments confirm the superiority and effectiveness of our approach. To encourage further research in scientific multi-literature representation learning, we will release our code and a new dataset from the biomedical domain.

## introduction with the increasing amount of scientific publications, researchers progressively turn to intricate natural language processing (nlp) tools for paper discovery and recommendation. however, most existing tools are tailored towards specific downstream tasks @xcite and thus their capabilities seem limited when presented with more complex tasks. this is attributed to the focus on either the content or topology information of scholarly documents, instead of both. an aspiring scholarly search engine offers researchers with the capacity to comprehend query topics and access multiple appropriate publications, or even more innovatively, a synthesized summary of the overall set of sources. such capabilities have the potential to revolutionize the modern research process @xcite . unfortunately, existing methods only focus on single document representation learning (sdrl) and the recommended paper are selected based on the rank of similarity between the learned representation and the embedding of input query.while language models, such as bert family @xcite @xcite and gpt family @xcite , has seen tremendous progress in understanding scientific documents @xcite and graph-based approaches like graphformer @xcite , specter @xcite have studied the relatedness of a scientific paper in its citation networks, little effort has been devoted to exploring scientific literature representation learning at the multi-document level (mdrl). in this paper, we extend the community's attention of representation learning to multi-document level and explore how to learn both semantic and relatedness information from content and topological networks. intuitively, knowledge entities and text content of the given paper can provide latent semantic information. for example, we could simply infer a paper belongs the biomedical domain by knowing its entities such as "diabetes", "gpx1" (a gene), etc. however, there is a risk of incorrect inference when a computer science paper takes biomedical instances as examples. same situation for using text content only, one can classify a paper simply by reading its abstract. unfortunately, an nlp model will be hallucinated due to the close lexical similarity of papers from different categories. thus, how to jointly learn from both entities and paper content can be critical to provide latent semantic information. typically, entity-relation triplets will be modeled in the form of a graph which poses a challenge of balancing the learning between graph neural networks (gnns) and language models (lms). we novelly introduce wasserstein distance to enable gnn and lm in parallel and interactively learn from each other while ensuring the learning of the latent semantic information. by leveraging the multi-document relations, e.g., citation graph, graph neural networks can simply enable each scientific paper to learn from its neighbors via the message passing mechanism. for example, @xcite proposed a citation graphbased memorization model (cgsum) to incorporate the information of both the source paper and its references to obtain summarization. @xcite utilized a disentangled representation based model disencite to encode paper according to both textual contexts and structure information for citation recommendation and generation. these works share resemblances to our method since they also take multi-document as input. however, our work aims to explore output representation equipped with latent semantic and rich relatedness information for each document, as well as a unified embedding for the entire input. this approach enables the learned representations not to be limited to specific tasks. inspired by the effective masking strategy @xcite , we propose to use a graph neural network equipped with a novel masking strategy to enable each document to learn the relatedness of its neighbors. to this end, we propose smrc 2 which learns the scientific multi-literature representation from both content information under wasserstein constraint and citation information via graph structure. the main contributions of our work are a threefold: ## related work representation learning most existing representation learning works in nlp focus on capturing the dependency in word-level, token-level and sentence-level @xcite @xcite . with the success in previous work, some efforts have been made to extend the learning paradigm to document-level @xcite . however, these methods suffer from the massive noises introduced by irrelevant words/sentences. several context-based and graph-based models are proposed @xcite to reduce noises in the training stage. given a document and its salient entities, for example, graph-based models can easily build a graph where the entities are nodes and their relations are edges to derive/infer the knowledge within this graph. unfortunately, these methods lack the consideration of rich relatedness in citation networks. recent works @xcite @xcite push the boundary forward by including citation information. these works are orthogonal to ours as the input to our model is also multi-literature with citation relations, but the difference is that they leverage citation information for enhancing the source document's representation while our work is forcing each document to learn both semantic and relatedness information and at the same time, output a unified representation for the entirety. further, research work @xcite @xcite @xcite proposed to tackle the text-attributed graph learning task (tag) are also similar to our approach which aims at better schema for jointly graph and language model learning. however, our work goes beyond these approaches by explicitly integrating semantic and relatedness information in scientific multi-literature representation. multi-document as input in real-world scientific document applications, obtaining information from multiple documents is a common requirement. researchers may want to know the whole story of a topic without reading every paper related to it, which falls under the subfield of nlp known as multi-document summarization (mds). while other multi-document tasks, such as multidocument reading comprehension, which aims to understand the content of multiple documents, are also of interest, our focus in this paper is on mds to evaluate the learned multi-document rep-resentation. extractive methods usually produce a summary by selecting the ranked sentences from the given document set @xcite @xcite . for example, @xcite treats mds as a sequence classification problem where each sentence is visited in sequential order and then adopts an rnnbased model to decide whether the sentence should be included in the summarization or not. @xcite conceptualizes mds to induce a multi-root dependent tree representation of the documents. however, similar sentences may be close in the vector space and share close scores which would cause redundancy problem @xcite . recently, graph-based approaches which aim to extract salient textual units from documents based on graph structure representations of sentences are proposed to eliminate this @xcite . different from these works, our method leverages document representation with latent semantic information as nodes and their citation relations as edges to construct the multi-document level graph. then, the message passing mechanism can be resorted to equipping relatedness information for the final representation which can be applied in multi-document tasks. ## methodology problem formulation before going further, we first give preliminary definitions of concepts in this paper. let @xmath0, ......, p n } denotes a paper group with n papers and g @xmath1, a, x ) is the multi-doc graph of p with each node v @xmath2, @xmath3, 1} n×n is the adjacency matrix and @xmath4. moreover, given f ge as the graph encoder, f lm as the language model, and f gd as the graph decoder, our goal here is to obtain the document representation h i as well as the multi-document representation h m as follow: overview as can be seen in figure 1 , smrc foot_0 consists of two modules: semantic representation learning module and relatedness representation learning module. in the semantic learning module, smrc 2 first in parallel obtain the entity-relation graph embedding via a gnn encoder and the abstract text 2 embedding via a language model. these two levels of information are then concatenated to form the initialized node features, along with their citation relations, to create a multi-doc graph. to learn rich relatedness information, we randomly mask some of their citation relations and train a gnn-based decoder to recover them based on the node features using the message passing mechanism. by leveraging both the latent semantic and rich relatedness information, smrc 2 is capable of not only single document but also multi-document applications. ## experiments in this section, we empirically evaluate the performance of our learned document representation h i and the unified multi-doc representation h m . ## conclusion as a conclusion, we have advanced the field of representation learning to the multi-doc level for scientific literature, with the introduction of smrc 2 . this novel approach effectively combines two elements -semantic content and topological relatedness, to create a superior learning space. to address the imbalance learning problem that occurs during the semantic learning stage, we introduced wasserstein distance. to our best knowledge, this is the first attempt to introduce wd for multi-view content learning. the extensive experiments and ablation study that we conducted validate the effectiveness of the proposed model. additionally, we released a new dataset to encourage the community to further explore the multi-document representation learning. these contributions advance research in scientific literature mining and provide a useful resource for the community to further investigate this new but important topic.