Knowledge Graph Generation From Text (2022)

Abstract: In this work we propose a novel end-to-end multi-stage Knowledge Graph (KG) generation system from textual inputs, separating the overall process into two stages. The graph nodes are generated first using pretrained language model, followed by a simple edge construction head, enabling efficient KG extraction from the text. For each stage we consider several architectural choices that can be used depending on the available training resources. We evaluated the model on a recent WebNLG 2020 Challenge dataset, matching the state-of-the-art performance on text-to-RDF generation task, as well as on New York Times (NYT) and a large-scale TekGen datasets, showing strong overall performance, outperforming the existing baselines. We believe that the proposed system can serve as a viable KG construction alternative to the existing linearization or sampling-based graph generation approaches.

## introduction automatic knowledge graph (kg) construction is an active research area aiming at representing the information present in abundant textual corpora in a more organized, structured and compressed form, which can be efficiently utilized in a variety of downstream applications, including reasoning, decision making, question answering, to name a few. however, this is a challenging problem due to the inherent non-unique graph representation (graph with n nodes can have n ! equivalent adjacency matrices), complex node and edge structure (node set is not fixed and edges are not binary), large output spaces (for graph with n nodes the system may need to output up to n 2 edges to specify its structure), lack of efficient architectures specialized for graph-structured generation output and limited parallel training data. the related problem of generating text from a given kg is generally more widely studied, with many suggested architectures and approaches. ## method in this section we cover the details of the proposed approach, first describing the functionality of the node generation in section 2.1, followed by the edge generation in section 2.3 and the discussion on edge imbalance problem in section 2.4. in fig. 2 we summarize all the architectural choices of the grapher system. the branches marked with a red cross denote the setups which in our earlier evaluations did not show advantage over the neighboring branch, e.g., the focal loss underperformed the sparse edge training for the text nodes combined with edge generation head. the branches with green check marks are the ones we select for further evaluation. the bold dark green check marks show two best performing systems across multiple experiments. in what follows, we now show the details of these choices. figure 2 : grapher architectural choices. ✕ -setups that did not show advantage or did not perform well during preliminary evaluations, ✓ -selected for further evaluation , ✔ -best performing system ## data to evaluate grapher's performance and compare it to the baselines, we use three datasets: two smallscale datasets: webnlg+ 2020 (ferreira et al., 2020) and nyt @xcite , and a largescale tekgen dataset from @xcite . ## experiments in this section we provide details about the model setups for evaluations, describe the scoring metrics, and present the results for both datasets. ## conclusion in this work, we proposed grapher, a novel multistage kg generation system, that separates the overall graph generation into two steps. in the first step, the nodes are generated from the input text using a pretrained language model. the resulting node features are then used for edge generation to construct the output graph. we proposed several architectural choices for each stage. in particular, graph nodes can either be generated as a sequence of text tokens or as a set of query-based feature vectors decoded into tokens through generation head (e.g., gru). edges can be either generated by a gru decoding head or selected by a classification head. we also addressed the problem of skewed edge distribution, where the token/class corresponding to the missing edge is over-represented, leading to inefficient training. for this, we proposed to use of either the focal loss, or the sparse adjacency matrix. the experimental evaluations showed that grapher matched state-of-the-art performance on smaller webnlg dataset, and showed strong overall performance, outperforming existing baselines, on nyt and tekgen datasets, serving as a viable alternative to the existing baselines.