Text-To- KG Alignment: Comparing Current Methods on Classification Tasks (2023)

Abstract: In contrast to large text corpora, knowledge graphs (KG) provide dense and structured representations of factual information. This makes them attractive for systems that supplement or ground the knowledge found in pre-trained language models with an external knowledge source. This has especially been the case for classification tasks, where recent work has focused on creating pipeline models that retrieve information from KGs like ConceptNet as additional context. Many of these models consist of multiple components, and although they differ in the number and nature of these parts, they all have in common that for some given text query, they attempt to identify and retrieve a relevant subgraph from the KG. Due to the noise and idiosyncrasies often found in KGs, it is not known how current methods compare to a scenario where the aligned subgraph is completely relevant to the query. In this work, we try to bridge this knowledge gap by reviewing current approaches to text-to-KG alignment and evaluating them on two datasets where manually created graphs are available, providing insights into the effectiveness of current methods. We release our code for reproducibility.

## introduction there is a growing interest in systems that combine the implicit knowledge found in large pretrained language models (plms) with external knowledge. the majority of these systems use knowledge graphs (kg) like conceptnet @xcite or freebase @xcite and either inject information from the graph directly into the plm @xcite @xcite @xcite or perform some type of joint reasoning between the plm and the graph, for example by using a graph neural network on the graph and later intertwining the produced representations @xcite @xcite . beyond their competitive performance, these knowledgeenhanced systems are often upheld as more interpretable, as their reliance on structured information can be reverse-engineered in order to explain predictions or used to create reasoning paths. one of the central components in these systems is the identification of the most relevant part of a kg for each natural language query. given that most kgs are noisy and contain idiosyncratic phrasings, which leads to graph sparsity @xcite , it is non-trivial to align entities from text with nodes in the graph. despite this, existing work often uses relatively simple methods and does not isolate and evaluate the effect of this component on the overall classification pipeline. furthermore, due to the lack of datasets that contain manually selected relevant graphs, it is not known how well current methods perform relative to a potential upper bound where the graph provides a structured explanation as to why the sample under classification belongs to a class. given that this problem applies to a range of typical nlp tasks, and subsequently can be found under a range of different names, such as grounding, etc., there is much to be gained from reviewing current approaches and assessing their effect in isolation. in this paper, we address these issues by providing an overview of text-to-kg alignment methods. we also evaluate a sample of the current main approaches to text-to-kg alignment on two downstream nlp tasks, comparing them to manually created graphs that we use for estimating a potential upper bound. for evaluation, we use the tasks of binary stance prediction @xcite , transformed from a graph generation problem in order to get gold reference alignments, and a subset of the choice of plausible alternatives (copa) @xcite that contain additional ex- ## background combining text with structured knowledge is a long-standing challenge in nlp. while earlier work focused more on the text-to-kg alignment itself, using rule-based systems and templates, recent work often approaches the problem as a part of a system intended for other nlp tasks than the alignment itself, such as question answering @xcite , language modelling @xcite ) and text summarization @xcite . as a consequence, approaches to what is essentially the same problem, namely to align some relevant subspace of a large kg with a piece of text, can be found under a range of terms, such as: retrieval @xcite @xcite , extraction @xcite , kg-totext-alignment @xcite , linking @xcite , grounding @xcite , and mapping @xcite . although it is natural to use multiple of these terms to describe a specific technique, we argue that it would be beneficial to refer to the task itself under a common name and propose the term text-to-kg alignment. the following sections formalise the task and discuss current approaches found in the literature. ## kg and datasets this section explains the data used in our own experiments. conceptnet as our knowledge graph, we use conceptnet @xcite -a generaldomain kg that contains 799, 273 nodes and 2, 487, 810 edges. the graph is structured as a collection of triples, each containing a head and tail entity connected via a relation from a pre-defined set of types. explagraphs explagraphs @xcite is originally a graph generation task for binary stance prediction. given a belief and argument pair (b,a), models should both classify whether the argument counters or supports the belief and construct a structured explanation as to why this is the correct label. an example of this can be seen in figure 3 . the original dataset provides a train (@xmath0, as well as a test set that is kept private for evaluation on a leaderboard. the node labels have been written by humans using free-form text, but the edge labels are limited to the set of relation types used in conceptnet. we concatenate the train and validation split and partition the data into a new train, validation and test split with an 80-10-10 ratio. @xcite , copa-sse adds semi-structured explanations created by human annotators to 1500 samples from balanced copa @xcite -which is an extension to the original copa dataset from @xcite . in this task, given a scenario as a premise, models have to select the alternative that more plausibly stands in a causal relation with the premise. an example with a manually constructed explanation graph can be seen in figure 4 . as with explagraphs, copa-sse uses free-form text for the head and tail entities of the triples and limits the relation types to the ones found in conceptnet. ## alignment approaches as mentioned, the general procedure for grounding text to a graph is three-fold: we first have to identify entities mentioned in the text, then link them to entities in the graph, and lastly construct a graph object that is returned to the inference model as additional context to be used together with the original text. for qa the text aligned with the graph is typically a combination of the question and answer choices. as our two downstream tasks are not qa, and also different from each other, we have to rely on different pre-processing techniques than previous work. the following section presents the implementation of three different text-to-kg alignment approaches that we compare against manually created graphs. an illustration of the different approaches applied to the same text sample can be seen in figure 2 . ## graph quality the following section provides an analysis of the quality of the different approaches when used to align graphs for both explagraphs and copa-sse. as can be seen from the tables, the approach based on the path generator (pg) from wang et al. (2020) (approach 3) returns fewer triples than the other approaches for both explagraphs and copa-sse. when using the entities from approach 2 as the start and end points, denoted by the abbreviation approach 3, the number of triples containing some form of alignment error is over twenty percent. when using the gold annotation as the start and end point of the pg, abbreviated approach 3-g, this goes down a bit but is still considerably higher than the approaches based on lexical overlap. approach 2 is able to identify some well-formatted triple in all of the cases for both tasks, while approach 1 fails to retrieve anything for five percent of the samples in copa-sse and two percent for explagraphs. in order to get some notion of semantic similarity between the different approaches and the original context they are meant to be a structural representation of, we calculate the cosine similarity between the context and a linearized (see section 4.4 for details on this procedure) version of the graphs. the scores can be found in we now present experiments where we compare the discussed approaches to text-to-kg alignment for explagraphs and copa-sse. as our plm, we use bert @xcite for all experiments. we use the base version and conduct a hyperparameter grid search for both tasks. we do the same search both with and without any appended graphs as the former naturally makes it easier to overfit the data, especially since both explagraphs and copa-sse are relatively small in size. the grid search settings can be found in appendix a.2 and the final hyperparameters in appendix a.3. we run all experiments over ten epochs with early stopping on validation loss with a patience value of five. as few-sample fine-tuning with bert is known to show instability @xcite , we run all experiments with ten random seeds and report the mean accuracy scores together with standard deviations. we use the same random seeds for both tasks; they can be found in appendix a.4. we find that the experiments are highly susceptible to seed variation. although we are able to match the performance of some previous work for the same plm on some runs, this does not hold across seeds. consequently, we also perform outlier detection and removal. details on this procedure can be found in appendix a.5. ## results for both tasks, having access to manually created graphs improves performance significantly. ## discussion the most striking result is perhaps the performance of approach 3-g on copa-sse. we hypothesise that this can be explained by the fact that annotators probably used exact spans from both the premise and the correct alternative from the text in their graphs, and consequently, they provide a strong signal as to why there is a relation between the premise and the correct answer choice and not the wrong one. this is easily picked up by the model. for explagraphs, which is a text classification problem, this is not the case: the appended graph might provide some inductive bias, but it does not provide a direct link to the correct choice, as the task is to assign a label to the whole sequence, not to choose the most probable sequence out of two options. this conclusion is further supported by the observation that appending the manually constructed graphs in their entirety has a much larger effect on copa-sse than explagraphs. furthermore, for copa-sse, as pointed out in the key finding of our experiments is that having access to structured knowledge relevant to the sample at hand, here represented by the gold annotations, provides a significant increase in performance even with a simple injection technique and judging by today's standards, a small pre-trained language model. they also show that for datasets of low sample sizes, such as explagraphs and copa-sse, the results are susceptible to noise. as the approaches based on lexical overlap are within the standard deviations of the experiments without any appended graphs, it is not possible to conclude that they add any useful information to the model. based on figure 6 , we think it is fair to conclude that these methods based on lexical overlap only provide a signal that has no relation to the correct label. as to why the approaches based on lexical matching do not have any effect here but reportedly have an effect in previous work on qa, there is one major reason that has not been discussed so far: namely that both datasets require knowledge that is not represented in conceptnet. as shown by @xcite , matching the task with the right kg is important. it is reasonable to question whether or not conceptnet, which aims to represent commonsense and world knowledge, does indeed contain information useful for deciding whether or not an argument counters or supports a belief, in the case of explagraphs, or if it can aid in the selection of the most likely follow-up scenario to a situation, in the case of copa-sse. in figure 5 , both the approaches based on lexical overlap (1 & 2) align the same exact graph with the text context, and judging from the result, it is pretty clear that the aligned graph has little to offer in terms of guiding the model towards the most likely follow-up. ## conclusion in this work, we find that the process of identifying and retrieving the most relevant information in a knowledge graph is found under a range of different names in the literature and propose the term text-to-kg alignment. we systematise current approaches for text-to-kg alignment and evaluate a selection of them on two different tasks where manually created graphs are available, providing insights into how they compare to a scenario where the aligned graph is completely relevant to the text. our experiments show that having access to such a graph could help performance significantly, and that current approaches based on lexical overlap are unsuccessful under our experimental setup, but that a generative approach using a plm to generate a graph based on manually written text as start and end entities adds a significant increase in performance for multiple-choice type tasks, such as copa-sse. for the approaches based on lexical overlap, we hypothesise that the lack of performance increase can be attributed to the choice of knowledge graph, in our case conceptnet, which might not contain any information useful for solving the two tasks.