Can Knowledge Graph Embeddings Tell Us What Fact-checked Claims Are About? (2020)

Abstract: The web offers a wealth of discourse data that help researchers from various fields analyze debates about current societal issues and gauge the effects on society of important phenomena such as misinformation spread. Such analyses often revolve around claims made by people about a given topic of interest. Fact-checking portals offer partially structured information that can assist such analysis. However, exploiting the network structure of such online discourse data is as of yet under-explored. We study the effectiveness of using neural-graph embedding features for claim topic prediction and their complementarity with text embeddings. We show that graph embeddings are modestly complementary with text embeddings, but the low performance of graph embedding features alone indicate that the model fails to capture topological features pertinent of the topic prediction task.

## introduction analysing claims shared on social media is of growing interest, from social/political sciences to artificial intelligence (ai). such analyses are often performed with respect to a specific set of topics (e.g. "immigration" or "abortion") that allow carrying out targeted studies of trends, understanding/quantifying hidden biases @xcite , discovering stances towards those topics @xcite or their underlying falsehood propagation patterns @xcite . fact-checking portals offer a wealth of information about claims, their truth values and their sources. to analyse claims about a given topic, scientists need (1) access to heterogeneous repositories of claims and (2) the prior knowledge of which entities are mentioned in claims that belong to a topic (as defined by thematic keywords in each portal). as a (partial) response to (1), recent work has presented claimskg-a large dynamic knowledge graph (kg) of fact-checked claims harvested from various fact-checking portals (like politifact.com) and their metadata (e.g. truth values, authors, sources, links to dbpedia) @xcite (cf. figure 1 ). 1 claimskg includes thematic keywords provided by the fact-checking portals (e.g. "elections" or "taxes"). however, using them to filter claims by topic is problematic as: (1) not all claims are annotated; (2) the keywords are very heterogeneous (granularity or level of abstraction; e.g. "economy" vs. "kim kardashian"); ( 3 ) there is no standardization within or across portals; ## claim topic classification dataset we present our semi-automatic approach to build a gold standard dataset of annotated claims for topic classification. since claimskg covers a wide range of different topics, annotating a random sample of claims would not yield a sufficient number of claims per topic. thus, we identified a set of 7 topics that have a high number of claims in claimskg and are relevant for claim-related studies: "healthcare " (1777), "taxes" (1519), "elections" (1074), "crime" (947), "education" (1263), "immigration" (1147) and "environment" (567). we then automatically identify claims potentially referring to these topics using the keywords assigned by the fact-checking sites. first, we mapped all keywords to common high-level concepts in two upper level taxonomies: the thesoz thesaurus of social sciences @xcite and the unesco thesaurus 2 employing a dictionary-based entity linking approach (the concepts are noted as topic in figure 1 ). we then extracted a random subset of claims that are linked to at least one of the chosen topics through their keywords. note that one claim 2 http://vocabularies.unesco.org/thesaurus can correspond to several concepts thus creating a multi-label dataset. to validate and complete the semi-automatically assigned labels, we finally asked 5 annotators to re-annotate the dataset and assign the claims to all applicable topics. this gold standard, composed of 629 annotated claims, has a krippendorff's α annotator agreement (masi distance) @xcite of 0.75 which is a reasonably high agreement but also shows that the task is not trivial. for example, consider the claim "nobody is leaving memphis. that's a myth." uttered by a city councilman, with the keywords "population" and "census" assigned by the fact-checking site. 3 at first glance, none of the selected topics seems to apply. however, the claim review explains that this claim had been uttered in context of a debate concerning the fear that a proposed onetime tax for schools might make people leave the city with this claim defending the tax. thus, this claim may be interpreted as being about "taxes" and even "education", depending on how much of the pragmatic context is taken into account. in the final dataset foot_2 the topic distribution is the following: "healthcare" (25%), "taxes" (21%), "elections" (17%), "crime" (16%), "education" (13%), "immigration" (12%) and "environment" (10%). ## representation learning and evaluation pipeline graph embedding models. we train foot_3 a can-decom/parafac model with n3 regularization (cp-n3) @xcite . 6 we computed a model for claimskg (ckg) and a variant without keywords (ckg-kw) needed in the ablation studies. the link prediction performance, reported in ## evaluation and discussion comparison and combination of graph and text embeddings. we explore the performance of graph embedding vs. text embedding features and whether there is any complementarity of the two. we train and evaluate a ridge classifier (bayesian ridge regressor used as a classifier) 9 as per section 3 by using [(1) ckg] graph embedding features (claim left-hand side vector), [(2) tedr, (3) tegpt2] text embedding features (pooled token vectors) from distilroberta (dr) and gpt2, [(1) & ( 2 ), ( 1 ) & (3)] the combination of both (concatenation), as reported in the first segment of the baseline being the basis for the gold annotations prior to human annotation, it is expected to achieve a very high performance: given the poor performance of graph embedding features alone, it is likely that the model fails to capture these equivalence structures effectively. impact of neighbourhood features. the lhs claim embeddings did not capture much useful information for the task. given the local nature of the link prediction training criterion, do we need to consider the embeddings of the neighbourhood to find useful features that capture the equivalence structures of the baseline? for each neighbour (author, date, sources, mentions in review and claim), we retrieve the rhs and relation vectors. we aggregate by (1) flat concatenation (flat concat.); (2) concat. of triple vectors (claim lhs×relation×neighbour rhs -triple concat.). ablation studies. for the link prediction models, the most informative features arise from the claim/keyword/topic equivalence structures, as they are used to generate the graph baseline (81% accuracy). to understand if those structures are captured beyond relying on classification performance, we investigate three settings: (1) text embeddings of keywords only (kw only) (2) graph embedding without the keyword subgraph (no keywords, no topic concepts, in green in figure 1 -ckg no kw); (3) text embedding of all text fields (claim, review headline, author, keywords, date). discussion. we have been able to determine, as hypothesized that most of the useful information learned by the link prediction graph embeddings comes from the subgraph pertaining to keywords (green nodes in figure 1 ), however the overall resulting classification performance with only embedding features is low (with or without neighbourhood), especially compared to the baseline. one hypothesis could be that the structure of the keyword subgraph is captured to some extent in the embeddings of claims and in the neighbourhood, but since the link prediction performance itself is low compared to standard graphs, there is only some part of the structure that the graph embedding model manages to capture. of course, the size of the topic classification dataset plays a role in the classification performance, however if the representations learned on ckg (which is in no-way a small dataset by link prediction standards) were able to capture the relevant structures, we should be able to reach results closer to the baseline and to text embedding features (on the same dataset). in the setting of this controlled topic classification task, the structures in question are the equivalence cliques between claims, keywords and topic concepts, which are more complex than the direct links that the local link prediction objective is meant to capture. although recent advanced in link prediction make models capable of capturing specific formal properties of a relation (transitive, reflexive, anti-symmetric, etc.) in multi-relational graphs, they do not go beyond direct links. given that such models are increasingly used to infer new relations in complex kgs (e.g., in biomedical informatics), this is a significant limitation of using these approaches for the inference of complex relations or for a downstream classification task. ## conclusion and future work we evaluated the effectiveness of claim embeddings as features in a topic classification dataset, produced specifically to allow probing how specific features impact classification performance. we evaluate several strategies for feature retrieval from graph embeddings and combine them with text embedding features (flair + distilroberta/gpt2). we found a small complimentary between the features, however, the low accuracy resulting from using graph embeddings alone (compared to the baseline) and the ablation studies show that the graph embedding model's reliance on a local link prediction objective likely limits the ability of the model to capture more complex relationships (e.g. equivalence cliques between claims, keywords and topic concepts). this echoes some of the open-problems identified in the 2019 graph representation learning workshop at neurips @xcite . given that link prediction models are increasingly used with complex kgs to infer new relations (kg completion), this limitation is something to keep in mind and should drive researchers working on knowledge graphs to explore more general graph representation learning approaches such as graph neural networks or random-walk approaches.