Fine-grained Fact Verification with Kernel Graph Attention Network (2020)

Abstract: Fact Verification requires fine-grained natural language inference capability that finds subtle clues to identify the syntactical and semantically correct but not well-supported claims. This paper presents Kernel Graph Attention Network (KGAT), which conducts more fine-grained fact verification with kernel-based attentions. Given a claim and a set of potential evidence sentences that form an evidence graph, KGAT introduces node kernels, which better measure the importance of the evidence node, and edge kernels, which conduct fine-grained evidence propagation in the graph, into Graph Attention Networks for more accurate fact verification. KGAT achieves a 70.38% FEVER score and significantly outperforms existing fact verification models on FEVER, a large-scale benchmark for fact verification. Our analyses illustrate that, compared to dot-product attentions, the kernel-based attention concentrates more on relevant evidence sentences and meaningful clues in the evidence graph, which is the main source of KGAT’s effectiveness. All source codes of this work are available at https://github.com/thunlp/KernelGAT .

## introduction online contents with false information, such as fake news, political deception, and online rumors, have been growing significantly and spread widely over the past several years. how to automatically "fact check" the integrity of textual contents, to prevent the spread of fake news, and to avoid the undesired social influences of maliciously fabricated statements, is urgently needed for our society. recent research formulates this problem as the fact verification task, which targets to automatically verify the integrity of statements using trustworthy corpora, e.g., wikipedia @xcite . for example, as shown in figure 1 , a system could first al jardine is an american rhythm guitarist claim verification supports refutes not enough info evidence reasoning alan charles jardine (born september 3, 1942) is an american musician, singer and songwriter who cofounded the beach boys. he is best known as the band's rhythm guitarist, and for occasionally singing lead vocals on singles. retrieve related evidence sentences from the background corpus, conduct joint reasoning over these sentences, and aggregate the signals to verify the claim integrity @xcite @xcite . there are two challenges for evidence reasoning and aggregation in fact verification. one is that no ground truth evidence is given; the evidence sentences are retrieved from background corpora, which inevitably contain noise. the other is that the false claims are often deliberately fabricated; they may be semantically correct but are not supported. this makes fact verification a rather challenging task, as it requires the fine-grained reasoning ability to distinguish the subtle differences between truth and false statements @xcite . this paper presents a new neural structural reasoning model, kernel graph attention network (kgat), that provides more fine-grained evidence selection and reasoning capability for fact verification using neural matching kernels @xcite . given retrieved evidence pieces, kgat first constructs an evidence graph, using claim and evidence as graph nodes and fullyconnected edges. it then utilizes two sets of kernels, one on the edges, which selectively summarize clues for a more fine-grained node representation and propagate clues among neighbor nodes through a multi-layer graph attention; and the other on the nodes, which performs more accurate evidence selection by better matching evidence with the claim. these signals are combined by kgat, to jointly learn and reason on the evidence graph for more accurate fact verification. in our experiments on fever @xcite , a large-scale fact verification benchmark, kgat achieves a 70.38% fever score, significantly outperforming previous bert and graph neural network (gnn) based approaches @xcite . our experiments demonstrate kgat's strong effectiveness especially on facts that require multiple evidence reasoning: our kernel-based attentions provide more sparse and focused attention patterns, which are the main source of kgat's effectiveness. ## related work the fever shared task @xcite aims to develop automatic fact verification systems to check the veracity of human-generated claims by extracting evidence from wikipedia. the recently launched fever shared task 1.0 is hosted as a competition on codalab foot_0 with a blind test set and has drawn lots of attention from nlp community. existing fact verification models usually employ fever's official baseline @xcite with a three-step pipeline system @xcite : document retrieval, sentence retrieval and claim verification. many of them mainly focus on the claim verification step. @xcite concatenates all evidence together to verify the claim. one can also conduct reasoning for each claim evidence pair and aggregate them to the claim label @xcite @xcite . twowingos @xcite further incorporates evidence identification to improve claim verification. gear @xcite formulates claim verification as a graph reasoning task and provides two kinds of attentions. it conducts reasoning and aggregation over claim evidence pairs with a graph model @xcite @xcite . @xcite further employs xlnet @xcite and establishes a semantic-level graph for reasoning for a better performance. these graph based models establish node interactions for joint reasoning over several evidence pieces. many fact verification systems leverage natural language inference (nli) techniques @xcite @xcite @xcite to verify the claim. the nli task aims to classify the relationship between a pair of premise and hypothesis as either entailment, contradiction or neutral, similar to the fever task, though the later requires systems to find the evidence pieces themselves and there are often multiple evidence pieces. one of the most widely used nli models in fever is enhanced sequential inference model (esim) @xcite , which employs some forms of hard or soft alignment to associate the relevant sub-components between premise and hypothesis. bert, the pre-trained deep bidirectional transformer, has also been used for better text representation in fever and achieved better performance @xcite @xcite . the recent development of neural information retrieval models, especially the interaction based ones, have shown promising effectiveness in extracting soft match patterns from query-document interactions @xcite @xcite @xcite . one of the effective ways to model text matches is to leverage matching kernels @xcite , which summarize word or phrase interactions in the learned embedding space between query and documents. the kernel extracts matching patterns which provide a variety of relevance match signals and shows strong performance in various ad-hoc retrieval dataset @xcite . recent research also has shown kernels can be integrated with contextualized representations, i.e., bert, to better model the relevance between query and documents @xcite . ## kernel graph attention network this section describes our kernel graph attention network (kgat) and its application in fact verification. following previous research, kgat first constructs an evidence graph using retrieved evidence sentences @xmath0, . . . , e p , . . . , e l } for claim c, and then uses the evidence graph to predict the claim label y (sec. 3.1 and 3.2). as shown in figure 2 , the reasoning model includes two main components: evidence propagation with edge kernels (sec. 3.3) and evidence selection with node kernels (sec. 3.4). ## experimental methodology this section describes the dataset, evaluation metrics, baselines, and implementation details in our experiments. ## evaluation result the experiments are conducted to study the performance of kgat, its advantages on different reasoning scenarios, and the effectiveness of kernels. ## case study the first evidence verifies that "al jardine is an american musician" but does not enough information about whether "al jardine is a rhythm guitarist". the edge kernels from kgat accurately pick up the additional information evidence (1) required from evidence (2): "rhythm guitarist". it effectively fills the missing information and completes the reasoning chain. interesting, "al jardine" also receives more attention, which helps to verify if the information in the second evidence is about the correct person. this kernel attention pattern is more intuitive and effective than the dot-product attention in gat. the later one scatters almost uniformly across all tokens and hard to explain how the joint reasoning is conducted. this seems to be a common challenge of the dot-product attention in transformers @xcite . ## conclusion this paper presents kgat, which uses kernels in graph neural networks to conduct more accurate evidence selection and fine-grained joint reasoning. our experiments show that kernels lead to the more accurate fact verification. our studies illustrate the two kernels play different roles and contribute to different aspects crucial for fact verification. while the dot-product attentions are rather scattered and hard to explain, the kernel-based attentions show intuitive and effective attention patterns: the node kernels focus more on the correct evidence pieces; the edge kernels accurately gather the necessary information from one node to the other to complete the reasoning chain. in the future, we will further study this properties of kernel-based attentions in neural networks, both in the effectiveness front and also the explainability front.