F ar F etched: Entity-centric Reasoning and Claim Validation for the G reek Language based on Textually Represented Environments (2022)

Abstract: Our collective attention span is shortened by the flood of online information. With FarFetched, we address the need for automated claim validation based on the aggregated evidence derived from multiple online news sources. We introduce an entity-centric reasoning framework in which latent connections between events, actions, or statements are revealed via entity mentions and represented in a graph database. Using entity linking and semantic similarity, we offer a way for collecting and combining information from diverse sources in order to generate evidence relevant to the user’s claim. Then, we leverage textual entailment recognition to quantitatively determine whether this assertion is credible, based on the created evidence. Our approach tries to fill the gap in automated claim validation for less-resourced languages and is showcased on the Greek language, complemented by the training of relevant semantic textual similarity (STS) and natural language inference (NLI) models that are evaluated on translated versions of common benchmarks.

## introduction motivation: the wider diffusion of the web since the dawn of web 2.0 has enabled instantaneous access to an expanding universe of information. the entire nature of news consumption has shifted dramatically, as individuals increasingly rely on the internet as their major source of information. while people access, filter and blend several websites into intricate patterns of media consumption, this wealth of information contained in billions of online articles inevitably creates a poverty of attention and a need to efficiently allocate this attention among the many sources that may absorb it. verifying whether a given claim coheres with the knowledge hidden in the vast amount of published information is a fundamental problem in nlp, taking into account that the arrival of new information may weaken or retract the initially supported inference. the problem is more apparent in less-resourced languages that lack the necessary linguistic resources for building meaningful nlp applications. approach and contribution: farfetched is a modular framework that enables people to verify any kind of textual claim based on the incorporated evidence from textual news sources. it combines a series of processes to periodically crawl for news articles and annotate their context with named entities. given a user claim, farfetched derives a relevant subset of the stored content based on its semantic similarity with the provided claim, thus being able to reason about its validity in an nli setting (figure 1 ). while the proposed framework focuses on the less-resourced greek language, its modular architecture allows the integration of pretrained models for any language. moreover, it is capable of topic-agnostic, evidence-aware assessment of arbitrary textual claims in a fully automated manner, without relying on feature engineering, curated sources and manual intervention. the main contributions of this work are summarized as follows: a) to formalize, develop and evaluate a claim validation and reasoning approach based on the aggregated knowledge derived from the continuous monitoring of news sources, and b) to train, evaluate and share sota models for the sts and nli downstream tasks for the greek language that support the core functionalities of our framework. 1 2 related work our work comprises functionalities comparable to those of fact checking frameworks, targeting the assignment of a truth value to a claim made in a particular context @xcite . for most related approaches @xcite @xcite @xcite the evidence to support or refute a claim is derived from a trustworthy source (e.g. wikipedia, crowdsourced tagging or expert annotators). interesting deviations are declare @xcite that searches for web articles related to a claim considering their inbetween relevance using an attention mechanism, and claimeval @xcite , based on firstorder logic to contextualise prior knowledge from a set of the highest page-ranked websites. farfetched can be distinguished from the aforementioned works by four major points: a) evidence collection is disentangled from manual annotation but relies on a constantly updating feed of news articles instead; b) claim validation based on the accumulated evidence relies on the effective combination of entity linking and attention-based models; c) our approach provides interpretable reasoning based on the aggregated evidence of multiple sources without assessing their truthfulness as opposed to most fact checking frameworks; and d) the outcome of the process is dynamic as the con-tinuous integration of new information may lead to a shift in the verdict of the validated claim. recent advances in the field of event-centric nlp have introduced event representation methods based on narrative event chains @xcite , knowledge graphs @xcite , qa pairs @xcite or event network embeddings @xcite to capture connections among events in a global context. our method relies on an entity-centric approach instead, where the identified entities are used as connectors between events, actions, facts, statements or opinions, thus revealing latent connections between the articles containing them. a few similar approaches have been proposed for combining world knowledge with event extraction methods to represent coherent events, but rely either on causal reasoning to generate plausible predictions @xcite or on qa models that require the accompanying news source to be provided along with the user's question @xcite . the latest advances regarding the technological concepts that comprise our methodology are provided below: entity linking (el) resolves the lexical ambiguity of entity mentions and determines their meanings in context. typical el approaches aim at identifying named entities in mention spans and linking them to entries of a kg (e.g. wikidata, dbpedia) thus resolving their ambiguity. recent methods combine the aforementioned tasks using local compatibility and topic similarity features @xcite , pagerank-based wikification @xcite -used also in farfetched-or neural endto-end models that jointly detect and disambiguate mentions with the help of context-aware mention embeddings @xcite . the recent interest for encapsulating diverse semantic sentence features into fixed-size vectors has resulted in sota systems for semantic textual similarity (sts) based on supervised cross-sentence attention @xcite , deep averaging networks (dan) @xcite or siamese and triplet bert-networks @xcite to acquire meaningful sentence embeddings that can be compared using cosine similarity. the latter approach is leveraged in our case to train an sts model for the greek language using transfer learning. finally, the task of natural language inference (nli) -also known as recognizing textual entailment (rte)-associates an input pair of premise and hypothesis phrases into one of three classes: contradiction, entailment and neutral. ferreira and vlachos, 2016 modeled fact checking as a form of rte to predict whether a premise, typically part of a trusted source, is for, against, or observing a given claim. sota nli models typically rely on transformer variants with global attention mechanisms @xcite , siamese network architectures @xcite (also used in farfetched to train a greek nli model), autoregressive language models for capturing long-term dependencies @xcite and denoising autoencoders @xcite . ## limitations we acknowledge that farfetched is possible to encounter errors in 3 main areas; these limitations are briefly addressed below. entity linking: highly ambiguous entities and name variations pose challenges to any entity linking method. since we claim that our approach is entity-centric, a wrong entity annotation may lead to irrelevant candidate evidence sequences and increase the probability of "neutral" nli verdicts. moreover, the tunable sensitivity of the integrated wikification module implies a trade-off between a precision-oriented and a recall-oriented strategy, the latter resulting in more annotated articles, but also being prone to false-positive annotations. evidence construction: this initial version of our approach relies solely on the sts comparison between the evidence and the claim, based on a shortest path approach as discussed in section 3.2.4. in cases that involve a larger number of entities in the user claim, calculating the shortest path between the alternating entity-section nodes can be computationally cumbersome. moreover, there is no guarantee that the shortest path is able to capture the most relevant candidate evidence sequences; to this end, outputting the top n best candidates is considered, providing a user with an overview of the extracted news excerpts together with their nli outcome. finally, neither a temporal evaluation of the evidence with regard to the claim nor a distinction between opinions and facts is considered; all candidates are treated as equal. natural language inference: recognizing the entailment between a pair of sentences partially depends on the tense and aspect of the predications. tense plays an important role in determining the temporal location of the predication (i.e. past, present or future), while the aspectual auxiliaries signify an event's internal constituency (e.g. whether an action is completed or in progress). while the work of @xcite indicates that language models substantially encode morphosyntactic information regarding tense and aspect, they are unable to reason based only on these properties. to this end, claims with a high presence of such semantic properties should be avoided. ## conclusions in this work, we presented a novel approach for claim validation and reasoning based on the accumulated knowledge from the continuous ingestion and processing of news articles. farfetched is able to evaluate the validity of any arbitrary textual claim by automatically retrieving and aggregating evidence from multiple sources, relying on the pillars of entity linking, semantic textual similarity and natural language inference. we showcased the effectiveness of our method on the fever benchmark as well as on diverse scenarios and acknowledged its limitations. as byproducts of our work, we trained and opensourced an nli and an sts model for the lessresourced greek language, achieving state-of-theart performance on the xnli and sts2017 bench-marks respectively. while our framework fills the gap in automated claim validation for greek, its modular architecture allows it to be repurposed for any language for which the corresponding models exist. for future work, we intend to address the limitations of our method mentioned in section 5, focusing primarily on an optimal entity linking setting, as well as on a more robust strategy for constructing relevant candidate evidence sequences.