Neural Machine Translation for Fact-checking Temporal Claims (2022)

Abstract: Computational fact-checking aims at supporting the verification process of textual claims by exploiting trustworthy sources. However, there are large classes of complex claims that cannot be automatically verified, for instance those related to temporal reasoning. To this aim, in this work, we focus on the verification of economic claims against time series sources. Starting from given textual claims in natural language, we propose a neural machine translation approach to produce respective queries expressed in a recently proposed temporal fragment of the Datalog language. The adopted deep neural approach shows promising preliminary results for the translation of 10 categories of claims extracted from real use cases.

## introduction false and misleading information spreading on media negatively impacts our society by affecting people opinions and behaviours. to oppose such phenomena, fact-checking is the process aiming at verifying the correctness of facts in a piece of text, i.e., claims @xcite . to this end, claims have to be debunked against trustworthy structured and non-structured sources. in this work, we report on our effort on statistical claims in collaboration with a national central bank in europe. specifically, we focus on claims about economic events and trends that should be checked against temporal data sequences, namely time series, the standard format for time-based data in this domain. fact-checking organizations debunk such claims applying a typically manual process to extract claims from text, retrieving pieces of evidence from relevant sources, and finally, reaching a verdict using such evidence. given the large amount of information produced and shared online, this process cannot guarantee that an adequate number of economic claims is debunked. to enable scalability, statistical claims could be automatically verified to support checkers in their work. in order to be interpretable to the human, not only should the automatic verification provide a label for the given claim, but also a clear explanation of the adopted reasoning process as well as the evidence used to draw the conclusion. declarative languages offer a valid solution to both challenges. mature data modeling languages balance computational complexity and expressive power, addressing the scalability challenges. interpretability is a major advantage of declarative languages, with well-known semantics applied to the ground data in a deductive, top-down approach (hansen and @xcite . different solutions translate text formats to executable high-level languages, e.g., sql queries, typically through neural architectures @xcite . while these approaches exhibit promising results, they do not effectively translate claims including temporal aspects into an executable form. this depends on their limited capability in capturing the temporal logic of claims. example. consider a central bank that offers a service to fact-check economic claims. reference information is available in the standard format of time-series. the bank has data about economic measures, such as inflation, public debt, employment rate, interest rates, and about events such as economic crises, and salary increments. the example in figure 1 shows the inflation and public debt metrics with key-value tables. while some claims can be easily written as sql queries for their verification, for most of them this is very difficult, or unfeasible in some cases. consider the claims (i) "inflation in the last 10 years has been lower than 3%" and (ii) "every time inflation is above 2% for 10 years, public debt surpasses 3000b". a possible sql query to verify claim (i) through counter-example detection is reported in the left-hand side of figure 1 . for claim (ii), a script should evaluate, for each year in the dataset, the value of public debt and then check a condition over inflation for the previous 10 years. although such expression can be written in sql for one given year, our claim requires checking the condition for every possible year, which turns out to be unfeasible or extremely laborious and inefficient in sql. with datalogmtl @xcite , a datalog fragment that incorporates temporal operators, we can represent our expression as: ⊥:= -[0,10] inflationgt(y), @xmath0.02, public_debt(x), @xmath1 the rule intercepts the cases in which claim (ii) is violated. in fact, it is triggered whenever the body (right-hand side) is satisfied, meaning that the inflation is found greater than 2% for the 10 preceding years but the public debt is below or equal to 3000b. specifically, the logical premise of our claim ("every time inflation is above 2%") is checked through the temporal operatorϱ , i.e., "always true in the interval ϱ = [t -10y, t]", so that for the rule to be triggered at a point in time t, the logical conclusion of our claim must be false, i.e., public_debt(x), @xmath2. the evaluation of the rule checks the claim and produces humanunderstandable counter-examples (figure 1 ). in this paper, we propose a neural machine translation approach to fact-check numerical temporal claims. in particular, our approach synthesizes temporal data queries (rules) from input claims for the verification of temporal claims. while there have been efforts to use neural architectures for generating sql scripts @xcite , to the best of our knowledge, there is no support to translate textual claims into datalog rules @xcite . the main contributions of our approach are: our model is generated based on a dataset of manually crafted claim-query pairs inspired by economic newspapers and the federal reserve dataset @xcite . finally, a validation phase attests the ability of our model to automatically synthesize newly generated claims unseen during fine-tuning. ## language and templates in this section we present the datalogmtl formalism, the reference data model, and a categorization of claim-query pairs based on their semantics. temporal datalog. we adopt datalog metric temporal logic (datalogmtl), a temporal logic-based language that has recently received great attention from the ai and database communities @xcite @xcite . datalogmtl extends datalog with metric temporal logic operators over the rational timeline. we consider datalogmtl rules of the form ⊥ := a 1 , . . . , a k , for @xmath3, where all a i are literals that follow the grammar with ϱ being a non-negative interval, and p an atom (on a predicate having either functional or infix notation) over a tuple of terms (i.e., variables and constants) τ . the conjunction of a is the rule body and commas denote the logical and (∧). given a database d, the semantics of a data-logmtl rule is defined through interpretations. an interpretation m specifies for each time point t, whether a ground atom p (a) from d is true at t, in which case we write m, t |= p (a)@t. the @ symbol denotes the time reference for an atom. an interpretation that satisfies all atoms of the body triggers the rule. the box minusϱ operator checks if a ground atom is continuously true in the interval ϱ, that is, m, t |=ϱ a iff m, s |= a for all s, with @xmath4. the diamond minus ♢ϱ operator checks if a ground atom is true at least once in the interval ϱ, that is, m, t |= ♢ϱ a iff m, s |= a for some s, with @xmath5. data model. we rely on existing data sources containing time series, each representing an economic metric and following the key-value format, where key represents a time instant and value the corresponding measure. time series with multiple attributes can also be treated after a pre-processing phase which restructures them into multiple tables having replicated key values. instances of our data model straightforwardly derive from widely adopted on-line economic libraries such as the federal reserve economic data (fred) and the datasource exposed by the central bank in our project. query templates. we distinguish temporal claims according to their semantics and organize them into templates, providing for each of them the corresponding datalogmtl rule. our rules can be efficiently executed on a datalogmtl engine @xcite . in ## translating claims to datalogmtl in the context of neural machine translation @xcite pre-trained language models (plms) neural networks based on transformers excel at handling natural language (nl) sequences in understanding and generation tasks @xcite . these models are typically pre-trained on large text corpora in an unsupervised manner, for example by predicting the missing tokens in each sentence. many natural language processing (nlp) tasks can benefit from the nl understanding and generation capabilities of plms by means of fine-tuning with task-specific data through transfer learning. fine-tuning is performed in a supervised manner, by providing the labelled input. our approach fine-tunes the t5 model to use its text-to-text capabilities for our text-to-rule translation task. to this end, we generate and augment a fine-tuning set of textual input (claims) and output (rules) labels to specialize the t5 model. starting from a template-based categorization of the rules ( we exploit the set of templates and the time series to augment the fine-tuning dataset. for each claim, we create a set of variants having the same semantics but different syntax. we replace the actual values of economic metrics, events, numeric values, temporal and comparison text snippets with specific placeholders. we show the process for claims belonging to templates 3 (1,2) e 2 (3,4): we replace their rules with placeholders: we then replace the placeholders with the actual input values from our data sources: we apply the same process to the rules: ## training and validation results we describe: (i) how we obtained the fine-tuned t5 model and (ii) how we used it to automatically translate textual claims into rules. our results are based on the exact string matches accuracy of predicted datalogmtl rules against the ground truth. traditional bleu and rouge metrics are not used here since they may lead to good accuracy also for minimal variations of the predicted rules resulting to drastically different execution outcomes. model fine-tuning. following our translation approach we generated 60k ⟨nl, dl⟩ pairs uniformly distributed among the 10 templates after undersampling the most populated classes. after splitting the dataset into training and test sets and setting the architecture parameters, we fine-tune a t5-base a salary increment has been observed during the last 3 years ⊥ := -[0,3] salary increment(x), @xmath6, true@[today] 8 metric at given epoch implies if inflation for the last ten years was higher than 0.02, we have a salary increment required event ⊥:= salary increment(x), @xmath7, -[0,10] inflationgt(z), @xmath8.02 9 event at given epoch implies till to 5 years after a stock market crash, inflation remained below 0.02 metric at a single instant ⊥:= inflation(x), @xmath9.02, ♢ -[0,5] stock market crash(z), @xmath10, if inflation is lower than 0.02, there is a salary increment instant imply required event ⊥:= salary increment(y), @xmath11,♢ -[0,3] salary increment(z), @xmath12, inflation(x), @xmath13.02 ## conclusion this work shows that numerical temporal claims can be computationally verified exploiting time series. while preliminary results are promising, one major challenge is the lack of training data for the rule generation module. templates are an effective solution, but the coverage of the system depends on the number of deployed templates. we are also working on how to create data explanations for true claims, as now we only show counter-examples. one road is to identify examples by perturbing the datalog rule, as studied for sql queries @xcite . we are looking at how to improve the validation step by comparing the effects of the generated rules, instead of their syntactical equivalence and by testing a real-world benchmark of claims.