Time Matters: An End-to-End Solution for Temporal Claim Verification (2024)

Abstract: Automated claim verification plays an essential role in fostering trust in the digital space. Despite the growing interest, the verification of temporal claims has not received much attention in the community. Temporal claim verification brings new challenges where cues of the temporal information need to be extracted, and temporal reasoning involving various temporal aspects of the text must be applied.In this work, we describe an end-to-end solution for temporal claim verification that considers the temporal information in claims to obtain relevant evidence sentences and harnesses the power of a large language model for temporal reasoning. We curate two datasets comprising a diverse range of temporal claims to learn time-sensitive representations that encapsulate not only the semantic relationships among the events, but also their chronological proximity.Experiment results demonstrate that the proposed approach significantly enhances the accuracy of temporal claim verification, thereby advancing current state-of-the-art in automated claim verification.

## introduction the proliferation of false information, or "fake news," continues to pose a challenge with potentially severe implications. computational claim verification has been proposed as a viable solution to this issue, leveraging technology to verify textual claims against a set of evidence sentences that either support or contradict these claims. however, there is still a considerable gap when it comes to verifying temporal claims which are statements associated with a specific time or duration. for effective verification of temporal claims, we need to retrieve evidence that focus not just on the semantic coherence between the claim and potential evidence, but more importantly, the temporal context so that the timeline is aligned between the claim and the evidence. consider the temporal claim "matteo renzi was a full-time undergraduate student in singapore in 2006". this claim can be refuted if we find evidence like "matteo renzi served as president of the province of florence from 2004 to 2009..." since it is highly unlikely for someone to serve as a president while concurrently undertaking a fulltime undergraduate degree in a different country. existing claim verification methods that employ traditional evidence retrieval based on lexical or semantic matching might overlook this evidence sentence and conclude that there is not enough info (nei) to verify the claim. consider another temporal claim "henry condell published his first folio in 1623 and performed several plays for his career in 1620.". this claim has two events "published his first folio" and "performed several plays" which are associated with two distinct dates, "1623" and "1620", respectively. for the temporal claim to be true, we need to verify that both events are supported by the evidence sentences. on the other hand, if we have evidence that shows one of the events is false, then the entire claim becomes false. for example, if we have the evidence sentence "henry condell ended his stage career in 1619.", then we can refute the event that he performed several plays in 1620, and conclude that the temporal claim is false. by analyzing the claim and evidence sentence at the event-level rather than the whole sentence, we can link the time references to their respective events and retrieve relevant evidence sentences. we describe an end-to-end solution for temporal claim verification by taking into account the temporal information in the claim to retrieve relevant evidence sentences. we identify events in both the claim and evidence sentences and associate the time-related information to the corresponding events. with this, we can assign a higher score to evidence sentences that align more closely with the claim events. the top ranked evidence sentences form the context for large language model (llm) to reason and determine the claim veracity. existing claim verification datasets such as fever and feverous have limited temporal claims. as such, we create two new temporal claim verification datasets comprising of a diverse range of temporal claims. experiment results on multiple datasets demonstrate that the proposed solution surpasses state-of-the-art claim verification methods, is robust, and can handle real-world claims. ## related work research on evidence-based claim verification typically formulates the problem as a natural language inference task, and classifies whether the evidence sentences support or refute the claim @xcite . gear @xcite uses a graph attention network to capture the semantic interaction between evidence sentences. kgat @xcite introduces kernels to measure the importance of the evidence and conduct fine-grained evidence propagation. @xcite incorporates external knowledge to inject commonsense knowledge into the model. unifee @xcite focuses on improving evidence retrieval on structured evidence by constructing a unified evidence graph and employing graph network to facilitate interactions between claims and evidence. several works have attempted to take into account temporal information for claim verification. @xcite considers the published date of the claim and evidence sentences, and re-ranks the sentences based on the proximity of their published dates to that of the claim. @xcite verifies economic claims against time series sources which are in tabular format. this work only deals with structured sql data and does not handle evidence in natural language. itr @xcite exploits the temporal proximity between the claim's publication date and evidence's publication date to create time representations for temporal reasoning. these works do not consider temporal expressions in the claim and evidence. ## proposed solution figure 1 shows our proposed temoral aware claim verification (tacv) solution. given a temporal claim, we extract claim events with their associated temporal expressions from the claim. to obtain more information about the claim, we use a sequence-to-sequence entity linking model genre @xcite to retrieve documents from sources such as wikipedia articles. each sentence from the retrieved documents is sent to the event extraction module to obtain evidence sentence events. we pair the extracted claim events with the evidence sentence events to create temporal-aware representations. this step facilitates the identification of the top-k most relevant evidence sentences, which are deemed potentially useful for verifying the claim event. utilizing the top-k evidence sentences as context, the framework harnesses the temporal reasoning capabilities of large language models (llms) to ascertain whether the evidence supports or refutes the claim event, or if the evidence is insufficient for verification. finally, these labels are aggregated to obtain the final label for the input claim. ## performance study we evaluate the effectiveness of the tacv framework for temporal claim verification. we show that tacv performs well not only on the new temporal t-fever and t-feverous datasets, but also on the standard benchmark fever and feverous datasets as well as the real world liar dataset @xcite . tacv uses huggingface's implementation of bert base to encode the tokens in the extracted events. for the temporal-aware representation encoding, a transformer with two layers and eight heads, having a dimension of 768, is used. the training is conducted over five epochs with a batch size of 8, and learning rate of 5e-6. we apply the adamw (loshchilov and hutter) optimizer with a fixed weight decay and select the best performing model for evaluation on the test set. sensitivity experiments. we examine the performance of tacv as we vary the number of top-k relevant evidence sentences for temporal reasoning. figure 3 shows the label accuracy and fever score for different k values on the t-fever and t-feverous validation datasets. we see that the optimal performance is attained when @xmath0, and @xmath1. as such, we use the top-3 sentences in t-fever, and the top-5 sentences in t-feverous with the highest relevance scores to form the context for the llm to output the label of each claim event. comparative experiments. we compare tacv with state-of-the-art evidence-based claim verification baselines: kgat @xcite , @xcite , itr @xcite , unifee @xcite . since itr assumes evidence sentences are given as input, we use the evidence sentences retrieved by our tacv as input to itr for fair comparison. we also validate the ability of tacv to handle the original synthetic general claims in fever and feverous, as well as real-world claims in liar which comprises of statements compiled from poli-tifact.com. for each claim in liar, we feed the claim sentence into bing search to retrieve the top-2 articles and all the sentences from these articles are used as potential evidence sentences. in among the results, tacv performs the worst on the feverous dataset which contains 54% nontemporal claims and 46% temporal claims. we randomly sample 25 temporal and 25 non-temporal claims to conduct a more detailed error analysis. manual inspection reveals that 90% of the error was due to the inability to extract structured evidence such as tables. incorrect temporal reasoning by llm contributed 10%, even when the correct evidence was retrieved. ## case studies ## conclusion we have introduced a new framework for temporal claim verification that addresses the growing challenge posed by misinformation in real-world settings, particularly in information-heavy industries such as media, finance, and legal sectors. our end-to-end solution can be seamlessing integrated into existing workflows to verify temporal claims where the accuracy of time-sensitive information is crucial. we have developed two temporal datasets that serve as evaluation benchmarks and resources for further research in temporal claim verification. experimental results have demonstrated the effectiveness of temporal-aware representations, which lead to marked performance improvements over state-of-the-art methods across multiple datasets, including the real world liar dataset. future research includes handling more complex sentence structures with implicit temporal expressions.