The Truth, The Whole Truth, and Nothing but the Truth: A New Benchmark Dataset for H ebrew Text Credibility Assessment (2023)

Abstract: In the age of information overload, it is more important than ever to discern fact from fiction. From the internet to traditional media, we are constantly confronted with a deluge of information, much of which comes from politicians and other public figures who wield significant influence. In this paper, we introduce HeTrue: a new, publicly available dataset for evaluating the credibility of statements made by Israeli public figures and politicians. This dataset consists of 1021 statements, manually annotated by Israeli professional journalists, for their credibility status. Using this corpus, we set out to assess whether the credibility of statements can be predicted based on the text alone. To establish a baseline, we compare text-only methods with others using additional data like metadata, context, and evidence. Furthermore, we develop several credibility assessment models, including a feature-based model that utilizes linguistic features, and state-of-the-art transformer-based models with contextualized embeddings from a pre-trained encoder. Empirical results demonstrate improved performance when models integrate statement and context, outperforming those relying on the statement text alone. Our best model, which also integrates evidence, achieves a 48.3 F1 Score, suggesting that HeTrue is a challenging benchmark, calling for further work on this task.

## introduction our society is struggling with an unprecedented amount of falsehoods, hyperboles, and half-truths @xcite . false information, commonly termed 'fake news', is now viewed as one of the greatest threats to democracy, journalism, and freedom of expression. distributing false content has become a significant concern in recent years when it is claimed to have helped change public opinion in the us elections of 2016 @xcite . our economies are not immune to the spread of fake news either, with fake news being connected to stock market fluctuations and large trades. for example, fake news claiming that barack obama, the 44th president of the united states, was injured in an explosion, wiped out $130 billion in stock value @xcite . also, recent studies have shown that false content reaches "farther" and a wider audience of readers than real content @xcite . even if a correction of the false information reaches the misinformed audience, simply providing the correct information is ineffective, as continued reliance on misinformation is likely when the misinformation conforms to a person's preexisting belief system, yet the correction does not @xcite . retracting misinformation that runs counter to a person's worldview can ironically even strengthen the to-be-corrected information, a phenomenon known as the worldview backfire effect @xcite . addressing the spread of fake news necessitates a comprehensive approach to assess the credibility of information. credibility assessment, distinct yet sometimes used interchangeably with fake news detection, involves evaluating the veracity of statements or claims made by individuals, particularly those in influential positions. while fake news explicitly entails deception, credibility assessment encompasses a broader spectrum, scrutinizing the authenticity and reliability of information regardless of the intent behind it. manual credibility assessment is not sufficient in effectively combating the rapid spread of false information, leading to a pressing need for alternative approaches. in particular, there is a need for automatic credibility assessment tools, which leverages statistical and machine learning methods to analyze and determine the credibility of statements or news as they come in, thereby avoiding the human-labor bottleneck. to effectively combat fake news and assess the credibility of information, there is a pressing need for comprehensive datasets that facilitate the development and evaluation of automatic detection tools. existing datasets primarily focus on english language content @xcite , leaving a substantial gap in the study of other languages, including hebrew, which known for its unique linguistic characteristics @xcite . additionally, many available datasets lack crucial supplementary information such as context, semantic features, and metadata associated with the statements in question. including these features is essential for building robust nlp models, enhancing their understanding and accuracy in credibility assessment tasks. the need for such fake-news detectors is particularly pressing in areas of which socio-political situations may be sensitive yet may have world-wide effects -such as the middle east. to mitigate this, we introduce the hetrue dataset, the first-ever hebrew fake-news detection benchmark. the hetrue dataset is a unique hebrew resource composed of 1021 statements made by politicians. each statement is meticulously labeled for truthfulness and supplemented with context information. the dataset is further augmented with metadata and semantic features, which have been validated and annotated by nonpartisan fact-checkers, compiled with the international fact-checking network (ifcn). while the hetrue dataset addresses a significant gap in hebrew language resources for credibility assessment, its comprehensive set of features also raises the bar for similar datasets in other languages, including english. utilizing this novel benchmark, in this paper we focus mainly on linguistic-based approaches for automatic credibility assessment -that is, one that relies solely on the claim, or the claim's context as input for detection. linguistic approaches are frequently utilized to tackle this task @xcite , as they can operate without additional resources or data. this makes these approaches more scalable and straightforward to implement in real-time applications with minimal effort and budget. these approaches hinge on the assumption that individuals tend to express themselves differently, whether verbally or in writing, when conveying false information compared to true information. @xcite asserts that these differences stem from a sense of stress, manifesting in a decrease in cognitive integration capacity, precision, organiza-tion, and prioritization. these challenges result in a change in the normal elements of the speaker's language. prior works have found that these linguistic signals can also specifically assist in distinguishing between truth and falsehood in politicians' statements @xcite . we hypothesize that linguistic signals indicative of a statement's credibility exist in hebrew and that these signals correlate with the credibility assessments made by professional journalists, who collaborated with us to create our gold-standard benchmark. to provide a broader view, we have also explored alternative methods, which will be discussed in the following sections. as @xcite first introduced, achieving the 'holy grail' in end-to-end credibility assessment systems, also referred to as fact-checking, requires a fully automatic platform. this platform should consist of three main components: a claim-spotter to detect claims that need validation in real time, a claim-checker to evaluate the veracity of these claims, and a fact-check reporter to justify the evaluations with convincing evidence. this system would make determinations by analyzing the claim's text, source, claimant profile, and retrieving evidence from databases of previously checked claims. our work advances towards this "holy grail," focusing on a fully automated credibility assessment system for hebrew texts. the unique features of our dataset foot_0 and our experiments could also benefit credibility assessment in other languages. we call on the computing and journalism communities to join in on this pursuit, given the importance and timeliness of this task. the contribution of our paper is, hence, manifold. first, we introduce hetrue, the first publicly available hebrew dataset for fake news detection. this dataset comprises 1021 statements from politicians and public figures, manually labeled by professional journalists, and serves as a valuable resource for researchers investigating fake news detection in hebrew. to the best of our knowledge, this is the first claim assessment dataset that accompanies each claim with its context, alongside semantic and metadata features. second, we conduct an extensive experimental analysis, exploring a variety of modeling types and dataset setups. to establish a baseline performance for the task, we employ several methods including a hand-crafted feature-based model, a recurrent neu-ral network (rnn) with static embeddings, and a model extending alephbert-a hebrew monolingual pre-trained encoder @xcite . our experiments with these models demonstrate the presence of linguistic cues in hebrew text that can distinguish between credible and non-credible statements. we provide an analysis of the key features from the hand-crafted feature model, emphasizing their significance. thirdly, we demonstrate the importance of integrating context and evidence in credibility assessment. our novel model, which combines statement context and an evidence-retrieval mechanism, results in significant performance enhancements. we further compare our results with metadata-based models, revealing that our linguistic approach outperforms these models, and demonstrating that the addition of metadata features to a linguistic (handcrafted feature model) does not have an additive effect on the model's performance foot_1 . ## related work credibility assessment is an interdisciplinary field that involves evaluating the believability, trustworthiness, reliability, accuracy, fairness, and objectivity of claims or statements @xcite . it is closely related to fact checking and deception detection, and focuses on assessing the quality of information and the level of trust that can be placed with respect to claims. the goal of credibility assessment is to differentiate between credible and non-credible claims, without necessarily determining the root cause of false information, such as whether it is due to incorrect factual claims or the intent of the speaker @xcite . there are three categories of approaches for credibility assessment and fake news detection: linguistic (text-based), evidence-retrieval, and metadatabased, which typically incorporate machine learning techniques for training effective and accurate classifiers @xcite . linguistic approaches attempt to identify clues in the text in order to verify veracity @xcite . the work of @xcite posits that changes in certain linguistic characteristics can signal an attempt to conceal writing style, aiding in the detection of deceptive texts. this research present the effectivness of specialized set of lying-detection features including quantity metrics (such as syllable and word counts), vocabulary and grammatical complexity, and an analysis of the usage of words denoting uncertainty, specificity, and expressiveness. on the other hand, @xcite focus on linguistic traits more frequently associated with deceptive behavior. their examination of 242 transcripts revealed that individuals inclined to deception tend to produce lengthier texts, utilize a greater number of sensory expressions (such as references to sight or touch), and exhibit a preference for using pronouns that shift focus away from themselves and towards others, in comparison to when they are being truthful. @xcite incorporates various psycholinguistic features derived from liwc @xcite , to detect persuasive and biased language. in our work, we systematically adapt and extend these methodologies to assess statement credibility in hebrew. the second approach is evidence retrieval, which aims to find sources supporting or refuting the claim. one strategy involves the construction of knowledge graphs, facilitating the identification of relationships between textual segments and graph instances @xcite . alternatively, preestablished knowledge bases can be employed to garner pertinent evidence, subsequently leveraging models that capitalize on this retrieved information for text classification @xcite . the third approach is metadata-based, and it is often used alongside other methods. it involves analyzing text-related information like publication date, claimant's demographics, and source type @xcite . additionally, with the rise of fake news on social networks @xcite , behavioral analysis becomes important to identify unreliable patterns through user or message interactions @xcite . in terms of the credibility levels to be predicted, the work by @xcite was first to introduce the task of credibility assessment in scale of 1-6, in their work with the liar dataset @xcite . they used lstm with glove word embedding and showed that stylistic cues can help determine the truthfulness of text. @xcite applied cnn and bilstm models to the same dataset and task, aiming to extract patterns from short statements and understand the unique behaviors of source speakers using different dataset attributes, underscoring the importance of including speakers' profile information for fake news classifi-cation. @xcite incorporated emotional signals for credibility assessment, which presented improved performance, relying on the fact that those signals can play a vital role in detecting false information @xcite . recent progress in this field encapsulates a variety of methodologies. @xcite , for instance, present a semi-supervised deep learning (ssdl) pipeline employing an attention rnn-based model. additionally, models like x-fact (gupta and srikumar, 2021) emphasize the integration of evidence with attention architecture. a distinctive approach is taken by fakeflow @xcite , which focus on news articles and study the importance of information flow to detect fake news. context plays a pivotal role in understanding and verifying claims. however, its inclusion in datasets is rare, and also varies significantly @xcite . some datasets, such as those created by @xcite and @xcite , incorporate context derived from related threads. in these instances, a claim is contextualized with a set of pertinent posts, often originating from the same thread, to compensate for the limited context within an individual post. on the other hand, @xcite adopt a distinctive strategy, considering the segments of an entire article to analyze information flow. our work uniquely provides context for sentence-level claims, drawing directly from preceding text or speech. this approach captures the immediate discourse surrounding the claim, highlighting the critical role of textual interactions in credibility assessment. we believe this direct integration of granular context at the sentence level is a novel contribution to the field. while english remains the predominant language for fake news detection datasets, other languages are often underrepresented. as observed, most efforts to date, such as those by @xcite ; wang (2017), have extracted realworld claims from dedicated english-based websites like politifact. however, gupta and srikumar (2021) are notable exceptions, having curated claims from 25 languages. their multilingual baselines, including models like claim only, attentionbased evidence aggregator, and augmenting metadata model, have established a promising foundation for multilingual fact-checking. another initiative is by @xcite , who compiled a dataset of 219 arabic statements. both these stud-ies employed google's evidence retrieval to bolster claim veracity modeling. yet, their scope appears limited, especially in terms of integrating context and a rich set of metadata/semantic features. no hebrew-specific dataset for fact verification or fake news detection existed until our contribution, focusing on real-life statements. towards hebrew fake-news detection. the availability of labeled benchmark datasets is crucial for building statistical approaches for automatic fake news detection in new languages. english has been extensively studied with a large amount of annotated data, while other, less-researched languages, have limited or no annotated data available @xcite . as of yet, no publicly available hebrew dataset on fake news detection or claim credibility assessment is publicly available. furthermore, a limited number of works exist in the field of automatic credibility and falsehood detection in hebrew. the studies of @xcite @xcite examine the linguistic differences between truthful and deceptive discourse in hebrew and aim to develop a primary test for the cognitive and emotional processes involved in deception @xcite . although these studies do not analyze natural inputs and instead focus on laboratory-created inputs, they offer valuable insights into the cognitive and emotional processes involved in deception in the hebrew language. in their corpus, comprising 48 pairs of stories told by 48 subjects, they found several distinguishing features: leveraging insights from these hebrew-focused studies, alongside the english-based works, we enhance credebility assessment in hebrew. our work extends the existing knowledge base, constructing comprehensive linguistic feature-based models and evaluating their effectiveness on our task. while our contribution heralds a significant leap in hebrew-centric datasets for fact verification, it is not confined to that. it further includes a wide range of experiments and the introduction of empirical evaluation of novel models tailored explicitly for this task: context-based model (cbm), attentionbased evidence aggregator (attn-ae) and context and evidence model (ccem). ## conclusions our study introduces hetrue, a unique and rich dataset designed for automatic credibility assessment in hebrew. this dataset stands out not only as the first of its kind in hebrew, but also as the first to incorporate a comprehensive set of features including the statement context, the statement itself, the journalist's version, and additional semantic and metadata features -annotated by professional journalists in a rigours process. the findings from our experiments underscore the effectiveness of linguistic approaches in the challenging task of credibility assessment. our results consistently demonstrate enhanced performance when context is integrated into the models across various setups. furthermore, we show that incorporating evidence-retrieval mechanisms alongside context further contributes to the models' performance. despite the advancements made, credibility assessment, particularly in morphologically rich languages like hebrew, remains a challenging task. while pre-trained language models like aleph-bert show promise in effectively understanding and generalizing language even with limited context, the quest for better accuracy continues. future research will aim to refine and further improve the methodologies introduced in this study. finally, the hetrue dataset, with its unique combination of features, holds potential for a broader range of nlp applications such as stance classification, argument mining, topic modeling, and rumor detection. this broad utility further underscores the value of our contribution and opens new pathways towards more sophisticated fake news detection methodologies.