Lost in Translation, Found in Spans: Identifying Claims in Multilingual Social Media (2023)

Abstract: Claim span identification (CSI) is an important step in fact-checking pipelines, aiming to identify text segments that contain a check-worthy claim or assertion in a social media post. Despite its importance to journalists and human fact-checkers, it remains a severely understudied problem, and the scarce research on this topic so far has only focused on English. Here we aim to bridge this gap by creating a novel dataset, X-CLAIM, consisting of 7K real-world claims collected from numerous social media platforms in five Indian languages and English. We report strong baselines with state-of-the-art encoder-only language models (e.g., XLM-R) and we demonstrate the benefits of training on multiple languages over alternative cross-lingual transfer methods such as zero-shot transfer, or training on translated data, from a high-resource language such as English. We evaluate generative large language models from the GPT series using prompting methods on the X-CLAIM dataset and we find that they underperform the smaller encoder-only language models for low-resource languages.

## introduction social media platforms have become a prominent hub for connecting people worldwide. along with the myriad benefits of this connectivity, e.g., the ability to share information instantaneously with a large audience, the spread of inaccurate and misleading information has emerged as a major problem @xcite . misinformation spread via social media has far-reaching consequences, including the potential to sow chaos, to foster hatred, to manipulate public opinion, and to disturb societal stability @xcite . claims play an integral role in propagating fake news and misinformation, serving as the building blocks upon which these deceptive narratives are formed. in their argumentation theory, toulmin (2003) described a claim as "a statement that asserts something as true or valid, often without providing sufficient evidence for verification." such intentional or unintentional claims quickly gain traction over social media platforms, resulting in rapid dissemination of misinformation as was seen during recent events such as the covid-19 pandemic @xcite and brexit (bastos and mercea, 2019). to mitigate the detrimental impact of false claims, numerous factchecking initiatives, such as politifact and snopes, dedicate substantial efforts to fact-checking claims made by public figures, organizations, and social media users. however, due to the time-intensive nature of this process, many misleading claims dodge verification and remain unaddressed. to address this, computational linguistic approaches have been developed that can assist human factcheckers @xcite @xcite @xcite . recently, @xcite introduced the task of claim span identification (csi), where the goal is to identify textual segments that contain claims or assertions made within the social media posts. the csi task serves as a precursor to various downstream tasks such as claim verification and check-worthiness estimation. while efforts have been made in combating misinformation in different languages @xcite , research in identifying the claim spans has so far been limited to english. previously, @xcite have manually extracted covid-19 claim spans from twitter in english. however, the landscape of fraudulent claims goes beyond covid-19 and twitter. in this work, we aim to bridge these gaps by studying the task of multilingual claim span identification (mcsi) across numerous social media platforms and multiple languages. to the best of our knowledge, this is the first attempt towards identifying the claim spans in a language different from english. we design the first data curation pipeline for the task of mcsi, which, unlike @xcite , does not require manual annotation to create the training data. we collect data from various fact-checking sites and we automatically annotate the claim spans within the post. using the pipeline, we create a novel dataset, named x-claim, containing 7k real-world claims from numerous social media platforms in six languages: english, hindi, punjabi, tamil, telugu, and bengali. figure 1 showcases a few examples from our dataset. we report strong baselines for the mcsi task with state-of-the-art multilingual models. we find that joint training across languages improves the model performance when compared to alternative cross-lingual transfer methods like zero-shot transfer, or training on translated data, from a highresource language like english. in this work, we make the following contributions: ## related work efforts to combat misinformation and fake news have focused on claims in various sources. the existing body of work in this area can be broadly categorized into the following major groups: claim detection @xcite @xcite , claim check-worthiness @xcite , claim span identification @xcite , and claim verification @xcite . being the precursor of several other downstream tasks, claim detection has garnered significant attention. various methods have been proposed to tackle claim detection, aiming to identify statements that may contain claims @xcite @xcite . in response to the escalating issue of false claims on social media, there has been a surge in the development of claim detection systems specifically designed to handle text from social media platforms @xcite @xcite . recently, @xcite introduced the task of claim span identification where the system needs to label the claim-containing textual segments from social media posts, making claim detection systems more explainable through this task. while most existing methods to combat fake news are primarily tailored for english @xcite @xcite , in recent times, there has been a surge in interest regarding the advancement of factchecking techniques for various other languages. claimrank @xcite introduced an online system to identify sentences containing checkworthy claims in arabic and english. the check-that! lab has organized several multilingual claim tasks over the past five years, progressively expanding language support and garnering an increasing number of submissions @xcite @xcite @xcite . in their latest edition, @xcite featured factuality tasks in seven languages: english, german, arabic, italian, spanish, dutch, and turkish. gupta and srikumar (2021) introduced x-fact, a comprehensive multilingual dataset for factual verification of realworld claims in 25 languages. unlike that work, here we focus on extracting the claim from a social media post, rather than fact-checking a claim. figure 2 : our two-step methodology to create the x-claim dataset for the multilingual claim span identification task. the top row shows the data collection (section 3.1) from a fact-checking website. the bottom row illustrates the automated annotation step (section 3.2) in which, first, the most similar post sentence (ps) is selected, and then, the claim span is created with the help of a normalized claim (nc). we use bertscore-recall @xcite for sentence selection and awesome-align @xcite for word alignment between nc and ps2. the task of claim span identification remains unexplored due to the lack of datasets in other languages. @xcite developed a dataset of 7.5k manually annotated claim spans in tweets, named curt; all the tweets and claim spans in that dataset are in english. additionally, while there has been interest in claims in other languages, there is a notable lack of progress on indian languages. here, we aim to bridge this gap. ## dataset we follow a two-step pipeline to develop our dataset: (i) data collection and (ii) automated annotation. we present a high-level overview of our proposed data creation methodology in figure 2 . below, we explain these steps in detail. ## results we carry out an exhaustive empirical investigation to answer the following research questions: r1. does the model benefit from joint training with multiple languages? (section 5.1) r2. do we need training data in low-resource languages when we have abundant data in highresource languages? foot_4 (section 5.2) r3. can large language models (llms) such as gpt-4 identify the claims made in multilingual social media? (section 5.3) r4. how does the automatically annotated x-claim dataset compare to prior manually annotated datasets like curt? (section 5.4) ## error analysis in this section, we qualitatively analyze the errors made by the best-performing multilingual mdeberta model. to provide insights on how llms can be improved for this task, we also discuss the errors made by gpt-4 in its bestperforming setting of 10-shot in-context learning. we analyzed the predictions on the test examples in english and hindi, and we report the kinds of errors made by the two language models in ## conclusion and future work we proposed a novel automated data annotation methodology for multilingual claim span identification. using it, we created and released a new dataset called x-claim, which consists of realworld claim spans, and social media posts containing them, collected from numerous social media platforms in six languages: english, hindi, punjabi, tamil, telugu, and bengali. using state-ofthe-art multilingual models, we established strong baselines based on encoder-only and generative language models. our experiments demonstrated the benefits of multilingual training when compared to other cross-lingual transfer methods such as zero-shot transfer, or training on the translated data, from a high-resource language like english. we observed lower performance for gpt-style generative llms when compared to smaller finetuned encoder-only language models and we discussed their error analysis in the spirit of improving the llms on this task. our work opens many important research questions: (1) how to obtain real-world claims without relying on fact-checkers analysis? (2) how to improve the understanding of llms about claims and social media in low-resource languages? (3) how to automatically curate multiple check-worthy claims made in the post? (4) how to improve the evaluation metric for the mcsi task? and (5) how to expand the csi task to other lowresource languages? we plan to address these research questions in future work.