Empowering the Fact-checkers! Automatic Identification of Claim Spans on T witter (2022)

Abstract: The widespread diffusion of medical and political claims in the wake of COVID-19 has led to a voluminous rise in misinformation and fake news. The current vogue is to employ manual fact-checkers to efficiently classify and verify such data to combat this avalanche of claim-ridden misinformation. However, the rate of information dissemination is such that it vastly outpaces the fact-checkers’ strength. Therefore, to aid manual fact-checkers in eliminating the superfluous content, it becomes imperative to automatically identify and extract the snippets of claim-worthy (mis)information present in a post. In this work, we introduce the novel task of Claim Span Identification (CSI). We propose CURT, a large-scale Twitter corpus with token-level claim spans on more than 7.5k tweets. Furthermore, along with the standard token classification baselines, we benchmark our dataset with DABERTa, an adapter-based variation of RoBERTa. The experimental results attest that DABERTa outperforms the baseline systems across several evaluation metrics, improving by about 1.5 points. We also report detailed error analysis to validate the model’s performance along with the ablation studies. Lastly, we release our comprehensive span annotation guidelines for public use.

## introduction the swift acceleration of online social media (osm) platforms has led to tremendous democratized content creation and information exchange. consequently, these platforms serve as ideal breeding grounds for malicious rumormongers and talebearers, abetting a colossal upsurge of misinformation. such misinformation manifests in many ways, including bogus claims, fabricated information, and rumors. the massive covid-19 'infodemic' (naeem and bhatti, 2020) is one such malignant byproduct that led to the rampant spread though the recent literature reflects extensive work done in claim detection @xcite @xcite , limited forays have been made in claim span identification i.e., recognizing the argumentative components of a claim @xcite . in the recent past, commendable work has been done on span-level argument unit recognition pertaining to other computational counterparts under the umbrella of am, such as hate speech @xcite , toxic language @xcite etc. such study, however, has eluded the realm of claims, owning to the lack of quality annotated datasets. this heralds a specialized corpus creation on claim span identification. to this end, we propose curt (claim unit recognition in tweets), a large-scale, claim span annotated twitter corpus. we also present several baseline models for solving claim span identification as a token classification task and evaluate them on curt. furthermore, we introduce claim descriptions, which are generic prompts aimed to assist the model in focusing on the most significant regions of the input text using explicit instructions on what to designate as a 'claim'. they are elucidated later in detail. finally, we benchmark our dataset with daberta (description aware roberta), a plugand-play adapter-based variant of roberta @xcite , endeavored to infuse the pre-trained language model (plm) with the description information. empirical results attest that daberta outperforms the conventional baselines and generic plms for our task consistently across various metrics. contributions. through this work, we make the following tangible contributions: we propose the novel task of claim span identification that aims to identify argument units of claims in the given text. ## related work claims on social media. the prevailing research on claims could be cleft into three categories -claim detection @xcite @xcite , claim checkworthiness @xcite , and claim verification @xcite @xcite . @xcite pioneered the efforts in claim detection by introducing the aawd corpus. subsequent studies largely relied on using linguistically motivated features such as sentiment, syntax, context-free grammars, and parse-trees @xcite @xcite . recent works in claim detection have engendered the use of large language models (lms). chakrabarty et al. (2019) re-enforced the power of fine-tuning, as their ulmfit lm, fine-tuned on a large reddit corpus of about 5m opinionated claims, showed notable improvements in claim detection benchmark. gupta et al. (2021) proposed a generalized claim detection model for detecting claims independent of its source. they handled structured and unstructured data in con-junction by training a blend of linguistic encoders (pos and dependency trees) and a contextual encoder (bert) to exploit the input text's semantics and syntax. as lms account for significant computational overheads, sundriyal et al. (2021) addressed this quandary and proposed a lighter framework that attempted to fabricate discernible feature spaces. the checkthat! lab's clef-2020 shared task (barrón-cedeno et al., 2020) has garnered the attention of several researchers. williams et al. (2020) won the task by fine-tuning the roberta (liu et al., 2019) accentuated by mean pooling and dropout. nikolov et al. (2020) ranked second with their out-of-the-box roberta vectors supplemented with twitter meta-data. span identification. zaidan et al. (2007) introduced the concept of rationales, which highlighted text segments that supported their label's judgment. trautmann et al. (2020) released aurc-8 dataset with token-level span annotations for the argumentative components of stance along with their corresponding label. mathew et al. (2021) proposed a quality corpus for explainable hate identification with token-level annotations. the semeval community has initiated fine-grained span identification concerning other domains of argument mining such as toxic comments @xcite and propaganda techniques @xcite . these shared tasks amassed many solutions constituting transformers @xcite , convolutional neural networks @xcite , data augmentation techniques @xcite , and ensemble frameworks @xcite . @xcite resembled the closest study to ours, wherein they compiled a corpus of around 1.2k biomedical tweets with claim phrases. in summary, existing literature on claims concentrates entirely on sentence-level claim identification and does not investigate on eliciting fine-grained claim spans. in this work, we endeavor to move from coarse-grained claim detection to fine-grained claim span identification. we consolidate a large manually annotated twitter dataset for claim span identification task and benchmark it with various baselines and a dedicated description-based model. ## dataset over the past few years, several claim detection datasets have been released @xcite . however, none of these corpora come with claim-based rationales that quantify a post as a claim. to bridge this gap, we propose curt (claim unit recognition in tweets), a large scale twitter corpus with tokenlevel claim span annotations. ## proposed methodology in this section, we outline daberta and its intricacies. the main aim is to seamlessly coalesce critical domain-specific information into pre-trained language models (plm). to this end, we introduce description infuser network (descnet), a plug-and-play adapter module that conditions the lm representations with respect to the handcrafted descriptions. the underlying principle behind this theoretical formalization is to link a claim span to a claim description to guide the model on what to focus on explicitly. as shown in figure 2 , desc-net houses two sub-components, namely, compositional de-attention block (coda) and interactive gating mechanism (igm). the particulars of each component are delineated in the following sections. claim descriptions. before delving into coda and igm, we first examine claim descriptions, which are the cornerstone of the proposed model. claim descriptions are handcrafted templates that guide the model where to concentrate its focus. the inclusion of claim description encourages the model to focus on the most essential phrases in the input tweet, which may be thought of as guided attention that leads to increased performance. we judiciously curated our claim descriptions in accordance with the annotation guidelines for claims and non-claims offered by @xcite . in ## experiments and results baseline models. we employ the following baseline systems. ▷ cnn+crf: a convolutional neural network (cnn) trained with glove @xcite and a crf head on top. ▷ bil-stm+crf @xcite : a sequence labeling model comprising bidirectional long short-term memory (bilstm) and crf layer. ▷ bert @xcite : a bidirectional transformerinspired auto-encoder language model fine-tune for our span identification task. ▷ distilbert @xcite : a smaller, faster, and lighter version of bert fine-tune on our dataset for the task at hand. ▷ spanbert @xcite : an enhanced version of the bert trained on span prediction objective. ▷ roberta @xcite : a robustly optimized bert approach, roberta, is a variant of bert with improved training methodology. we fine-tune it on our dataset. ▷ nlrg (chhablani et al., 2021): a system proposed at semeval-2021 task 5 on toxic span detection @xcite . it is a combination of spanbert and roberta where the former model is used for predicting the span start and end, while the latter is used for token classification. ▷ hitsz-hlt @xcite : the system topped the semeval-2021 task on toxic span detection. they approached the task as a combination of sequence labeling and span extraction and proposed an ensemble of three bert-based models. evaluation metrics. in concordance with @xcite , we evaluate the performance of all the systems, based on token-level precision (p), recall (r), and f1 scores. to further put a lens over how the models fare for different token types, we calculate the micro-level precision, recall, and f1 score for each of the 'b', 'i', and 'o' tokens. 3 lastly, to quantify the number of tokens included in the spans, we also report the dice similarity coefficient (dsc) @xcite . performance comparison. we summarize our collated results in ablation study. hyper-parameter tuning. we utilize the base version of roberta @xcite to propose daberta. the model is trained end-to-end using the adam optimizer (kingma and ba, 2014), learning rate of 4e-5, and batch size of 32 for 20 epochs with early stopping if the dice score does not improve after 5 epochs. we used the nvidia tesla v100 32 gb gpu. the hyper-parameter tuning is done with respect to the validation dataset. figure 4 reflects the effect of integrating the adapter descnet at different layer of roberta. it is observed that the performance consistently increases as the integration is done at a higher level of roberta layers. this is admissible as studies on probing the plm layers suggest that different layers encode distinct linguistic properties @xcite . furthermore, evidence by @xcite suggests that the lower layers of a language model encode the syntactic information, whereas the higher layers capture the complex semantics. as we strive to employ deep semantic interaction between the plm representations and the claim descriptions, our results are consistent with their findings. error analysis. in this section, we manually analyze the errors the models are prone to make. ## conclusion through this systematic research, we introduced the novel task of claim span identification, which is valuable on various fronts. we conducted an evidence-based document retrieval experiment, demonstrating that employing claim spans retrieves more relevant evidence than using the entire tweet. furthermore, as there exists no specialized corpus for claim span identification, we compiled curt, a large-scale twitter corpus consisting of around 7.5k tweets annotated with token-level claim spans. we showed convincing results using various token classification baselines on our dataset. moreover, we benchmarked curt with daberta, an adapter-based variant of roberta, that encapsulates critical domain-specific information into the pre-trained model via claim descriptions. through extensive qualitative, quantitative, and empirical results, we illustrated how daberta outperforms the other models on different fronts. lastly, we also developed an extensive set of annotation guidelines and released them for further research.