LESA : Linguistic Encapsulation and Semantic Amalgamation Based Generalised Claim Detection from Online Content (2021)

Abstract: The conceptualization of a claim lies at the core of argument mining. The segregation of claims is complex, owing to the divergence in textual syntax and context across different distributions. Another pressing issue is the unavailability of labeled unstructured text for experimentation. In this paper, we propose LESA, a framework which aims at advancing headfirst into expunging the former issue by assembling a source-independent generalized model that captures syntactic features through part-of-speech and dependency embeddings, as well as contextual features through a fine-tuned language model. We resolve the latter issue by annotating a Twitter dataset which aims at providing a testing ground on a large unstructured dataset. Experimental results show that LESA improves upon the state-of-the-art performance across six benchmark claim datasets by an average of 3 claim-F1 points for in-domain experiments and by 2 claim-F1 points for general-domain experiments. On our dataset too, LESA outperforms existing baselines by 1 claim-F1 point on the in-domain experiments and 2 claim-F1 points on the general-domain experiments. We also release comprehensive data annotation guidelines compiled during the annotation phase (which was missing in the current literature).

## introduction the concept of a claim lies at the core of the argument mining task. @xcite , in his argumentation theory, described the term 'claim' as 'an assertion that deserves our attention'; albeit not very precise, it still serves as an initial insight. in recent years, @xcite described a 'claim' as 'a disputed statement that we try to support with reasons.' the predicament behind the claim detection task exists given the disparity in conceptualization and lack of a proper definition of a claim. the task of claim detection across different domains has garnered tremendous attention so far owing to an uprise in social media consumption and by extension the existence of fake news, online debates, widely-read blogs, etc. as an elementary example, claim detection can be used as a precursor to fact-checking; wherein segregation of claims aids in restricting the corpus that needs a fact-check. a few examples are shown in most of the existing works are built upon two fundamental pillars -semantic encapsulation @xcite and syntactic encapsulation @xcite . they mainly focus on adapting to texts from similar distributions or topics or both. secondly, they often exercise against wellstructured and laboriously pre-processed formal texts owing to the lack of a labeled corpus consisting of unstructured texts. as a result, claim detection from unstructured raw data still lies under a relatively less explored umbrella. motivation: claims can be sourced from a variety of sources, e.g., online social media texts, microblogs, wikipedia articles, etc. it is, however, crucial to pay special attention to claims observed on online social media (osm) sites @xcite . twitter, being a major osm platform, provides the perfect playground for different ideologies and perspectives. over time, twitter has emerged as the hub for short, unstructured pieces of text that describe anything from news to personal life. most individuals view and believe things that align with their compass and prior knowledge, aka conformity bias @xcite -users tend to make bold claims that usually create a clash between users of varied opinions. at times, these claims incite a negative impact on individuals and society. as an example, a tweet that reads "alcohol cures corona" can lead to massive retweeting and consequential unrest, especially in times of a pandemic, when people are more vulnerable to suggestions. in such cases, automated promotion of claims for immediate further checks could prove to be of utmost importance. an automated system is pivotal since osm data is far too voluminous to allow for manual human checks, even if it was an expert. at the same time deploying separate systems contingent on the source of a text is inefficient and moves away from the goal of attaining human intelligence in natural language processing tasks. an ideal situation would be a framework that can effectively detect claims in the general setting. however, a major bottleneck towards this goal is the unavailability of an annotated dataset from noisy platforms like twitter. we acknowledge this bottleneck and, in addition to proposing a generalised framework, we develop a qualitative annotated resource and guidelines for claim detection in tweets. proposed method: there exists several claim detection models; however, the downside is that most of them are trained on structured text from a specific domain. therefore, in this work, we propose lesa, a linguistic encapsulation and semantic amalgamation based generalized claim detection model that is capable of accounting for different text distributions, simultaneously. to formalize this, we divide the text, contingent upon their structure, into three broad categories -noisy text (tweets), semi-noisy text (comments), and nonnoisy text (news, essays, etc.). we model each category separately in a joint framework and fuse them together using attention layers. since the task of claim detection has a strong association with the structure of the input, as argued by @xcite , we leverage two linguistic properties -part-of-speech (pos) and dependency tree, to capture the linguistic variations of each category. subsequently, we amalgamate these features with bert @xcite for classification. we evaluate lesa on seven different datasets (including our twitter dataset) and observe efficient performance in each case. moreover, we compare lesa's performance against various state-of-theart systems for all seven datasets in the general and individual settings. the comparative study advocates the superior performance of lesa. summary of the contributions: we summarize our major contributions below: we propose a generalized claim detection model, lesa, that identifies the presence of claims in any online text, without prior knowledge of the source and independent of the domain. to the best of our knowledge, this is the first attempt to define a model that handles claim detection from both structured and unstructured data in conjunction. we evaluate lesa against multiple state-of-theart models on six benchmark claim detection datasets and our own twitter dataset. comparison suggests lesa's superior performance across datasets and the significance of each model component. reproducibility: code and dataset is publicly available at https://github.com/lcs2-iiitd/ lesa-eacl-2021 . appendix comprises of detailed dataset description, annotation guidelines, hyperparameters, and additional results. ## related work in the past decade, the task of claim detection has become a popular research area in text processing with an initial pioneering attempt by @xcite . they worked on mining claims from discussion forums and employed a supervised approach with features based on sentiment and word-grams. @xcite proposed a context dependent claim detection (cdcd) approach. they described cdc as 'a general, concise statement that directly supports or contests the given topic.' their approach was evaluated over wikipedia articles; it detected sentences that include cdcs using context-based and contextfree features. this was followed by ranking and detecting cdcs using logistic regression. @xcite proposed context-independent claim detection (cicd) using linguistic reasoning, and encapsulated structural information to detect claims. they used constituency parsed trees to extract structural information and predicted parts of the sentence holding a claim using svm. although their approach achieved promising results, they also used a wikipedia dataset which was highly engineered and domain dependent. @xcite used six disparate datasets and contrasted the performance of several supervised models. they performed two sets of experiments -in-domain cd (trained and tested on the same dataset) and cross-domain cd (trained on one and tested on another unseen dataset). they learned divergent conceptualisations of claims over cross-domain datasets. @xcite proposed the first unsupervised approach for claim detection. they hypothesised a "claim sentence query" as an ordered triplet: @xmath0. according to the authors, a claim begins with the word 'that' and is followed by the main concept (mc) or topic name which is further followed by words from a pre-defined claim lexicon (cl). this approach would not fit well for text stemming from social media platforms owing to a lack of structure and the use of 'that' as an offset for claim. in recent years transformer-based language models have been employed for claim detection. @xcite used over 5 million selflabeled reddit comments to fine-tune their model. however, they made no attempt to explicitly encapsulate the structure of a sentence. recently, the clef-2020 shared task @xcite attracted multiple models which are tweaked specifically for claim detection. @xcite bagged the first position in the task using a fine-tuned roberta @xcite model with mean pooling and dropout. first runner up of the challenge, @xcite used logistic regression on various meta-data tweet features and a roberta-based prediction. cheema et al. (2020), the second runner up, incorporated pretrained bert embeddings along with pos and dependency tags as features trained using svm. traditional approaches focused primarily on the syntactic representations of claims and textual feature generation, while recent neural methods leverage transformer models. with lesa, we attempt to learn from the past while building for the future -we propose encapsulating syntactic representations in the form of pos tags and dependency sequences along with the semantics of the input text using transformer-based bert @xcite . another key observation has been the use of highly structured and domain-engineered datasets for training the existing models in claim detection. in the current age of alarming disinformation, we recognise the augmented need for a system that can detect claims in online text independent of its origin, context or domain. therefore, in addition to considering texts from different online mediums, we incorporate, for the first time, a self-annotated large twitter dataset to the relatively structured datasets that exist in this field. ## proposed methodology traditionally, the narrative on claim detection is built around either syntactic @xcite or semantic @xcite properties of the text. however, given our purview on the integration of both, we propose a combined model, lesa , that incorporates exclusively linguistic features leveraged from part-of-speech (pos) tags and dependency tree (dep) as well as semantic features leveraged from transformer-based model, bert @xcite . by the virtue of digital media, we generally deal with texts from three kind of environments: (a) a controlled platform where content is pre-reviewed (e.g., news, essays, etc.); (b) a free platform where authors have the freedom to express themselves without any restrictions on the length (e.g., online comments, wikipedia talk pages); and (c) a free platform with restrictions on the text length (e.g., tweets). the texts in the first category is usually free of any grammatical and typographical mistakes, and thus belong to the non-noisy category. on the other hand, in the third case, texts exhibit a significant amount of noise, in terms of spelling variations, hashtags, emojis, emoticons, abbreviations, etc., to express the desired information within the permissible limit, thus it belongs to the noisy class. the second case is a mixture of the two extreme cases and hence, constitutes the semi-noisy category. we employ three pre-trained models representing noisy, semi-noisy, and non-noisy data for both pos and dependency-based features. the intuition is to leverage the structure-specific linguistic features in a joint framework. domain adaptation from a structured environment to an unstructured one is non-trivial and requires specific processing. therefore, to ensure generalization, we choose to process each input text from three different viewpoints (structure-based segregation), and intelligently select the contributing features among them through an attention mechanism. we use it to extract the pos and depbased linguistic features. subsequently, we fuse the linguistic and semantic features using another attention layer before feeding it to a multilayer perceptron (mlp) based classifier. the idea is to amalgamate diverse set of features from different perspectives and leverage them for the final classification. a high-level architectural diagram is depicted in figure 1 . we design parallel pillars for each viewpoint (right side of figure 1 ) such that the noisy pillar contains pre-trained information from the noisy source and so on. when the common data is passed through the three pillars we hypothesize each pillar's contribution dependending on the type of input data. for example, if the data source is from a noisy platform, we hypothesize that the noisy pillars will have more significance than the other two viewpoints. we demonstrate this effect in ## datasets a number of datasets exist for the task of claim detection in online text @xcite ; however, most of them are formal and structured texts. as we discussed earlier, osm platforms are overwhelmed with various claim-ridden posts. despite the abundance of tweets, literature does not suggest any significant effort for detecting claims in twitter; arguably, the prime reason is the lack of a large-scale dataset. recently, a workshop on claim detection and verification in twitter was organized under clef-2020 @xcite . it had two subtasks related to claim identification with separate datasets. the first dataset consists of 1, 060 covid-19 tweets for claim detection; whereas, the second one comprises of another 1, 000 tweets for claim retrieval. in total, there were 2, 069 annotated tweets of which 1, 704 had claims and 365 were our annotation clef-2020 non-claim claim non-claim 301 47 claim 64 550 unfortunately, the aforementioned limited instances are insufficient to develop an efficient model. therefore, we attempt to develop a new and relatively larger dataset for claim detection in osm platforms. we collected ∼ 40, 000 tweets from various sources @xcite @xcite @xcite and manually annotated them. we additionally included claim detection datasets of @xcite and @xcite and reannotated them in accordance with our guidelines. during the cleaning process, we filtered a majority of tweets due to their irrelevancy and duplicacy. to ensure removal of duplicates, we performed manual checking and exhaustive preprocessing. data annotation: to annotate the tweets, we extend and adapt the claim annotation guidelines of @xcite . the authors targeted and annotated only a subset of claims, i.e., factually verifiable claims. they did not consider personal opinions, sarcastic comments, implicit claims, or claims existing in a sub-sentence or sub-clause level. subsequently, we propose our definition of claims and extrapolate the existing guidelines to be more inclusive, nuanced and applicable to a diverse set of claims. our official definition for claims, adopted from oxford dictionary foot_1 , is to state or assert that something is the case, with or without providing evidence or proof. we present the details of annotation guidelines in @xcite . following the guidelines, we annotated the collected tweets, and to ensure coherence and conformity, we re-annotated the tweets of @xcite and @xcite . it is intriguing to see the differences and similarities of the two guidelines; therefore, we compile a confusion matrix for clef- dataset text noisy twr @realdonaldtrump does ingesting bleach and shining a bright light in the rectal area really cure #covid19? have you tried it? is that what killed kim jong un? #trumpisalaughingstock #trumpisaloser semi-noisy oc *smacks blonde wig on axel* i think as far as diz is concerned, he is very smart but also in certain areas very dumb --witness the fact that he didn't notice his apprentices were going to turn on him, when some of them (cough vexen cough) aren't exactly subtle by nature. wtp not to mention one without any anonymous users talking in capital letters !!!!!!!! non-noisy mt tax data that are not made available for free should not be acquired by the state. pe i believe that education is the single most important factor in the development of a country. vg when's the last time you slipped on the concept of truth? wd the public schools are a bad place to send a kid for a good education anymore. other datasets: since we attempt to create a generalized model that is able to detect the presence of a claim in any online text, we accumulate, in addition to the twitter dataset, six publicly available benchmark datasets: (i) online comments (oc) containing blog threads of livejournal @xcite , (ii) wiki talk pages (wtp) @xcite , (iii) german microtext (mt) @xcite , (iv) persuasive student essay (pe) @xcite , (v) various genres (vg) containing newspaper editorials, parliamentary records and judicial summaries, and (vi) web discourse (wd) containing blog posts or user comments @xcite . all datasets utilised in this paper contain english texts only. for german microtexts (mt), we used the publicly available english translated version published by mt's original authors @xcite ). the same was utilized by @xcite . the datasets are formed by considering text at the sentence level. for example, in persuasive essays (pe) dataset, each essay is broken into sentences and each sentence is individually annotated for a claim. considering the structure of the input texts in these datasets, we group them into three categories as follows: noisy (twitter), semi-noisy (oc, wtp), non-noisy (mt, pe, vg, wd). we list one example from each dataset in ## experimental setup for all datasets besides twitter, we use the train, validation, and test splits as provided by ukp lab foot_2 . a mutually exhaustive 70:15:15 split was maintained for twitter dataset. we compute pos embeddings by learning word2vec skip-gram model @xcite on the tri-gram foot_3 pos sequence. for the skip-gram model, we set context @xmath1, embedding @xmath2, and discard the pos sequence with @xmath3. subsequently, we compute dependency embeddings with @xmath4. please note that the choice of using bi-lstm, as oppose to transformers, for extracting the pos features is empirical foot_4 . the outputs of the pos and dependency embedding layers are subsequently fed to a bil-stm and globalaveragepooling layers, respectively. their respective outputs are projected to a 32-dimensional representation for the fusion. we employ huggingface's bert implementation for computing the tweet representation. the ## experimental results in all aforementioned experiments, we use our pre-defined concept of three viewpoints, i.e., noisy, semi-noisy and non-noisy. therefore, for completeness, we also construct a combined viewpoint which does not contain any structure-specific pillar in pos or dep branches. the results from this ablation experiment are reported in lesa (combinedview) row. we observe that the combined-view results are inferior to the variant with separate viewpoints for each component (c.f. second last and last row of ## conclusion in this paper, we addressed the task of claim detection from online posts. to do this, we proposed a generic and novel deep neural framework, lesa, that leverages the pre-trained language model and two linguistic features, corresponding to the syntactic properties of input texts, for the final classification. additionally, we tackled the texts from distinct sources for the claim detection task in a novel way. in particular, we categorized the input text as noisy, non-noisy, and semi-noisy based on the source, and modeled them separately. subsequently, we fused them together through an attention module as the combined representation. one of the major bottlenecks of claim detection in online social media platforms is the lack of qualitative annotation guidelines and a sufficiently large annotated dataset. therefore, we developed a large twitter dataset of ∼ 10, 000 manually annotated tweets for claim detection. in addition to our twitter dataset, we employed six benchmark datasets (representing either semi-noisy or non-noisy input channels) for evaluation of the proposed model. we compared the performance of lesa against four state-of-the-art systems and two pre-trained language models. comparison showed the superiority of the proposed model @xmath5. as a byproduct of the study, we released a comprehensive guideline for claim annotation.