T r C laim-19: The First Collection for T urkish Check-Worthy Claim Detection with Annotator Rationales (2020)

Abstract: Massive misinformation spread over Internet has many negative impacts on our lives. While spreading a claim is easy, investigating its veracity is hard and time consuming, Therefore, we urgently need systems to help human fact-checkers. However, available data resources to develop effective systems are limited and the vast majority of them is for English. In this work, we introduce TrClaim-19, which is the very first labeled dataset for Turkish check-worthy claims. TrClaim-19 consists of labeled 2287 Turkish tweets with annotator rationales, enabling us to better understand the characteristics of check-worthy claims. The rationales we collected suggest that claims’ topics and their possible negative impacts are the main factors affecting their check-worthiness.

## introduction in 2013, world economic forum (wef) listed massive digital misinformation as one of the top global risks likely to occur in 10 years 1 . unfortunately, we witnessed many unpleasant incidents due to misinformation spread over internet such as massive stock price changes foot_1 , gunfights foot_2 , and others. since the start of covid-19 pandemic, we have also observed many incidents showing the value of true information and how misinformation about health issues can be deadly (e.g., misusing disinfectants to prevent coronavirus after donald trump suggested injecting disinfectants as treatment 4 ). in order to prevent the negative outcomes of misinformation, many fact-checking websites emerged all over the world in the last decade @xcite . the fact-checking websites manually investigate veracity of claims and share their findings with their readers. while they play an important role in the combat against misinformation, their precious journalistic effort is not enough to reduce spread of misinformation and its negative outcomes. while making a claim is so easy, investigating its veracity is highly time consuming, taking around one day @xcite . furthermore, @xcite report that misinformation spread eight times faster than true information. hence, we need effective solutions to help human fact-checkers and to reduce the negative impact of misinformation. as @xcite outline, the first task of a fact-checking system is to detect whether a statement contains a check-worthy claim or not. considering the massive amount of messages shared on social media platforms, check-worthy claim detection models help human fact-checkers to filter out unimportant claims and use their valuable time to detect veracity of the most important claims. a number of researchers worked on this problem (e.g., @xcite @xcite ) and shared tasks for check-worthy claim detection have been organized @xcite @xcite . while researchers showed great interest in factchecking, the available resources are still limited and the vast majority of the studies focused on english. regarding the task of detecting check-worthy claims, the only available labeled datasets are for english and arabic @xcite . however, as wef notes in its aforementioned report, misinformation is a global problem affecting all countries. misinformation can also spread internationally. for instance, during 2019 european elections, same or similar stories have been shared in different languages across european countries @xcite . hence, in order to have an effective combat against spread of misinformation, we need research studies for a wide range of languages. in this work, we focus on turkish and introduce trclaim-19, which is the very first labeled turkish tweets with the rationales of annotators for checkworthy claim detection task. turkish is a particularly important language for fact-checking studies because @xcite report that 49% of internet users in turkey coincide with at least one fake news in a week, which is higher than all other countries investigated in their study. furthermore, being a member of altaic language family, turkish language has different linguistic features than other languages studied for fact-checking, such as being an agglutinative language and having flexible word order structure in sentences. in addition to developing a useful resource for the research community, we also seek answers for the following research questions. in particular, we have first crawled turkish tweets for 344 days in 2019, tracking important events happened in turkey such as local elections, earthquake in istanbul, and military operation in syria. eventually, we gathered around 225 millions turkish tweets. subsequently, we crawled 765 claims fact-checked by two turkish fact-checking websites. next, for each claim, we retrieved three tweets from our tweet crawl using lucene search engine library foot_4 . each retrieved tweet has been labeled by three separate annotators. for each tweet, we asked annotators whether it is relevant to the respective claim, and whether it contains a check-worthy claim. inspired by @xcite 's study, we also asked their rationale for the tweets labeled as check-worthy. the contributions of our work are as follows. ## related work a number of researchers created annotated datasets for check-worthy claim detection. to our knowledge, @xcite created the first check worthiness dataset using transcripts of the u.s. presidential debates. they invited journalists, professors and university students to label the data. each sentence in debates is labeled by at least two annotators. they used a three-scale label which are "non factual", "unimportant factual" and "check-worthy factual". in total, they labeled 1571 sentences. following @xcite , us political debates have been used in many other studies. @xcite constructed a check-worthy detection dataset using us primary and presidential debates. they identified 9 reputable news portals and fact-checking websites (e.g., polifact, cnn and factcheck.org), and considered statements which are fact-checked by at least one of these websites, as check-worthy. @xcite also applied the same method to build their dataset using again us debates. they report that the agreement between fact-checking websites is low: only one sentence is fact-checked by all nine websites they listed while 880 statements are fact-checked by only one website. in a follow-up study, @xcite introduce claimrank for detecting check-worthy claims in english and arabic. however, the arabic data they use is just the translation of the us election debates. clef check that! lab have been organizing shared tasks for detecting check-worthy claims since 2018. in 2018, similar to other datasets, they used us debates and political speeches, and considered statements fact-checked by factcheck.org as check-worthy @xcite . for arabic check-worthy detection task, they just used translations of the english dataset. in 2019, the organizers of the lab used the extension of the previous year's dataset @xcite . in 2020, clef check that! lab 7 organizers constructed datasets using tweets and again political debates @xcite . the english tweet dataset consists of 962 tweets about covid-19 pandemic. the arabic dataset consists of 7500 tweets about 15 trending topics at the time of crawling among arab social media users such as covid-19 and trump peace plan 8 . overall, the vast majority of the existing datasets are english and use u.s. political speeches and debates. trclaim-19 distinguishes from the existing data resources as follows. 1) tr-claim19 is the first collection for turkish check-worthy claims. 2) trclaim-19 is the first collection providing with the rationales of annotators. 3) we collect labels for each tweet from three annotators, enabling us to analyze subjectivity of the task and different rationales for the same check-worthy claims. ## tweet collection social media platforms play an important role in spreading misinformation. therefore, we focus on claims spread on twitter. in order to gather tweets with check-worthy claims, we crawled turkish tweets between january 1, 2019 and december 31, 2019 9 using twitter api. we used keyword tracking approach enabling us to gather tweets about important topics. in 2019, there were many major events happened in turkey, such as local elections 10 , an earthquake in istanbul 11 , a military operation in syria 12 , and others. therefore, we used keywords related to these major events such as "sec ¸im" (election), "terrör" (terror), "ekonomi" (economics), "mülteci" (refugee), "suriye" (syria), politician names (e.g., "erdogan"), and others. we used a dynamic keyword list, instead of a static one, such that we updated the list whenever a new important event happened. for instance, after the earthquake in istanbul in september 26, 2019, we added the word "deprem" (earthquake) in our keyword list. at the end of our long crawling process, we have collected around 225 millions turkish tweets. the distribution of tweets crawled for each month is shown in figure 1 . the number of tweets collected 9 crawling had stopped only for three weeks in july 2019 10 www.bbc.com/news/world-europe47764393 11 https://en.wikipedia.org/wiki/2019_ istanbul_earthquake 12 www.aa.com.tr/en/operation-peacespring/operation-peace-spring-starts-inn-syria-erdogan/1607147 between january and june is much higher than other months because local elections were hold on 31 st march and 24 th of june (repeated elections for istanbul) and twitter was one of the main platforms for political discussions. ## tweet selection as we have collected tweets more than we can annotate, we have to select a sample among them. a random sampling could cause many not-checkworthy tweets. inspired by test collection construction methodology in the field of information retrieval @xcite , we first collected claims fact-checked by fact-checking websites (section 4.1) and then used each claim as a search query to retrieve tweets which are similar to claims we collected (section 4.2). using tweets similar to claims also allow us to seek answer for our research question rq-2. now we explain our tweet selection process in detail. ## annotation process two authors of this paper and 7 volunteers annotated the tweets we selected. all annotators are native speakers of turkish language and have degrees at various disciplines including law, computer science, dentistry, history, and business administration. their ages are between 22 and 35. we believe that having annotators with diverse backgrounds allows us to capture different thoughts and perspectives about check-worthy claims. before starting the annotation process, we explained the dataset and the annotation task to each volunteer. in the annotation interface, annotators are able to see tweets to be labeled and also respective claims used to retrieve them. annotators are asked the following three questions in the annotation task. we did not put any restrictions to define their rationale not to induce any bias. annotators used a free-style text form to type their rationale behind their check-worthiness judgment. we only requested annotators to do their best to be consistent in their rationale definition and use the same text for the same rationale to better analyze their rationales in the post-process of the data. each tweet has been annotated by three annotators separately. we divided the participants into 3 groups. then each annotator group labeled a separate set of tweets. the sizes of tweet sets were determined based on availability of each annotator group because they did this annotation task voluntarily. in particular, the number of tweets assigned to each group are, 384, 427, and 1476. in order to prevent bias in the annotations, the annotators have not seen labels of others until the labelling process ends. in total, we collected 2287x3=6861 relevance and check-worthiness judgments. ## analysis of trclaim-19 ## baseline results in this section, we provide performance results of four models on our dataset in order to provide reference baselines for future studies. we first randomly selected 635 claims (out of 765) gathered from teyit and dp, and used 1900 tweets retrieved using these selected claims for training. the remaining 387 tweets are used for testing. the baseline models we use are as follows. we use scikit toolkit foot_10 for both svm-bow and lr-bow models with default parameters. we adjust all models in two settings: binary and multiclass classification. in binary classification we use aggregated labels (i.e., "check-worthy" and "not check-worthy") based on majority voting for each tweet. in multiclass classification, we use total number of check-worthy labels for each tweet, yielding a quaternary (4-point) scale of labels (i.e., 0, 1, 2, and 3). we rank tweets based on their check-worthiness using our baseline models. we report average precision (ap), r-precision (r-p), and precision with different cutoff values (p@k) for binary classification. for multiclass classification, we report ndcg scores. based on ap, p@1, p@5, and p@10, m-bert yields the highest scores. berturk model outperforms others based on r-p and ndcg metrics. as expected, bert based models outperform lr-bow and svm-bow models in most of the cases. however, their scores are close in many cases, and svm-bow and lr-bow slightly outperform m-bert and berturk models in various cases. this might be because of the size of the training dataset. future studies might explore weak supervision or cross lingual transfer learning to better fine tune bert models. ## conclusion in this work, we introduce trclaim-19, which is the first annotated dataset for turkish check-worthy claims. we first crawled 225m turkish tweets in 2019 by tracking keywords about important events happened in turkey. in order to select the tweets to be annotated, we first crawled 765 claims factchecked by two turkish fact-checking websites. then we used these claims as search queries to select 3 tweets to be annotated for each claim. in total, we collected annotations for 2287 turkish tweets. in addition to check-worthy annotations, we also collected rationales behind their judgments. furthermore, we provide performance results of baseline models on trclaim-19 for future studies. in our analysis of trclaim-19, we have the following observations. firstly, annotators have low agreement on check-worthiness of claims, suggesting that we need graded judgments, instead of bi-nary judgments. secondly, in many cases, nonexperts disagree with experts on check-worthiness of claims even though they knew that they have been fact-checked by experts. thirdly, annotators might have different rationales for labeling a claim as check-worthy. fourthly, the rationales we collect suggest that topics of claims are the main factors affecting their check-worthiness. in particular, claims about economics, politics, and society are likely to be check-worthy. lastly, the rationales suggest that claims about negative events are more likely to be check-worthy than claims about positive events. in the future, we plan to extend to our work covering other languages. besides being useful data resources, they will also allow us to compare rationales across different languages and cultures. in addition, we plan to apply a think-aloud methodology to collect rationales, allowing us to better understand the thought process of annotators. furthermore, collecting rationales for not-check-worthy claims might be useful to understand disagreement between annotators.