Empathy and Hope: Resource Transfer to Model Inter-country Social Media Dynamics (2021)

Abstract: The ongoing COVID-19 pandemic resulted in significant ramifications for international relations ranging from travel restrictions, global ceasefires, and international vaccine production and sharing agreements. Amidst a wave of infections in India that resulted in a systemic breakdown of healthcare infrastructure, a social welfare organization based in Pakistan offered to procure medical-grade oxygen to assist India - a nation which was involved in four wars with Pakistan in the past few decades. In this paper, we focus on Pakistani Twitter users’ response to the ongoing healthcare crisis in India. While #IndiaNeedsOxygen and #PakistanStandsWithIndia featured among the top-trending hashtags in Pakistan, divisive hashtags such as #EndiaSaySorryToKashmir simultaneously started trending. Against the backdrop of a contentious history including four wars, divisive content of this nature, especially when a country is facing an unprecedented healthcare crisis, fuels further deterioration of relations. In this paper, we define a new task of detecting supportive content and demonstrate that existing NLP for social impact tools can be effectively harnessed for such tasks within a quick turnaround time. We also release the first publicly available data set at the intersection of geopolitical relations and a raging pandemic in the context of India and Pakistan.

## introduction the covid-19 pandemic started in late 2019 @xcite and as of this writing is still ongoing. several factors -geopolitical, economic, social among others -dramatically influenced health outcomes around the world. in this paper, we focus on the ongoing (as of @xcite infection wave in india @xcite . after aggressive initial steps to successfully curb the spread of the virus, case counts exploded in india towards the end of april 2021. the rapidity of the spread overwhelmed the healthcare infrastructure in the country. a widespread shortage of medical-grade oxygen @xcite , overworked medical staff, and full capacity emergency rooms became the norm in major population centers. the crisis was heavily discussed on social media and the associated hashtags were among the most discussed twitter trends globally. in pakistan, a neighboring country that fought four wars with india over the past seven decades @xcite , a significant volume of tweets expressed solidarity with the indian populace primarily through two hashtags -#indianeedsoxygen and #pakistan-standswithindia. in addition, the hashtag #endi-asaysorrytokashmir started trending in pakistan. the tweets using this hashtag were primarily divisive and often referenced a long-running territorial dispute at the heart of india-pakistan relations. amidst a far-reaching and rapidly progressing pandemic, divisive content of this nature negatively impacts the mental well-being of the affected population and can contribute to strained relations. hashtag based filtering, while extremely effective, cannot solely identify supportive content. for instance, users can hijack trending hashtags and post content that violates the spirit of the hashtag (see in this paper, we present a method to automatically detect supportive content from the tweet text (excluding hashtags, mentions, emojis, and urls). our minimally supervised approach combines multiple soft signals -a hope speech classifier that detects peace-seeking content @xcite , and an empathy-distress classifier trained on a well-known empathy-distress data set @xcite . we further demonstrate superior performance in presence of supervision and release an annotated data set in this important humanitarian domain. model reusability is a major challenge in nlp applications @xcite . we see our paper as preliminary evidence that nlp methods for positive impact research are not isolated efforts, and solutions arising from adjacent tasks can be re-purposed to tackle newer challenges. nlp for positive impact: our work can be described by the following two broad themes specific to this workshop -online well-being & positive information sharing and case studies for nlp for social good. in order to create a positive impact, we believe a research contribution needs to satisfy a subset of the following conditions: (1) a problem domain with a high societal impact; (2) resourcesharing to facilitate scientific progress; and (3) a research theme that spawns a rich line of follow-up work. our paper has the following contributions: social: we analyze the bilateral relationship between countries with a contentious history amidst a raging pandemic. our work is at the intersection of two important themes -geopolitical relations and healthcare crises. we show a significant out-pouring of support and solidarity between the two nations' online communities in the context of the pandemic. barring a few recent efforts @xcite , there is little literature on web manifestation of the india-pakistan relationship co-occurring with other crises. to the best of our knowledge, this is the first analysis of social media text interactions between india and pakistan amidst a pandemic. resource: we present a data set of tweets exploring geopolitical relations between historic adversaries amidst a health crisis. publicly available data sets expressing empathy and distress are scarce @xcite . beyond our immediate objective of detecting supportive tweets, this data set may be useful in answering several other research questions. the reusability argument: we present a compelling case study that nlp for positive impact applications are not isolated tasks. rather, multiple existing resources can be combined to tackle a new challenge in a fast turnaround time setting. ## task in this paper, we consider the task of detecting supportive content. supportive behavior in language has been previously studied. for example, a aaai-2020 shared task focused on detecting disclosure and supportiveness from written accounts of casual and confessional conversations @xcite . our task is slightly different in the sense that we are interested in detecting content where speakers are supporting a country/people severely affected by a healthcare crisis. we define supportive content to be either expressing empathy, distress, or solidarity. our def- empathy our hearts go out to our neighbours who are facing unprecedented misery. pakistani people are praying for you . . . distress i am a pakistani but seriously this is heartbreaking what i am seeing from few days about india.we are enemies but this is about humanity,if we unite in this pandemic we both countries can fight together and can win this battle together,peace . . . solidarity as a human we all are together pray for india and for all people all over the world who are suffering from covid may allah pak save us from this dangerous covid-19 stop hating start praying our definition for not-supportive content does not have a similar psychological grounding. our annotators observed that the not-supportive content in this specific context, primarily ( 1 ) expressed politically motivated hate; (2) demonstrated a warmongering attitude; (3) expressed schadenfreude; (4) mentioned politically contentious issues; and (5) expressed unrelated content such as product promotion etc. ## resource we use two existing resources for our work. next, we present a short description of these resources. ## data our data set, t , consists of 309,394 tweets posted by 150,289 unique users collected between 21 april 2021 and 04 may 2021. the top trending hashtags in pakistan for april 22 and april 23 were retrieved from https://getdaytrends.com/ and all associated tweets were obtained using the twitter api foot_1 . other closely related trending hashtags were also included (e.g., #indianeedsoxygen and #indianeedoxygen, or #pakistanstandswithindia and #pakistanstandwithindia). additional details are in generating country labels for tweets: the twitter api bundles geographic location (coordinates) with tweets. in addition, we utilized a weak signal -if a user's twitter handle contains an india or pakistan flag emoji, then we assume their tweets originated in india or pakistan respectively. in the cases where the location information and our signal are both present, we notice no inconsistency, indicating our weak country signal is robust. ## related work social media response to the ongoing pandemic has received significant research attention: (1) health misinformation (memon and carley, 2020; @xcite , ( 2 ) polarization @xcite , (3) disease modeling @xcite , etc. counterhate measures along the line of counterspeech research @xcite @xcite to combat anti-asian hate @xcite , and community blame @xcite has been studied. our work contrasts with existing literature in three ways: (1) we analyze bilateral relations of nuclear adversaries amidst a raging pandemic; (2) we release a novel data set for wider use exploring related research questions; and (3) we present a new method that combines recent nlp for positive impact advances in a new, timely, and important task. while the political volatility between india and pakistan has been extensively studied by social scientists @xcite @xcite , barring few recent lines of work @xcite @xcite , social media interactions between the civilians of india and pakistan has received little or no attention. all recent work on indian and pakistani social media @xcite @xcite hashtags and punctuations are removed as a preprocessing step. hashtag total india pakistan #indianeedsoxygen 145,975 26,383 19,748 #indianeedoxygen 24,488 5,049 2,400 #pakistanstandswithindia 96,226 12,331 21,583 #pakistanstandwithindia 17,406 2,772 3,790 #endiasaysorrytokashmir 25,081 87 8,022 #indiasaysorrytokashmir 557 15 169 all 309,733 46,651 55,712 while @xcite introduced a novel task of detecting hostility-diffusing, peace seeking hope speech and considered comments on relevant youtube videos as the data set, @xcite is the first work on analyzing web-manifestation (twitter) of political polarization between the two countries and how political parties factor in these discussions. our work leverages two existing resources: (1) a hope speech classifier introduced in @xcite ; and (2) a well-known empathydistress data set @xcite . as already mentioned, our work differs in a key way that we re-purpose these resources for a new nlp for positive impact task: detecting supportive tweets in the context of social media discussions during a national healthcare crisis. our work also draws inspiration from recent findings about mining stance from hashtags @xcite . ## methods, results, and discussion research question: does sampling tweets containing h supportive hashtags alone suffice? we first investigate if hashtag-based filtering alone guarantees supportive tweets with a high probability. we randomly sample 1,000 tweet texts from t supportive and manually annotate them. our annotators are provided only the tweet texts, i.e., the body of the tweet excluding hashtags, urls, mentions, and emojis. three annotators fluent in english, hindi, and urdu, and well-versed with the geopolitical events between india and pakistan first independently annotated these tweets and achieved a fleiss' κ score of 0.76 indicating moderate agreement. next, disagreements are resolved through a follow-up adjudication process and a higher fleiss' κ score of 0.86 is reached. of the randomly chosen 1,000 tweets 444 tweets, i.e., 44.4% were marked positive. this result indicates that solely relying on supportive hashtag will not do better than chance and underscores the importance of sophisticated methods. in addition, we randomly sampled 1,000 tweet texts from t supportive ∪t not-supportive as our test set (denoted as d eval ). throughout our annotation process, whenever consensus label is absent, following standard literature @xcite , we consider the majority label as the gold-standard label. annotator subjectivity is a well-studied research as already described, the hope speech classifier is designed for a different scenario of detecting peace-seeking, hostility diffusing content from social media discussions generated during a conflict. our current task of detecting supportive tweets, although related, is not identical. furthermore, the classifier is trained on a different social media platform, youtube, that allows unstructured text without any length restriction, whereas twitter allows unstructured text but imposes a length restriction. similarly, the empathy-distress classifier is trained on a different data set of user responses to news events. hence, a pertinent research question is if the hope speech classifier or the empathy-distress classifier is any good in differentiating between supportive and not-supportive tweets. we first start with a simple experiment to il-lustrate that the resources provide useful signal. let @xmath2, y } such that x ∼ t supportive and y ∼ t not-supportive , i.e., s consists of tweet pairs x, y where x and y are randomly drawn from the pool of tweets with supportive and not-supportive hashtags, respectively. let p h (z) and p e (z) denote the predicted hope speech and empathy-distress probabilities of tweet z. we compute: it is unsurprising that r h has a much higher value than r e . the hope speech classifier is trained on a data set relevant to a recent india-pakistan conflict and thus has a substantial overlap in domain. hence, a general nature of positive dialogue may indicate a desire to put things behind and help each other. in contrast, the empathy-distress classifier is trained on a broad, diverse, data set of user responses to news events and has no overlap with the current domain. however, when we rank tweets from t supportive by the classifier's probability, we notice that top predictions are of extremely high quality in both cases. we annotate top 1,000 unique tweets from t supportive ranked by p h (.) and obtain 950 positives. similarly, top 1,000 unique tweets from t supportive ranked by p e (.) yield 899 positives upon manual annotation. moreover, the two classifiers complement each other as among the top 1,000 unique tweets from the hope speech classifier and the top 1,000 unique tweets from the empathy-distress classifier had minimal overlap (62 samples). this annotation task also yielded a substantially higher fleiss' κ score (0.8068) without any follow-up adjudication process indicating that the chosen samples have less ambiguity than our earlier experiment that involved annotating randomly selected tweets from t supportive . our results thus indicate existing resources can be harnessed for informed sampling yielding high-quality positives. research question: how to leverage existing resources to design an effective classifier to detect supportive tweets? we utilize two existing resources, a hope speech classifier from @xcite , and an empathy-distress data set from @xcite . we first train an empathy-distress classifier on the empathy-distress data set that can classify tweets as exhibiting empathy or distress, or not. our pipeline utilizes the hope speech and empathy-distress classifiers and constructs a weakly labeled data set where the positive examples exhibit themes like empathy, distress, support, and solidarity -the supportive speech, and the negative examples exhibit themes like controversy, whataboutism, and hostility -the not-supportive speech. the two classifiers are used to label tweets and the positive class probability is used to rank all the tweets in the set t supportive ∪ t not-supportive yielding two ranked lists. d + informed contains all tweets using any of the top 1,000 tweets in both ranked lists (2,000 in total, 1,938 unique) are considered positive samples, and a set of negative samples, d - informed , is constructed by randomly sampling 500 tweets each from the bottom 80% of both ranked lists (1,000 in total, 1,000 unique). the full data set construction pipeline is presented in algorithm 1. the trained model is denoted as m informed . earlier research has reported hashtags as an effective way to obtain weak labels @xcite . we contrast m informed against a baseline that uses hashtags alone as a source of weak labels and contains the identical number of (weakly labeled) positives and negatives as d informed . essentially, any tweet belonging to t supportive is considered a positive and any tweet belonging to t not-supportive is considered a negative. positives and negative examples are randomly sampled from these sets and a data set with the same proportions as d informed is constructed. the trained model is denoted as m hashtag . we train our classifiers using bert @xcite (bert-base-uncased) using the transformers library @xcite and a 90/10 train/validation split. in addition, since english social media content from the indian subcontinent exhibits a variety of disfluencies @xcite , and since the svm baseline has been successfully applied to the original hope speech detection task @xcite , we include an svm baseline as well that uses tf-idf vectors as document feature representations. the trained models are evaluated on d eval , 1000 randomly sampled tweets from t supportive ∪ t not-supportive . note that hashtags, urls, emojis, mentions, and punctuation are removed from the tweets prior to training. ## ethical and societal implications while the setting discussed in the paper involves humanitarian tasks, the techniques can be trivially adapted with the explicit objective to censor empathetic content. in many recent conflicts in the indian subcontinent, such systems can have adverse social effects, and thus particular care is needed before these systems are deployed. also, languagespecific features can sometimes cause syntactically similar but semantically opposite content to be surfaced underscoring the need for a human-in-theloop setting before such systems are deployed for social media content moderation tasks. finally, our classifier relies on a black box hope speech classifier and thus runs the risk of propagating possible biases from the black box model. further case studies need to be considered before deployment and we welcome a thorough investigation of our released data set from the scientific community. ## conclusions in this paper, we present a task and associated resources for a vital domain -geopolitical relations against the backdrop of a raging pandemic. we release a data set of tweets discussing the oxygen crisis and healthcare system collapse in india due to a covid-19 wave. our data set is geographically diverse and connects several diverse themes -a long acrimonious history between two neighboring countries that involves four wars and a recent bilateral relations breakdown, a raging pandemic that has claimed several hundred thousand lives within a few weeks and is still ongoing. our analysis reveals a strong humanitarian streak that prioritizes health and well-being over past geographical or ethnic disputes. we then re-purpose existing resources designed for adjacent tasks like hope speech and empathy distress detection and utilize these to identify supportive tweets. our experiments reveal that nlp for positive impact tasks can utilize existing adjacent resources to rapidly bootstrap solutions.