Automatized Detection and Annotation for Calls to Action in L atin- A merican Social Media Postings (2022)

Abstract: Voter mobilization via social media has shown to be an effective tool. While previous research has primarily looked at how calls-to-action (CTAs) were used in Twitter messages from non-profit organizations and protest mobilization, we are interested in identifying the linguistic cues used in CTAs found on Facebook and Twitter for an automatic identification of CTAs. The work is part of an on-going collaboration with researchers from political science, who are investigating CTAs in the period leading up to recent elections in three different Latin American countries. We developed a new NLP pipeline for Spanish to facilitate their work. Our pipeline annotates social media posts with a range of linguistic information and then conducts targeted searches for linguistic cues that allow for an automatic annotation and identification of relevant CTAs. By using carefully crafted and linguistically informed heuristics, our system so far achieves an F1-score of 0.72.

## introduction we here report on a nlp pipeline designed to automatically identify and annotate "calls-to-action" (ctas). we focus specifically on the mobilization of potential voters via social media as part of an on-going collaboration with partners from political science. ctas are of interest for political science because recent years have seen social movements and political parties working with ctas via social media as an effective tool for social mobilization @xcite . guided by the interests of our project partners from political science @xcite , who are investigating the expression of grievances in the context of ethnic political parties, we focus on facebook and twitter posts leading up to the recent (2020) elections across latin america and aim to support their research by automatically detecting and annotating ctas. we faced several challenges in doing this. for one, we cannot build on very much previous nlp work. although initial work has been done to identify ctas for russian @xcite , no research had as yet been done on the linguistic expression of ctas in spanish. we were also faced with severe issues of data scarcity -in 31,229 social media postings contained in the corpus, only 2,542 were found to contain ctas. in addition, the corpus collected by our partners is inherently unbalanced, as it differs across countries and elections. as such we decided to implement a primarily rule-based nlp pipeline for the automatized detection and annotation of ctas. our pipeline is based on previous efforts focusing on english and german and is designed particularly to identify deep morphosyntactic, semantic and pragmatic linguistic features that are relevant for analysis at the discoursal level @xcite and for the linguistic framing of utterances (cf. druckman's frames of communication @xcite ). while other studies have focused on analyzing the effects of such ctas on voters @xcite , in this use case we are concerned with how these mobilizations are linguistically expressed. we therefore aim at automatically identifying ctas via linguistic cues. ## related work as already noted, there is limited previous work we can build on. @xcite showed that twitter messages from non-profit organizations framed as a cta were retweeted more often and generated more interaction between users than other messages, while they were simultaneously the least used strategy for messages. @xcite worked on historical data of bolotnaya protests (2012) in russia to demonstrate the possibility of automatically detecting ctas in social media posts. their classification task yielded an f1score of 0.77, thus showing that ctas in russian can be successfully detected automatically to some degree. while their focus lies on the detection of ctas in a protest setting, where movements use social media to mobilize people and convince them to join the protests, we are focusing on ctas that are targeting voter mobilization. ## characterization of calls to action our specific use case is the analysis of social media posts published in the period leading up to an election and aimed at mobilizing voters. we therefore restricted an identification of ctas to sentences in social media posts that directly or indirectly call upon the addressees for political participation in an election setting. typical examples are (1) and ( 2 ). whereas the ctas in the first two examples are directly encoded and expressed via an imperative, the more indirect example (3) also falls under our definition of a cta. here, the message to politically participate is covertly encoded and does not involve a direct use of an imperative. in order to automatically identify such indirect ctas, it is necessary to implement linguistically informed rules (see §6). we exclude ctas that aim at getting readers to click on a specific link, support an organization with donations or attend events or demonstrations, even though they are linguistically similarly framed as direct ctas. this exclusion is motivated by the research interests of our partners, who are not interested in these other types of ctas, but in voter mobilization. ## nlp pipeline lians for the automatic detection and annotation of ctas we used lians, a rule-based nlp pipeline. lians (linguistic annotation service) has been built on the basis of visargue @xcite , a nlp pipeline initially designed for analyzing linguistic features in spoken dialogs and debates in english and german. we built a spanish version of the pipeline in the context of the cta project. 1 the automated linguistic feature identification in visargue is realized in a rule-based fashion: for each feature, a list of relevant cues (lexical items and constructions) is defined by language experts, and rules are created for the disambiguation according to the context they are found in. this has a clear advantage over a naive and decontextualized application of static word lists. the linguistic features covered by visargue were selected by experts of theoretical and computational linguistics and are strongly grounded in theoretical linguistic insights. we used the existing features and categories as a blueprint for the spanish lians. carefully crafted feature sets and disambiguation rules were added to ensure the reliable annotation of ctas. the main workflow of lians consists of two steps: 1) preprocessing; 2) annotation with prior disambiguation. we first convert the raw posts into standardized xml files, which are then further preprocessed using the stanza nlp kit @xcite . stanza conducts sentence splitting, tokenization and lemmatization on the input files, and adds pos-tags, morphological features and dependency relations of each token (lexeme) as xml attributes. lians further adds discourse unit splitting. approximating the definition of a basic discourse unit @xcite , each clause is defined as one discourse unit (du), e.g., if a sentence is comprised of one matrix clause and one embedded clause, the sentence is defined as containing two dus. after preprocessing, rule-based annotation is applied on the basis of pre-defined disambiguation rules for linguistic features as in the overall corpus is still in the process of being compiled by our political science partners @xcite , and currently consists of around 30,000 posts on twitter and facebook published by political parties, indigenous associations and presidential candidates before and during the three most recent elections in latin america, which took place in bolivia (18 october 2020), ecuador (7 february 2021) and @xcite . ## methodology all posts in the dataset were annotated with lians using the features available for the spanish version. to only extract dus and sentences that thematically deal with elections from the dataset, we manually compiled a list of voting-related keywords such as democracia 'democracy' and elecciones 'elections', including their synonyms and semantically related nouns, verbs and adjectives. corresponding dus and sentences were then extracted. direct ctas can be detected very straightforwardly based simply on the identification of imperative forms, so one easy way of proceeding is to search for instances of imperatives of verbs like votar 'vote' as well as the participle of votar 'voting'. in a further step, we searched for sentences which contain imperatives and election-related keywords from our hand-defined list. in this way, we are able to detect ctas as in example (2). additionally, sentences containing the election date (e.g. este 7 de febrero), or count downs to the election day (faltan 5 días para) were also considered as ctas since they aim at reminding voters about when to exercise their right to vote. moving on to more indirect and emotive ctas, we established rules that search for the noun voto 'vote' preceded by the pronoun tú 'your' or nuestro 'our' to identify posts that directly aimed at motivating voters through the use of more appellative expressions by directly addressing them. ctas as in (4) are often followed by a phrase describing the expected (positive) change that may occur after voting for a specific candidate or voting in general. ## results and discussion a randomly selected subset of 800 facebook and 200 twitter posts from ecuador was manually annotated by two spanish speaking annotators, both co-authors of the paper, in order to evaluate the results from our nlp pipeline. we decided to evaluate on data from ecuador, as the majority of our corpus comes from there. out of 800 facebook posts 156 (16 implicit and 140 explicit) and 29 (10 implicit and 19 explicit) of 200 tweets were identified as ctas. with the help of a more detailed analysis of the linguistic features found in ctas we found that 49 % of them include at least one an overview of our results for the automatic detection of explicit and implicit ctas across the whole corpus is presented in overall our results show that lians can help to annotate small corpora that are unsuitable for machine learning approaches due to their small size and unbalanced nature, therefore reducing the manual annotation effort for our collaboration partners. ## limitations the results of our evaluation show that there is still some room for improvement. first, the currently unbalanced nature of the corpus is not just a problem for machine learning approaches, it also poses challenges for the development of the nlp pipeline, as regional language varieties of all countries must be considered equally when creating the annotation rules. second, our annotations rely on the morphological features provided by the stanza nlp kit. this means that our pipeline struggles when the morphological analysis delivered by stanza is erroneous. for example, this was the case for voto 'vote', which can be a verb or a noun. all instances of voto were analyzed as nouns by stanza, which makes it difficult to identify dus such as voto por un futuro mejor 'vote for a better future'. we adjusted the pipeline to correct for errors of this type. ## conclusion and future work in conclusion, we have implemented an automated approach to identify explicit and implicit electionrelated ctas in spanish social media. while a few previous studies have looked at the identification of ctas in tweets related to protests, to the best of our knowledge our work is the first to look at the linguistic patterns that can be found in such attempts at voter mobilization. our next steps are, first, to annotate more cta posts from peru and bolivia in order to create a more balanced gold standard corpus; and, second, to create our own balanced corpus. third, we aim to expand our set of rules to allow especially more implicit ctas to be annotated automatically. we plan on adding a more sophisticated scoring system that assigns an aggregated score to the identified cta cues. thus, the classification as a cta of a post will depend on how many linguistic cues that indicate a cta are present and how heavily they are weighted. the word vota 'vote', for instance, should receive a greater weight than a modal like tenemos que 'we have to', since the former is a clear cue for a cta, while the latter could also be used in other political contexts. we are confident that we can improve our overall results even further with these extensions of the pipeline and the introduction of a sophisticated scoring system. finally, we also intend to experiment with machine learning approaches once the data set has grown large enough via boot-strapping through cta classifcations and annotations via our pipeline.