It’s about What and How you say it: A Corpus with Stance and Sentiment Annotation for COVID -19 Vaccines Posts on X / T witter by B razilian Political Elites (2025)

Abstract: This paper details the development of a corpus with posts in Brazilian Portuguese published by Brazilian political elites on X (formerly Twitter) regarding COVID-19 vaccines. The corpus consists of 9,045 posts annotated for relevance, stance and sentiment towards COVID-19 vaccines and vaccination during the first three years of the COVID-19 pandemic (2020-2022).Nine annotators, working in three groups, classified relevance, stance, and sentiment in messages posted between 2020 and 2022 by local political elites. The annotators underwent extensive training, and weekly meetings were conducted to ensure intra-group annotation consistency. The analysis revealed fair to moderate inter-annotator agreement (Average Krippendorf’s alpha of 0.94 for relevance, 0,67 for sentiment and 0,70 for stance). This work makes four significant contributions to the literature. First, it addresses the scarcity of corpora in Brazilian Portuguese, particularly on COVID-19 or vaccines in general. Second, it provides a reliable annotation scheme for sentiment and stance classification, distinguishing both tasks, thereby improving classification precision. Third, it offers a corpus annotated with stance and sentiment according to this scheme, demonstrating how these tasks differ and how conflating them may lead to inconsistencies in corpus construction, as a results of confounding these phenomena — a recurring issue in NLP research beyond studies focusing on vaccines. And fourth, this annotated corpus may serve as the gold standard for fine-tuning and evaluating supervised machine learning models for relevance, sentiment and stance analysis of X posts on similar domains.

## introduction social media platforms, such as x (formerly twitter), play a crucial role in analyzing public discourse on policy issues, particularly due to their widespread adoption by both the general public and politicians @xcite @xcite . due to its prominence in public discourse and widespread adoption, x has become an essential tool for monitoring public opinion (e.g. @xcite @xcite . however, the massive volume of user-generated data makes manual analysis impractical, underscoring the need for annotated corpora to improve accuracy in tasks like filtering relevant content, sentiment analysis and stance detection. annotated datasets not only allow for the training of algorithms to recognize subtle patterns but also enable benchmarking, validation, and adaptation to emerging topics like vaccine hesitancy. for public health applications, such as tracking covid-19 discourse, rigorously labeled corpora are indispensable. one approach to analyzing social media discussions is sentiment analysis, a subfield of natural language processing (nlp) that aims to automatically classify the emotional tone of a given text @xcite . over the years, it has played a crucial role in examining public discourse across diverse domains, including politics (e.g. @xcite , consumer behavior (e.g. @xcite , financial markets (e.g. @xcite , and healthrelated discussions, such as the covid-19 pandemic and vaccines (e.g. @xcite @xcite . in sentiment analysis, texts are typically classified as expressing positive, negative, or neutral sentiment. however, sentiment analysis is not well-suited to capture the stance individuals take when reacting to policy questions. instead, stance detection studies in nlp are directed at identifying an author's position regarding a specific proposition or predefined target @xcite @xcite . unlike sentiment analysis, which assesses the overall sentiment of a text, stance detection determines the opinion expressed toward a particular entity or topic, with or without sentiment demonstrations. typically, it categorizes documents as favorable, unfavorable, or neutral in relation to the target. this approach has been applied in various contexts, including product reviews (e.g. @xcite , political debates (e.g. @xcite @xcite @xcite , and fake news detection (e.g. @xcite . distinguishing between these tasks is essential for a refined analysis of opinion and tone in online debates, including discussions on x @xcite despite the importance of sentiment and stance classification, resources for these tasks remain scarce in languages other than english, particularly for brazilian portuguese @xcite @xcite @xcite . in this context, the development of an annotated corpus in portuguese, focusing on posts about covid-19 vaccines, constitutes a significant contribution. such a corpus facilitates the automatic filtering and classification of stance and sentiment, improving our understanding of social media discussions and offering researchers a valuable resource to further explore this topic. however, our approach departs from conventional frameworks in sentiment analysis and stance detection by introducing a third classification: "unclear". this category is distinct from "neutral" -widely adopted in existing research -as it accounts for instances where content neither adopts a neutral stance nor conveys any discernible sentiment or position. specifically, we classify cases as "unclear" when ambiguity or insufficient context makes it impossible to determine a definitive stance or sentiment. rather than interpreting such cases as an absence of stance, we frame them as uncertain or indeterminate. to bridge this gap, we introduce a curated and annotated corpus of posts concerning covid-19 vaccines and vaccination in portuguese, that measures both sentiment and stance classification. we present a corpus of x posts foot_0 posted by brazilian political elites (i.e., candidates officially endorsed by their parties in local elections) between 2020 and 2022. the corpus is annotated for relevance, stance and sentiment. in total, 9,045 posts were annotated for relevance. out of these, 5,937 posts (65,64%) were classified as relevant and received annotations for stance and sentiment. the remainder of this article is organized as follows. section 2 reviews related research on stance detection, sentiment analysis, and studies concerning covid-19 vaccine discourse. section 3 details the collection and filtering of the corpus of posts, as well as the annotation guidelines and protocol. section 4 presents and discusses the annotation results and the final corpus. finally, section 5 outlines our conclusions and directions for future research. ## related work during the covid-19 pandemic, researchers employed nlp methods to investigate shifts in public discourse during critical phases such as lockdown measures, vaccine distribution, and policy changes using both unsupervised and supervised machine learning techniques (e.g. @xcite @xcite @xcite @xcite . by harnessing x data, these studies systematically analyze real-time emotional fluctuations and shifts in public narratives, thereby illuminating collective behavioral patterns and societal dynamics. however, while automated tools like vader 2 and textblob are widely employed for sentiment classification in the covid-19 domain (e.g. @xcite @xcite @xcite @xcite , there remains a notable scarcity of humanannotated corpora specifically tailored to vaccinerelated discourse or other covid-19 topics (positive examples are @xcite @xcite , especially to languages other than english. to a lesser extent, and somewhat surprisingly, stance detection studies have been pivotal in analyzing public attitudes toward vaccines, lockdown measures, and government responses during the covid-19 pandemic. @xcite conducted a systematic review of research employ- ## material and methods to construct this corpus, we collected posts from 2020 to 2022 posted by candidates running for mayor in all brazilian capitals during the 2020 municipal elections. x profiles were selected based on the candidates' registration and certification by the brazilian superior electoral court (tse). out of the 295 candidates running for mayor in the 26 state capitals, we identified existing accounts for 258 (87.5%). among these, 89 (30.17% out of 295) profiles were inactive during the analyzed period foot_1 , and 35 (12% out of 295) accounts did not publish content relevant to our research topic. our final sample consisted of 143 (48.5%) mayoral candidates. we utilized then x's rest api foot_2 to collect posts published by these accounts dur-ing 2020 and 2021. however, due to changes in the api usage policies in 2022, we employed twscrape foot_3 , a dedicated python package designed for collecting x data. to refine the dataset, we filtered all posts published by these candidates using a keyword-based selection process foot_4 . as highlighted by @xcite , this method is preferable to alternative approaches, such as subjective categorization, as it provides researchers with greater control, ensures reproducibility, and can be adapted for use across different media platforms. the set of keywords used in this study was developed in several test trials based on observations of spelling variations, term frequency, and usage. orthographic and spelling issues were addressed after a preliminary analysis of common variations used by x users. as an additional measure, we also accounted for capitalization of terms. subsequently, we identified posts that, however containing keyword terms, did not refer to covid-19 vaccines or vaccination. after the filtering by keywords, we randomly sampled 3,015 posts per year for manual annotation. ## results and discussion along the 61 annotation rounds, inter-annotator agreement, in terms of krippendorff's alpha, was calculated for relevance, sentiment and stance, as shown in figure 1 . following each round, group supervisors conducted meetings with the annotation teams to address any questions or issues raised by the annotators. as it turns out, relevance achieved the highest overall agreement between annotators, with an average alpha score of 0.94 and minimal variation across rounds. this task benefited from a more balanced dataset, fewer conflicts among annotators, and a larger total number of observations. both sentiment and stance attained moderate agreement. the average alpha was 0.67 for sentiment and 0.70 for stance. however, both tasks showed significant variability along the rounds. this variability can be attributed to the imbalance between classes at the stance and sentiment classification. as a result, there was greater disagreement in the content being analyzed in each round, especially for the minority categories (unfavorable and unclear). for stance, 78.6% of the three-year sample were classified as favorable towards covid-19 vaccines. however, there are 17.4% posts were an opinion towards vaccines to protect against sars-cov-2 are not self-evident based on the message. an example of discourse that is unclear are publications that ex- figure 3 shows the distribution of stance categories along the years. the favorable class is ma-jority for all years with its percentage increasing from 59.7% in 2020 to 86.3% in 2021 and 84.2% in 2022. the unclear class comes second with 33.8% in the first year, and decreasing to 12.1% in 2021 and 10.7% in 2022 .as can be noticed in the figure, the proportion on messages not expressing a favorable or unfavorable position drops with covid-19 vaccines becoming available to brazilians (1°s emester of 2021). the unfavorable class is minority in all years. in contrast to the uptick in favorable posts in 2021, there is a sharp decrease in unfavorable tweets during this year. these messages comprised 6.6% of the dataset in 2020, 1.6% in 2021, with an uptick to 5.1% in 2022. lastly, figure 4 shows the proportion of sentiment categories along the three years. in 2020, 59.6% of the posts were interpreted as negative, 35.1% expressed positive feelings and 5.2% were neither positive or negative, which we refer to as unclear. as it turns out, there is an inversion in the year vaccines are introduced (2021) when compared to its preceding year, whereby the positive class becomes the majority class (48.8%) and there is a decrease in negative posts (now 44.2%), and unclear remains the minority with 7.1% of the publications. with the arrival of vaccines and the loosening of social distancing restrictions, there remained a relatively limited supply of vaccines, a slow rollout and the largest number of covid-19 deaths during the omicron and delta waves. in this context, the proportion of negative posts keeps similar along the years as politicians continue to express frustration, and sadness. in 2022, positive posts were 54.4% of the sample, negative were 38.6% and unclear were 7%. a final point to note is the stark differences between stance and sentiment distributions. stance presents a very unbalanced distribution with most posts being favorable towards covid-19 vaccination and vaccines. unfavorable posts are a rare occurrence, but there are a considerable percentage of messages that are not clear on the position towards covid-19 vaccines. on the other hand, for sentiment, the dataset is more balanced between negative and positive posts. it is rare for local political elites to fail to use emotions in their discourse towards covid-19 vaccines. ## conclusions and future work this study detailed the development of an annotated corpus of posts in brazilian portuguese, posted by brazilian political elites focusing on covid-19 vaccines and vaccination. the corpus was first classified according to each post's relevance to the topic. relevant posts were then further annotated with respect to stance (favorable, unfavorable, and unclear) and sentiment (positive, negative, and unclear). the creation of this corpus addresses significant gaps in the literature, due to the scarcity of resources in brazilian portuguese and the lack of curated datasets in this language related to vaccines, particularly in the context of covid-19. furthermore, the study presents a reliable annotation scheme that distinguishes between sentiment analysis and stance detection. the analysis of the annotated corpus provides evidence that measurement error can occur due to two problems. (i) if a relevance rule is not applied, scholars may be annotating data that are not specific to the topic even if keywords are present; and (ii) when sentiment and stance tasks are not separately considered, class conflation may introduces bias. the annotation process involved nine annotators, divided into three groups, who analyzed 9,045 posts published between 2020 and 2022. the process included extensive training, weekly meetings and supervision to resolve conflicts to ensure consistency. the analysis of annotation the results revealed fair to moderate agreement among annotators, as indicated by a 0.94 overall krippendorf's alpha of for relevance, 0.67 for sentiment and 0.70 for stance. this annotated corpus can serve as a gold standard for training and, amongst other things, evaluating machine learning models. in future research, we are planning to explore whether the patterns reported in this study also apply to discourse on covid-19 vaccines when specifically focusing on children and adolescents, or vulnerable populations, such as the elderly and those who have other chronic illnesses. we also plan to use this corpus to further expand the classification of covid-19 vaccine discourse to national political elites and other x messages on the same domain. the annotated corpus is publicly available on github under a creative commons license (cc by-nc-sa 4.0) foot_12 , ensuring that future research can build upon this work while respecting the ethical standards for data sharing.