Detecting Calls to Action in Multimodal Content: Analysis of the 2021 G erman Federal Election Campaign on I nstagram (2024)

Abstract: This study investigates the automated classification of Calls to Action (CTAs) within the 2021 German Instagram election campaign to advance the understanding of mobilization in social media contexts. We analyzed over 2,208 Instagram stories and 712 posts using fine-tuned BERT models and OpenAI’s GPT-4 models. The fine-tuned BERT model incorporating synthetic training data achieved a macro F1 score of 0.93, demonstrating a robust classification performance. Our analysis revealed that 49.58% of Instagram posts and 10.64% of stories contained CTAs, highlighting significant differences in mobilization strategies between these content types. Additionally, we found that FDP and the Greens had the highest prevalence of CTAs in posts, whereas CDU and CSU led in story CTAs.

## introduction in this study, we experiment with the automated classification of calls to action (ctas) from the 2021 german instagram campaign to advance the understanding of mobilization in social media election campaigns. our primary goal is to determine the efficacy of several computational approaches for binary classification of the presence or absence of ctas in instagram posts and stories from the 2021 federal election in germany. to this end, we fine-tuned a bert model @xcite , experimented with synthetic training data to enhance the model, and contrasted these approaches with zero-and few-shot prompting using openai's gpt-4 model family. through our study, we aim to address the three gaps in computational text analysis for the social sciences identified by @xcite : 1) we experiment with a non-english language, 2) we evaluate all classifications against human annotations for external validation @xcite , and 3) we investigate the potential of llms for overcoming the specialization before integration gap. the 2021 election marked a shift in germany's political landscape, with the long-serving chancellor angela merkel stepping down. the key parties in the race included the cdu/csu, spd, greens, fdp, afd, and the left. in 2021, instagram was used by almost the same share of the german population as facebook and was particularly popular among younger users under the age of 30 @xcite . about half of the candidates had profiles on instagram, with notable differences between parties @xcite . we are interested in the front-runner and party accounts and how they utilized ctas on instagram to gain insight into their mobilization and audience engagement strategies. understanding these strategies reveals how political actors use instagram to engage voters. thus, our secondary goal is to use the cta classifications to contrast mobilization strategies between instagram stories and posts, filling a gap as ephemeral stories have often been overlooked. therefore, we want to answer the following research questions: rq1a which of the currently available model variants, when tested with few-shot and zero-shot prompts, achieves the highest performance in automated detection of ctas in german-language instagram content? rq1b does incorporating synthetic training data enhance the performance of a fine-tuned bert model in detecting ctas in germanlanguage instagram content? rq1c when comparing the best-performing gpt and bert models, what are the performance differences in detecting ctas between different types of instagram content (stories vs. posts) and text types (ocr vs. caption vs. transcript)? rq2 how does the usage of ctas vary between different types of instagram content (stories vs. posts) and between different political parties? ## the corpus we collected two types of instagram content: permanent posts that may include multiple images or videos with a caption and stories that typically consist of a single image or video. captions in posts represent the primary textual content on instagram, varying in length and often featuring hashtags. while captions are the primary text elements, many images and videos incorporate embedded text or spoken words. for our computational analysis, we deconstructed each instagram post and story into smaller units to analyze text in various forms: captions, embedded text (through optical character recognition, ocr), and speech (transcriptions) for video audio. this approach resulted in up to two text documents per image and up to three documents per video. as a post can contain multiple images, this leads to a maximum of 3 • n images documents per post, plus an additional document for the caption. in contrast, instagram stories typically comprise a single image or video, resulting in one ocr document and an optional transcription document per story. see ## methods we have operationalized ctas as a binary variable, indicating their presence or absence in documents, simplifying our model's classification process. each social media post or story is analyzed by decomposing it into several text documents, enabling the computational analysis of multimodal data. to answer questions on a post/story level, we assign 'true' for an entire post or story if call to action is marked as 'true' in any of the associated documents. this section defines ctas, describes our annotation study, and the prompt engineering and model training steps. ## results in the first part of this section, we will answer our primary questions rq1a-c regarding the computational classifications through the external evaluation based on human annotations. at the end of the section, we will answer our secondary interest rq2, uncovering the differences between stories, posts, and parties. ## discussion our experiments confirm the efficacy of large language models for the binary classification of calls to action in social media election campaigns. overall, the gpt-4 models performed well in zero-and few-shot settings. regarding cohen's kappa, there is a strong agreement between language model classifications and ground truth labels. fine-tuning the gbert-large bert model, however, exceeds the performance of the llms. the relatively low number of 1,388 human-annotated a closer look at the classification quality on a text-and post-type level reveals problems with classifying story transcripts. ctas in these documents account for only 5.16% of the overall training data. this highlights the potential for further improvements in data augmentation using synthetic documents: a qualitative inspection of synthetic training data generated based on transcripts revealed less similarity to original transcripts than to, for example, post captions. improving the synthetic data prompt to generate more realistic transcripts might improve the classification performance for this type of text while increasing the linguistic variance across synthetic training data might further increase the overall classification performance. striving for the best possible annotation quality, we chose the gbert-w/-synth-cta for our classification task. however, training a robust classification model takes several steps, from annotation through hyperparameter tuning to the final evaluation. conversely, the gpt-4 models are readily available, and prompt engineering was comparatively uncomplicated in our context. with decreasing prices, evolving models, and the availability of open-source alternatives, like llama 3, this study further confirms the utility of large language models for computational social science tasks and political science analyses. after applying the model, we uncovered significant differences between political actors' use of ctas in stories and posts. we found a slightly higher prevalence of ctas across posts compared to previous studies @xcite , which may be attributed to our sample: we collected data close to election day, ctas have been shown to increase closer to election day @xcite . our study contributes to the study of election campaigns mainly by uncovering a significant difference between posts and stories and between parties. the greens, for example, have been highlighted before as the party with the highest prevalence of ctas across their posts. at the same time, we found the party's stories contain the lowest number of ctas relative to the number of stories posted. overall, the use of ctas in stories was low, which contrasts with towner and muñoz's (2024) observations of the 2018 u.s. gubernatorial election, raising questions about what other elements or content constituted political stories in the 2021 election.