Fine-Grained Detection of Solidarity for Women and Migrants in 155 Years of G erman Parliamentary Debates (2024)

Abstract: Solidarity is a crucial concept to understand social relations in societies. In this paper, we explore fine-grained solidarity frames to study solidarity towards women and migrants in German parliamentary debates between 1867 and 2022. Using 2,864 manually annotated text snippets (with a cost exceeding 18k Euro), we evaluate large language models (LLMs) like Llama 3, GPT-3.5, and GPT-4. We find that GPT-4 outperforms other LLMs, approaching human annotation quality. Using GPT-4, we automatically annotate more than 18k further instances (with a cost of around 500 Euro) across 155 years and find that solidarity with migrants outweighs anti-solidarity but that frequencies and solidarity types shift over time. Most importantly, group-based notions of (anti-)solidarity fade in favor of compassionate solidarity, focusing on the vulnerability of migrant groups, and exchange-based anti-solidarity, focusing on the lack of (economic) contribution. Our study highlights the interplay of historical events, socio-economic needs, and political ideologies in shaping migration discourse and social cohesion. We also show that powerful LLMs, if carefully prompted, can be cost-effective alternatives to human annotation for hard social scientific tasks.

## introduction solidarity is a crucial concept for understanding how societies achieve and maintain stability and cohesion @xcite , and it plays a critical role in shaping policies @xcite . traditionally, solidarity relied on common identity and reciprocity, potentially excluding out-groups like migrants @xcite and reinforcing social boundaries and hierarchies @xcite . however, growing diversity and increasing socio-economic, political, religious, and cultural complexities of modern societies (kymlicka, 2020) call such traditional forms of solidarity into question. simultaneously, recent populist groupbased exchangebased compassionate empathic based on... based on... shared identity valuing for contributions support for marginalized groups respect for diversity perceived differences imbalance in contribution denying help to vulnerable disregard of groups' differences ## related work our work connects to (i) computational social science (css) (ii) analysis of political data (parliamentary debates) and (iii) the emergent field of analysis of social solidarity using nlp approaches. ## data we obtain data from two sources: (i) open data, covering bundestag (en.: federal diet) protocols from 1949 until today; and (ii) reichstagsprotokolle covering reichstag (en.: imperial diet) protocols until 1945. 1 we use the ocr-scanned version from @xcite . links to data, models, etc. used are in appendix d. for the reichstag data, we apply preprocessing steps similar to @xcite (e.g., removal of ocr artifacts), but keep german umlauts, capitalization, and punctuation. we automatically split the data into individual sittings and collect metadata like the date, period and number of each sitting, which we manually check and correct. additionally, we removed interjections and split the text into sentences using nltk @xcite , resulting in 19.1m sentences. we release this dataset of plenary protocols from german political debates (deuparl) consisting of 9,923 sittings from 1867 to 2022 on github. 2 to select keywords, we (i) train a word2vec model @xcite on our dataset to identify words with vector representations similar to migrant (en.: migrant) and frau (en.: woman); (ii) manually expand this list with intuitively relevant terms; (iii) from both lists, we filter for those which appear at least 200 times in the dataset. this resulted in 32 keywords for migrant and 18 keywords for frau. these include general terms like migrant, immigrant and frau to period-specific terms, such as vertriebene (en.: expellees) and bürgerkriegsflüchtlinge (en.: civil war refugees), or social roles, such as mütter (en.: mothers) and hausfrauen (en.: housewives). see the full list of keywords, and further preprocessing in appendix a. for a detailed keyword distribution across the dataset, see fig. 11 and fig. 12 in the appendix. using these keywords, we extract 58k main sentences (instances) for migrants and 131k for women from deuparl, expanding each with three preceding and three following sentences for context, resulting in a total of (i) 463k sentences (9.79m tokens) for migrants and (ii) 1.58m sentences (32.82m tokens) for women. fig. 2 shows the number of instances over time. 3 fig. 7 in the appendix shows yearly relative frequencies of sentences with terms related to women and migrants in the entire dataset. it is notable that both frau and migrant terms represent a minor fraction of the discourse, typically under 0.02. periodic spikes in mentions likely align with historical and societal changes, such as post-wwii for migrant. ## data annotation to obtain ground truth data for model training and evaluation, we annotated 2864 instances with five annotators (all student assistants, with specializations in social science or computer science). the annotation was performed over a duration of nine months. in the first three months, we iteratively refined the annotation guidelines and monitored the inter-rater agreement (measured by cohen's kappa) until inter-rater agreement converged (see section 4.2 for exact scores) and annotators began annotating independently. ## models and experiments to determine the most cost-effective model (both in terms of performance and costs) for our large-scale sociological analysis, we evaluate various models, including llama-3-70b-instruct, gpt-4-1106-preview, base and instructionfinetuned gpt-3.5-turbo-0125, across high-level and fine-grained (anti-)solidarity annotation tasks in achieving human-level performance. once the quality of the models is assured, we apply the best performing model -gpt-4 -large-scale to determine trends in section 7. ## results results on the test sets are shown in conversely, bert achieves lower f1 scores across all categories, performing comparably to the gpt-3.5 versions on high-level tasks but struggling with fine-grained tasks and ambivalent categories like mixed stance, likely due to challenges from underrepresented labels such as anti-solidarity categories for women, even after over-sampling (see fig. 4 , as well as overall, the human upper bound consistently outperforms all models. gpt-4 is the closest but still falls behind by 0.11 to 0.14 and 0.05 to 0.12 points in fine-grained and high-level tasks respectively for women and migrants. llama-3-70b follows, with gaps of 0.24 to 0.29 and 0.24 to 0.13 points in finegrained and high-level tasks, while bert shows the largest discrepancies. these results suggest there is a gap compared to human understanding, especially in more complex annotation tasks. error analysis for the error analysis, we compare the human annotations and zero-shot predictions of gpt-4 for both target groups on the test set, using the confusion matrices for high-level labels shown in fig. 3 , as well as for fine-grained level labels in fig. 6 provided in the appendix. we also consider explanations provided by gpt-4. observed errors align with those noted during human annotation, with solidarity and anti-solidarity rarely confused (1% of cases). confusion primarily occurs between (anti-)solidarity subtypes and none, as well as mixed stance, as the model seems to look for stronger indications of solidarity despite instructions to consider even slight expressions of it (see examples 1 and 2 in there is also notable confusion between the solidarity subtypes, with the most frequent confusion between group-based and compassionate solidarity, likely because of the presence of multiple category characteristics within the texts (see example 3 in ## analysis in this section, we analyze how the solidarity discourse in the german parliament developed over 155 years, using automatic, fine-grained annota-tions from the best-performing model, gpt-4 zeroshot. however, due to cost constraints, we limit the annotations to a) data concerning migrants, i.e., the target group for which our models achieved higher scores and b) a sample of 18,300 instances from the overall 58k instances concerning migrants. we draw the sample proportionally for the time spans in the original data. this selection includes all records with political party information (see appendix b for details on political parties data extraction and list of parties included in the analysis). the cost of this automatic annotation was around 500 euro. ## concluding remarks this study set out to i) provide a high-quality dataset of (anti-)solidarity annotations in german parliamentary debates, ii) from an nlp perspective, evaluate the ability of llms to assist in largescale social and political analyses over extended periods, and iii) from a css perspective, uncover long-term shifts in solidarity trends in germany, laying groundwork for future sociological studies. concerning i), we invested 1000+ annotation hours and 18k euro to provide a data set of 2,864 manually annotated text snippets following the sociological framework of @xcite . regarding ii), our findings indicate that gpt-4 outperforms other models (llama-3-70b, gpt-3.5 base, fine-tuned, and bert) in reproducing human annotations. while other models demonstrate proficiency in identifying high-level categories, they exhibit limitations in handling more nuanced categories, specific to sociological theory. gpt-4, though challenged by ambiguity, handles complex tasks even in a zero-shot setting. contrary to previous research in css that suggested smaller fine-tuned models like bert performed well @xcite , our study finds that larger models are more effective our hard css task, possibly due to the carefully designed prompts based on human expertise. our observations align with @xcite , who noted that while llms have not yet reached the quality of human analysis in classification tasks within css, large instruction-tuned llms are preferable. however, we observe that the open-source model llama-3 exhibits inferior performance compared to the non-open-source gpt-4. in terms of iii), our analysis of german parliamentary debates from 1865 to 2022 reveals shifts in attitudes toward migrants influenced by labor demands, migration waves, and socio-political changes. the evolution from group-based to compassionate solidarity marks a shift towards economic pragmatism and issues of redistribution in migration discourse, likely spurred by rising global humanitarianism @xcite . despite traditional solidarity dominance, the resurgence of anti-solidarity since wwii, peaking in 2022, illustrates deepening political polarization. this underscores the ongoing tension between xenophobic tendencies and liberal ideals in germany @xcite , reflecting the complex challenges of integration and multiculturalism in a major migrant destination. these findings lay a strong foundation for further sociological research using llms. future work could examine how shifts in political discourse impact policy-making and public opinion, exploring correlations with civic solidarity. comparative studies across countries with varying migration histories may reveal factors influencing (anti-)solidarity. additionally, exploring the effects of such rhetoric on related policies, like migration or equal opportunity, could yield further insights. from an nlp perspective, future work can explore a broader range of llms, including those fine-tuned on german data, and explore the effect of using highlighting and explanations to improve classification.