GESIS - DSM at P erpective A rg2024: A Matter of Style? Socio-Cultural Differences in Argumentation (2024)

Abstract: This paper describes the contribution of team GESIS-DSM to the Perspective Argument Retrieval Task, a task on retrieving socio-culturally relevant and diverse arguments for different user queries. Our experiments and analyses aim to explore the nature of the socio-cultural specialization in argument retrieval: (how) do the arguments written by different socio-cultural groups differ? We investigate the impact of content and style for the task of identifying arguments relevant to a query and a certain demographic attribute. In its different configurations, our system employs sentence embedding representations, arguments generated with Large Language Model, as well as stylistic features. final method places third overall in the shared task, and, in comparison, does particularly well in the most difficult evaluation scenario, where the socio-cultural background of the argument author is implicit (i.e. has to be inferred from the text). This result indicates that socio-cultural differences in argument production may indeed be a matter of style.

## introduction argument retrieval is a well-established task in computational argumentation @xcite : given a query or question, e.g. on "should we have free trade agreements?", the task aims to retrieve topically relevant arguments. topically relevant arguments can be heterogeneous, even for arguments with the same stance (for vs. against) -i.e. someone can focus on quality of the products as essential to free trade, while another may focus on international collaboration. these different perspectives in arguments can @xmath0. the perspective argument retrieval task @xcite argues that the socio-cultural background (e.g. gender, religion) can be taken into account when selecting relevant arguments. sociocultural information can determine the dynamics of argumentation at different levels. for example, socio-demographics can be used to approximate the stance of an argument about a specific topic (e.g., women, pro abortion); additionally, the specific arguments used to support a stance can be a correlate of a specific socio-cultural attribute (religious being against abortion because it is a sin). differences between arguments of different socio-cultural groups have mostly been researched as differences in argument content, meaning the semantic differences in the arguments: which phrases, aspects, and points are mentioned. for instance, @xcite analyze how social groups differ in the social bias of their arguments, i.e., male users using terms that indicate a gender bias. however, previous research indicated that argument style may be different between socio-cultural groups as well. with argument style, we mean how something is said, e.g. features of form and not content like complexity of words, length of words and sentences, grammatical tenses, or pronoun usage. such stylistic features have mostly been studied from the perspective of argument persuasiveness @xcite but also have been used to analyze socio-demographic differences in deliberation processes, i.e., women referring more to others than men in their arguments @xcite . while the shared task does not address the issue of persuasiveness of the arguments directly, the intuition that "an argument i would write is one that is likely to resonate stronger with me", builds a potential bridge between socio-demographic retrieval and persuasion research (e.g., consider el @xcite on empowering vs. challenging arguments). additionally, diversity (providing arguments with different perspectives on the same query) can be employed to maximize the reception of a certain argument. also, such a diversity of perspectives in recommendations can be beneficial, for instance for citizens in democracies @xcite . in approaching the task of argument retrieval for specific socio-cultural profiles, we can also resort to findings from other tasks, such as author profiling @xcite . this task aims to predict author characteristics from user-generated text, with these characteristics often having a sociodemographic nature, (i.e., gender or age). successful approaches use semantic content as well as style to profile authors of user-generated texts @xcite . our approach the perspectivist argument retrieval task raises the question of how socio-cultural groups differ in their arguments for a given query. these differences can be semantic, i.e., groups may differ in what they say in their arguments, or stylistic, i.e., groups may differ in how they formulate them. we first explore this distinction in the shared task data in section 3. we then describe our approaches to ranking arguments according to socio-cultural specific relevance in section 4: one is based on semantic content similarity using 'prototypical' arguments generated with a large language model (llm). the other uses stylistic features. our final method places third overall in the shared task, and, in comparison, does particularly well in the most difficult scenario, the one in which the socio-cultural background of the argument author is implicit (i.e. has to be inferred from the text). it implements a three-step pipeline, using semantic information in a ranking step and stylistic information to classify whether arguments are relevant for a given socio-cultural group. our results indicate that the stylistic differences in the arguments of different socio-cultural groups are more relevant to the task of retrieving relevant arguments than semantic differences in our setup. we publicly release our code for the experiments and analyses. 1 2 task: data and evaluation scenarios the question at the core of the task is: can we find the arguments that members of a given sociocultural target group would write for this query? details on the size and train/dev/test splits of the three cycles is provided in evaluation scenarios the systems are evaluated on three scenarios: 1. argument retrieval without consideration of socio-cultural differences (baseline). 2. argument retrieval for a specified sociocultural attribute. information about any other attribute could be used to diversify the set of retrieved arguments (explicit scenario). 3. argument retrieval for a specified socio-cultural attribute. in contrast to explicit, information about other attributes is hidden (implicit scenario). system evaluation is based on both relevance and diversity of the selected arguments (in terms of socio-cultural attributes) to promote the diversity of opinions. ## data analysis: content or style? in the development of our pipeline, data analysis played a crucial role. in the following, we summarize core findings for content and style. these analyses were conducted on the cycle 1 corpus. semantic content differences firstly, we assess whether there are arguments that multiple sociocultural groups share and find that ∼ 11% of the arguments appear with the same argument text for at least two different socio-cultural profiles. the same argument may, for instance, be produced by a non-religious man and a roman-catholic woman. while this is expected in a natural setting, as groups may share views and thus arguments, this raises the question of differentiation between socio-cultural groups, in particular for the implicit scenario. secondly, we cluster sentence-bert representations of arguments using the k-means algorithm and evaluate the resulting clusters against the ground-truth socio-cultural groupings, as well as stance and topic. per socio-cultural attribute, we run one clustering with k equaling the number of groups in the attribute (e.g. for residence @xmath1, as there are the two groups city and countryside). we find that all of the socio-cultural groups have an adjusted rand score @xcite @xmath2.1 with the respective clustering, indicating virtually no overlap of the clusters with the groups. only the topic shows a relatively higher adjusted rand score (for an overview of the results of our clustering experiments, see appendix e). these analyses indicate that there is little semantic distinction between arguments of different sociocultural groups, at least in the present semantic representation space. ## system description as discussed in the introduction, we want to assess the impact of content and style on the perspective argument retrieval performance. we thus divide the problem into two steps covering these aspects, with an additional filtering step. get group for the subsequent steps. 2) semantic argument ranking. find the top k arguments that are semantically closest to the query. 3) final candidates retrieval. option (a) stylistic relevance classification: select only those from the pre-selected k arguments with a stylistic fingerprint indicating a relevant argument given the target socio-cultural group. option (b) generated argument re-ranking: per query and socio-cultural attribute, generate an argument and find the top k arguments that are semantically closest to the generated argument. for the non-perspectivist baseline scenario, only step 2), semantic argument ranking, is executed. for the explicit perspectivism scenario, all three steps are executed. finally, for the implicit perspectivism scenario, we execute solely steps 2) and 3) as no socio-cultural information is accessible. in what follows, we detail the operationalization of these three steps. ## discussion of results the results of our different systems over the three test sets are displayed in ## conclusion and future work we present our approach to the perspective argument retrieval shared task 2024. our proposed method implements a three-step pipeline, leveraging semantic information in a ranking step and stylistic information to classify whether arguments are relevant for a given socio-cultural group. the performance of this approach, in particular for the implicit scenario, showcases the potential of including stylistic information for the task of perspectivist argument retrieval. this raises several questions for future research. especially with regard to the third test set, in which the perspectives of politicians and voters were flipped, we argue that investigating the reasons for differences in production and perception of arguments of different socio-cultural groups, e.g. through semantic or stylistic differences, is of interest. consequently, how to combine this information in retrieval scenarios should be investigated.