Overview of P erspective A rg2024 The First Shared Task on Perspective Argument Retrieval (2024)

Abstract: Argument retrieval is the task of finding relevant arguments for a given query. While existing approaches rely solely on the semantic alignment of queries and arguments, this first shared task on perspective argument retrieval incorporates perspectives during retrieval, ac- counting for latent influences in argumenta- tion. We present a novel multilingual dataset covering demographic and socio-cultural (so- cio) variables, such as age, gender, and politi- cal attitude, representing minority and major- ity groups in society. We distinguish between three scenarios to explore how retrieval systems consider explicitly (in both query and corpus) and implicitly (only in query) formulated per- spectives. This paper provides an overview of this shared task and summarizes the results of the six submitted systems. We find substantial challenges in incorporating perspectivism, especially when aiming for personalization based solely on the text of arguments without explicitly providing socio profiles. Moreover, re- trieval systems tend to be biased towards the majority group but partially mitigate bias for the female gender. While we bootstrap per- spective argument retrieval, further research is essential to optimize retrieval systems to facilitate personalization and reduce polarization.

## introduction in argument retrieval, the objective is to extract arguments that match a given query, such as a question or topic. existing research defines the relevance and ordering of candidate arguments differently. in the simplest case, arguments are extracted based on the semantic relevance of the query. more sophisticated methods consider the quality of the arguments, suitable counterarguments @xcite , or arguments that answer comparative figure 1 : this example entry shows which information we consider for this shared task. first, we incorporate the semantic information as the text of queries and arguments. secondly, we use the demographic and sociocultural properties (perspective) of argument authors or users, including age, gender, or political attitude. addressing this research gap, we introduce the shared task on perspective argument retrieval ( § 2). incorporating the perspective of authors and readers (figure 1 ), we aim to foster personalization by retrieving arguments that match individual perspectives beyond their semantic alignment and reduce polarization by enabling individuals to compare and contrast arguments from their own and other perspectives. therefore, we present a novel multi-lingual dataset ( § 3) providing demographic and socio-cultural (socio) profiles of argument authors or readers for german, french, and italian. in this context, relevant arguments are semantically aligned with a given query and match the specific socio variables provided in the query. we use three scenarios (figure 2 ) to disentangle the effect of perspectivism: with this shared task, we aim to examine how retrieval systems can exploit the latent influence of demographic and socio-cultural profiles, such as age or political attitude, and how they are biased regarding over-or underrepresented groups (like different age groups). current approaches in computational argumentation tend to prioritize majority groups and marginalize minority groups @xcite . to fulfill these objectives, we adopt a fine-grained and comprehensive evaluation protocol and assess the performance of submitted argument retrieval systems in two tracks: relevance and diversity ( § 4). the retrieval system should provide top-k arguments that are highly relevant to the query and simultaneously diversify varying demographic and socio-cultural profiles. therefore, we rely on prior work enforcing diversification across stances in retrieved arguments @xcite . with this shared task and results from the six participating teams, we address the following research questions: rq1: can argument retrieval systems encode socio-cultural variables? results ( § 6) reveal substantial challenges in encoding perspectives and successfully achieving personalization. systems struggle to capture fine-grained socio-linguistic features without explicit profile mentions. moreover, there is a lack of suitable metrics for evaluating relevance, diversity, and fairness. rq2: are argument retrieval systems biased regarding specific socio-cultural variables? while retrieval systems primarily follow the corpus bias, in-depth analysis ( § 7) shows that they balance gender bias but increase age group bias. rq3: how do argument retrieval systems generalize when switching the perceiving perspective from authors to readers? perceiving perspective causes substantial performance drops ( § 6), as readers select arguments according to their political standing (attitude and important issue) but not regarding their demographic ones, like age or denomination, catholic or protestant ( § 3). contributions with this shared task, we establish the task of perspective argument retrieval and present a novel dataset covering explicitly and implicitly expressed perspectives from argument authors and readers. a comprehensive evaluation of the submitted systems underscores the challenge of this task as the system struggles to incorporate the fine-grained linguistic influence of demographic and socio-cultural variables. further, while these systems mostly replicate the dataset bias, they partially overcome gender bias. these insights call for further research to incorporate perspectivism successfully and fairly, aiming for systems providing personalization. ## perspective argument retrieval argument retrieval is the task of finding top-k relevant arguments y within a corpus c for a given query q @xcite . we formulate perspective argument retrieval as an expansion of argument retrieval to perspectivism @xcite when finding best-matching arguments. by considering demographic and socio-cultural (socio) variables, we account for latent aspects of argumen-tation beyond semantic features, such as age, occupation, or political attitude. concretely, this shared task proposes three scenarios (figure 2 ) to evaluate how argument retrieval systems can account for perspectivism. ## data in the following, we outline the data used for this shared task, involving the data source ( § 3.1), the used demographic and socio-cultural variables ( § 3.2), the composition of the argument corpus and the queries ( § 3.3). ## evaluation we employ a two-folded evaluation to comprehensively measure the retrieval quality for all three scenarios. concretely, we distinguish between relevance and diversity. relevance with relevance, we focus on the ability of a retrieval system to select relevant candidates given the query, for example, all arguments addressing the queried issue for the baseline scenario or arguments that additionally match specific socio-cultural properties for explicit or implicit perspectivism. following previous work @xcite @xcite , we use the normalized discounted cumulative gain (ndcg@) and precision@ metric. compared to precision, ndcg has the advantage of taking the position of the ranked items into account. therefore, the metric places greater emphasis on higherranked arguments. diversity using diversity, we account for the influence of perspectivism in the evaluation by measuring whether a retrieval system proposes balanced arguments regarding the stance distribution and the authors' diverse socio-cultural properties (such as gender or political attitude). following previous studies regarding fairness in argument retrieval systems @xcite , we calculate alpha-ndcg@ for each available property separately and average them afterwards. this metric accounts for relevance and diversity by assessing whether an item is relevant and introduces a new perspective compared to the previous one. consider the following example: a system retrieved a list of arguments relevant to a given issue, and we aim to evaluate diversity for political attitude. the metric would prefer the arguments to be sorted like this ['liberal', 'conservative', 'left', 'conservative'] over ['conservative', 'conservative', 'liberal', 'left']. an optimal ranking ensures that all relevant perspectives for a corresponding socio-cultural variable are represented among the top-ranked arguments. note that these conditioned properties are withheld when evaluating diversity since we condition specific socio-demographic properties in the query for explicit or implicit perspectivism (scenarios 2 and 3). as a second metric, we look at the normalized discounted kl-divergence, introduced as a metric of unfairness @xcite . this metric evaluates the fairness of the ranking by comparing the distribution of a protected group (e.g. what is the proportion of arguments by female authors when looking at the property 'gender') in the top-k ranked items against a gold standard proportion (what is the proportion of arguments by female authors in the whole corpus?). the divergence is calculated at specified cut-off points and then averaged, with each point discounted by the logarithm of its rank, to assess how well the ranking reflects the representation of the protected group. in this case, the relevance of an argument is not considered; instead, the metric can reveal biases against specific groups. for example, it can show whether systems disproportionately favor dominant groups in the top arguments. final ranking we evaluate the performance at four different values of k [4, 8, 16, 20] and calculate the average performance across these k values. this evaluation is conducted for the three scenarios across three different test sets, resulting in 9 scores for relevance and 9 for diversity. the final rank is determined by averaging these nine scores. ## submissions in the following, we summarize the baseline ( § 5.1) and the submitted systems ( § 5.2). further, we elaborate on the unique ideas incorporated by the participants. ## results in the following section, we discuss the results of the submitted systems focusing on rq1. additional detailed discussions regarding single scenario, evaluation cycle, and top-k are in appendix § a.2 and § a.3. relevance and diversity agree but not with fairness. considering perspectivism is difficult. we analyze the performance differences between the three scenarios. figure 5 and figure 6 reveal that considering no socio variable (scenario one) performs the best across all test sets of the three evaluation cycles. comparing scenarios one with two and three (considering perspectivism explicit or implicit) highlights the challenges of incorporating socio variables in the retrieval stage. this becomes even more apparent when comparing scenarios two and three. while considering socio variables in the query and corpus (scenario two) results in higher performance, it crucially requires more computing. in contrast, the more efficient approach of considering socio variable only in the query (scenario three) causes significant performance degradation. thus, existing retrieval systems show crucial limitations in taking into account perspectivism, either explicit or implicit. particularly, they need the signal of the socio variable within the query and corpus. further analysis of the participating teams reveals that the implicit difference between arguments of distinct socio variables is more stylistic than semantic. as a result, we see the need to build retrieval systems accounting for such fine-grained socio-linguistic variations to consider perspectivism accurately and efficiently. ## analysis in the following, we focus on rq2 and examine whether retrieved argument candidates are biased regarding socio-cultural groups and if submitted systems compensate for such biases. we focus on age and gender, known for which recent work found substantial bias in argumentation. specifically, @xcite show that common argumentation sources (e.g. debating corpora) exhibit substantial bias regarding young ages and european-american males. further, holtermann et al. ( 2022 ) shows that fine-tuning lms on argumentative data increases stereotypical bias, even if lms exhibited a counter-stereotypical bias before tuning. as shown in figure 3 , our dataset is biased towards specific groups, such as male and/or young authors. we establish a random baseline by randomly sampling 20 topic-relevant arguments for every query of the implicit scenario across 10 different seeds and average the number of arguments retrieved for each group. similarly, we average the performance metric. systems are biased regarding majority groups. we examine the 20 most relevant arguments, count how many represent the distinct group, and compute the standard deviation for each system towards the random baseline. a negative deviation indicates that the system further reduces the representation of that group, meaning the group is less represented in the top arguments compared to its underlying distribution in the corpus. conversely, a positive deviation indicates increased representation. in the case of a majority group, the system amplifies the bias. figure 7 shows the shift in representation for gender, comparing the 2019 and 2023 test sets. we observe most systems (including the sbert baseline) reducing the male bias. however, the top retrieved arguments still overrepresent male authors by a large margin, as the deviation is not more than one argument. interestingly, one team reinforced the male bias in the 2019 dataset with a slight positive deviation. however, that system slightly outperformed the other teams in increasing the representation of other gender categories in the 2023 dataset (positive value for no gender specified). figure 8 focuses on different age groups and shows that all systems reinforce a bias regarding young ages. this is particularly true for the 2023 dataset, where systems systematically retrieve fewer arguments written by older age groups than randomly sample arguments. this supports general findings in nlp that older age groups are underrepresented in data and models. comparing the two middle-aged groups reveals that 35-49 is better represented than 50-64 for 2019. since both age groups occur approximately equally frequently in the corpus, this indicates a stronger age bias, with the older group being significantly less wellrepresented. while these findings suggest that systems are biased toward representing the majority group, they mitigate this bias more effectively for the female gender category. systems partly mitigate gender but not age bias. we compute each group's deviation from the system performance to the random baseline performance. if there is no bias, the deviation for a system should be the same for each group. for gender, figure 9 shows all systems reduce the bias regarding the majority group (male gender category). for nonbinary and unspecified gender, the performance pattern is similar to representation: one system shows slight bias improvement, while the others are slightly more biased than the baseline. the female group's performance improved for the 2019 dataset compared to the random baseline but not for the 2023 dataset. we assume that the sbert model has potentially seen more topics from the 2019 election and detected sub-issue-specific differences within known topics. for example, the model could have identified specific frames used more frequently by males than females. for age, systems seem to agree more with the dataset distribution: younger age groups have fewer declines or even improvements (in 2023) compared to older age groups (figure 17 in appendix). again, systems perform the worst on the 50-64 age group. ## conclusion with the shared task on perspective argument retrieval, we explore for the first time how argument retrieval systems align socio-cultural properties beyond topic relevance. analyzing the submissions shows that semantic content alone does not distinguish between different socio-cultural groups adequately. instead, incorporating additional classification tasks or features is crucial for accurately matching arguments to socio-cultural characteristics. the subsequent analysis shows that systems overrepresent arguments from majority groups. however, they partially mitigate these biases, such as gender bias. by publishing data reflecting authors' and readers' perspectives, this shared task represents an initial step towards advancing argument retrieval regarding perspectivism. this facilitates the investigation of personalization and polarization and addresses social bias and fairness in computational argumentation.