Sövereign at The Perspective Argument Retrieval Shared Task 2024: Using LLM s with Argument Mining (2024)

Abstract: This paper presents the Sövereign submission for the shared task on perspective argument retrieval for the Argument Mining Workshop 2024. The main challenge is to perform argument retrieval considering socio-cultural aspects such as political interests, occupation, age, and gender. To address the challenge, we apply open-access Large Language Models (Mistral-7b) in a zero-shot fashion for re-ranking and explicit similarity scoring. Additionally, we combine different features in an ensemble setup using logistic regression. Our system ranks second in the competition for all test set rounds on average for the logistic regression approach using LLM similarity scores as a feature. In addition to the description of the approach, we also provide further results of our ablation study. Our code will be open-sourced upon acceptance.

## introduction although the world wide web is full of content, search engines nowadays still lack support for extracting information regarding argument retrieval @xcite . argument retrieval addresses the issue of retrieving relevant arguments from a corpus based on a specific query @xcite . further issues arise for particular perspectives, i.e., an argument might only be relevant in a special situation with certain restrictions. the shared task on "perspective argument retrieval" @xcite , investigates these challenges by including sociocultural properties/factors (e.g. political interests, occupation, age, and gender) in a multilingual setup (see figure 1 for illustration). the data includes documents in german, french, and italian. motivated by the abilities of large language models (llms; cf. @xcite , we investigate methods to leverage them for this task. we consider two approaches: implicitly re-ranking the argument candidates, and explicitly computing rele-figure 1 : cross-lingual perspective argument mining: relevant arguments are marked in green, and irrelevant arguments are marked in red or orange. in the orange one, demographic properties match, yet the texts are not relevant. the english translations (using google translate 2 ) for the query and the arguments are denoted as en t . vance scores for candidate arguments. first, we use the cosine similarity of sentence bert @xcite between the encoded arguments and the query to retrieve the nearest neighbors (arguments) as candidates similar to the baseline approach by @xcite . note that the given arguments also contain topic labels and socio-cultural factors (e.g. in scenario 2, section 3), which we also benefit from. we then supply the query and the retrieved candidates to an llm and ask it to re-rank the arguments. in our second method, the llm is asked to produce a score for a given query-argument pair. we further train a logistic regression classifier using several initial similarity scores as features and use the computed feature weights in an ensemble fashion to compute a final relevance score. our llm scoring based method shows improved performance for scenario 1 and 2, while the llm re-ranking performs competitively in scenario 3, on the development set. therefore, we submit the results obtained with the logistic regression using the llm scoring as the final solution to the competition. the name of our team in the leaderboard of the organizers is "sövereign". the contributions of this paper are as follows: ## related work in this section, we briefly describe the existing studies that we take into consideration while developing our proposed approach. ## task description in this shared task, the goal is to retrieve multilingual arguments gathered from the voting recommendation platform foot_0 . for a description of the dataset, we refer to @xcite . the key challenge here is to consider socio-cultural factors during retrieval. for the shared task 2024 we submit systems for all the three competition scenarios: for each evaluation round, the data consists of a set of queries and a set of candidates/arguments. the set of queries includes the query text, and for scenarios 2 and 3 it also contains an explicitly given socio-cultural/demographic attribute. the set of candidates contains the argument text, a "stance" parameter ("favor" or "against"), and a "topic" parameter. the retrieval performance is measured using: a) relevance: ndcg@k and pr@k (precision @ k), and b) diversity: αndcg@k and kldiv@k (kullback-leibler divergence @ k), where k is the rank of retrieved arguments. ## methodology in this section, we present two approaches for the argument mining task. the first method applies the llm directly to rank the arguments, the second integrates llm scores as a feature for a logistic regression model. in the next subsection, we introduce scores utilized in both approaches. ## experimental setup we use the mixtral-8x7b-instruct-v0.1 foot_1 llm model by mistralai foot_2 with the default parameters using huggingchat foot_3 . this model comes with a lenient license and offers a good balance between performance and model size 7 . by using the huggingchat framework, we explicitly make the model exchangeable, and we expect increased performance by using larger models. we refrain from the model fine-tuning and apply it as a zero-shot. regarding sbert, we use the pre-trained model paraphrase-multilingual-mpnet-base-v2 foot_5 , likewise the baseline from the organizers @xcite . we trained the logistic regression classifier using the scikit-learn foot_6 framework on the training dataset with the default parameters. to choose the solution for the final evaluation round, we test our approaches on the development set and submit the test set ranking using the bestperforming algorithm. scenario 1: default argument retrieval here, sbert is already a very strong baseline. logistic regression achieves better scores for ndcg, pr, and αndcg and @xmath0. scenario 2: explicit perspectivism results are shown in appendix scenario 3: implicit perspectivism in this scenario, both approaches perform almost on par; the llm re-ranking methods perform better than other approaches, as shown in ## results on the test sets in this section, we present the test results of our approach from logistic regression. these results are evaluated and shared by task organizers. ## conclusion we present sövereign, an llm re-ranking approach for perspective argument retrieval. we show an investigation of two llm utilizations, a) implicit re-ranking, and b) explicit relevance scoring. the explicit relevance scoring methods achieve better scores for explicit perspectivism when used in an ensemble with other similarity features, i.e., sbert, topic, and socio-cultural (if applicable). in scenario 3, implicit perspectivism, llm reranking performs better than the llm scoring. we believe this might be due to the formulations of the prompts in scenario 3: ranking prompt emphasizes socio-cultural property, directly. in future work, we would like to explore more utilization methods of llms in this task, e.g., trying different prompts. the data additionally contains "stance" attributes, which we omitted to use for our submission, but might be an important feature. we also plan to try other llm models and improve the results for test set 3 by classifying the topic from the query and matching it with the topics from the arguments.