Stance-Aware Re-Ranking for Non-factual Comparative Queries (2023)

Abstract: We propose a re-ranking approach to improve the retrieval effectiveness for non-factual comparative queries like ‘Which city is better, London or Paris?’ based on whether the results express a stance towards the comparison objects (London vs. Paris) or not. Applied to the 26 runs submitted to the Touché 2022 task on comparative argument retrieval, our stance-aware re-ranking significantly improves the retrieval effectiveness for all runs when perfect oracle-style stance labels are available. With our most effective practical stance detector based on GPT-3.5 (F₁ of 0.49 on four stance classes), our re-ranking still improves the effectiveness for all runs but only six improvements are significant. Artificially “deteriorating” the oracle-style labels, we further find that an F₁ of 0.90 for stance detection is necessary to significantly improve the retrieval effectiveness for the best run via stance-aware re-ranking.

## introduction argument retrieval is the task of identifying and ranking text passages or documents based on their topical relevance to an argumentative query and based on their argumentativeness (i.e., the presence and quality of arguments). current argument search engines like args.me @xcite or argumentext @xcite ) mainly focus on retrieving pro and con arguments on socially relevant and potentially controversial topics like 'nuclear energy' or 'plastic bottles' but they do not directly target to find pros and cons for the different options in "everyday" non-factual comparisons like 'which city is better, london or paris?'. such information needs were in the focus of the comparative argument retrieval task at the touché 2022 lab @xcite . given a query with two comparison objects (e.g., the london vs. paris example), the goal was to retrieve results that contain arguments for or against either object. many participants of the task improved over a bm25 baseline @xcite by using neural (re-)ranking models like colbert @xcite or mono-and duot5 @xcite , and by taking estimated argument quality into account. still, none of the participants successfully exploited stance information for the ranking (i.e., whether a result expresses a stance on the comparison objects or not) even though stance detection was also offered as a subtask at touché. we close this gap and, as our first contribution, suggest a simple stance-aware re-ranking approach that can be applied to the retrieval results for any comparative query: rank documents that do not express a stance on the comparison objects below any documents that do. in an evaluation on the 26 runs submitted to the touché 2022 task, we find that our re-ranking significantly improves the retrieval effectiveness of all runs when using the task's official ground truth stance labels (i.e., assuming a "perfect" oracle-style stance detector). when instead using the participants' stance predictions, hardly any run's effectiveness can be improved as the participants' stance detectors are not effective enough (f 1 ≤ 0.31 on the four classes 'pro first object', 'pro second object', 'both equal', and 'no stance'). as our second contribution, we thus target a better practical stance detection effectiveness and compare three approaches: (1) a fine-tuned sentiment-prompted roberta model @xcite , (2) a zero-shot stance detector based on a pre-trained flan-t5 model @xcite , and (3) gpt-3.5 @xcite with fewshot prompting. among these, the gpt-3.5-based stance detector is the most effective with an f 1 of 0.49. using the stances detected with gpt-3.5, our stance-aware re-ranking can again improve the retrieval effectiveness of all 26 runs but only 6 of the improvements (23%) are significant. in further experiments, we artificially perturb the ground truth stance labels to analyze what stance detection effectiveness is necessary to significantly improve the retrieval effectiveness of the best run via stanceaware re-ranking and find that an f 1 of 0.90 is required. our code and data are publicly available. 1 2 re-ranking scenario: touché 2022 our re-ranking scenario is that of the touché 2022 shared task on comparative argument retrieval. given one of 50 non-factual comparative queries, relevant text passages from a collection of about one million passages should be retrieved and ranked, and (optionally) their stances be detected. for our experiments, we use the 26 runs (ranked lists of results) submitted to the task, as well as the relevance + quality assessments and the stance labels that the task organizers provided @xcite . in the task, the retrieval effectiveness of the submitted runs was evaluated using ndcg@5 @xcite for topical relevance and for argument quality, and the stance detection effectiveness was evaluated using macro-avg. f 1 on the four stance classes. ## stance-aware re-ranking interestingly, none of the touché participants successfully used stance information in their retrieval approaches. this is somewhat surprising as, intuitively, helpful retrieval results for non-factual comparative queries should express some stance towards the comparison objects (either favoring one of the objects or stating that both are equal). our suggested re-ranking approach thus simply moves all results that do not express a stance to the end of a ranking (i.e., below any result that expresses a stance), while preserving the relative order of the documents that express a stance. ## initial re-ranking experiments in our initial experiments, we re-rank the top-5 results of each of the 26 runs submitted to touché based on the task's ground truth stance labels (i.e., assuming "perfect" oracle-style stance detection) or based on the participants' detected stances. following the touché setup, we report ndcg@5 scores for relevance and for quality and refer to the runs by their team names (e.g., aldo nadi or captain l.). ## improving the stance detection targeting better practical stance detection, we compare three approaches: (1) a fine-tuned sentimentprompted roberta model @xcite , (2) a zero-shot stance detector based on a pretrained flan-t5 model @xcite , and (3) gpt-3.5 (brown et al., 2020) with few-shot prompting. following touché, we use macroavg. f 1 to compare the detection effectiveness (class distribution: 'pro first object' 19%, 'pro second object' 13%, 'both equal' 20%, 'no stance' 48%). for the roberta-based detector, we fine-tune a roberta model using the sentiment-prompting ## testing limits with simulated stances to analyze the (potential) impact of stance detectors that are more effective than our currently most effective practical approach (gpt-3.5-based; f 1 of 0.49), we gradually artificially deteriorate the ground truth stances as follows. from the passages with ground truth stance labels, we iteratively randomly select one without replacement and sample a stance label from the ground truth label distribution (o 1 : 19%, o 2 : 13%, =: 20%, ⊥: 48%; a sampled label for a passage could be the same as in the ground truth) until the f 1 of the perturbed ground truth falls below a desired stopping threshold. using this process, we simulate "stance detectors" with f 1 scores of 0.95, 0.9, 0.85, . . . , 0.25, 0.2. for each threshold, we run the process ten times with different random seeds to obtain ten perturbed ground truths per target f 1 score. the ten perturbed ground truths are then each used to re-rank a run's retrieval results and the resulting ten ndcg@5 scores are averaged-to somewhat smooth out possible randomization effects. as an example, column 'simul.' in to clarify whether there is a relationship between stance detection f 1 and retrieval effectiveness improvement, figure 1 shows the effectiveness scores when re-ranking the top-3 teams' best runs with the perturbed ground truths of different target stance detection f 1 scores. one can clearly observe that an increasing stance detection f 1 yields increased retrieval effectiveness improvements (relevance and quality; trends similar for other runs and teams). the minimally needed stance detection f 1 so that the respective stance-aware re-ranking significantly improves an original run is shown in fig-ure 2 (runs given by their initial ndcg@5 scores). as for the relevance-wise improvements, one can observe a clear trend that runs with a better initial effectiveness require better stance detection to yield significant improvements. for the relevancewise best runs, even almost perfect stance detection f 1 scores of 0.9 or 0.95 are needed to yield significant relevance-wise improvements. as for the quality-wise improvements, no clear trend is observable. two "outliers" of runs with a good initial effectiveness only require some rather low stance detection f 1 for significant improvements, but many runs with quite different initial quality-wise effectiveness require pretty high f 1 scores. interestingly, the quality-wise best run aldo nadi a can never be significantly improved, even with perfect oracle-style stance labels. ## conclusion we have proposed a simple stance-aware re-ranking approach for non-factual comparative queries that just moves results that do not express a stance on the comparison objects below any results that do. for all 26 runs submitted to the touché 2022 task on comparative argument retrieval, our re-ranking can significantly improve the retrieval effectiveness when using the official touché stance labels (i.e., assuming a "perfect" oracle-style stance detector). then again, re-ranking based on the stances detected by the task participants (f 1 ≤ 0.31) hardly improves any run. we thus experimented with other stance detectors to achieve better practical stance effectiveness. using our most effective detector (gpt-3.5-based; f 1 of 0.49), the re-ranking can again improve the retrieval effectiveness for all 26 runs but only 6 of the relevance-wise and 12 of the quality-wise improvements are significant. in a final experiment with controlled perturbation of the ground truth stances, we found that better stance detection effectiveness tends to yield better re-ranking effectiveness and that a stance detection f 1 of 0.90 is necessary to significantly improve the relevance-wise most effective run. substantially improving the practical stance detection effectiveness thus is an interesting direction for future work that could also be the basis for a diversified result presentation: splitting the results into three separate lists for 'pro first object', 'pro second object', and 'both equal'. besides, our reranking approach does not yet consider any potential confidence scores of a stance detection model and also no potentially predicted stance "magnitude". developing stance detectors that assign a confidence or stance magnitude might actually be helpful to further improve the stance-aware reranking (e.g., to rank results with high-confidence stances above the ones with low confidence).