Do Large Language Models Rank Fairly? An Empirical Study on the Fairness of LLM s as Rankers (2024)

Abstract: The integration of Large Language Models (LLMs) in information retrieval has raised a critical reevaluation of fairness in the text-ranking models. LLMs, such as GPT models and Llama2, have shown effectiveness in natural language understanding tasks, and prior works such as RankGPT have demonstrated that the LLMs have better performance than the traditional ranking models in the ranking task. However, their fairness remains largely unexplored. This paper presents an empirical study evaluating these LLMs using the TREC Fair Ranking dataset, focusing on the representation of binary protected attributes such as gender and geographic location, which are historically underrepresented in search outcomes. Our analysis delves into how these LLMs handle queries and documents related to these attributes, aiming to uncover biases in their ranking algorithms. We assess fairness from both user and content perspectives, contributing an empirical benchmark for evaluating LLMs as the fair ranker.

## introduction the emergence of large language models (llms) like gpt models @xcite and llama2 @xcite ) marks a significant trend in multiple fields, ranging from natural language processing to information retrieval. in the ranking challenges, llms have shown demonstrated performance. research, such as rankgpt @xcite and prp @xcite , highlights the proficiency of gpt models in delivering competitive ranking results, surpassing traditional neural ranking models in precision and relevance. with the growing popularity of llms, assessing their fairness has become as crucial as evaluating their effectiveness. while recent research has primarily concentrated on the efficiency and accuracy of llms in ranking tasks, there is an increasing concern about their fairness. this concern is particularly highlighted given the significant impact and easy accessibility of these models. prior studies in natural language processing @xcite @xcite and recommendation systems @xcite have shown the un-fair treatment towards underrepresented groups by llms. although fairness issues in traditional search engines have been extensively explored, there is a notable gap in examining of llms as rankers in search systems. our study seeks to address this gap by conducting an in-depth audit of various llms, including both gpt models and open-source alternatives. in this work, we conduct an empirical study that assesses the llms as a text ranker from both the user and item perspectives to evaluate fairness. we investigate how these models, despite being trained on vast and varied datasets, might unintentionally mirror social biases in their ranking outcomes. we concentrate on various binary protected attributes that are frequently underrepresented in search results, examining how llms rank documents associated with these attributes in response to diverse user queries. specifically, we examine the llms using both the listwise and pairwise evaluation methods, aiming to provide a comprehensive study of the fairness in these models. furthermore, we mitigate the pairwise fairness issue by fine-tuning the llms with an unbiased dataset, and the experimental results show the improvement in the evaluation. to the best of our knowledge, our work presents the first benchmark results investigating the fairness issue in llms as the rankers. in summary, this paper makes the contribution as follows: ## llm fair ranking we define the set of queries in our dataset as q, consisting of m queries, and the set of items as d, comprising n items. for each query @xmath0, there exists a list of item candidates d (q) from d. we represent each i-th query-item pair with a text token vector x (q) i and an associated relevance score y (q) i . importantly, the item candidates in d are annotated with a binary attribute indicating their classification as either belonging to a protected group or a non-protected group. this attribute, such as gender or race, is crucial as it highlights the potential exposure bias present in the ranking prediction process. next, we present our evaluation benchmark dataset and introduce two fairness evaluation methods: listwise and pairwise evaluation. ## results and analysis in our benchmark, we carefully evaluate the popular llms including gpt-3.5, gpt-4, llama2-13b, and mistral-7b @xcite . this section details our analysis of their performance across both listwise and pairwise evaluations. ## enhancing fairness with lora we employed lora @xcite to fine-tune the mistral-7b model. our approach involves creating a balanced training dataset with equal representation of responses from both protected and unprotected groups. this balanced dataset aims to steer the model towards fairer rankings when evaluating pairs of relevant or irrelevant items from diverse groups. the implementation of the lora module is facilitated using the peft @xcite package. aligning with the parameter-efficient methodology outlined in the original lora, our study specifically focuses on adapting attention weights. to simplify and enhance parameter-efficiency, we opted to freeze other parameters. in our case, we set the optimal rank to 1, deeming a low-rank adaptation matrix as adequate. the chosen learning rate is 0.003, and the batch size is set at 4. these configurations were selected based on considerations specific to our study. the dataset, comprising approximately 140,000 item pairs randomly sampled for each trec dataset, facilitate comprehensive training. the process, conducted on an nvidia a100 80gb, needs approximately 30 hours. we split the queries for training and testing, using 80% for training and the remaining 20% for testing. the results of fine-tuning mistral-7b with lora are illustrated in figure 4 . post-tuning, there is a noticeable reduction in consistent responses from the model when queried twice with reversed item orders. this indicates an increase in response variability, which is a positive indicator of fairness, as less predictability in responses can mitigate systematic bias. the improvement in fairness is further supported by figure 4b , where the outcomes post-lora fine-tuning show ratios approaching 1.0, indicating a more equitable treatment of protected and unprotected groups by the model. ## conclusion the empirical study and in-depth analysis provided in this research study have revealed the intricate biases presented in large language models (llms) when evaluated for fairness through listwise and pairwise methods. while listwise evaluations painted a picture of relative fairness, a deeper investigation via pairwise evaluations uncovered subtler and more profound biases that often favored certain protected groups. the implementation of lora fine-tuning on the mistral-7b model yielded encouraging strides to rectify these biases, demonstrating enhanced fairness in the model's output. going forward, our efforts will pivot towards further improving ranking performance with targeted ranking loss functions, while concurrently addressing fairness more holistically through refined prompting strategies, aiming for an optimal balance between utility and equity in broad llmbased ranking applications.