A Study of Implicit Ranking Unfairness in Large Language Models (2024)

Abstract: Recently, Large Language Models (LLMs) have demonstrated a superior ability to serve as ranking models. However, concerns have arisen as LLMs will exhibit discriminatory ranking behaviors based on users’ sensitive attributes (gender). Worse still, in this paper, we identify a subtler form of discrimination in LLMs, termed implicit ranking unfairness , where LLMs exhibit discriminatory ranking patterns based solely on non-sensitive user profiles, such as user names. Such implicit unfairness is more widespread but less noticeable, threatening the ethical foundation. To comprehensively explore such unfairness, our analysis will focus on three research aspects: (1) We propose an evaluation method to investigate the severity of implicit ranking unfairness. (2) We uncover the reasons for causing such unfairness. (3) To mitigate such unfairness effectively, we utilize a pair-wise regression method to conduct fair-aware data augmentation for LLM fine-tuning. The experiment demonstrates that our method outperforms existing approaches in ranking fairness, achieving this with only a small reduction in accuracy. Lastly, we emphasize the need for the community to identify and mitigate the implicit unfairness, aiming to avert the potential deterioration in the reinforced human-LLMs ecosystem deterioration.

## introduction large language models (llms), represented by chatgpt @xcite have empowered ranking tasks @xcite , which is important in filtering overload information to users @xcite . however, ensuring that llms do not pose ethical risks becomes crucial. recently, various evaluation methods have been introduced to assess the degree of discrimination in llms @xcite @xcite , showing that llms frequently exhibit pronounced ranking discriminatory behaviors against explicit sensitive attributes, such as gender @xcite . although a massive amount of work focuses on addressing unfairness when explicitly using sensitive attributes in ranking tasks @xcite , our investigation reveals the persistence of implicit ranking unfairness: llms even generate substantial discriminatory ranking behaviors when using non-sensitive yet personalized user profiles (e.g., user names). these profiles are commonly used as identifiers of gender and race since humans often draw stereotypical conclusions based on them @xcite @xcite . implicit ranking unfairness in llms highlights new and more urgent risks towards llms-based ranking application (e.g., recommendation) because (1) such unfairness is often inconspicuous because it only depends on non-sensitive user profiles; and (2) such unfairness is more widespread since these non-sensitive user profiles can be easily acquired and used by existing platforms, such as user names or email addresses. to comprehensively analyze the problem, in this paper, we will focus on three research aspects regarding implicit ranking unfairness in llms. firstly, we propose an evaluation method to investigate how serious the implicit ranking unfairness is in existing llms. specifically, following the practice in @xcite , we design a ranking task prompt template (figure 1 ). then we give substantial empirical evidence to confirm the existence of implicit ranking unfairness. finally, we find that the degree of implicit ranking unfairness is nearly 2-4 times more serious than explicit unfairness, and the unfairness is caused by collaborative information. empirical evidence is in section 4). secondly, since this implicit unfairness is more severe and more hidden, we aim to investigate the reasons behind its occurrence. specifically, we identify that the llms can probe sensitive attributes exclusively from these personalized and non-sensitive user profiles. then we also show that the word embeddings of certain non-sensitive user profiles are more closely aligned with the sensitive attribute. such phenomena contribute to the collection of unfair datasets during the pre-training phases (see evidence in section 5). finally, we aim to propose a method to mitigate such implicit ranking unfairness. previous research proposed to mitigate user unfairness either by employing privacy policies that hide sensitive attributes @xcite @xcite , utilizing certain prompts to instruct llms to disregard sensitive attributes @xcite or add counterfactual sample to enhance fairness @xcite . however, they show limited effectiveness in mitigating implicit ranking unfairness (see section 6). in this paper, we propose a fair-aware data argumentation method to mitigate such unfairness. specifically, we incorporate counterfactual samples that contain certain implicit attributes to help the model produce fair ranking results. due to the massive and noisy characteristic of the non-sensitive features, we employ a pair-wise regression method to choose hard and informational non-sensitive features to conduct data argumentation. the experiments demonstrate that our method outperforms the existing methods on two ranking datasets. ## preliminary in this section, we will formulate the llms-based ranking tasks and the implicit ranking unfairness concept formally. ## evaluation settings in this section, we will describe our evaluation settings including the datasets and some details. ## implicit unfairness of llms in this section, we aim to evaluate the implicit unfairness. note that we average the different chat-gpt versions and llama 2 @xcite results to conduct the analysis. ## implicit ranking unfairness traceback in this section, our objective is to investigate why the implicit ranking unfairness exist. ## implicit ranking unfairness mitigation in this section, we propose a fair data augmentationation method to mitigate implicit ranking unfairness. we employ the 2sls procedure (kmenta, 2010) to remove the noise in non-sensitive attributes. after that, we can conduct data augmentation effectively by utilizing the top-n different feature sets that exhibit the most serious unfair behaviors in ranking. ## experiments in this section, we will conduct experiments to show the effectiveness of our methods. ## related work recently, researchers have discovered that llms can exhibit discriminatory behaviors @xcite . in previous discrimination evaluation settings, researchers often measure stereotype sentence pairs that only differ in the sensitive attribute. for example, they often adapt terms "male" and "female" @xcite @xcite and for race, they often substitute terms "black", "white" and "asian" @xcite . among the allocational harms, previous studies found that llms often exhibit discrimination against certain groups. for example, salinas et al. (2023 @xcite discovered that llms will generate discriminatory content for disadvantaged gender. @xcite show recommendation outcomes may discriminate against certain groups, see also @xcite . in our research, we mainly utilize the counterfactual fairness concept to measure the implicit ranking unfairness of llms-based recommendation. there are some works that try to mitigate unfairness problems in llms. for example, rlhf @xcite and rlaif @xcite try to utilize reinforcement learning to align llms with human values. generally, to address the imbalance in the original dataset against certain groups, some work @xcite @xcite ) create matched pairs (e.g., male or female) to ensure a more equitable dataset and other methods @xcite add non-toxic examples for groups. other approaches @xcite suggest the use of down-weighting samples containing social group or discriminated information as a re-sampling strategy. while some method proposes to utilize the prompt-tuning method to learn a fair-aware prompt @xcite . moreover, other studies @xcite propose to filter out and remove discriminated or taxonomic content from datasets. ## conclusion in conclusion, our findings show that llms exhibit serious implicit ranking unfairness. this implies that, even when sensitive attributes are not explicitly provided, llms can still exhibit discriminatory ranking behaviors. regarding the root causes, we find that llms' capability to deduce sensitive attributes from non-sensitive attributes contributes to intrinsic discriminatory knowledge. finally, we propose to mitigate such unfairness effectively by utilizing fair-aware data augmentation. we emphasize the necessity of identifying and moderating implicit ranking unfairness in existing llms.