Understanding behaviour of large language models for short-term and long-term fairness scenarios (2023)

Abstract: Large language models (LLMs) have become increasingly accessible online, thus they can be easily used to generate synthetic data for technology. With the rising capabilities of LLMs, their applications span across many domains. With its increasing use for automating tasks, it is crucial to understand the fairness notions harboured by these models. Our work aims to explore the consistency and behaviour of GPT3.5, GPT-4 in both short-term and long-term scenarios through the lens of fairness. Additionally, the search for an optimal prompt template design for equalized opportunities has been investigated in this study. In the short-term scenario for the German Credit dataset, an intervention to a key feature recorded an increase in loan rejection rate by 37.15% for GPT-3.5 and 49.52% for GPT-4. In the long-term scenario for ML fairness gym, adding extra information about the environment to the prompts has shown no improvement to the prompt with minimal information in terms of final credit distributions. However, adding extra features to the prompt has increased the profit rate by 6.41% (from 17.2% to 23.6%) compared to a baseline maximum-reward classifier with compromising group-level recall rates.

## introduction with the rising capabilities of llms, their applications in many domains have been proposed by different studies. for example, for improving autonomous driving @xcite @xcite , for generating lending decisions and credit scores @xcite , and for gathering legislation details @xcite . however, there is a low number of studies that investigate their implications on the different groups. for the zero-shot scenario, we use the german credit dataset @xcite , a tabular dataset * * equal contribution by all authors with 20 features of 1000 individuals. the features describe the personal and financial information for accessing the risk of lending. the llms, instructed as staff of a bank, decide whether or not to grant a loan to a candidate based on the values of these features. we analyzed these decisions to understand the default fairness of llms. the critical deciding features are found, and interventions in those features can flip the decisions of the llm. we compared the consistency between the important features derived empirically and the llms' claimed important features. to explore how decisions affect the dynamics of the environment for long-term settings, we use ml fairness gym (d'amour et al., 2020) which consists of simulations of several long-term scenarios such as lending, attention allocation, and college admissions. it is a framework for simulating the long-term effects of machine learning (ml) in societal contexts. with increasing emphasis on ml fairness, it's crucial to understand the discrepancies between zero-shot fairness decisions and their long-term outcomes. the impact of prompt designs on the fairness of llms has been explored. additionally, we extracted long-range trajectories created by agents trained with proximal policy optimization (ppo) @xcite , which are then paired and fed into llms for fair preferences without any explicit fairness definition. long-term implications of fairness remain to be a relatively unexplored area, even less so with the llms use cases. german credit dataset has been used extensively as a benchmark dataset for shortterm fairness, however only a few studies @xcite @xcite applied llms and focused on their performance in long-term fairness scenarios. in this study, the intervention of features for altering the response outcome has been investigated. the key contributions of this paper: currently, there is no unified method for the definition of fairness, and hence multiple criteria have been proposed. these criteria exist across groups as well as individual levels of fairness. some of the group-level metrics consist of demographic parity, which measures the extent of independence of predictions of membership in a sensitive group, and equalized odds, which makes sure that the model performs equally well for all groups. individual fairness criteria are proposed in @xcite and further in @xcite , where two distance metrics are measured. one consists of a distance between the characteristics of the samples, to see how similar two samples are, and the other is a distance between the predictions to see how differently these samples have been treated. machine learning has been used in credit scoring systems where fairness is a crucial criterion. moreover, classification scenarios which generally consist of a sensitive variable are prone to fair or biased decisions depending on the algorithm and data used. for long-term fairness scenarios, @xcite , measure fairness in pay-as-you-go systems, @xcite use soft and hard interventions, for long-term fairness scenarios for sequential decision-making. hu and chen (2018)'s work on achieving long-term fairness consists of using short-term interventions for the labor market. lackner (2020) proposes the use of voting, i.e. taking previous decisions into account for further decisions. si @xcite , consider the scenario of dynamic resource allocation where fairness is required at every time slot, as well as in the long-term context over a period of time. simulation studies for lending allocation, attention allocation, and college admissions by d' @xcite show that long-term fairness dynamics are hard to assess and present a framework for using agents for simulations. with the rising popularity of llm-based applications, the evaluation of llms has been evolving around the consistency and robustness of responses, the fairness metrics of static scenarios, and the reason for bias. with more results revealing the logical flaws, lack of consistency and robustness @xcite @xcite , and reasons for biases across tasks @xcite @xcite , the fairness research community has been actively creating methods to de-bias the llms. the basic bias evaluation method is to analyze the bias associations in the content generated by the model in response to the input prompts @xcite , while some work has developed benchmark datasets to test for social bias. gpt-3 is declared socially biased and it is validated by prompt completion and co-occurrence tests @xcite . the authors test the association between gender and occupation, and in 83% of 388 occupations prompts are generated with text related to male identifiers. the general prompt construction includes task instructions, context descriptions, feature descriptions, and questions. studies @xcite show that although chatgpt still has a problem with fairness. however, there is no existing work in explaining long-term fairness. ## conclusion for short-term fairness, the feature of "credit history comment" is shown to be a critical feature for the lending decision of llms. the comment of "delay in paying off in the past" for the feature of "credit history comment" has shown a significant effect on lending decisions, with the loan rejection rate increasing from 39.94% to 89.46% after interventions on results for gpt-4. the comment of "critical account other credit existing (not at this bank)" for the feature of "credit history comment" has shown a significant effect on lending decisions, with the loan rejection rate increasing from 38.75% to 75.79% after interventions on results for gpt-3.5. since llms are relying heavily on this feature, this implies that candidates with a single delay history or critical account would be very unlikely to receive a loan regardless of the improvement of values in other dimensions (e.g. improvement in occupation, acquiring more properties, or increase of income class, etc.). for long-term fairness, the evaluation of fairness has shown heavy bias toward the positional arrangement of the options in the prompt, which implies the limitation of llms for providing consistent preferences based on multivariate long-term trajectories. by embedding the llms as decision agents in the lending environment, it is shown to be important for the prompt to include the minimum and maximum bounds of the values, and the default rates for providing decisions that cause the most equalized final credit distributions while avoiding the overly preserved behavior of infrequent lending, as shown in the feature set with the profit objective in place of the dynamic default rate as profit objective factor. this shows the importance of providing a dynamic objective definition rather than a fixed objective definition for the decisions of llms. for the llm as an agent experiment, adding extra features to the prompt has increased the profit rate by 6.41% (from 17.2% to 23.6%) compared to the baseline maximum-reward classifier with compromising group-level recall rates yet improved precision rates. this implies a potential commercial incentive for users to tailor the prompt design to increase profit, which would sacrifice the precision equality of fairness metrics while overlooking or neglecting the simplest prompt design that provides the optimal final credit distributions across groups for minimizing group-level discrepancy.