REQUAL - LM : Reliability and Equity through Aggregation in Large Language Models (2024)

Abstract: The extensive scope of large language models (LLMs) across various domains underscores the critical importance of responsibility in their application, beyond natural language processing. In particular, the randomized nature of LLMs, coupled with inherent biases and historical stereotypes in data, raises critical concerns regarding reliability and equity. Addressing these challenges are necessary before using LLMs for applications with societal impact. Towards addressing this gap, we introduce REQUAL-LM, a novel method for finding reliable and equitable LLM outputs through aggregation. Specifically, we develop a Montecarlo method based on repeated sampling to find a reliable output close to the mean of the underlying distribution of possible outputs. We formally define the terms such as reliability and bias, and design an equity-aware aggregation to minimize harmful bias while finding a highly reliable output. REQUAL-LM does not require specialized hardware, does not impose a significant computing load, and uses LLMs as a blackbox. This design choice enables seamless scalability alongside the rapid advancement of LLM technologies. Our system does not require retraining the LLMs, which makes it deployment ready and easy to adapt. Our comprehensive experiments using various tasks and datasets demonstrate that REQUAL-LM effectively mitigates bias and selects a more equitable response, specifically the outputs that properly represents minority groups.

## introduction in the ever-evolving realm of advanced technologies, large language models (llms) have quickly emerged as versatile tools, extending their influence far beyond the boundaries of natural language processing (nlp). many of the traditionally challenging tasks with decades of research in various * this work was supported in part by nsf 2107290. example 1: (part 1) selecting a subset of candidates from a pool, based on a set of criteria is common across multiple applications ranging from journalism, to college admissions and job hiring. for example, consider the hr department of a sales company who wants to select a set of candidates for the performance award based on multiple criteria such as sales and customer-satisfaction. passing the performance information of the employees, they can ask the llm to select the candidate set. llms are sequential randomized approaches based on estimations learned from large textual datasets. in particular, based on the prompt and the sequence of tokens generated so far, each word (token) in the dictionary is assigned a probability. then, the next token is generated probabilistically (proportional to the probabilities of the top-k or top-p%) using the parameter temperature. consequently, the output may vary when the llm is queried again. as a result, a valid concern, particularly for a decision maker, is whether they should rely on the llm's output for taking action. in settings similar to example 1, the reliability question is further significant, since a method to combine the performance criteria has not been specified, while small changes in the combination details may significantly change the output @xcite . another challenge that makes a single query to the llms unreliable arises for the symmetric settings, where the ordering between the input does not matter, i.e., shuffling the input should not impact the output. for instance, in example 1 the ordering based on which the employees are passed to the llm should not impact the output. conversely, llms receive an input as a (ordered) sequence. as a result, as it was observed in @xcite , the output of the llms for symmetric problems vary when the input is shuffled. we also observed the same behavior in our experiments on a subset selection task, where the entities that are placed at the beginning of the list had a higher chance of being returned as the output. to resolve these issues we introduce requal-lm that, instead of relying on a single query to an llm, follows a monte-carlo method @xcite based on repeated sampling. particularly, viewing each llm output as a sample from the underlying distribution of possible outputs, it identifies the centroid of a collection of samples as its estimation of the mean of the distribution, and returns the closest output to the centroid as the most reliable one. to further clarify this, let us consider example 1 once again. example 1: (part 2) observing the dependency of the llm output with the input ordering, and to possibly consider various combinations of performance criteria, the hr department does not rely on a single output of the llm. instead requal-lm enables issuing multiple queries to the llm, each time shuffling the list of the employees. it then returns the "closest-to-centroid" of the obtained samples as the most reliable output. while being effective in practice, data-driven technologies have been heavily criticized for machine bias @xcite , and llms are not an exception when it comes to bias. as a result, another valid concern when using llms for decision making is neutrality: to ensure that impact of historical biases and stereotypes are minimized and that values such as diversity are promoted. example 1: (part 3) the hr department would likes to maximize diversity in the selected awardees. in particular, they would like to prevent selecting a male-only list of employees. requal-lm allows specifying two or more demographic groups and it minimizes the output bias (measured as the cosine-similarity difference of its output's embedding with different groups' representations). llms are among the fast-growing technologies, with new and advanced versions regularly emerging, while many of these systems are "black-box". our system design is not dependent on a specific llm, which makes it a ready-to-apply wrapper that works on top of any of the current and future closed-source and open-source llms. requal-lm does not require pre-training or fine-tuning, is task-agnostic, and can handle non-binary demographic groups. in the following, first in § 2 we carefully discuss the problem setting, introduce notations, and formally define terms such as reliability and bias. next, in § 3 we review the architecture of requal-lm, and develop our methodology for finding an equitable centroid and return the closest output to it, the one that is both equitable and reliable. the experimental evaluations, related work, and the discussions of the benefits and limitations of requal-lm are provided in § 4, § 5, § 6, and § 8, respectively. 2 preliminaries -(input) task: we consider a task, such as subset selection, sentence completion, assembling a team of experts, etc., described as a prompt: llm. -(input) demographic groups: we assume the existence of at least one sensitive attribute (e.g., sex) that specify the demographic groups @xmath0, • • • , g ℓ } (e.g., {male, female}). the demographic groups are used to specify the output bias. our system, requal-lm, is agnostic to the choice (but limited to the performance) of the embedding model, and can adapt any stateof-the-art text embedding technique. without loss of generality, we use instructor -a method for generating task-specific embeddings in accordance with provided instructions @xcite . given two text phrases o i and o j and their corresponding embeddings v ⃗ i and v ⃗ j , the similarity between o i and o j is measured as the cosine similarity between their embeddings, i.e., s im (o i , o j ) = cos ∠(v ⃗ i , v ⃗ j ). similarly, the distance between o i and o j is defined as ∆(o i , o j ) = 1 -s im (v ⃗ i , v ⃗ j ). definition 1 (reliability). given a prompt i, let o i be the universe of possible-to-generate outputs for i. furthermore, let ξ be the probability distribution of outputs for i. that is, ∀@xmath1, p r ξ (o) is the probability that o is generated for i. let µ ⃗ ξ be the mean of ξ in the embedding space. then the reliability of an output @xmath2. that is, let @xmath3, t o 2 , • • • t o |o| ⟩ sequentially generated by the llm. at each iteration i, let p r(t o i ) be the probability of generating the token t o i . then p r ξ (o) can be computed as the product of its token probabilities. that is, p r the bias of an output o for a prompt i is computed as the maximum similarity disparity of the demographic groups with o. formally, bias is sometimes inherent to the task at hand and is not harmful. for example, when the task involves summarizing or rephrasing a paragraph that is particularly written about a specific gender, the resulting output tends to be naturally biased towards that gender. we call this type of output bias as the inevitable bias. formally, we say a bias level ε is inevitable if there is no valid output @xmath4. in other words, for any output o ′ where β(o) < ε, we can say o ′ / ∈ o i . therefore, we define the inevitable bias as β n (i) = min @xmath5. we consider any bias that is not inevitable, discriminatory. harmful stereotypes are in this category. we call this type of output bias as the harmful bias. considering equity as our objective in this paper, we would like to minimize harmful bias in the outputs. the harmful bias of an output can be computed by subtracting its bias from the inevitable bias, i.e., β h (o) = β(o)β n (i). after defining the terms and notations, we are able to formulate our problem: given a task presented in the form of a prompt i, and including the demographic groups g, the objective is to identify an output @xmath6, such that it maximizes ρ(o) and minimizes β h (o). ## related work language models have gained popularity due to their proficiency at comprehending human language. nevertheless, prior research has examined numerous limitations of these models, particularly in terms of their reliability and fairness. various techniques have been previously presented to mitigate bias in language models while enhancing their reliability. in this literature, drop out is a regularization technique adopted to mitigate gender bias @xcite . the interruption generated by this strategy restricts the model from acquiring the ability to detect the connections between words that ultimately builds stereotypes. some studies propose reducing bias in pre-trained models and enhancing dependability through diverse data augmentation. this involves incorporating data points that cover various demographics @xcite @xcite . additionally, there are studies that focus on mitigating bias in word representation using post-processing techniques @xcite , as well as in sentence representation @xcite and context representa-tions @xcite . nevertheless, certain algorithms necessitate the process of retraining the model @xcite or finetuning @xcite . weighted sampling to improve fairness in classification tasks has been studied before @xcite but, to the best of our knowledge, this paper is the first to use repeated sampling for fairness (and reliability) in the context of llms. perhaps the most similar paper to our work is @xcite (called pair-ranker in our experiments), that uses pairwise comparison between the llm outputs to rank them. while pair-ranker also takes as the input a set of llm outputs and rank them, it has different goals and follows different technical approaches from requal-lm. also, pair-ranker has a significantly higher query cost, compared to requal-lm: pair-ranker issues an extra o(m 2 ) calls to the llm to rank the outputs, while requal-lm does not issue any additional calls other the m calls to collect the outputs. ## benefits in the following, we list some of the advantages of requal-lm, compared to the existing approaches. ## conclusion large language models exhibit remarkable versatility due to their ability to understand human language and generate content across various domains, languages, and tasks. however, responsible usage of llms calls to first understand and minimize the potential harms of these technology. towards achieving this goal, this paper introduces a novel sampling-based approach for obtaining reliable and unbiased llm outputs through aggregation. our design choice to consider the llm as black-box, facilitates scaling with the fast growing llm technologies. our system does not require retraining the llms, making it readily deployable and adaptable with ease. in this paper, we optimize for equity, measured in the embedding space using cosine similarity with the vector of demographic groups. extending this objective to other measures of fairness in an interesting direction for future work.