CURATRON : Complete and Robust Preference Data for Rigorous Alignment of Large Language Models (2024)

Abstract: This paper addresses the challenges of aligning large language models (LLMs) with human values via preference learning (PL), focusing on incomplete and corrupted data in preference datasets. We propose a novel method for robustly and completely recalibrating values within these datasets to enhance LLMs’ resilience against the issues. In particular, we devise a guaranteed polynomial time ranking algorithm that robustifies several existing models, such as the classic Bradley–Terry–Luce (BTL) model and certain generalizations of it. To the best of our knowledge, our present work is the first to propose an algorithm that provably recovers an \epsilon -optimal ranking with high probability while allowing as large as O(n) perturbed pairwise comparison results per model response. Furthermore, we show robust recovery results in the partially observed setting. Our experiments confirm that our algorithms handle adversarial noise and unobserved comparisons well in LLM preference dataset settings. This work contributes to the development and scaling of more reliable and ethically aligned AI models by equipping the dataset curation pipeline with the ability to handle missing and maliciously manipulated inputs.

## introduction large language models (llms) are highly advanced artificial intelligence (ai) systems capable of understanding, interpreting, and generating languages. the integration of ai chatbots like chatgpt into our daily lives and businesses has had a profound impact on both society and industries @xcite . however, the success of gpts/llms depends not only on their ability to generate responses and perform tasks well but also on their alignment with human values and expectations. the prevalent method for aligning ai/llms currently involves preference learning (pl) through human feedback. however, gathering human feedback is slow and expensive and often results in incomplete or imperfect data @xcite . furthermore, participants may intentionally provide inaccurate or harmful feedback due to malicious intentions, as pointed out by @xcite . these factors can lead to unintended consequences in estimating rankings from preference datasets from models such as btl. they pose a considerable challenge in ensuring the integrity and reliability of the preference datasets used for aligning llms, especially when scaling up the alignment process with large-scale responses and participants. approaching the issues, we consider the following learning problem: suppose there are 𝑛 responses we wish to order based on a notion of comparison, between every pair of responses, with probabilistic outcomes. further, we are given a set, ℵ = {(𝑖, 𝑗, {𝑦 𝑘 𝑖 𝑗 })}, consisting of 𝐾 independent pairwise comparison outcomes, denoted by {𝑦 𝑘 𝑖 𝑗 } ∈ {0, 1}, 𝑘 ∈ @xcite , between pairs of responses (𝑖, 𝑗) ⊆ [𝑛] × [𝑛], a significant proportion of which might be corrupted by an adversary. in this passive learning setting, our contributions are as follows. we give a generic definition of (additive) adversarial noise and show that if it is not accounted for, the quality of the estimated ranking can be quite poor. to address this, we develop an efficient and correct ranking method called robust preference data for rigorous alignment (roratron), which is robust against adversarial noise. under certain assumptions, we prove that our method guarantees high-probability learnability with a small margin of error. we also devise a method called complete robust preference data for rigorous alignment (curatron) to handle the scenario where not all pairs are compared, and the observed pairwise data is adversarially corrupted. ## partially observered adversarial setting in this section, we consider the partially observed and adversarially corrupted comparison results setting. both factors can be modeled in a unified manner by setting the corresponding missing entries of the preference matrix to zero (or a specific constant to account for numerical stability). we present our robust ranking algorithm for this setting in algorithm 2 -this essentially involves using the 'optspace' matrix completion algorithm of @xcite followed by using the robust pca algorithm of @xcite as sub-routines. we now derive the recovery guarantees as follows. theorem 2 (provably good estimation of ranking in btl model in the presence of adversarial noise as well as missing data). consider a similar notation as in theorem 1 but let @xmath0. let ω ⊆ [𝑛] × [𝑛] be a set of compared response pairs. assume ω is drawn uniformly from all subsets of [𝑛] × [𝑛] of size |ω| such that |ω| ≥ 𝐶 ′′ 𝑛 log(𝑛) and let the sparse noise satisfy ∥s∥ ∞ ≤ δ 𝑤 log(𝑛) 𝐶 δ 𝑛 where δ 𝑤 := min 𝑖, 𝑗 𝑤 𝑖 -𝑤 𝑗 . let the number of comparisons per pair be 𝐾 ≥ 𝑐𝑛 4 /δ 𝑤 . then with probability at least 1 -2/𝑛 3 , algorithm 2 returns a ranking that satisfies dist( 𝜎, p) ≤ 𝜖. remark 3 (robust estimation of btl model in the partially observed case). for the btl model, theorem 2 says 𝑂 (𝑛 log 𝑛) pairs suffice to estimate the btl model, which matches bounds from @xcite . further, even in this incomplete comparison data case, we are able to tolerate uniformly random additive sparse noise with its maximum absolute entry scaling as the order of the btl 'score-gap' divided by the number of responses up to logarithmic factors, ie, 𝑂 (δ 𝑤 /𝑛). proof. from lemma 3, we have 𝜓(p) = 1w ⊤ -w1 ⊤ for the btl model where 𝜓 is the logit function. clearly, in this case, 𝜓(p) is a real skew-symmetric matrix of rank 𝑟 = 2. since it is skew-symmetric, its eigenvalues, which are the roots of its charateristic polynomial, are of the @xmath1, and hence, 𝜎 min (𝜓(p)) = 𝜎 max (𝜓(p)), ie, the condition number of 𝜓(p), 𝜅 = 1. now, we recall the spectral-lower bound from corollary 2 of @xcite , ## experiments we now perform simulations in order to understand the performance of our robust ranking approach in practice in both general and llm preference dataset settings. ## conclusion our study examines how missing information and distorted feedback can impact llms, potentially compromising their performance in terms of alignment with human values. we have proposed a robust algorithm for provably correct and efficient ranking responses in the btl, lr, and general binary choice models. this robust ranking data is then input in the pl step. further, we also handled the partially observed setting, wherein only some response pairs are compared, by integrating matrix completion techniques into our robust learning algorithm. in all cases, we provided statistical and computational guarantees using novel techniques. through our comprehensive analysis, we hope to contribute to the ongoing discussion on ai safety by helping to create and scale llms/agi models that align with human values and expectations. some future research directions include tightening the recovery results for partially observed settings under weaker conditions (possibly using noisy-case extensions of @xcite ), exploring other notions of adversarial noise, and understanding the minimax optimal rates for ranking estimators under various noise models. we also plan to study the parametric non-active pairwise ranking setting, studying lower bounds and practical algorithms in the active setting similar to @xcite . furthermore, it would be interesting to investigate whether we can extend this approach to solve the entity corruption problem in retrieval models, as shown in @xcite . another research direction could be defining an alignment framework that expands dpo to various objective functions based on rank centrality @xcite . finally, we aim to examine the relationship between robust pl and model capacity, as this can shed light on the trade-offs between model complexity and generalization performance.