Unsupervised Contrast-Consistent Ranking with Language Models (2024)

Abstract: Language models contain ranking-based knowledge and are powerful solvers of in-context ranking tasks. For instance, they may have parametric knowledge about the ordering of countries by size or may be able to rank product reviews by sentiment. We compare pairwise, pointwise and listwise prompting techniques to elicit a language model’s ranking knowledge. However, we find that even with careful calibration and constrained decoding, prompting-based techniques may not always be self-consistent in the rankings they produce. This motivates us to explore an alternative approach that is inspired by an unsupervised probing method called Contrast-Consistent Search (CCS). The idea is to train a probe guided by a logical constraint: a language model’s representation of a statement and its negation must be mapped to contrastive true-false poles consistently across multiple statements. We hypothesize that similar constraints apply to ranking tasks where all items are related via consistent, pairwise or listwise comparisons. To this end, we extend the binary CCS method to Contrast-Consistent Ranking (CCR) by adapting existing ranking methods such as the Max-Margin Loss, Triplet Loss and an Ordinal Regression objective. Across different models and datasets, our results confirm that CCR probing performs better or, at least, on a par with prompting.

## introduction language models have been shown to store plenty of facts and have powerful reasoning capacities @xcite . ranking tasks require both of these skills: multiple items have to be put in relation based on a pointwise item representations listwise loss objective comparison criterion. we are posing the question: what is the best approach to elicit a model's ranking knowledge and in-context ranking capacities without supervision? knowing the answer to this question would allow us to uncover knowledge gaps, outdated information and existing biases before applying the language model. once we trust a model, we could then put this best approach to action for solving in-context ranking tasks. a natural starting point for unsupervised ranking is prompting. in §2, we explore different task formulations: pairwise, pointwise and listwise prompting as outlined in fig. 1 . in the pairwise setting, any two items are compared and pairwise results are converted into a global ranking post-hoc. in pointwise prompting, the model assigns a score to each item individually. the listwise approach tasks the model to directly decode the entire ranking. for either approach, constrained decoding is essential to ensure the output can be converted into a ranking that includes all items. yet, even with constrained decoding and calibration, we find that prompting often leads to inconsistent rankings. for this reason, we turn to the model-internal representations of ranking tasks and their items in §3. we train a "probing model" with different unsupervised ranking objectives to find a latent ordering direction in the items' vector representations. @xcite recently proposed the contrast-consistent search (ccs) method to find a direction in a language model's activation space that distinguishes truthful from false statements @xcite . this is achieved with a loss that imposes a logical constraint: the representation of a statement and its negation must be mapped to opposite (contrastive) poles. ranking tasks share similar properties: we can convert a ranking task into multiple pairwise comparisons and train a probe to find a "ranking direction" that allows ranking one item higher than the other consistently across all pairs. this has one significant advantage over the original ccs method for factual statements-instead of requiring a training set of multiple yes-no questions, we can source all pairwise permutations from a list of items which allows training the probe on a single ranking task. we extend and adapt the binary ccs method to contrast-consistent ranking (ccr) by exploring pairwise ( §3.1), pointwise ( §3.2) and listwise ( §3.3) approaches as illustrated in fig. 1 . pairing items in the prompt and obtaining the vector representations of all pairs is computationally expensive. moreover, binary, contrastive poles may not be ideally suited for ranking tasks where the distances between items are not unit-length. similar to the pointwise prompting approach, we instead embed each item individually, e.g., "the size of the us is [mask], the size of china is [mask], ...". we then pair the items represented by the activations of the [mask] tokens in the loss function. in particular, we propose variants of the well-known max-margin and triplet loss by including a consistency and confidence component. as a final adjustment, we mitigate the limitation that pairwise and pointwise objectives do not guarantee transitivity: item a may be ranked above b, b above c, but c above a, creating a circular contradiction. to address this, we introduce an unsupervised ordinal regression objective for listwise ccr probing. our experiments in §4 confirm that ccr probing outperforms prompting with a deberta @xcite and gpt-2 @xcite model across 6 datasets. among the ccr probing methods, the triplet loss variant performs best on average. even for a much larger mpt-7b (mosaicml, 2023) model, ccr probing performs at least on a par with prompting. yet, ccr probing has the advantage of better control, reliability and interpretability as we discuss in §5. ## prompting for rankings prompting is an accessible way to test a language model's ranking knowledge @xcite . we experiment with three different prompt types outlined in tab. 1: pairwise, pointwise and listwise prompting @xcite . all prompt types contain at least one item to be ranked, a criterion to rank on, and what we refer to as comparison token. in every setting, we rely on some form of "constrained decoding" (for decoder-only) or "constrained mask-filling" (for encoder-only models). in essence, we restrict the vocabulary to a list of candidates and select the tokens with the highestscoring logits. pairwise prompting. itempair p: is {item a} more in terms of ranking criterion than {item b}? yes / no-between any two items, the language model is tasked to make ranking decisions which are then converted into a ranking post-hoc as elab-prompt type emb calls loss / model datapoints ccr probing pointwise prompting. itemsingle s: on a scale from 0 to 10, the ranking criterion of {item} is x-in pointwise prompting, the language model ranks one item at a time. if two items are assigned the same rank (i.e., the same candidate token from the list @xmath0, 1, 2, . . . , 10}), we break the tie via sorting by the tokens' logit scores. listwise prompting. itemlist l optional: context. order by ranking criterion. options: "a" {item a}, "b" {item b}... the correct ordering is: x-for listwise prompting, we apply a step-wise approach: we let the model select the highest-scoring item from the list of candidates @xmath1, b, ...}, remove this token from the list and append it to the prompt. we repeat the process until the candidate list is exhausted. importantly, the ordering of the candidate options in the prompt poses a "positional bias" @xcite . therefore, we randomly shuffle the ordering of the options and repeat the listwise prompting multiple times. ## unsupervised probing for rankings querying a language model's knowledge via prompting, we limit ourselves to prompt design and evaluating the tokens' logit scores. in contrast, probing accesses the information contained within a language model more directly by operating on latent vector representations. conventionally, probing involves training a "diagnostic classifier" to map the vector representations of an utterance to a target label of interest (e.g., tense, gender bias, etc.) in a supervised fashion. the goal typically is to measure what information is contained within a language model @xcite . while the motivation of this work is closely related, we focus on an unsupervised probing variant and consider supervised probing only as a performance upper bound for validation purposes in §4.5 and §5. contrast-consistent search (ccs). @xcite propose contrast-consistent search (ccs), an unsupervised probing method which seeks to train a probe to satisfy logical constrains on the model's activations. instead of labels, ccs requires paired prompts in the form of yes-no questions: both statements x + i and x - i are fed to a language model and the activations of the model's last hidden layer corresponding to the "yes" and "no" token, x + next, the paired vectors are projected to a score value s i via the probe f θ (x i ) = σ(θ t x i +b) which is trained using the origccs loss objective: origccs comprises two terms: the consistency term encourages f θ (x + i ) and f θ (x - i ) to sum up to 1. the confidence term pushes the scalars away from a deficient f θ (x + i ) = f θ (x - i ) = 0.5 solution, and instead encourages one to be close to 0 and the other to be close to 1. thus, the origccs objective promotes mapping true and false statements to either 0 or 1 consistently, when the probe is trained on multiple yes-no questions. foot_0 from yes-no questions to rankings. origccs relies on logical constraints to identify a true-false mapping in the models' activations. we argue that ranking properties can similarly be expressed as logical constraints which are discernable by a probing model. in fact, the pairing of yes-no statements in eq. ( 1 ) resembles the itempair prompt type presented in tab. 1. one advantage of ranking tasks is that we can source many pairwise comparisons from a single ranking task which reduces the need for a training set of different yes-no questions. in the original ccs paper, it has been shown that a training set of as few as 8 pairwise comparisons can be enough for good test set performance. a ranking task of 8 items allows for 28 comparisons when considering all pairwise combinations and even 56 comparisons when considering all pairwise permutations. we adapt binary ccs to contrast-consistent ranking (ccr) by gradually modifying three components of the original method: in §3.1, we start by changing only the prompt. in §3.2, we explore pointwise ccr probing which requires changing the loss function in addition. finally, in §3.3, we also alter the probing model to propose a listwise regression approach. importantly, all ccr approaches are unsupervised and involve training a linear probing model whose number of parameters is held constant across settings to allow for a fair comparison. ## discussion to scrutinize our results, we explore settings with a train-test split, and discuss the interpretability considerations of ccr probing. ranking direction across tasks. instead of training our probes on a single ranking task, we train them on a training set of multiple rankings and evaluate on a held-out set. to this end, we use 4-fold cross-validation which allows comparing ccr probing against supervised probing in a fair setup. this setup is more similar to the experiments in the original ccs paper @xcite and thus rests on a similar hypothesis: is there are a more universal "ranking direction" in the activations of a language model that holds across ranking tasks? fig. 6 in the appendix presents the results of this k-fold validation experiment. firstly, our probes identify ranking properties that exist across different ranking tasks. this particularly holds for ranking tasks that resemble each other more closely as in scalaradj or reviews. secondly, ccr probing does not fall far behind supervised probing. since this is especially evident for datasets with fewer ranking tasks, we hypothesize that ccr probing is less likely to overfit and instead exploits general ranking properties. interpretability. besides performance, another argument for ccr probing is control and post-hoc interpretability offered by the parametric probe. in fig. 4 for instance, we plot the ranking scores s @xmath2. this allows us to inspect the distances between items projected onto the latent ranking scale. the predictions and parameters are deterministic opposed to prompt-based generations from stochastic decoding methods. on a more abstract level, we relate multiple language model queries through a surrogate model that projects the language model's outputs to a shared ranking scale. this paper builds upon contrast-consistent search (ccs) @xcite , which has inspired multiple other follow-up works: some explore calibrated versions of ccs @xcite , others adapt ccs to order-invariant, multi-class settings @xcite , compare different ccs objective functions @xcite and elicit inference-time interventions to increase truthfulness @xcite . @xcite raise concerns that ccs may not "discover knowledge", but instead simply latches onto to most salient features. we argue that our ccr approach is less affected by this concern as we are mainly focused on achieving good predictive performance in unsupervised ranking tasks by making consistent measurements across multiple prompts. to this end, we test our method against regression-based variants in §3.3 and evaluate on an unseen held-out set in §5. pairwise and listwise prompting have been explored in different tasks @xcite @xcite , but is most frequently focused on document retrieval @xcite . pairwise (ranknet) @xcite and listwise (listnet) @xcite ranking approaches have also been compared outside of language model prompting. we additionally explore pointwise prompting @xcite and find that, counter-intuitively, pointwise often outperforms listwise prompting. to move beyond prompting, we propose an expansion of the ccs method to rankings. ccs and ccr are concep-tually different to "contrast consistency" which refers to contrastive data perturbations @xcite . they are also different to "contrastive decoding" @xcite which contrasts log-probabilities between an expert and an amateur model. instead, our ccr probing approach is strongly influenced by unsupervised ranking @xcite and probing of semantic, ordinal axes @xcite @xcite . ## conclusion we analyze the ranking capabilities of language models by comparing pairwise, pointwise and listwise prompting techniques and find that listwise prompting is less computationally expensive, but more susceptible to mistakes. we then propose an unsupervised probing method termed contrast-consistent ranking (ccr). ccr learns an affine mapping between a language model's activations and a model-inherent ranking direction. especially for smaller language models, ccr outperforms prompting, is easier to control, less susceptible to prompt design and more interpretable. we see a lot of potential in in-context probing for making consistent measurements with language models.