Measuring the Inconsistency of Large Language Models in Preferential Ranking (2024)

Abstract: Despite large language models’ (LLMs’) recent advancements, their bias and hallucination issues persist, and their ability to offer consistent and preferential rankings remains underexplored. This study investigates the capacity of LLMs to provide consistent ordinal preferences, a crucial aspect in scenarios lacking absolute answers. We introduce a formalization of consistency based on order theory, outlining criteria such as transitivity, asymmetry, reversibility, and independence from irrelevant alternatives. Our diagnostic experiments on selected state-of-the-art LLMs reveal their inability to meet these criteria, indicating a strong positional bias and poor transitivity, with preferences easily swayed by irrelevant alternatives. These findings highlight a significant inconsistency in LLM-generated preferential rankings, underscoring the need for further research to address these limitations.

## introduction expressing one's preferences in an ordinal manner is a widespread and informative practice in human reasoning and communication @xcite . by evaluating and comparing available options, individuals can make more informed decisions and communicate their values to others more effectively. in the domain of natural language processing (nlp), human preferential feedback serves as a valuable data type for aligning language models with human inclinations @xcite . recent advances in large language models (llms) have prompted researchers to investigate the potential of llms in complex ranking-based tasks -such as recommendation @xcite , web search @xcite , and text relevance comparison @xcite -traditionally handled by task-specific models. moreover, given that human annotation and evaluation now, provide a preferential ranking on following 4 objects: , , , and . ## conclusion and future work to conclude, we have formalized the consistency measurements in preferential ranking tasks by designing corresponding criteria and metrics. through diagnostic experiments, we have evaluated some of the most advanced llms, uncovering severe inconsistencies and positional biases that are prevalent across all models, among other observations. our study raises general awareness of discrepancies in llms and signals a call for future research efforts. specifically, we highlight two areas of interest: the development of a non-mcqa benchmark for consistency measurement and the creation of mitigation methods to enhance the consistency of llms in ranking-based tasks.