Language Model Council: Democratically Benchmarking Foundation Models on Highly Subjective Tasks (2025)

Abstract: As Large Language Models (LLMs) continue to evolve, evaluating them remains a persistent challenge. Many recent evaluations use LLMs as judges to score outputs from other LLMs, often relying on a single large model like GPT-4o. However, using a single LLM judge is prone to intra-model bias, and many tasks – such as those related to emotional intelligence, creative writing, and persuasiveness – may be too subjective for a single model to judge fairly. We introduce the Language Model Council (LMC), where a group of LLMs collaborate to create tests, respond to them, and evaluate each other’s responses to produce a ranking in a democratic fashion. Unlike previous approaches that focus on reducing cost or bias by using a panel of smaller models, our work examines the benefits and nuances of a fully inclusive LLM evaluation system. In a detailed case study on emotional intelligence, we deploy a council of 20 recent LLMs to rank each other on open-ended responses to interpersonal conflicts. Our results show that the LMC produces rankings that are more separable and more robust, and through a user study, we show that they are more consistent with human evaluations than any individual LLM judge. Using all LLMs for judging can be costly, however, so we use Monte Carlo simulations and hand-curated sub-councils to study hypothetical council compositions and discuss the value of the incremental LLM judge.

## introduction as large language models (llms) continue to advance, evaluating their outputs remains a significant challenge. manual human evaluations are timeconsuming and expensive, motivating the need for automatic metrics @xcite and evaluation methods (e.g. @xcite @xcite . conventional model evaluations rely on closed-ended questions that can be "people can find it hard to maintain long distance relationships, not because they don't value you, but because life can be hectic in unique ways." "the first response is slightly better. both of them demonstrate empathy and provide advice. however, the first is more concise and direct." checked automatically such as mmlu @xcite . however, these static benchmarks are vulnerable to data contamination @xcite and are often misaligned with human preferences in real-world, open-ended contexts @xcite . for evaluating open-ended responses, automatic metrics like bleu @xcite , rouge @xcite , and bleurt @xcite are common. these require reference responses that may be expensive to collect and yet may still fail to reflect human preferences beyond a quality threshold @xcite . arena-based methods enable reference-free evaluation by comparing llms in head-to-head matchups. the outcome of a battle between the responses of two llms can be evaluated based on objective measures @xcite , ratings from strong model judges like gpt-4 @xcite , or human judges @xcite , with each outcome contributing to metrics like win rates or elo scores @xcite . strong llm judges like gpt-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans, in a generaldomain, arena-based setting @xcite . unfortunately, it has also been observed that evaluator models tend to have their own biases, for example recognizing and preferring their own outputs over those of other models @xcite , which can be mitigated by using a committee of llm judges (e.g. @xcite . in a landscape of diverse llm judges, can we account for each model's opinions and still establish a definitive ranking amongst them? this paper aims to contribute a unique perspective in the pursuit of a fully decentralized approach to evaluating llms. when ranking llms on competencies that may be too subjective for a single model to judge fairly, we investigate the dynamics and benefits of an inclusive evaluation network where all models contribute equally to the ranking process. inspired by democratic organizations in human society, we introduce the language model council (lmc), a framework for collective evaluation for a group of llms. the lmc operates in three stages: (1) a test set is formulated with equal participation from all council members, (2) the test set is administered to all council members to complete, and (3) the responses are evaluated by the council as a collective jury. our main contributions are as follows: we release all data, code, leaderboard, and web demo at https://llm-council.com . ## related work subjective tasks and label variability in humans have received increased attention in the nlp community @xcite @xcite . llms also exhibit variability in their outputs as they inherit inconsistencies and biases from human data @xcite @xcite @xcite . while llm judges might not replicate human disagreement patterns exactly @xcite , their differences in judgment are increasingly viewed as reflecting valid dissent. notably, when humans disagreed with gpt-4, they considered its judgments reasonable in 75% of cases and sometimes revised their own answers @xcite . ## case study: using the lmc to rank llms on emotional intelligence the lmc framework consists of three stages: (1) test set formulation, (2) response gathering, and ## results and findings qwen-110b outranks gpt-4o in an unexpected upset. like other benchmarks, larger models within the same family tend to outrank their smaller or older versions. however, unlike other benchmarks, qwen-1.5-110b (#20 on chatbot arena) scores highest on our ei task, followed by gpt-4o (#1 on chatbot arena). 9 this is surprising, as qwen-1.5-110b does not typically outperform gpt-4o. one possible reason for this outcome is the use of qwen-1.5-32b as the reference model (appendix c). because all of qwen-1.5-110b's responses are compared to the responses of a strictly smaller variant of the same family, this could result in an outsized advantage for qwen-1.5-110b in the evaluation overall. this raises an interesting possibility of successor bias in arena-style evaluations: the choice of reference model may inadvertently favor its successors in the same arena. judges prioritize actionability, clarity, and structure when expressing preferences. using chainof-thought (cot) prompting, judges provided detailed reasoning for their preferences. we analyzed 1,000 reasoning traces and identified common themes (appendix f). judges disfavor models that produce responses significantly shorter than the suggested word limit. despite a suggested 250-word limit, some models generated much shorter responses, even though decoding parameters allowed for longer output. models that adhered to the limit, using 220+ words on average, performed better, while all of the models in the bottom four positions averaged less than 200 words. notably, gemini-1.5-pro placed solely by budget constraints. given the cost of human evaluations (c9/hour), the number of models studied had to be limited. however, these nine models were carefully chosen to ensure diversity in model size, openness, and company origin. consistent judging, neutral voting patterns, and lower contrarianism correlate with higher separability. judges with more consistent votes and neutral voting patterns tend to achieve higher separability. judges with high conviction -those that express strong preferences (@xmath0, suggesting that overly strong votes tend to introduce noise. for example, claude-3-opus, despite expressing strong preferences only 0.1% of the time, achieves the second-highest separability (72.6%). higher contrarianism, which measures how often a judge disagrees with the majority decision, also correlates with lower separability. qwen-1.5-72b, the most contrarian judge, has only 38.9% separability. llms exhibit slight self-bias, but this effect is mitigated within the full council. out of the 20 llms in our study, 12 exhibited positive selfenhancement bias, meaning they rated themselves more favorably than the council's overall score for them. when self-graded battles are excluded, the overall rankings remain similar (figure 9 ). this indicates that while individual models carry selfenhancement bias, the ensembling of the lmc effectively neutralizes these biases. agreement among lmc members, among humans, and between the lmc and humans is similar. figure 3 shows that the rankings produced by lmc members and humans are consistent, with small variations. both groups agree on the top-performing models and the lowest-ranked ones. the level of agreement between the lmc and humans is roughly the same as the agreement within human evaluators (51.9%). the lmc's rankings align more closely with human evaluations than other benchmarks or individual judges. the lmc achieves the high-e q -b e n c h c h a t b o t a r e n a c h a t b o t a r e n a e q m m l u l m c -e i ( o u r s ) h u m a n s t u d y 1 2 3 4 5 6 7 8 9 rank qwen1.5-110b-chat gpt-4o-2024-05-13 claude-3-opus qwen1.5-32b-chat llama-3-70b-chat claude-3-haiku mixtral-8x7b llama-3-8b-chat gpt-4-0613 est correlation with human-established rankings, outperforming other benchmarks, including those from similar domains like the ei-specific subset of chatbot arena (appendix e) and eq-bench (paech, 2023) (figure 4 ). while a few individual llm judges, like dbrx-instruct, achieve similar correlation scores (e.g., 0.917), they do so with significantly lower separability (50.5% vs. 90.5% for the lmc). this suggests that the lmc's collective judgment not only aligns more closely with human preferences but also provides clearer distinctions between model performance, making it a more reliable method for ranking models in our case study. additional findings. the 20x20 llm interactions generate a wealth of data, some of which is not included in the main paper for brevity. additional insights can be found in appendix a. ## discussion one of the most important questions about using a language model council is whether it is worth the trouble. collecting more opinions costs more energy, and parsing, tallying, and storing them add logistical costs. what is the value of another opinion, much less everyone's opinion? from a cost perspective, a closely related question is how many examples should be judged in the first place. a smaller test set not only lowers costs but also simplifies the task for any human reviewers -reading and evaluating 10 examples is far more manageable than 10,000. if we assume that the main goal is to establish good relative rankings, we narrow the focus of quantifying the success of a council by measuring the significance and stability of the final ranking. for significance, we look at separability (as in the main experiment), which measures how well models can be distinguished based on non-overlapping confidence intervals. for stability, we create a new metric, mean expected rank variance (merv), defined as the expected ordinal swing of the average respondent's rank (appendix g). a merv of 3 means an average respondent's rank is expected to change up to 3 positions in a new trial. lower merv indicates that the benchmark has more stable rankings, with merv of 0 signifying perfect deterministic-like stability. ## conclusion in this paper, we introduce the language model council (lmc), a flexible, decentralized evaluation framework for ranking llm agents through democratic participation. applying the lmc to an emotional intelligence task with 20 llms, we demonstrate that the lmc can produce highly separable rankings that align more closely with human judgments than other benchmarks or individual judges. through both monte carlo simulations and hand-curated sub-councils, we find that while larger councils provide benefits, they are incrementally diminishing, and the majority of key qualities-such as ranking significance, stability, and alignment with human evaluations-can be achieved with a smaller, well-chosen ensemble of judges. as humans increasingly rely on llms to evaluate other llms, we hope the lmc framework, along with insights from our case study, offers a valuable foundation for developing reliable yet inclusive llm evaluations, even for highly subjective tasks.