B o N Appetit Team at L e W i D i-2025: Best-of-N Test-time Scaling Can Not Stomach Annotation Disagreements (Yet) (2025)

Abstract: Test-time scaling is a family of techniques to improve LLM outputs at inference time by performing extra computation. To the best of our knowledge, test-time scaling has been limited to domains with verifiably correct answers, like mathematics and coding. We transfer test-time scaling to the LeWiDi-2025 tasks to evaluate annotation disagreements. We experiment with three test-time scaling methods: two benchmark algorithms (Model Averaging and Majority Voting), and a Best-of-N (BoN) sampling method. The two benchmark methods improve LLM performance consistently on the LeWiDi tasks, but the BoN method does not. Our experiments suggest that the BoN method does not currently transfer from mathematics to LeWiDi tasks, and we analyze potential reasons for this gap.

## introduction supervised learning typically assumes a single fixed label per example. however, prior work documents substantial interpretative variability in human annotations, with annotators often disagreeing on labels @xcite @xcite , especially for subjective natural language processing (nlp) tasks @xcite . @xcite and @xcite argue that this variability is informative rather than problematic and @xcite suggests that variability should be explicitly integrated into the annotation processes. the shared task learning with disagreement (lewidi) 2025 @xcite tackles this opportunity and provides four datasets with annotator-level metadata and label variation. we document the datasets in detail in subsection 3.6. the datasets support two different tasks: (1) perspecivist task: predicting the label of each individual annotator. (2) soft-label task: predicting the distribution of human annotations for a single in the previous iteration of the lewidi shared task @xcite , many teams trained encoder-based models like bert @xcite directly on soft-labels. however, the innovations in generalist large language models (llms) and the rise of "reasoning" capabilities @xcite @xcite motivated us to answer the following question: to answer this question, we turn to test-time scaling methods, like bon sampling, which improve the llms performance by spending more compute per problem @xcite . these methods have been very successful in mathematics and coding, but have not been applied yet to nlp tasks with annotation disagreement, as far as we know. in this paper, we take established test-time scaling methods and apply them to the lewidi tasks. our contributions are: ## method nomenclature: a dataset is a collection of problem instances (problem in short). we sample a reasoning llm n times to solve a problem. each sample contains a prediction and a cot. a prediction could be text, a soft-label or a list of integers (perspectivist task). our test-time scaling method is not novel but rather a combination of methods already established in the literature. our innovation is to apply it to a new domain: the lewidi-2025 tasks. we refer to the method as bon sampling with step-wise scores, or just bon sampling in short. the method, shown in figure 1 , consists of three steps: (1) a reasoning llm generates n samples for a problem. (2) a judge llm scores each cot-step in each sample for correctness. (3) we choose the sample with the best score for the final prediction. we explain all method details in subsection 3.1 and subsection 3.2. ## results in ## conclusion we present a systematic evaluation of three testtime scaling methods on the lewidi tasks. our key findings are: (1) our prediction diversity metric correlates with test-time scaling performance and problem difficulty on the lewidi soft-label task.