Soft Self-Consistency Improves Language Models Agents (2024)

Abstract: Generations from large language models (LLMs) can be improved by sampling and scoring multiple solutions to select a final answer. Current “sample and select” methods such as self-consistency (SC) rely on majority voting to score answers. However, when tasks have many distinct and valid answers, selection by voting requires a large number of samples. This makes SC prohibitively expensive for interactive tasks that involve generating multiple actions (answers) sequentially. After establishing that majority voting fails to provide consistent gains on such tasks, we demonstrate how to increase success rates by softening the scoring criterion. We introduce Soft Self-Consistency (SOFT-SC), which replaces SC’s discontinuous scoring with a continuous score computed from model likelihoods, allowing for selection even when actions are sparsely distributed. SOFT-SC improves both performance and efficiency on long-horizon interactive tasks, requiring half as many samples as SC for comparable or better performance. For a fixed number of samples, SOFT-SC leads to a 1.3% increase over SC in absolute success rate on writing bash programs, a 6.6% increase on online shopping (WebShop), and a 4.7% increase for an interactive household game (ALFWorld). Finally, we show that SOFT-SC can be applied to both open-source and black-box models.

## introduction the performance of large language models (llms) can be greatly improved by generating multiple samples and scoring their answers before making a final selection. one popular and effective "sample and select" approach is self-consistency (sc; @xcite , which leverages chain-ofthought prompting @xcite to generate multiple solutions for each input query and then determines the final answer via a majority vote. while sc has demonstrated consistent benefits on question-answering datasets, we find it provides minimal gains in several interactive settings where llms act as agents to generate a sequence of actions. sc's selection mechanism relies on exact match in order to tally votes, i.e., it scores answers based on their frequency. however, in interactive domains, multiple distinct and valid answers -in this case, actions -can be generated at each step. this diminishes the effectiveness of sc over actions because the likelihood of generating identical actions decreases as the number of plausible options grows. for instance, a model tasked with predicting bash commands based on user queries has a very large action space (all bash commands) and could generate semantically equivalent commands that differ in their surface form (e.g., ls -ltr vs ls -trl). 2 therefore, deriving a signal from voting in llm-agent domains would require sampling a large number of actions at each step throughout a lengthy trajectory, reducing efficiency and making sc prohibitively expensive (cf. fig. 1 ). we hypothesize that relaxing the strict scoring criterion from votes tallied by exact match to a continuous score will address the shortcomings of sc in two ways: (i) improving task performance in sparse action spaces; and (ii) increasing sample efficiency, i.e., higher success rates with fewer samples. we propose soft self-consistency (soft-sc), a continuous relaxation of exact-match sample and select methods. unlike match-based voting, soft-sc handles cases without a unique majority answer. crucially, for a white-box model, soft-sc incurs no additional cost and requires no external tests or metrics, as the probabilities used are already produced. finally, we show that soft-sc can be used to rescore black-box models' outputs and can be integrated into an efficient variant of sc. we test soft-sc on three diverse interactive domains: bash @xcite , @xcite , and alfworld @xcite . ## results and discussion unless mentioned otherwise, we report average performance on 3 random seeds for each test set. for the same number of samples k, soft-sc outperforms sc. in soft-sc exhibits better scaling with k. in soft-sc effectively scales with model size. as we scale up the size of the lm, we find that soft-sc continues to provide improvements over sc. fig. 2 shows the scaling trends for codellama models ranging from 7b to 70b parameters on bash and webshop with a fixed @xmath4. for each lm, soft-sc always outperforms sc. furthermore, soft-sc often allows smaller lms to outperform larger members of the same model class, e.g., codellama-13b with soft-sc outperforms codellama-34b with sc. this points to additional efficiency gains from soft-sc, as it can allow smaller models to replace larger ones. soft-sc improves black-box models more than sc. soft-sc requires access to token probabilities to score actions. however, the most performant llms are typically black-box models, often with limited or no access to logits @xcite @xcite . in fig. 3 4 related work sample and select methods for llms. ensembling via voting over or aggregating outputs @xcite can improve a classifier's performance. @xcite apply this paradigm to improve llms on reasoning tasks, introducing self-consistency (sc). we find that the majority voting used in sc is not suited for llm-agent domains because the llm's generations may not exactly match when the action space is large. @xcite generalize sc by prompting the llm to determine consistency. however, llms still struggle to determine consistency in interactive domains where the task is partially observable @xcite . in contrast to soft-sc, past work examining re-ranking strategies in code generation @xcite or reasoning @xcite ) rely on external test cases or model-based metrics to score responses. ## conclusion after establishing the shortcomings of standard voting-based sc in interactive tasks, we introduced soft-sc, which relaxes the exact-match scoring function used by sc to a continuous score. on three commonly used interactive benchmarks, we showed that soft-sc results in improved performance and increased efficiency. we also show that soft-sc is compatible with both white-box and black-box models and that it can be integrated into a more efficient adaptive variant of self-consistency. finally, we find that a well-calibrated model is not required for soft-sc to work well, and that logits outperform verbalized confidence scores.