Nash C o T : Multi-Path Inference with Preference Equilibrium (2024)

Abstract: Chain of thought (CoT) is a reasoning framework that can enhance the performance of large language models (LLMs) on complex inference tasks. In particular, among various studies related to CoT, multi-path inference stands out as a simple yet effective improvement. However, there is no optimal setting for the number of inference paths. Therefore, we have to increase the number of inference paths to obtain better results, which in turn increases the inference cost. To address this limitation, we can utilize question-related role templates to guide LLMs into relevant roles, thereby increasing the possibility of correct inferences for each path and further reducing dependence on the number of inference paths while improving reasoning accuracy. However, placing LLMs into specific roles may reduce their reasoning diversity and performance on a few tasks where role dependence is low. To alleviate the excessive immersion of the LLM into a specific role, we propose Nash CoT by constructing a competitive system on each path that balances the generation from role-specific LLMs’ and the general LLMs’ generation, thereby ensuring both effective role adoption and diversity in LLM generation further maintaining the performance of multi-path inference while reducing the requirement of the number of inference paths. We evaluate Nash CoT across various inference tasks, including Arabic Reasoning, Commonsense Question Answering, and Symbolic Inference, achieving results that are comparable to or better than those of multi-path CoT with the equal number of inference paths.

## introduction large language models (llms) have profoundly revolutionized the field of natural language processing (nlp) @xcite @xcite ; openai, * equal contribution. in order to reduce the dependency of multi-path inference on the number of inference paths while further enhancing its effectiveness, we can cast llms into a relevant role for a specific question. for instance, we can utilize a template such as "you're a mathematician, and • • • " to cast the llm in the role of a mathematician. this approach can improve the accuracy of each path's answer generated by the llm, thereby reducing the multi-path cot's reliance on the number of paths, potentially improving the llm's reasoning. however, excessively casting the llm into a specific role using templates may compromise its robustness (diversity), thereby adversely affecting ## preference equilibrium in mini-batch inference the experimental results of self-consistency indicate that multi-path reasoning can unleash the reasoning capability of llms. however, multipath inference lacks a theoretical foundation for determining the optimal number of inference paths, requiring us to increase the number of paths to ensure performance, and further increase the computational resource consumption. to reduce the number of paths required for multi-path inference, we utilize the concept of nash equilibrium to locally construct a binary gaming system in multipath inference. specifically, the preference of each valid inference path of the llm needs to achieve nash equilibrium with the preferences of the generation guided by the role-immersed llm. this approach increases the possibility of each path correctly answering the question while maintaining a certain level of robustness (diversity), thereby reducing the number of inference paths required by self-consistency. preference model. given text input x, and the sampled predictions (answers) y 1 , y 2 , and bradley-terry reward model r θ that has been deliberated by chen et al. we first define y 1 is preferred over y 2 as equation 1 : player templates : role templates for our llm that are shown the structure: {id(player): description.} mathematician: you are a mathematician, you excel at analyzing problems from a mathematical logical perspective and arrive at conclusions that align with your values. literary scholar: you are a literary scholar who has read a vast array of literary works. please consider the problem from the perspective of a literary scholar. philosophical: you are a philosopher, your knowledge base includes a wealth of philosophical knowledge. you enjoy approaching problems from a philosophical perspective and arriving at conclusions that align with your values. geographer: you are a geographer with a deep understanding of geographical knowledge. please approach the given problem from the perspective of a geographer. furthermore, drawing from the definition in @xcite , we define one policy (llm for decision making) π 1 as more preferred over another policy π 2 as: preference equilibrium. we first discuss why we should construct a gaming system: to increase the accuracy of each inference path, we can select the most suitable template (example 3) for the given question, immersing the llm into a templaterelated role. this approach helps llms to solve problems more effectively, thereby reducing the number of inference paths needed. however, this may lead to some issues: if the llm is excessively immersed in a specific role, it may reduce the diversity of llm's generation, and further reduce llm's accuracy in solving tasks that have low role dependence. to address these problems, we locally build a bi-player gaming system that the preference of role-immersed llm (player 1) over the normal status of llm (player 2) is the pay-off of role-immersed llm, vice visa. if player 1 and player 2 reach nash equilibrium foot_1 , then the generation can balance the preference of both player 1 and player 2. definition 3.1 (preference equilibrium). given any two policys (players) π 1 and π 2 along with the pay-off of π 1 and π 2 as p(π 1 ≺ π 2 ) and p(π 2 ≺ π 1 ), we define the nash equilibrium of π 1 and π 2 as preference equilibrium. subsequently, we define the status that player 1 and player 2 reach nash equilibrium as preference equilibrium (definition 3.1). meanwhile, in theorem 3.1, we prove the existence of nash equilibrium in this system. in particular, the strategy of player 1 equal to player 2 is one solution that this system has reached preference equilibrium. theorem 3.1 (existence of preference equilibrium). given any two policy (player) π 1 and π 2 within the gaming system defined in definition 1, where π ∈ π. π 1 ≡ π 2 is a solution where the gaming system reaches nash equilibrium. ## experiments the goal of our experiment is to 1) showcase the performance advantage and effectiveness of nash cot. 2) showcase whether nash cot helps reduce inference paths and inference time. ## ablation study we further conduct extensive ablations to answer the following questions: 1) is there a correlation between the performance of nash cot and the setting of the number of inference paths and can the performance of nash cot be improved to surpass self-consistency by adjusting the number of inference paths? 2) does the template improve the accuracy of path predictions, and what impact does it have on experimental performance? 3) is the nash equilibrium the reason for the performance improvement observed in nash cot? as the number of inference paths increases, nash cot can obviously surpass self-consistency with fewer inference paths. to address question 1), we selected mistral-instruct (7b) and evaluated three different reasoning tasks, adjusting the n mini and n outer . as shown in figure 4 , as the number of loops increases, nash cot has a high possibility of outperforming self-consistency with fewer paths. however, different from self-consistency, the experimental results of nash cot do not show a monotonic (linear) relationship with the total number of paths. this indicates that there is a significant difference between nash cot and self-consistency. unlike nash cot, the experimental results of selfconsistency show a clear improvement in performance as the number of paths increases. ## conclusion we prove the existence of a nash equilibrium in preferences between role-immersed and normalstatus llm generations, and develop a gaming system within each path of the multi-path cot to introduce nash cot. experimental results show that nash cot can perform equally or even better than self-consistency while requiring fewer paths.