LM 2: A Simple Society of Language Models Solves Complex Reasoning (2024)

Abstract: Despite demonstrating emergent reasoning abilities, Large Language Models (LLMS) often lose track of complex, multi-step reasoning. Existing studies show that providing guidance via decomposing the original question into multiple subproblems elicits more robustness in LLM reasoning – a decomposer generates the subproblems, and a solver solves each of these subproblems. However, these techniques fail to accommodate coordination between the decomposer and the solver modules (either in a single model or different specialized ones) – the decomposer does not keep track of the ability of the solver to follow the decomposed reasoning. In this paper, we propose LM2 to address these challenges. LM2 modularizes the decomposition, solution, and verification into three different language models. The decomposer module identifies the key concepts necessary to solve the problem and generates step-by-step subquestions according to the reasoning requirement. The solver model generates the solution to the subproblems that are then checked by the verifier module; depending upon the feedback from the verifier, the reasoning context is constructed using the subproblems and the solutions. These models are trained to coordinate using policy learning. Exhaustive experimentation suggests the superiority of LM2 over existing methods on in- and out-domain reasoning problems, outperforming the best baselines by 8.1% on MATH, 7.71% on JEEBench, and 9.7% on MedQA problems (code available at https://github.com/ LCS2-IIITD/Language_Model_Multiplex).

## introduction recent trends in solving complex reasoning tasks using large language models (llms) typically follow two different dominant approaches: (i) well-curated prompting techniques @xcite on llms of exorbitant * equal contribution size like @xcite , or (ii) finetuning a relatively smaller llm using domain-focused data @xcite @xcite . methods from the former category heavily rely on the proprietary llm being used and are prone to fail absolutely when employed with less powerful models. the latter category, though cost-effective compared to humongous llms, often loses in generalizability due to a narrow training domain. the chronicle of decomposed reasoning. a number of recent literature has pointed out that llms tend to perform better on complex reasoning tasks when the problem is decomposed into step-by-step subproblems @xcite @xcite . earlier techniques demonstrated the superiority by providing the model with examples containing the original problem decomposed into multiple sub-problems along with their answers @xcite . however, @xcite illustrated that decoupling the decomposer from the solver by finetuning a separate decomposer language model (lm) to coordinate with a larger solver lm is beneficial to simply prompting a single monolithic lm to decompose and solve. echoing their findings, @xcite also found that distilling decomposition abilities from a larger lm to a smaller lm is much more generalizable compared to decomposing the solver abilities directly. our contributions. however, a major bottleneck in existing methods of decomposer finetuning is the lack of tightness between the decomposersolver interactions. typically, the decomposition is done in a memoryless manner, with or without the solver's initial response; no strategy is employed to track whether the solver can follow the decomposed chain of reasoning. towards this very end, we propose a novel multi-llm coordination framework, language model multiplex (lm 2 ). lm 2 is built upon three separate lms, each dedicated to : how many distinct, non-equilateral triangles with a perimeter of 60 units have integer side lengths , , and such that , , is an arithmetic sequence? solver lm decomposer lm verifier lm sq: what is a, b, c in terms of common difference d? sa:since a, b, and c form an arithmetic sequence, we can express c in terms of a as @xmath0, where d is the common difference let be the common difference, so and we can assume that is positive in particular, can't be 0, because the triangle is not equilateral then the perimeter of the triangle is , so hence, the sides of the triangle are , 20, and these sides must satisfy the triangle inequality, which gives us solving for , we find , or therefore, the possible values of are 1, 2, , 9, which gives us possible triangles 1. we know that for a triangle with side lengths a, b, and c, the sum of the lengths of any two sides must be greater than the length of the third side. 2. since we are given that a, b, and c form an arithmetic sequence, we can express b as the average of a and c. this gives us . ... 6. since a, b, and c form an arithmetic sequence, we can express c in terms of a as @xmath1, where d is the common difference. ... 10. this gives us the side lengths of the triangle as @xmath2, @xmath3, and @xmath4. hence final answer is concepts: 1. triangle inequality 2. arithmetic progression related equations 1. let x,y,z be sides of triangle, then x+@xmath5, x+@xmath6, z+@xmath7. if x,y,z are in arithmetic progression then y-@xmath8:what is the value of a + c? sa:since we are given that a, b, and c form an arithmetic sequence, we can express b as the average of a and c. this gives us beyond the discourse of overall numbers, we perform in-depth ablation analyses to identify the roles of each component of the model. we observe that (i) the verifier lm and concept generated by the decomposer lm play a crucial role in generalizing out-of-distribution reasoning tasks like medqa, jeebench chemistry, etc.; (ii) finetuning the decomposer is crucial for better concept identification -finetuned llama-2 7b generates more effective conceptual requirements compared to even gpt-4; (iii) even while not using all the modular components of lm 2 , the prompt template of structured reasoning boosts the performance of the efficacy of explicitly generating intermediate reasoning steps over direct generation of the required answer was first demonstrated by @xcite . chain-of-thought prompting @xcite generalized the scratchpad learning of @xcite into an in-context learning regime using llms. chain-of-thought and its successors @xcite typically let the decomposition of a composite, multi-step reasoning problem remain implicit in the llm. @xcite demonstrated that instead, an explicit call to the llm to generate multiple smaller problems that are steps to answer the original query achieves more robust reasoning. similarly, @xcite proposed a promptingbased problem decomposition approach where the llm is asked to decompose a complex task using few-shot examples. however, this still burdens a single language model in handling both decomposition and solution. @xcite circumvented this challenge by distilling the decomposition abilities into a relatively smaller language model. their proposed method, daslam, utilizes two separate language models that coordinate with each other to solve complex reasoning problems. their findings suggest that finetuning the decomposer is more generalizable than finetuning the solver model. this has been further supported by @xcite recently. @xcite explored the distillation of decomposition abilities via offline reinforcement learning. @xcite proposed a programmatic retrieval augmentation framework, namely demonstrate-search-predict (dsp), for knowledge-intensive generation tasks. dsp relies on the coordination between a generative lm and a retrieval model through sophisticated programs. recent attempts have been made to incorporate dense verifiers (typically, a finetuned, bidirectional language model acting as a classifier) aiding a generative model towards robust, verifiable problem solving and text generation @xcite . different techniques for verification of lm-generated outputs have been proposed subsequently, such as self-verification @xcite , majority voting @xcite , etc. ## methodology our proposed method, lm 2 , is built upon the coordination of multiple lms to perform reasoning in a modular fashion. however, such coordination is not implicit in the pertaining stage of a model; instead, we seek to inculcate this ability via finetuning (parts of) the lm multiplex. to this end, lm 2 is built upon three functional components: a (preferably larger) solver model, a decomposer model, and a verifier model. for fine-grained control over the function of the different components of lm 2 , we make use of a structured, step-by-step input-output framework (see figure 1 ). the role of each of the modules in lm 2 is described as follows. ## experiments for all the experiments, lm 2 uses the openai text-davinci-003 model (hereafter mentioned as gpt-3.5) as the solver and llama-2 13b @xcite as the base models for the decomposer and the verifier. details including the hyperparameters, compute resources, evaluation strategy, etc. are provided in appendix a. training data curation. for the first stage of finetuning of the decomposer lm, we curated a dataset of 15, 396 question, concept, sub-question, subanswer tuples. the questions were taken from the train split of the math dataset @xcite . the questions were taken from the math dataset. for verifier lm finetuning, a dataset of 3, 674 question-answer-classification tuples was generated. details of the prompts used for each of these steps are provided in appendix b. baseline details. we compare lm 2 with five existing methods: chain-of-thought prompting (cot) @xcite , least-to-most prompting (l2m) @xcite , progressive hint prompting (php) @xcite , demonstrate-search-predict (dsp) @xcite , and daslam @xcite . the original setting of php requires an 8-shot prompting; however, since all other methods including lm 2 predict in the zero-shot setting, we use php in 1-shot for a fairer comparison. ablation study. we perform five types of ablation studies aimed at comprehensively understanding ## results we summarize the performance of lm 2 along with the baseline methods on the math and medqa datasets in can it improve on out-of-domain tasks? in both daslam and lm 2 , the solver model is kept frozen with the hope of retaining generalizability. however, the decomposer model in both methods (and the verifier in lm 2 ) are finetuned using mathematical reasoning problems. this raises the question of the generalizability of these finetuned components over problems other than mathematical reasoning. one of the most significant challenges with daslam is that it is not able to perform well on out-of-domain tasks like jeebench chemistry. we find that our method can surpass this limitation as can be seen in tables 1 (medqa) and 2 (jeebench chemistry). while daslam degrades the performance over cot on medqa, lm 2 achieves an absolute accuracy gain of 6.8 percentage points. how important is the verifier? next, we seek to investigate the relative importance of each component in our pipeline. we observe that the accuracy decreases substantially upon removing the verifier model (lm 2 \v in the middle third of we can see that there is a drop of 13.0% in chemistry versus 10.08% in physics and 3.4% in math subsets. the relative drop in accuracy with the ablation of the verifier is sharper with multi-answer, numeric, and integer answer questions. this makes sense given the computational reasoning requirement is higher in these problems and the verifier plays a crucial role in guiding the decomposer and the solver along the correct reasoning path. how important are the concepts? as can be seen from ## conclusion in this paper, we present lm 2 , a cooperative cohort of generative language models working together to solve complex reasoning problems. lm 2 utilizes a frozen solver model that is guided to solve reasoning problems by incrementally answering questions framed by a decomposer model and checked by the verifier model that is trained to coordinate with each other. we find that lm 2 proves its supremacy over existing methods over a variety of reasoning tasks, both in-domain and out-domain. we find that despite being trained using mathematical reasoning examples, our proposed structured response scheme along with the fine-grained verification strategy plays a crucial role in generalizing lm 2 to heavily out-of-distribution tasks like medical question answering and chemistry.