LLM 2: Let Large Language Models Harness System 2 Reasoning (2025)

Abstract: Large language models (LLMs) have exhibited impressive capabilities across a myriad of tasks, yet they occasionally yield undesirable outputs. We posit that these limitations are rooted in the foundational autoregressive architecture of LLMs, which inherently lacks mechanisms for differentiating between desirable and undesirable results. Drawing inspiration from the dual-process theory of human cognition, we introduce LLM2, a novel framework that combines an LLM (System 1) with a process-based verifier (System 2). Within LLM2, the LLM is responsible for generating plausible candidates, while the verifier provides timely process-based feedback to distinguish desirable and undesirable outputs. The verifier is trained with a pairwise comparison loss on synthetic process-supervision data generated through our token quality exploration strategy. Empirical results on mathematical reasoning benchmarks substantiate the efficacy of LLM2, exemplified by an accuracy enhancement from 50.3 to 57.8 (+7.5) for Llama3-1B on GSM8K. Furthermore, when combined with self-consistency, LLM2 achieves additional improvements, boosting major@20 accuracy from 56.2 to 70.2 (+14.0).

## introduction large language models @xcite @xcite have exhibited remarkable abilities across various tasks that span general assistance @xcite , coding @xcite , vision @xcite and more. however, they still occasionally produce undesirable outputs in many scenarios, e.g., reasoning and planning @xcite , factual consistency @xcite , and human value alignment @xcite , etc. we hypothesize these deficiencies stem from the fundamental design of llms. specifically, the nexttoken prediction objective optimizes llms to maximize the probability of human-generated strings empirically, with no explicit mechanism to distinguish between desirable and undesirable outputs. during the inference stage, llms autoregressively generate outputs token-by-token in a single pass, with no awareness of their errors. this procedure is reminiscent of system 1 in the dual-process theory, which postulates that thinking and reasoning are underpinned by two distinct cognitive systems @xcite @xcite . system 1 operates automatically and subconsciously, guided by instinct and experience. in contrast, system 2, thought to be unique to humans, is more controlled and rational, enabling deliberate thinking for difficult tasks, especially when system 1 may make mistakes @xcite . in this paper, we introduce llm2, which aims to empower llms with system 2 reasoning. as shown in figure 1 , llm2 integrates an llm (system 1) with a process-based verifier (system 2). during inference, the llm generates multiple candidates at each time step, and the verifier provides timely feedback on each candidate. by efficiently exploring the generation space based on the verifier's feedback, llm2 ultimately identifies more effective outputs. during the training stage, the process-based verifier is optimized with a pairwise comparison loss to distinguish between desirable and undesirable tokens. to obtain informative token pairs data for process-supervision, we propose a token quality exploration strategy that generates synthetic data based on the potential impact of tokens on the generated text. we evaluate llm2 on two representative mathematical reasoning datasets: gsm8k @xcite and math @xcite . with q: ... r: ... 3 + 5 = q: ... r: he writes 6 pages every week (a) synthetic process-supervision (b) process-based verifier (c) dual-process llm so he ... ... to make ... for each ... ... and he ... quarter 0.6 0.3 acc. •β month month llm verifier verifier llm q: ... r: he writes 6 pages every 8 1 . -s θ (@xmath0,x t -) pairwise comparison loss selected as x t -s θ (@xmath1,x t + ) 8 1 . -+ figure 1: an illustration of the training and inference stages of llm2. the training stage includes (a) synthetic process-supervision data collection and (b) the optimization of a process-based verifier. the inference stage involves (c) a dual-process llm for generation. the integration of system 2 reasoning, llm2 achieves substantial performance improvement across llama3 models ranging from 1b to 8b parameters. for instance, compared to the vanilla llama3-1b, llm2 significantly improves accuracy from 50.3 to 57.8 (+7.5) on gsm8k, and from 24.2 to 28.8 (+4.6) on math. combining llm2 with self-consistency further boosts the model's performance, enhancing major@20 accuracy from 56.2 to 70.2 (+14.0) on gsm8k. further analysis of the utilization of self-generated answers underscores the effectiveness and promising potential of synthetic process-supervision data. 2 method 2.1 dual-process llm we aim to build a dual-process llm (i.e., llm2), where an llm serves as system 1 for giving plausible proposals and a verifier functions as system 2 for deliberate thinking to refine and prevent mistakes introduced by system 1. specifically, we formalize this procedure as: therefore, the logits of π * at time step t, denoted as z * t , are computed as: the probability distribution π * (x t |@xmath2. this formulation allows π * to integrates seamlessly with various decoding strategies, depending on the use case. ## conclusion in this work, we introduce llm2, a framework that augments llms with a system 2-like reasoning process. by coupling an llm with a process-based verifier, llm2 proficiently differentiates between optimal and suboptimal outputs. the framework is empowered by synthetic processsupervision data generated via a novel token quality exploration strategy, which is instrumental in training the verifier. our empirical results and analyses confirm the efficacy of llm2 in enhancing llm performance.