Meta-Reasoning: Semantics-Symbol Deconstruction for Large Language Models (2024)

Abstract: Neural-symbolic methods have demonstrated efficiency in enhancing the reasoning abilities of large language models (LLMs). However, existing methods mainly rely on syntactically mapping natural languages to complete formal languages like Python and SQL. Those methods require that reasoning tasks be convertible into programs, which cater to the computer execution mindset and deviate from human reasoning habits. To broaden symbolic methods’ applicability and adaptability in the real world, we propose Meta-Reasoning from a linguistic perspective. This method empowers LLMs to deconstruct reasoning-independent semantic information into generic symbolic representations, thereby efficiently capturing more generalized reasoning knowledge. We conduct extensive experiments on more than ten datasets encompassing conventional reasoning tasks like arithmetic, symbolic, and logical reasoning, and the more complex interactive reasoning tasks like theory-of-mind reasoning. Experimental results demonstrate that Meta-Reasoning significantly enhances in-context reasoning accuracy, learning efficiency, out-of-domain generalization, and output stability compared to the Chain-of-Thought technique.

## introduction symbols serve as the primitive carrier through which humans can comprehend, articulate, and conceptualize the intricacies of both nature and society @xcite . from a cross-linguistic perspective, ideographic symbolic languages like arabic numerals, mathematical symbols, and emojis can transcend barriers to natural semantic understanding. they serve as a universal representation across ethnically diverse human languages @xcite cheng et @xcite @xcite , facilitating communication and comprehension on a global scale. in a specific mono-linguistic communication scenario, symbols inherently possess multiple referential meanings shaped by social and cultural properties @xcite . consequently, a single symbol can encapsulate diverse semantic representations. conversely, various semantic representations can converge onto the same symbol, forming a manyto-one relationship when abstracting referential meanings. this transformation opens avenues for transforming natural language reasoning into more generalized patterns, enabling efficient solutions. current reasoning paradigms of large language models (llms), such as chain-of-thought (cot) prompting @xcite @xcite , rely on multiple in-context learning demonstrations to perform well. however, the number of demonstrations is limited by llms' input capacity and inference cost, rendering it impractical to cover the distribution of specific task features exhaustively. therefore, we advocate a paradigm shift from infinite semantics systems to finite symbolic systems so that llms can acquire more generic knowledge with enhanced data learning efficiency, as shown in figure 1 . motivated by the insight above, we introduce meta-reasoning, a novel reasoning paradigm aimed at deconstructing the semantics of entities and operations in questions into generic symbolic representations. meta-reasoning enables llms to learn generalized reasoning patterns across various semantics-wrapped scenarios, enhancing learning efficiency and reasoning accuracy. we apply meta-reasoning to in-context learning by designing demonstrations integrating semantic resolution with the cot technique. this empowers llms to deconstruct questions and effectively capture more generalized knowledge autonomously. to assess the efficacy of our method, we conduct experiments on over ten datasets, spanning both conventional reasoning scenarios, which involve arithmetic, symbolic, and logical reasoning tasks, and interactive reasoning scenarios, which involve theory-of-mind reasoning. we mainly compare our method with the cot method upon gpt-3 and chatgpt. experimental results show that meta-reasoning consistently outperforms the few-shot-cot method across all tasks, demonstrating significant performance improvements. in the conventional reasoning scenarios, meta-reasoning achieves an average performance gain of +20% across all datasets with fewer demonstrations. in more complex interactive reasoning scenarios, meta-reasoning surpasses cot across all levels of theory-of-mind reasoning with just a single demonstration. moreover, meta-reasoning demonstrates remarkable out-of-domain generalization and output stability, indicating its scalability and user-friendly nature as a reasoning paradigm. to our knowledge, we are the first to establish an equivalence mapping from semantics to symbols within natural language. this innovation facilitates in-context learning for llms, significantly enhancing their capacity for generalized reasoning. we expect to extend the reasoning ability boundary of llms based on this research. ## preliminary: why meta-reasoning? meta-reasoning is an idealized reduction-based reasoning paradigm defined in this work, whose goal is to reduce the infinite semantic concepts in the world's languages to a finite symbolic system, thus allowing machines to generalize to many semantically wrapped problems through the acquisition of universal laws. this paradigm is best suited for such a reasoning scenario: the final reasoning results are independent of the particular semantic representations and are only related to the underlying reasoning skeletons. the core of meta-reasoning lies in semantic-symbolic deconstruction, which we simplify as semantic resolution. this process conveys the semantics of the original problem via symbols with generalized meanings, without affecting the final results. however, why deploying semantic resolution in llms is a key issue, we must consider the advantages it brings to the reasoning process. we explore this issue from two perspectives: (i) the human reasoning speed when responding to different questions, and (ii) the machine reasoning accuracy when responding to different questions. we select multiarith @xcite and gsm8k @xcite , two arithmetic datasets, and rephrase 100 questions in each dataset according to the semantic resolution rules that will be introduced in section 3.1, thereby creating metaquestions. subsequently, we distribute the original and meta-questions to both human volunteers and llms to obtain corresponding results of metrics. we assess the response speed of three human volunteers by measuring the total time taken from receiving the question to providing the answer. 1 as shown in figure 2 , human response speed significantly improves when solving meta-questions, particularly evident in gsm8k. this acceleration is attributed to the removal of unimportant semantic information in meta-questions, which enables quicker recognition of the reasoning skeleton by humans. moreover, the more concentrated distribution of human reaction times suggests ## meta-reasoning paradigm we have observed notable performance improvements in llms when tackling questions after semantic resolution in the last section. in this section, we formally introduce the meta-reasoning paradigm employed in llms. section 3.1 defines the specific rules for semantic resolution. then, we put this process through in-context learning for llms to imitate, and section 3.2 formalizes the demonstration design form of in-context learning. ## discussion we conduct ablation studies to examine the role of semantic resolution in the reasoning process. moreover, we compare our method with existing work in language programming @xcite , highlighting meta-reasoning's broader applicability across diverse scenarios. these extended analyses are shown in appendix c. ## related work our work is related to the research lines of neuralsymbolic methods and chain-of-thought reasoning. please refer to appendix f for full details. neural-symbolic methods in llms. symbolic learning (chen et al., 2021) significantly improves llms' reasoning performance. prior works focus on converting natural languages into programming languages @xcite and accessing external interpreters for execution @xcite ; or using symbolic tasks for post-tuning @xcite , leading to performance improvements. however, these symbols are well-defined formal languages completely independent of natural languages. our work jumps out of this framework and further enhances the efficiency of the symbolic methods. ## conclusion we propose meta-reasoning, a semantic-symbol deconstruction paradigm for reasoning. through the semantic resolution of the original questions, we enable llms to grasp meta-forms and general solutions for specific types of reasoning tasks. this approach requires fewer demonstrations to expand the upper limit of their reasoning accuracy, out-ofdomain generalization, and output stability.