Exploring Self-supervised Logic-enhanced Training for Large Language Models (2024)

Abstract: Traditional attempts to enhance the logical reasoning abilities of language models often rely on supervised fine-tuning, limiting their generalization to new tasks or domains. Large Language Models (LLMs), with their capacity to condense vast knowledge, can effectively tackle many tasks. Yet, our experiments reveal a gap in their performance on logical reasoning benchmarks when compared to state-of-the-art fine-tuning based models. To bridge this gap, we present LogicLLM, a first-of-its-kind, fully self-supervised framework for integrating logical reasoning capabilities into LLMs, and activating them via in-context learning. We apply this to two LLM series, FLAN-T5 and LLaMA, with parameter sizes from 3 billion to 33 billion. LogicLLM demonstrates its effectiveness through successful improvements on two logical reasoning benchmarks (ReClor and LogiQA-v2). Additionally, LogicLLM based on FLAN-T5-11B attains comparable results to ChatGPT, and evaluations with LLaMA-based models on three language understanding benchmarks (RACE, MMLU and Big-Bench-Hard) confirm that the improvements come without compromising the model’s general language understanding capabilities.

## introduction logical reasoning serves as a bedrock for negotiation, debate and writing, underpinning our ability to engage with complex cognitive tasks @xcite . an example of logic reasoning in natural language is shown in figure 1 . as the complexity of relations and expressions presented in this task defy straightforward conversion into symbolic or formal languages, perfecting logical reasoning within language models has proven to be a significant challenge @xcite . @xcite . the relations between different constituents, e.g., agriculture and development of andean society, include various predicates, and it is hard to be converted into logical form through either first-order logic or formal language. past attempts to incorporate logical reasoning into language models primarily focused on integrating knowledge about logic. for instance, @xcite employed graph neural networks to capture relational semantics, while @xcite used data augmentation to implement firstorder logic. these techniques, however, are constrained by their need for extensive annotated training data, which hinders the model's ability to generalize across different tasks due to disparities in data distribution and optimization objectives. conversely, recent breakthroughs in large language models (llms) like palm @xcite , llama @xcite , chat-gpt 2 , gpt-4 (openai, 2023), and bard 3 offer a promising alternative. these llms effectively encapsulate a vast array of knowledge and tackle diverse tasks with minimal specialization, guided by human instruction. despite their potential, our experiments on logical reasoning benchmarks revealed deficiencies in their logical reasoning capabilities as shown later in our experiments. contemporary efforts to fortify llms' specific capabilities fall broadly into two categories. the first employs external tools or apis @xcite @xcite @xcite , aiding llms in argument parsing and semantic understanding. yet, these tools' utility for logical reasoning remains limited due to the absence of a symbolic language for problem descriptions. the second category, instruction tuning, relies on data augmentation or enriched human feedback but struggles due to the scarcity of task-specific data and high annotation costs @xcite . in this work, we pivot away from these traditional methods and introduce logicllm, which performs selfsupervised logic-enhanced meta-training for llms. it tackles two primary challenges: 1) synthesising logic-consistent data from raw texts ensuring fully self-supervised training, and 2) effectively incorporating logic prior into llms while preventing learning problems, such as memorization, forgetting and generalization. to tackle the first challenge, logicllm emphasizes the necessity of understanding and exploiting fuzzy logical consistency. as mentioned previously, strict formal logic is often absent in natural language, we instead treat the relational consistency between different perspectives of relational expressions as an approximation to fuzzy logic consistency foot_2 . in fact, ensuring logical consistency in a discourse is a key requirement for text coherence and effective information conveyance @xcite . we devise a method that inspects the implicit intra-sentence relation of entity pairs at the discourse level to extract logically consistent examples from wikipedia articles (figure 2 ). specifically, we posit that direct and indirect relations of an anchor entity pair should be logically consistent, as they are derived from the "same" context. for the second challenge, logicllm adopts an auto-regressive objective optimizing on the logically consistent relation instances directly to make it seamlessly adapt to its pretraining objective. it tasks the model with generating the alternative per-spective (indirect or direct) given a direct or indirect description of the anchor entity pair. we further employ counterfactual data augmentation through entity replacement to enforce relation-centric reasoning, which not only avoids the model's tendency to merely recall results from memory but also ensures the preservation of the logic-enhanced aspect of the learning process. logicllm is task-agnostic and does not require any annotations, making it adaptable to various logical reasoning tasks. we have conducted experiments across two distinct llm series, flan-t5 @xcite and llama @xcite , encompassing a variety of parameter sizes. these experiments are designed to investigate two main questions: (1) can the logical reasoning capabilities be exclusively improved through self-supervised meta-training for llms, thereby circumventing the need for task-specific supervised fine-tuning? (2) how does the logic-enhanced meta training affect the llm's language understanding capabilities, i.e., does it suffer from forgetting or generalization issues? in response to the first question, our findings suggest that llms trained with the logicllm objective demonstrate superior performance on logical reasoning benchmarks, eliminating the need for further fine-tuning. our logicllm based on flan-t5-11b attain comparable results to chatgpt on two logic reasoning benchmarks, reclor @xcite and logiqa-v2 @xcite , highlighting the feasibility of enhancing logical reasoning abilities through self-supervised training alone. regarding the second question, our evaluations with llama-based models on three general language understanding benchmarks -race @xcite , mmlu @xcite and big-bench-hard (bbh) @xcite , confirm that the enhanced logical reasoning capabilities do not compromise the model's overall language understanding on mmlu and bbh. in fact, the learned logic ability appears to boost the model's performance in race. ## logicllm figure 2 shows the framework of logicllm. it involves three main steps: 1) logic-consistent data construction (section 3.1), which synthesises the logic-consistent data using relation discrimination between entity pairs; 2) counterfactual data augmentation (section 3.2), which augments the logicconsistent training data by entity sampling and replacement; 3) llm training (section 3.3), which performs continual training of llms using the training data generated by the previous two steps. ## experiment we integrate our pre-training approach into two prominent llms: llama ( better results are annotated in bold. † refers that the corresponding model is trained through qlora @xcite . and flan-t5 @xcite . these models boast parameter sizes ranging from 3 billion to 30 billion. to thoroughly evaluate the capability of llms from various angles, we have carefully selected five datasets representing three distinct categories. reclor @xcite and logiqa-v2 @xcite are two logical reasoning benchmarks sourced respectively from standardized graduate admission examinations and logical examination papers intended for reading comprehension. race @xcite is a reading comprehension task that assesses general reasoning abilities. mmlu @xcite is used for measuring the learned knowledge and massive multitask language understanding, and big-bench-hard (bbh) @xcite is a collection of multiple challenging tasks where llms fall behind human being. by employing mmlu and bbh, we aim to verify whether the logic-oriented metatraining negatively impacts the models' ability to generalize across a wide range of tasks. due to space limitation, more implementation details can be found in appendix a. ## conclusion in this paper, we have explored the feasibility and effectiveness of enhancing logical reasoning of llms via purely self-supervised training. we evaluate the performance based on two llm series, i.e., flan-t5 and llama. the experimental results on two logical reasoning benchmarks, logiqa-v2 and reclor, demonstrate the effectiveness of our method. and the performance on race, mmlu and big-bench-hard have also verified that the framework do not hurt the generalization of llms. finally, we have analyzed the factors relevant to logic during training, and the compability with supervised instruction tuning. we hope the analysis could bring new insights to future research. shuang zeng, yuting wu, and baobao chang. 2021. sire: separate intra-and inter-sentential reasoning for document-level relation extraction. in findings of acl, pages 524-534. acl. wanjun zhong, siyuan wang, duyu tang, zenan xu, daya guo, jiahai wang, jian yin, ming zhou, and nan duan. 2021. ar-lsat: investigating analytical reasoning of text. corr, abs/2104.06598. reclor logiqa-v2 model / dataset dev test dev test acc. acc. acc. acc. roberta-l. 62.6 55.6 59.8 57.0 merit (roberta-l) 69.4 61.6 62.6 59.3 merit (deberta-xxl) 80.6 78.1 --llama-7b 28.8 28.3 24.4 23.7 llama-13b 31.6 34.4 31.6 31.1 llama-33b 45.2 50.3 41.2 41.6 gpt-3.5-turbo 56.6 61.2 54.5 52.7 w/ cot 58.8 57.7 -53.1