A bstract M eaning R epresentation-Based Logic-Driven Data Augmentation for Logical Reasoning (2024)

Abstract: Combining large language models with logical reasoning enhances their capacity to address problems in a robust and reliable manner. Nevertheless, the intricate nature of logical reasoning poses challenges when gathering reliable data from the web to build comprehensive training datasets, subsequently affecting performance on downstream tasks. To address this, we introduce a novel logic-driven data augmentation approach, AMR-LDA. AMR-LDA converts the original text into an Abstract Meaning Representation (AMR) graph, a structured semantic representation that encapsulates the logical structure of the sentence, upon which operations are performed to generate logically modified AMR graphs. The modified AMR graphs are subsequently converted back into text to create augmented data. Notably, our methodology is architecture-agnostic and enhances both generative large language models, such as GPT-3.5 and GPT-4, through prompt augmentation, and discriminative large language models through contrastive learning with logic-driven data augmentation. Empirical evidence underscores the efficacy of our proposed method with improvement in performance across seven downstream tasks, such as reading comprehension requiring logical reasoning, textual entailment, and natural language inference. Furthermore, our method leads on the ReClor leaderboard at https://eval.ai/web/challenges/challenge-page/503/leaderboard/1347. The source code and data are publicly available at https://github.com/Strong-AI-Lab/Logical-Equivalence-driven-AMR-Data-Augmentation-for-Representation-Learning.

## introduction enabling pre-trained large language models (llms) to reliably perform logical reasoning is an important step towards strong artificial intelligence @xcite . however, data annotation for logical reasoning tasks is a difficult, time-consuming and costly process that has led to the scarcity of 1 https://eval.ai/web/challenges/ challenge-page/503/leaderboard/1347 2 amr-lda github repository large-scale logical reasoning datasets derived from natural language on the web. therefore, llms are usually trained on generic corpora or smaller logical reasoning datasets that lead to poor generalisation @xcite . automatic augmentation of logical reasoning data has the potential to enhance the generalisation and performance of llms on logical reasoning tasks. to address this challenge, we propose a logicdriven data augmentation method based on abstract meaning representation (amr). amr is a structural representation of the semantics and logical structure of text via a rooted directed acyclic graph (dag) @xcite . figure 1 shows an example of an amr graph. the amr graph can be easily modified by changing nodes or arguments to create logically equivalent or nonequivalent graphs. by taking advantage of the ease of logical manipulation of amr graph and of end-to-end conversion between natural language and amr graphs, our proposed data augmentation is not task-specific or template-dependent, and can generate logically equivalent and nonequivalent sentences that are diverse in their use of language. in order to improve the performance of llms on downstream tasks requiring logical reasoning, we investigate two different applications of the proposed logic-driven data augmentation for two different types of language models. in this paper, we describe models such as roberta @xcite and deberta @xcite as discriminative large language models, and models like gpt-3.5 (openai, 2023a) as generative llms. we improve the reasoning ability of discriminative large language models by applying contrastive learning to identify logically equivalent and nonequivalent sentence pairs generated using the proposed data augmentation before fine-tuning the model further on downstream tasks. in order to improve the performance of generative llms on logical reasoning 1 5914 ## related work logical reasoning is rigorous thinking to derive a conclusion based on a given premise @xcite . existing reasoning datasets reasoning can be categorised into two levels: sentence level, including tasks like natural language inference that assess if one sentence logically follows from another (e.g., mnli @xcite , rte @xcite , mrpc @xcite , qnli @xcite , qqp @xcite ); passage level, which requires logical deduction from given contexts, questions, and multiple choices (e.g., pararule @xcite , pararule-plus @xcite ) and reading comprehension tasks (e.g., reclor @xcite , logiqa @xcite ). we introduce an abstract meaning representation-based methodology for logic-driven data augmentation aimed at enhancing models' logical reasoning capabilities across these tasks. there are three primary methods for enhancing the capabilities of pre-trained language models in logical reasoning and general natural language understanding: 1) data augmentation with fine-tuning, exemplified by amr-da @xcite , which employs abstract meaning representation for paraphrasing, and lreasoner @xcite , which uses templates and syntax parsing for constructing logically equivalent sentences; 2) continual pre-training, with methods like merit @xcite integrates a meta-path strategy for discerning logical text structures and a counterfactual data augmentation strategy to preclude pre-training shortcuts. idol @xcite utilises six logical indicators @xcite to build a logic pre-training dataset from wikipedia, enhancing the logical reasoning capabilities of pre-trained models. 3) prompting, notably chain-of-thought prompting @xcite , to improve multi-step logical reasoning performance. our amr-lda surpasses lreasoner-lda by incorporating a broader range of logical equivalence laws, enabling the automatic construction of more precise logically equivalent sentences. our contrastive learning method enhance the performance of pre-trained models, including merit and idol, on logical reasoning tasks. additionally, our amr-based logic-driven prompt augmentation can improve large language models' logical reasoning capabilities, contrasting with the detrimental effects of cot prompting and amr-da. 2 5915 3 method ## conclusion the sparsity of web data related to logical reasoning constrains the advancement of large language models in their performance on logical reasoning tasks. existing methods for constructing logically equivalent sentences had been restricted to templates and specific datasets. our amr-lda considers more logical equivalence laws than existing methods do, and it does not reply on any ad-hoc templates. we applied amr-lda to fine-tuning discriminative llms and prompt augmentation of generative llms (gpt-3.5 and gpt-4), yielding better results than baseline methods on logical reasoning tasks.