Empower Nested B oolean Logic via Self-Supervised Curriculum Learning (2023)

Abstract: Beyond the great cognitive powers showcased by language models, it is crucial to scrutinize whether their reasoning capabilities stem from strong generalization or merely exposure to relevant data. As opposed to constructing increasingly complex logic, this paper probes into the boolean logic, the root capability of a logical reasoner. We find that any pre-trained language models even including large language models only behave like a random selector in the face of multi-nested boolean logic, a task that humans can handle with ease. To empower language models with this fundamental capability, this paper proposes a new self-supervised learning method Curriculum Logical Reasoning (Clr), where we augment the training data with nested boolean logic chain step-by-step, and program the training from simpler logical patterns gradually to harder ones. This new training paradigm allows language models to effectively generalize to much harder and longer-hop logic, which can hardly be learned through naive training. Furthermore, we show that boolean logic is a great foundation for improving the subsequent general logical tasks.

## introduction artificial intelligence has made a giant leap from perception to cognition, with powerful pre-trained language models (plms) @xcite @xcite @xcite @xcite , large language models (llms) @xcite @xcite demonstrating human-level comprehension and reasoning powers on a series of challenging tasks like commonsense reasoning @xcite , opendomain question-answering @xcite , an aeroplane flies along the four sides of a square at the speeds of 200, 400, 600 and 800 km/hr. find the average speed of the plane around the field? valjean allows javert to arrest him, but quickly escapes. it is hinted that valjean escaped with the help of a file hidden in a coin, an item he is later proved to possess. valjean returns to his house to pack his clothes and hides behind the door when javert comes looking for him. ## introducing nested boolean logic this section presents our method to introduce multinested boolean logic to existing data. we first present the notations. let x denote the input text, with its ground truth y, and p θ denote the classifier (e.g. a language model) with parameters ## benchmark to benchmark the multi-nested boolean logic, we construct a new dataset in this paper and following experiments are based on this. as apart from other datasets, it is composed of a series of subsets, representing different levels of logical complexity. we will release this benchmark for future research. ## empower nested boolean logic we present a new self-supervised learning manner. ## empirical results as opposed to the prior section, where we evaluate the model on each level of logical difficulty, in this section, we evaluate each model on boolkill u 1∼4 , u 5∼8 , and ũ5∼8 as an alternative. these sets cover the range from @xmath0. u 1∼4 is a simpler one and u 5∼8 and ũ5∼8 appear to be highly challenging, since we previously show that state-ofthe-art plms are almost powerless for the nested boolean logic beyond u 3 . we experiment on deberta-v3-base and deberta-v3-large. each model is trained for 3,000 steps with a batch size of 16 and learning rate of 2e-5 / 1e-5 for the base / large one. to verify clr, we report two experiments. in the first experiment, we compare different training settings and evaluate the models on boolkill. in the second, we leverage the boolean logic in boolkill to benefit other general logical tasks. ## related work the study of boolean operations is the fundamental requirement for a series of challenging tasks, e.g. arithmetical reasoning @xcite , commonsense reasoning @xcite , reading comprehension @xcite , dialogue comprehension @xcite . we concentrate on the multi-nested boolean logic by augmenting the text with boolean statements. previous studies analyze more general logical reasoning, e.g. rica @xcite , robustlr @xcite , fairr @xcite , by logical paraphrase or contrast sets. self-supervised learning methods typically generate learnable inputs on top of unlabeled corpora, e.g. by masking @xcite , insertion @xcite , sentence reordering @xcite , contrastive learning @xcite , while our method is by imposing a series of sentences to the suffix, actually generating learnable logic. we introduce curriculum learning @xcite , which allows the model to learn step by step to further facilitate self-supervised learning. curriculum learning is under-discussed in context of language processing @xcite . while deep neural networks are capable of handling very complex tasks, in reality they lean to exploit spurious cues @xcite @xcite , and can be powerless to very simple perturbations as a consequence. our work discloses that language models are poorly skilled at basic boolean logic. in parallel, studies show that language models can be easily fooled by some naive patterns within the text, e.g. lexical overlap @xcite , entity boundary @xcite , word order @xcite . we also release a challenging benchmark to evaluate boolean logical reasoning. there are a series of work focusing on constructing challenging logic, e.g. reclor @xcite , hotpotqa @xcite , anli @xcite . ## conclusion this paper provides a quantified analysis on the multi-nested boolean logic. we flag the deficiency in the state-of-the-art language models in terms of such basic capability, which will inevitably cause pitfalls in dealing with more complex reasoning tasks. for this, we propose curriculum logical reasoning, a new self-supervised learning method to empower language models with foundational logical capability. we also show that our idea can act as a cornerstone learning method for general logical reasoning.