APOLLO : A Simple Approach for Adaptive Pretraining of Language Models for Logical Reasoning (2023)

Abstract: Logical reasoning over text is an important ability that requires understanding the semantics of the text and reasoning through them to arrive at correct inferences. Prior works on pretraining language models to improve the logical reasoning ability require complex processing of training data (e.g., aligning symbolic knowledge to text), yielding task-specific data augmentation that is not easy to adapt to any general text corpus. In this work, we propose APOLLO, a simple adaptive pretraining approach to improve the logical reasoning skills of language models. We select a subset of Wikipedia for adaptive pretraining using a set of logical inference keywords as filter words. Further, we propose two self-supervised loss functions for training. First, we modify the masked language modeling loss only to mask specific parts-of-speech words that likely require higher-order reasoning to predict them. Second, we propose a sentence-level classification loss that teaches the model to distinguish between entailment and contradiction types of sentences. The proposed pretraining paradigm is both simple and independent of task formats. We demonstrate the effectiveness of APOLLO by comparing it with prior baselines on two logical reasoning datasets. APOLLO performs comparably on ReClor and outperforms baselines on LogiQA.

## introduction logical reasoning is an important ability of humans that helps us in making rational decisions based on known information. it is an important ability for text understanding across various downstream tasks, e.g., in open-domain question answering @xcite , machine figure 1 : motivation of selective masking. in random masking @xcite , a word is masked at random. predicting these words often require more of language understanding than higher-order reasoning (e.g., predicting "would" at the 2 nd [mask] place). in selective masking, a word is masked if its pos tag is from a specific set. these candidate words are marked in the blue box in the input sentence. filling these words requires more reasoning (e.g., to predict "more" at the 2 nd [mask] place instead of "less", which is also grammatically valid, the model needs a better understanding of the semantics of the sentence). there have been some recent works on improving the logical reasoning abilities of plms @xcite @xcite . these works typically generate a dataset containing symbolic structures such as logical graphs from text, logical contrast sets, etc., and then train the lm using custom loss objectives to learn logical reasoning abilities. while the performance improvements achieved by these methods are encouraging, the proposed solutions generally require complex data processing to generate the additional structural information (graphs, contrast data, etc.) required for training the model. for example, @xcite constructs synthetic context-answer pairs using the entity-level graph from wikipedia for training the model. further, the loss functions proposed in these works are very specifically designed in accordance with their respective data augmentation technique and widely differs from the typical masked language modeling loss used for lm pretraining @xcite . additionally, some of these works usually require task-specific design choices, which are not necessarily learning generalizable logical reasoning ability that is reusable across different task formats. for example, @xcite parses symbolic logical structures from the training data of a specific dataset, which might not generalize to a new dataset or task. overall, it is unclear if these highly specific inductive biases are indeed essential for improving the logical reasoning abilities in language models, or if a simpler approach is possible. on the other hand, prior works @xcite have shown that continual domainadaptive pretraining of plms leads to performance gains on downstream tasks. inspired by this, we propose apollo, a continual pretraining-based approach to inject logical reasoning abilities in language models that requires minimal data processing and loss function modifications. firstly, we present a simple way of selecting sentences for training a model that is more likely to involve logical implications. we achieve this by defining a set of logical inference keywords and selecting a subset of sentences from a large text corpus, each containing at least one of these keywords. we hypothesize that plms can learn logical reasoning capabilities more easily using such sentences since the premise/conclusions are explicitly stated. we note that in contrast to previous works @xcite , our method can select sentences from any general text corpus, eliminating the need for any domain-specific corpus. secondly, we modify the masked language modeling (mlm) loss @xcite to selectively mask specific words in the sentence, based on their parts-of-speech tags. prior works @xcite have shown the benefit of selective masking of words on task-guided fine-tuning. we hypothesize that masking words with parts-of-speech (pos) tags that are related to higher-order reasoning (such as adverbs, conjunctions, etc.) present more challenging masked positions for the plm to predict. for instance, in figure 1 , we observe that the words marked in blue boxes are more related to reasoning compared to the non-highlighted words that mainly involve knowledge about specific nouns or english grammar. lastly, we design a sentence-level classification loss to predict if the reasoning in the sentence describes an entailment in the reasoning process or a contradiction. this enables the model to better understand the differences between positive and negative implications in a sentence, thus improving logical reasoning. to test apollo, we evaluate it on two downstream logical reasoning tasks: reclor @xcite and logiqa @xcite , and compare it with other baselines. we achieve state-ofthe-art performance on logiqa and comparable performance on reclor. we demonstrate that our method generalizes across different model types. further, we show that using our proposed loss functions does not induce any catastrophic forgetting @xcite of the original language modeling skills. this demonstrates that our simple, continual pretraining approach is generalizable to different datasets and enables the plm to acquire strong logical reasoning abilities. overall, compared to prior works, our proposed pretraining paradigm for apollo 1) uses sentences from text corpus for training instead of complex data structures such as entity graphs, etc. 2) uses simple learning objectives that are closer to language modeling compared to the contrastive loss. 3) is agnostic to both task format and downstream datasets. 4) achieves state-of-the-art performance on logiqa. ## method in this section, we describe the details of our proposed approach. in apollo, we use a keywordbased selection strategy to collect a dataset of reasoning-related sentences called implication ( §2.1) and then continue training a pretrained model checkpoint jointly using two loss functions ( §2.2). this model is then fine-tuned on the training dataset of each task separately for evaluation. a detailed overview of the pipeline is shown in figure 2 . ## experimental setup in this section, we describe the details of the datasets on which we evaluate apollo, the baselines we compare it with, and some implementation details of our training procedure. ## conclusion in this paper, we proposed apollo, an adaptive pre-trained language model with logical reasoning abilities. we use a subset of wikipedia sentences for continued pretraining of the model using two self-supervised loss functions. the choice of the training dataset and loss functions are guided by the goal to include more reasoning-related sentences and training signals, respectively. through experiments on two logical reasoning datasets and ablation studies, we demonstrate the effectiveness of our proposed approach. overall, we show that apollo is a generalized solution to improving logical reasoning in language models. a key advantage of apollo is that the pretraining steps are independent of the dataset used to train the model and the downstream task format. this opens the scope to use a larger text corpus for training such as c4 @xcite . additionally, expanding on the keywords beyond positive and negative implications (for example, conditionals such as "if-then", "either-or", etc.) can also benefit the training pipeline.