IDOL : Indicator-oriented Logic Pre-training for Logical Reasoning (2023)

Abstract: In the field of machine reading comprehension (MRC), existing systems have surpassed the average performance of human beings in many tasks like SQuAD. However, there is still a long way to go when it comes to logical reasoning. Although some methods for it have been put forward, they either are designed in a quite complicated way or rely too much on external structures. In this paper, we proposed IDOL (InDicator-Oriented Logic Pre-training), an easy-to-understand but highly effective further pre-training task which logically strengthens the pre-trained models with the help of 6 types of logical indicators and a logically rich dataset LoGic Pre-training (LGP). IDOL achieves state-of-the-art performance on ReClor and LogiQA, the two most representative benchmarks in logical reasoning MRC, and is proven to be capable of generalizing to different pre-trained models and other types of MRC benchmarks like RACE and SQuAD 2.0 while keeping competitive general language understanding ability through testing on tasks in GLUE. Besides, at the beginning of the era of large language models, we take several of them like ChatGPT into comparison and find that IDOL still shows its advantage.

## introduction with the development of pre-trained language models, a large number of tasks in the field of natural language understanding have been dealt with quite well. however, those tasks emphasize more on assessing basic abilities like word-pattern recognition of the models while caring less about advanced abilities like reasoning over texts @xcite . in recent years, an increasing number of challenging tasks have been brought forward gradually. at sentence-level reasoning, there is a great variety of benchmarks for natural language inference like qnli @xcite and mnli @xcite . although the construction processes are different, nearly all these datasets evaluate models with binary or three-way classification tasks which need reasoning based on two sentences. at passage-level reasoning, the most difficult benchmarks are generally recognized as the ones related to logical reasoning mrc which requires questionanswering systems to fully understand the whole passage, extract information related to the question and reason among different text spans to generate new conclusions in the logical aspect. in this area, the most representative benchmarks are some machine reading comprehension datasets like reclor @xcite and logiqa @xcite . considering that there are quite few optimization strategies for the pre-training stage and that it is difficult for other researchers to follow and extend the existing methods which are designed in rather complex ways, we propose an easy-to-understand but highly effective pre-training task named idol which helps to strengthen the pre-trained models in terms of logical reasoning. we apply it with our customized dataset lgp which is full of logical information. moreover, we experimented with various pre-trained models and plenty of different downstream tasks and proved that idol is competitive while keeping models and tasks agnostic. recently, chatgpt attracts a lot of attention all over the world due to its amazing performance in question answering. thus, we also arranged an experiment to let idol compete with a series of llms (large language models) including it. the contributions of this paper are summarized as follows: figure 1 : a diagram illustrating the three steps of our method: (a) construct the logically rich dataset lgp from wikipedia, (b) further pre-train models to improve logical reasoning ability, and (c) answer logical reasoning mrc questions with the help of logical indicators appeared both in context and choices. see section 4 for more details on our method. ## ablation study in this section, we conducted a series of ablation experiments about the multiple logical indicators we used in both fine-tuning and pre-training phases. we evaluate the models based on roberta with 4 different seeds and report the average value. ## conclusion and future work in this paper, we proposed an easy-to-understand further pre-training method idol which fully exploits the logical information provided by 6 types of logical indicators and is proven effective on different pre-trained language models while keeping them competitive on many other kinds of downstream tasks. particularly, idol achieves state-ofthe-art performance on logical reasoning machine reading comprehension tasks. with respect to future work, we plan to leverage the sentence-level or passage-level logical features in the meantime and integrate it with idol to generate a stronger multi-task further pre-training method for improving the logical reasoning ability of pre-trained language models. moreover, we decide to redesign the idol task and find out whether logical indicators also play an important role in those generative pre-trained models as well. furthermore, we will explore the way of combining idol with prompting to find a better method to elicit the reasoning abilities of llms.