Bi-directional C ognitive T hinking Network for Machine Reading Comprehension (2020)

Abstract: We propose a novel Bi-directional Cognitive Knowledge Framework (BCKF) for reading comprehension from the perspective of complementary learning systems theory. It aims to simulate two ways of thinking in the brain to answer questions, including reverse thinking and inertial thinking. To validate the effectiveness of our framework, we design a corresponding Bi-directional Cognitive Thinking Network (BCTN) to encode the passage and generate a question (answer) given an answer (question) and decouple the bi-directional knowledge. The model has the ability to reverse reasoning questions which can assist inertial thinking to generate more accurate answers. Competitive improvement is observed in DuReader dataset, confirming our hypothesis that bi-directional knowledge helps the QA task. The novel framework shows an interesting perspective on machine reading comprehension and cognitive science.

## introduction machine reading comprehension (mrc) has made significant strides with an array of neural models rapidly approaching human parity on some benchmarks such as squad @xcite . however, existing methods are still in their infancy at the level of cognitive intelligence. recently, brain science and psychology provide an important basis for the development of brain-like computing and the simulation of human perception, thinking, understanding, and reasoning abilities @xcite . thinking is the generalization and indirect reflection of the human brain on the nature, interrelationships and internal regularities of objective things @xcite . two types of thinking are complementary in psychology: inertial thinking -from a previous to a subsequent stimulus -and reverse thinkingfrom a subsequent to a previous stimulus @xcite . inertial thinking @xcite is a conventional way of thinking, which thinks and solves problems from the previous ideas. reverse @xcite is a creative way of thinking, opposite to the inertial thinking. specifically, in the mrc task, the two types of thinking can be regarded as a process which reasons from questions (answers) to answers (questions). for example, as shown in fig. 1 , we can get the answer easily by locating the entities pregnant wowen and loquat. contrarily, the generative question, which can be reasoned by reading the answer and passage, describes two aspects, including can pregnant women eat loquat and what is the benefit to eat loquat for pregnant women. we hope that this ability of reverse reasoning can improve performance on reading comprehension tasks. previous methods @xcite @xcite ) only consider a obverse logical relationship, which is based on the given question and the passage. they ignore the reverse relationship between the given passage and the answer. although the work @xcite proposes a joint model that both asks and answers questions, it couples all the knowledge rather than decoupling modules, which is consistent with the concept of psychology. similarly, we hypothesize that the ability of reverse reasoning can help models achieve better qa performance. this example 1 passage: loquat is a kind of southern fruit with a unique flavor and is loved by people. loquat has a unique flavor and is rich in nutrients. pregnant women can eat it. it contains protein, fat, fruit acid, fructose and calcium, phosphorus, iron, sodium, potassium and other minerals. pregnant women eating loquat can increase appetite, relieve heat and relieve thirst. loquat can stimulate the secretion of digestive glands, and has a good effect on increasing appetite, helping digestion and absorption, and quenching thirst and relieving heat ... q: loquat, can pregnant women eat it ? a: pregnant women can eat loquat and pregnant women eating loquat can increase appetite, relieve heat and relieve thirst. is motivated partly by observations made in psychology that devising questions while reading can help students improve in reader-based processing of text @xcite . therefore, insights into solutions to the problem can be gained from human cognitive processes. complementary learning systems theory (clst) @xcite @xcite suggests that the human brain contains complementary learning systems that support the simultaneous use of many sources of information as we seek to understand an experienced situation. one of the systems acquires an integrated system of knowledge gradually through interleaved learning, including our knowledge of the meanings of words, the properties of frequentlyencountered objects, and the characteristics of familiar situations. it is just like inertial thinking that learns relationships between different things in the real world for a long time. the other system is a fast learning system similar to reverse thinking, which is targeted to focus on stimulating and enhancing infrequently-utilized circuit areas in the brain from another unusual perspective. in this paper, we propose the bi-directional cognitive knowledge framework (bckf). and the corresponding bi-directional cognitive thinking network (bctn) is designed to validate the effectiveness of the reverse thinking, as shown in fig. 2 , which will be introduced in detail in section 4. the contributions can be summarized as follows: ## bi-directional cognitive knowlwdge framework motivated by complementary learning systems theory (clst) @xcite @xcite finally, the understanding system guides the behavior of the model and its understanding of the language by combining inertial thinking and reverse thinking. ## model following the overview in fig. 2 , the proposed model that derives from our bi-directional cognitive knowledge framework consists of the following modules, and the training of the model contains two stages. the backward encoder models interactions between the answer and passage during the first stage (backward @xmath0, called reverse thinking training ( §4.2). the forward encoder, similar to the backward encoder with different parameters and inputs, retrains with the given passage and question during the second stage (forward @xmath1, called retraining with inertial thinking ( §4.3). the medial temporal lobes (mtl) system contains gate-reverse thinker, no-gate-inertial thinker and fusion layer. the gate-reverse thinker learns the reverse connections of the neurons from the negative side and determines the stimulus intensity of reverse thinking in memory. the no-gate-inertial thinker builds the obverse relationship. and the fusion layer combines the bi-directional knowledge to prepare for decoding. the soft decoder outputs an answer (question) sentence with soft-gated pointer-generator copying @xcite to synthesize the distribution on vocabulary and the distribution of tokens from the source input into a single output distribution. ## results in this section, we evaluate our model on the benchmark dataset: dureader dataset @xcite . pre-trained language model roberta (rb) and some state-of-art models are used as baselines to test the performance of the proposed bctn. we use the large pre-trained language model (-large) in the main experiments for getting a strong result. however, to increase training speed, the base model (-base) are utilized in our ablation study and other experiments. the experimental results and analyses validate the effectiveness of our model. ## conclusion in this paper, we present the bi-directional cognitive thinking network (bctn) which corresponding to the bi-directional cognitive knowledge framework (bckf) from the perspective of psychology. the bctn answers the question with bi-directional knowledge by simulating the inertial thinking and reverse thinking. and we decouple these two parts of knowledge rather than couple them with the same module. to determine the stimulus intensity of reverse thinking in memory, we consider the decoded tokens to calculate the score based on the gate mechanism. we show that the proposed bctn is very effective, it has competitiveness with the previous methods in literature on dureader with a single model. our future work will consider to use different datasets and design various models to simulate the behavior of our brain to capture human-level language understanding and intelligence. finally, we believe that our framework can generalize to other generative tasks, such as summarization and image caption.