F ACT - DRIVEN L OGIC R EASONING
Anonymous authors
Paper under double-blind review
A BSTRACT
Recent years have witnessed an increasing interest in training machines with reasoning ability, which deeply relies on accurate, clearly presented clue forms that are usually modeled as entity-like knowledge in existing studies.
However, in real hierarchical reasoning motivated machine reading comprehension, such one-sided modeling is insufﬁcient for those indispensable local complete facts or events when only "global" knowledge is really paid attention to.
Thus, in view of language being a complete knowledge/clue carrier, we propose a general formalism to support representing logic units by extracting backbone constituents of the sentence such as the subject-verb-object formed "facts", covering both global and local knowledge pieces that are necessary as the basis for logic reasoning.
Beyond building the ad-hoc graphs, we propose a more general and convenient fact-driven approach to construct a supergraph on top of our newly deﬁned fact units, beneﬁting from both sides of the connections between facts and internal knowledge such as concepts or actions inside a fact.
Experiments on two challenging logic reasoning benchmarks show that our proposed model, F OCAL R EASONER , outperforms the baseline models dramatically and achieves state-of-the-art results.
I NTRODUCTION
To understand human language, deep neural networks have been widely applied and achieved impressive benchmark results (Chen et al., 2016b; Sachan & Xing, 2016;
Seo et al., 2017; Dhingra et al., 2017;
Cui et al., 2017; Song et al., 2018;
Hu et al., 2019; Zhang et al., 2020a; Back et al., 2020; Zhang et al., 2020b; Hermann et al., 2015).
However, the core requirement among the natural language understanding, logic reasoning, cannot be simply solved by the current design philosophy of extracting statistical patterns from data (Shi et al., 2020).
In order to solve such a problem, there emerges an interest that accounts for human intuition about the entailment of sentences and reﬂects the semantic relations between sentential constituents (Iwa´nska, 1993).
In this paper, we focus on logic reasoning in the form of natural language understanding (NLU) as logic reasoning may be naturally embodied in such a task and natural language offers sufﬁcient enough clues for effective logic reasoning.
In detail, we concern about a logic reasoning question-answering (QA) task, where given passage, question, and candidate answer options, the model has to make a proper decision with its logic reasoning ability.
There are examples shown in Figure 1 from logic reasoning benchmark datasets, ReClor (Yu et al., 2020) and LogiQA (Liu et al., 2020). [Figure]
Recent neural models usually exploit a pretrained language model (PrLM) as a key encoder for effective contextualized representation.
According to diagnostic tests (Ettinger, 2020; Rogers et al., 2020), though PrLMs like BERT (Devlin et al., 2019) have encoded syntactic and semantic information after large-scale pre-training, they perform sensitivity to role reversal and struggles with pragmatic inference and role-based event knowledge, which are fundamental for reasoning whose major challenge is to uncover logical structures, and reason with the candidate options and questions to predict the correct answer.
It is difﬁcult for PrLMs to
Question
Passage
Answer
From this we know
[Figure]
[Figure] than [Figure]
Xiao
[Figure]
Sun
[Figure]
Sun
[Figure] Which one of the follow- ing statements, most seriously weakens the argument? A large enough comet colliding with Earth could have caused a cloud of dust that enshrouded the planet and cooled the climate long enough to result in the dinosaurs’ demise. A. Many other animal species from same era did not become extinct at the same time the dinosaurs did.
[Figure]
It cannot be determined from dinosaur skeletons whether the animals died from the effects [Figure] a dust
[Figure]
The consequences for vegetation
[Figure]
comet [Figure] Earth are [Figure]
understood.
[Figure]
√ √
[Figure]
capture the logical structure inherent in the texts since logical supervision is rarely available during pre-training.
Existing logic reasoning has shown serious dependence on knowledge-like clues. This is due to the lengthy, noisy text in human language which is though a natural carrier of knowledge but does not provide a clean, exact knowledge form. Thus, an increasing interest is in using graph networks to model the entity-aware relationships in the passages (Yasunaga et al., 2021; Ren & Leskovec, 2020; Huang et al., 2021; Krishna et al., 2020; Lv et al., 2020).
However, all these methods may insufﬁciently capture indispensable logical units from two perspectives.
First, they mostly focus on entity-aware commonsense knowledge, but pay little attention to those non-entity, non-commonsense clues (Zhong et al., 2021).
Second, when existing models extract predicate logic inside language into knowledge, they only exploit quite limited predicates like hasA and isA but ignore a broad range of predicates in real language.
From either of the perspectives, the existing methods actually only concern about those "global" knowledge that keeps valid across the entire data, without sufﬁcient "local" perception of complete facts or events in the given speciﬁc part of logic reasoning task.
We argue such insufﬁcient modeling on logic units roots from the ignorance of language itself being the complete knowledge/clue carrier.
Thus, we propose extracting a kind of broad facts according to backbone constituents of a sentence to effectively cover such indispensable logic reasoning basis, ﬁlling the gap of local, non-commonsense, non-entity, or even non-knowledge clues in existing methods as shown in Figure 2.
For example, these units may reﬂect the facts of who did what to whom , or who is what in Figure 3.
Such groups can be deﬁned as "fact unit" following Nakashole & Mitchell (2014) in Deﬁnition 1.
The fact units are further organized into a supergraph following Deﬁnition 2.
Deﬁnition 1 (Fact Unit)
Given an triplet T =
{ E 1 , P, E 2 } , where E 1 and E 2 are arguments (including entity and non-entity), P is the predicate, a fact unit F is the set of all entities in T and their corresponding relations. Intuitively, fact unit can be seen as a sort of proposition.
Deﬁnition 2 (Supergraph) A supergraph is a structure made of fact units (regarded as subgraphs) as the vertices, and the relations between fact units as undirected edges.
As shown in Figure 2, we regard the deﬁned fact as the results of syntactic processing, rather than those from semantic role labeling (SRL) as in previous study, thus the proposed fact also extends the processing means in existing work.
Correspondingly, in this work, we propose a fact-driven logical reasoning model, called F OCAL R EASONER , which builds supergraphs on top of fact units as the basis for logic reasoning, to capture both global connections between facts and the local concepts or actions inside the fact.
Our model F OCAL R EASONER is evaluated on two challenging logic reasoning benchmarks including ReClor, LogiQA, one dialogue reasoning dataset Mutual, for verifying the effectiveness and the generalizability across different domains and question formats.
R ELATED W ORK
Machine Reading Comprehension Recent years have witnessed massive researches on Machine
Reading Comprehension (MRC) whose goal is training machines to understand human languages, which has become one of the most important areas of NLP (Chen et al., 2016b; Sachan & Xing, 2016;
Seo et al., 2017; Dhingra et al., 2017;
Cui et al., 2017; Song et al., 2018;
Hu et al., 2019;
Zhang et al., 2020a; Back et al., 2020; Zhang et al., 2020b).
Despite the success of MRC models on various datasets such as CNN/Daily Mail (Hermann et al., 2015), SQuAD (Rajpurkar et al., 2016),
RACE (Lai et al., 2017) and so on, researchers began to rethink what extent does the problem been solved.
Nowadays, there are massive researches into the reasoning ability of machines.
According to (Kaushik & Lipton, 2018; Zhou et al., 2020; Chen et al., 2016a), reasoning abilities can be broadly categorized into (1) commonsense reasoning (Davis & Marcus, 2015; Bhagavatula et al., 2019;
Talmor et al., 2019; Huang et al., 2019); (2) numerical reasoning (Dua et al., 2019); (3) multi-hop reasoning (Yang et al., 2018) and (4) logic reasoning (Yu et al.,
; Liu et al., 2020), among which logic reasoning is essential in human intelligence but has merely been delved into.
Natural Language
Inference (NLI) (Bowman et al., 2015; Williams et al., 2018; Nie et al., 2020) is a task closely related to logic reasoning.
However, it has two obvious drawbacks in measuring logic reasoning abilities.
One is that it only has three logical types which are entailment, contradiction and neutral .
The other is its limitation on sentence-level reasoning.
Hence, it is important to research more comprehensive and deeper logic reasoning abilities.
comet
Earth coreference colliding comet dust caused comet demise result dust planet enshrouded dust climate cooled same entity
(Qiu et al., 2019; Ding et al., 2019; Chen et al., 2019b).
logic reasoning in MRC There are two main kinds of features in language data that would be the necessary basis for logic reasoning: 1) knowledge : global facts that keep consistency regardless of the context, such as commonsense, mostly derived from named entities; 2) non-knowledge : local facts or events that may be sensitive to the context, mostly derived from detailed language.
Existing works have made progress in improving logic reasoning ability (Yasunaga et al., 2021;
Ren & Leskovec, 2020; Huang et al., 2021; Krishna et al., 2020; Zhong et al., 2021;
Wang et al., 2021).
However, these approaches are barely satisfactory as they mostly focus on the global facts such as typical entity or sentence-level relations, which are obviously not sufﬁcient.
In this work, we strengthen the basis for logic reasoning by unifying both types of the features as "facts".
Different from previous studies that focus on the knowledge components, we propose a fact-driven logic reasoning framework that builds supergraphs on top of fact units to capture both global connections between entityaware facts and the local concepts or events inside the fact.
M ETHODOLOGY
In this section, we present a fact-driven approach for logic reasoning and the overall architecture of the model is shown in Figure 4.
The framework can be divided into three steps as following.
We ﬁrst extract fact units from raw texts via syntactic processing to construct a supergraph.
Then it performs reasoning over the supergraph along with a logical fact regularization.
Finally, it aggregates the learned representation to decode for the right answer.
3. F ACT U NIT E XTRACTION AND S UPERGRAPH C ONSTRUCTION
Fact Unit Extraction.
Figure 5 illustrates our method for constructing a supergraph from raw text inputs.
The ﬁrst step is to obtain triplets that constitute a fact unit.
To keep the framework generic, we use a fairly simple fact unit extractor based on the syntactic relations.
Given a context consisting of multiple sentences, we ﬁrst conduct dependency parsing of each sentence.
After that, we extract the
M a x P oo li ng p a ss a g e q ue s t i o n o p t i o n
R o B
ER T a coreference ......
Fact-driven Reasoning Process
A n s w e r
P r e d i c t i o n
Logical Facts Regularization
C r o ss A tt en t i o n OQ-Pair
OQ-Pair OQ-Pair
OQ-Pair
C o - a tt en t i on
I n t e r a c t i on
[Figure]
Supergraph Construction.
With the obtained triplets, the fact units are organized in the form of Levi graph (Levi, 1942), which turns arguments and predicates all into nodes.
An original fact unit is in the form of F = ( V, E, R ) , where V is the set of the arguments, E is the set of edges connected between arguments, and R is the relations of each edge which are predicates here.
The corresponding Levi graph is denoted as F l =
( V L , E L , R L ) where V L = V ∪ R , which makes the originally directly connected arguments be intermediately connected via relations.
As for R L , previous works such as (Marcheggiani & Titov, 2017; Beck et al., 2018a) designed three types of edges R L = { default, reverse, self } to enhance information ﬂow.
Here in our settings, we extend it into ﬁve types: default-in, default-out, reverse-in, reverse-out, self , corresponding to the directions of edges towards the predicates. Detailed description for edge types can be found in Appendix A.
We construct the supergraph by making connections between fact units F l .
In particular, we take three strategies according to global information, identical concept and co-reference information.
We add a node V g initialized with the question-option representation and connect it to all the fact unit nodes.
The edge type is set as aggregate for better information interaction.
(2) There can be identical mentions in different sentences, resulting in repeated nodes in fact units.
We connect nodes corresponding to the same non-pronoun arguments by edges with edge type same .
(3) We conduct co-reference resolution on context using an off-to-shelf model 1 in order to identify arguments in fact units that refer to the same one.
We add edges with type coref between them.
The ﬁnal supergraph is denoted as S =
( F l ∪ V g , E ) where E is the set of edges added with the previous three strategies.
G RAPH R EASONING AND L OGICAL F ACT R EGULARIZATION
Graph Reasoning.
A natural way to model the supergraph is via Relational Graph Convolution
Networks (Schlichtkrull et al., 2018).
We ﬁrst feed the contexts, stated as [ CLS ] C [ SEP ] q || o
[ SEP ] , into a pre-trained model to get the encoded representation.
We initialize the nodes with averaged hidden states of its tokens because our triplets extraction performs in word-level.
For edges, we use a one-hot embedding layer to encode the relations.
Based on the relational graph convolutional network and given the initial representation h
i for every node v i , the feed-forward or the message-passing process with information control can be written as h ( l +1) i [Equation] [Equation]
[Equation]
w (
[Equation]
r h ( l ) j ) , where N r ( v i ) denotes the neighbors of node v i under relation r and c i,r is the number of those nodes.
w ( l ) r is the learnable parameters of layer l .
g ( l ) q is a gated value between 0 and 1.
Through the graph encoder F G ( . ) , we then obtain the hidden representations of nodes in fact units as { h F 0 , ...
h Fm } = F G ( { v L, 0 , ...v L,m } , E L ) . h
Fi is the node representation inside fact unit. They are then concatenated as the representation for supernode as h S 0 .
{ h 0 , ...
h m } = F G ( { h S 0 , ... h Sm } , E C ) . For node features on the supergraph, it is fused via the attention and gating mechanisms with the original representations of the context encoder H C .
We apply attention mechanism to append the supergraph representation to the original one ˜ H =
Attn ( H c , K f , V f ) , where { K f , V f } are packed from the learned representations of the supergraph.
We compute λ ∈ [0 , 1] to weigh the expected importance of supergraph representation of each source word λ 1 = σ (
W λ ˜ H + U λ H C ) , where W λ and U λ are learnable parameters.
H C and ˜ H are then fused for an effective representation
H =
H C + λ ˜ H ∈ R 4 × d .
For the application to the concerned QA tasks that require reasoning, options have their inherent logical relations, which can be leveraged to aid answer prediction.
Inspired by Ran et al.
(2019), we use an attention-based mechanism to gather option correlation information.
Speciﬁcally for an option O i , the information it get by interaction with option
O j is calculated as O ( j ) i = [ O qi − O qi Attn ( O qi , O qj ; v ); O qi ◦ O qi Attn ( O qi , O qj ; v )] , where O qi is the representation of the concatenation for the i -th option and question after the context encoder. Then the option-wise information are gathered to fuse the option correlation information ˆ O i =
tanh ( W c [ O qi ; { O ( j ) i } i (cid:54) = j ] + b c ) , where W c ∈ R d × 7 d and b c ∈ R d . For answer prediction, We seek to minimize the cross entropy loss by L ans = − log softmax ( W z C + b z ) l ∈ R 4 , where C is the combined representations of ˆ O and H .
Logical Fact Regularization.
Since the subject, verb, and object in a fact should be closely related with some explicit relationships, we design logical fact regularization technique to make the logical facts more of factual correctness.
Without loss of generality, we assume that in our settings, the summation of the subject vector and the relation vector should be close to the object vector as much as possible, i.e., v subject + v predicate → v object . Speciﬁcally, given the hidden states of the sequence h i where i = 1 , . . .
, L and L is the total length of the sequence, The regularization is deﬁned as L lfr =
(cid:80) mk
=1 (1 − cos( h sub k + h pred k , h obj k )) , where m is the total number of logical fact triplets extracted from the context as well as the option and k indicates the k -th fact triplet.
3. T RAINING O BJECTIVE
During training, the overall loss for answer prediction is L = α L ans + β L lfr , where α and β are two parameters.
In our implementation, we set α = 1 . 0 and β = 0 .
[Table]
E XPERIMENTS 4. E XPERIMENTAL S ETUP
We conducted the experiments on three datasets.
Two for specialized logic reasoning ability testing:
ReClor (Yu et al., 2020) and LogiQA (Liu et al., 2020) and one for logic reasoning in dialogues:
MuTual (Cui et al., 2020).
We take RoBERTa-large (Liu et al., 2019) and DeBERTa-xlarge (He et al., 2020) as our backbone models for convenient comparison with previous works.
We also compare our model with DAGN (Huang et al., 2021), a framework leveraging RoBERTa-large as the backbone and LReasoner (Wang et al., 2021), the previous state-of-the-art model on the leaderboard using DeBERTa-xlarge.
For more details on datasets and baseline models, one can refer to Appendix A.
4. I MPLEMENTATION D ETAILS
The overall model is end-to-end trained and updated by Adam (Kingma & Ba, 2015) optimizer with an overall learning rate 8e-6 for ReClor and LogiQA, and 4e-6 for MuTual.
The weight decay is 0 .
We set the warm-up proportion during training to 0 .
Graph encoders are implemented using
DGL, an open-source lib of python.
The layer number of the graph encoder is 2 for ReClor andfor LogiQA.
The maximum sequence length is 256 for LogiQA and MuTual, and 384 for ReClor.
The model is trained for 10 epochs with a total batch size 16 and an overall dropout rate 0 .
1 on
NVIDIA Tesla V100 GPUs, which takes around 2 hours for ReClor and 4 hours for LogiQA 2 .
4. R ESULTS
Tables 1 and 2 show the results on ReClor, LogiQA, and MuTual, respectively.
All the best results are shown in bold.
From the results, we have the following observations:
1) Based on our implemented baseline models (basically consistent with public results), we observe dramatic improvements on both of the logic
reasoning benchmarks, e.g., on ReClor test set, F OCAL
R EASONER achieves +4 .
2% on dev set and +3 .
3% on the test set.
F OCAL R EASONER also outperforms the prior best system LReasoner 3 , reaching 77 .
05% on the EASY subset, and 44 .
64% on the HARD subset.
The performance suggests that F OCAL R EASONER makes better use of logical
[Table] [Table]
[Table]
structure inherent in the given context to perform reasoning than existing methods.
More detailed ablations will be shown in Section 5.
2) Table 3 lists the accuracy of our model on the dev set of ReClor of different question types.
Results show that our model can perform well on most of the question types, especially "Strengthen" and "Weaken".
This means that our model can well interpret the question type from the question statement and make the correct choice corresponding to the question.
Our model also achieves comparable performance with the unpublished LReasoner.
It employs symbolic reasoning and data augmentation techniques, which is in a different research line from ours.
Without data argumentation, LReasoner shows relatively poorer results, showing that our fact-driven approach would be beneﬁcial compared with the symbolic-driven technique.
Compared with the neural methods for logic reasoning, symbolic approaches would rely heavily on dataset-related predeﬁned patterns which entail massive manual labor, potentially restricting the generalizability of models.
) On the dialogue reasoning dataset MuTual, our model achieves quite a jump compared with the
RoBERTa-base LM.
This veriﬁes our model’s generalizability on other downstream reasoning task settings
For the model complexity, our method basically keeps as simple as previous models like DAGN.
Our model only has 414M parameters compared with 355M in the baseline RoBERTa, and 400M in
DAGN which also employs GNN.
A NALYSIS 5. A BLATION S TUDY
To dive into the effectiveness of different components in F OCAL R EASONER , we conduct analysis by taking RoBERTa as the backbone on the ReClor dev set.
Tables 5-6 summarize the results.
[Table]
[Table]
Fact Units Variants Apart from our syntactically constructed fact units, there are two other ways in different granularities for construction.
We replace the fact units with named entities that are used in previous works like Chen et al. (2019a).
The statistics of fact units and named entities of ReClor and LogiQA are stated in Table 4, from which we can infer that there are indeed more fact units than named entities.
Thus using fact units can better incorporate the logical information within the context.
When replacing all the fact units with named entities, we can see from Table 5 that it signiﬁcantly decreases the performance.
We also explore the performance using semantic role labeling a similar way as in Zhong et al.
We can see that SRL, leveraging much more complex information as well as computation complexity, fails to achieve performance as good as our original fact unit
[Table] [Table]
Supergraph Reasoning:
The ﬁrst key component is supergraph reasoning.
We ablate the global atom and erase all the edges connected with it.
The results suggest that the global atom indeed betters message propagation, leveraging performance from 64 . 6% to 66 .
We also ﬁnd that replacing the initial QA pair representation of the global atom with only question representation hurts the performance.
In addition, without the logical fact regularization, the performance drops from 66 .
8% to 64 .
2% , indicating its usefulness.
For edge analysis, when
(1) all edges
are regarded
as a single type rather
than the original designed 8 types in total and (
) co-reference edges are removed, the accuracy drops to 63 .
7% and 64 .
8% , respectively.
It is proved that in our supergraph, edges link the fact units in reasonable manners, which properly uncovers the logical structures.
Interactions: We further experimented with the query-option-interactions setting to see how it affects the performance.
The results suggest that the features learned from the interaction process enhance the model.
Considering that the logical relations between different options are a strong indicator of the right answer, this means that the model learns from a comparative reasoning strategy.
5. E FFECTS OF F ACT U NITS N UMBERS
To inspect the effects of the number of fact units, we split the original dev set of ReClor and LogiQA into 5 subsets.
The statistics of the fact unit distribution on the datasets are shown in Table 7.
The numbers of fact units for most contexts in ReClor and LogiQA are in [3 , 6) and [0 , 3) , respectively.
Comparing the accuracies of RoBERTa-large baseline, prior SOTA LReasoner and
our proposed
F OCAL R EASONER in Figure 6, our model outperforms baseline models on all the divided subsets, which demonstrates the effectiveness and robustness of our proposed method.
Speciﬁcally, for ReClor,
F OCAL R EASONER performers better when there are more fact units in the context, while for LogiQA,
F OCAL R EASONER works better when the number of fact units locates in [0 , 3) and [9 , 12) .
The reason may lie in the difference in style of the two datasets.
However, all the models include ours struggle when the number of fact units is above certain thresholds, i.e., the logical structure is more complicated, calling for better mechanisms to cope with.
[0, 3) [3, 6) [6, 9) [9, 12) [12, ∞ )
[Figure]
NUMBER OF FACT UNITS
RoBERTa DAGN Focal Reasoner 37 . 35 .30 .
[ 0, 3) [ 3, 6) [ 6, 9) [ 9, 12) [ 12, ∞ )
ACCU R
AC Y
NUMBER OF FACT UNITS
RoBERTa DAGN Focal Reasoner
5. I NTERPRETABILITY : A C ASE S TUDY
[Table]
We aim to interpret F OCAL R EASONER ’s reasoning process by analyzing the node-to-node attention weights induced in the supergraph in
Figure 7.
We can see that our F OCAL R EASONER can well bridge the reasoning process between context, question and option.
Speciﬁcally, in the graph, "students rank 30%" attends strongly to "playing improve performance".
Under the guidance of question to select the option that weakens the statement and option interaction, our model is able to tell that "students rank 30% can play" mostly undermines the conclusion that "playing improves performance".
A recent survey in a key middle school showed that high school students in this school have a special preference for playing football, and it far surpasses other balls.
The survey also found that students who regularly play football are better at academic performance than students who do not often play football.
This shows that often playing football can improve students' academic performance.
Which of the following can weaken the above conclusion most?
A. Only high school students who are ranked in the top 30% of grades can often play football.
B. Regular football can exercise and maintain a strong learning energy.
C. Often playing football delays the study time.
D. Research has not proved that playing football can contribute to intellectual development
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
Fact Units
[Figure]
①
[Figure]
after Interaction
A A
B B
C
D
C
D
A: 4.
C: -12.3718 B:
[Figure]
D:
[Figure] [Figure]
C ONCLUSION
For logic reasoning arising from machine reading comprehension, it is well known that clear and accurate forms like global knowledge are crucial.
In this work, we make a ﬁnding that existing studies miss focusing on quite a lot of non-knowledge parts which is also indispensable for better reasoning.
Thus we propose extracting a general form called "fact unit" to cover both global and local logical units, hoping to shed light on the basis of structural modeling for logic reasoning.
Our proposed
F OCAL R EASONER not only better uncovers the logical structures within the context, which can be a general method for other sophisticated reasoning tasks, but also better captures the logical interactions between context and options.
The experimental results verify the effectiveness of our method.
A L EVI G RAPH C ONSTRUCTION
Levi graph transformation turns labeled edges into additional vertices.
There are two types of edges in a traditional Levi graph: default and reverse .
For example, an edge ( E 1 , R 1 , E 2) in the original graph becomes ( E 1 , default, R 1) , ( R 1 , default, E 2) , ( R 1 , reverse, E 1) and ( E 2 , reverse, R 1) .
However, the type of source and target vertices in the graph also matters (Beck et al., 2018b).
Speciﬁcally, previous works use the same type of edge to pass information, which may reduce the effectiveness.
Thus we propose to transform the default edges into default-in and default-out edges, and the reverse edges into reverse-in and reverse-out edges.
B D ETAILS FOR D ATASETS AND B ASELINE M ODELS
In this section, we describe the datasets and baseline models used in the experiments.
B. D ATASETS
ReClor ReClor contains 6,138 multiple-choice questions modiﬁed from standardized tests such as
GMAT and LSAT, which are randomly split into train/dev/test sets with 4,638/500/1,000 samples respectively.
It contains multiple logic reasoning types.
The held-out test set is further divided into
EASY and HARD subsets based on the performance of BERT-based model Devlin et al.
LogiQA LogiQA consists of 8,678 multiple-choice questions collected from National Civil Servants
Examinations of China and are manually translated into English by experts.
The dataset is randomly split into train/dev/test sets with 7,376/651/651 samples correspondingly.
LogiQA also contains various logic reasoning types.
MuTual MuTual has 8,860 dialogues annotated by linguist experts and high-quality annotators from
Chinese high school English listening comprehension test data.
It is randomly split into train/dev/test sets with 7,088/886/886 samples respectively.
There more than 6 types of reasoning abilities reﬂected in MuTual.
MuTual plus is an advanced version, where one of the candidate responses is replaced by a safe response (e.g., "could you repeat that?" ) for each example. B. B ASELINE M ODELS
DAGN explores passage-level discourse-aware clues used for solving logical reasoning QA.
Speciﬁcally, they leverage discourse relations annotated in Penn Discourse TreeBank 2.0 (PDTB 2.0) (Prasad et al., 2008) and punctuation as the delimiters to split the context into elementary discourse units (EDUs).
They are further organized into a logical graph and feed into a GNN to get the representation.
LReasoner is a symbolic-driven framework for logical reasoning of text.
It ﬁrstly identiﬁes the logical symbols and expressions explicitly for the context and options based on manually designed rules.
Then it performs logical inference over the expressions according to logical equivalence laws such as contraposition (Russell & Norvig, 2002).
Finally, it verbalizes the expressions to match the answer.
F OCAL R EASONER enjoys two major merits.
(1) Broader knowledge: Compared with DAGN which uses sententious knowledge such as logical connectives (e.g., becaus , however ), F OCAL R EASONER leverages a broader type of knowledge characterized by "fact unit", including global knowledge and local knowledge.
(2) More transferable: Compared with LReasoner which manually design rules to extract logical patterns and perform logical reasoning in a symbolic way, Focal Reasoner is neural-based and manual-free, which is more generalizable to other datasets.
C M ODEL C OMPLEXITY
In this section, we display the statistics for the parameters of F OCAL R EASONER and baseline models to demonstrate the model complexity and the effectiveness.
[Table]
From Table 8 we can see that the increase of parameters is no more than 15% compared with the baseline models, and is comparable with strong baseline model DAGN.
This indicates that, our model, being quite effective, is not a result of stacking complicated modules.
D V ARIANCES FOR F OCAL R EASONER W . R . T EXPERIMENT RESULTS
In this section
[Table]
we report the average and variances run on 5 random seeds for F OCAL R EASONER with different pre-trained language models.
[Table]
E F URTHER I NTERPRETATION
We change the example in Figure 7 a bit.
Speciﬁcally, we change the conclusion from "playing football can improve students’ academic performance" to "football players are held to higher academic standards than non-athletes".
We can observe that our model can select the relatively correct answer, which indicates that F OCAL R EASONER has some logical reasoning ability, instead of simple text matching and mining ability
[Table]
[Table]
[Figure]
A. Only high school students who are ranked in the top 30% of grades can often play football.
B. Regular football can exercise and maintain a strong learning energy.
C. Often playing football delays the study time.
D. Research has not proved that playing football can contribute to intellectual development.
[Figure]
students have preferences 2.
preference playing football 3.
it surpasses balls 4.
who play football 5.
students
[Figure]
[Figure]
!play football 7.
players higher standards 8.
[Figure]
!contribute development
[Figure]