Original: N AIL :
Revised: N AIL :

Original: A
Revised: A

Original: C
Revised: C

Original: HALLENGING
Revised: HALLENGING

Original: B
Revised: B

Original: ENCHMARK FOR NA
Revised: ENCHMARK FOR NA

Original: ¨ IVE
Revised: ¨ IVE

Original: LOGICAL REASONING
Revised: LOGICAL REASONING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Logical reasoning over natural text is an important capability towards human level intelligence.
Revised: Logical reasoning over natural text is an important capability towards human level intelligence.

Original: Existing datasets are either limited and inadequate to train and evaluate logical reasoning capability (e.g., LogiQA and ReClor), or not oriented for logical reasoning (e.g., SQuAD and HotpotQA).
Revised: Existing datasets are either limited and inadequate to train and evaluate logical reasoning capability (e.g., LogiQA and ReClor), or not oriented for logical reasoning (e.g., SQuAD and HotpotQA).

Original: In this paper, we focus on a speciﬁc category of logical reasoning, named na¨ıve logical reasoning , and propose a new large-scale benchmark, named N AIL , aiming to help models learn and evaluate na ¨ ıve logical reasoning capability.
Revised: In this paper, we focus on a speciﬁc category of logical reasoning, named na¨ıve logical reasoning , and propose a new large-scale benchmark, named N AIL , aiming to help models learn and evaluate na ¨ ıve logical reasoning capability.

Original: N AIL is sourced from standardized exams such as Chinese National Civil Servants Examination and Law School Admission Test.
Revised: N AIL is sourced from standardized exams such as Chinese National Civil Servants Examination and Law School Admission Test.

Original: Furthermore, to collect more data, we propose to imitate examples of standardized exams rather than designing them from scratch.
Revised: Furthermore, to collect more data, we propose to imitate examples of standardized exams rather than designing them from scratch.

Original: N AIL is available in both Chinese and English containing a total of 10 , 296 ∗ 2 instances.
Revised: N AIL is available in both Chinese and English containing a total of 10 , 296 ∗ 2 instances.

Original: Empirical results show that state-of-the-art neural models struggle on N AIL with very poor accuracy (the best result is 30.1% for N AIL and 36.2% for Chinese N AIL ), while human experts can perform 100% accuracy.
Revised: Empirical results show that state-of-the-art neural models struggle on N AIL with very poor accuracy (the best result is 30.1% for N AIL and 36.2% for Chinese N AIL ), while human experts can perform 100% accuracy.

Original: Further results indicate that human imitations can signiﬁcantly help models learn na ¨ ıve logical reasoning ability.
Revised: Further results indicate that human imitations can signiﬁcantly help models learn na ¨ ıve logical reasoning ability.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Current deep models have achieved near human-level performance on many tasks in NLP (Devlin et al., 2019; Liu et al., 2019), and more often than not, superﬁcial knowledge sufﬁces to solve the problems.
Revised: Current deep models have achieved near human-level performance on many tasks in NLP (Devlin et al., 2019; Liu et al., 2019), and more often than not, superﬁcial knowledge sufﬁces to solve the problems.

Original: To move towards human intelligence, we need to equip the models with logical reasoning capabilities (e.g., ability to draw logical conclusions from given statements), which is also a long sought-after goal of AI (Newell & Simon, 1956; McCarthy et al., 1960).
Revised: To move towards human intelligence, we need to equip the models with logical reasoning capabilities (e.g., ability to draw logical conclusions from given statements), which is also a long sought-after goal of AI (Newell & Simon, 1956; McCarthy et al., 1960).

Original: One related task is natural language inference whose goal is to assign the logical relationships ( contradicted , neutral and entailment ) to sentence pairs.
Revised: One related task is natural language inference whose goal is to assign the logical relationships ( contradicted , neutral and entailment ) to sentence pairs.

Original: To push the development of models in logical reasoning, the researchers have focused on more challenging reading comprehension tasks, which often require more complex reasoning as well as longer input.
Revised: To push the development of models in logical reasoning, the researchers have focused on more challenging reading comprehension tasks, which often require more complex reasoning as well as longer input.

Original: However, most existing reading comprehension datasets are not oriented for the logical reasoning (e.g., SQuAD and HotpotQA), with the exception of LogiQA (Liu et al., 2020), ReClor (Yu et al., 2020) and LR-LSAT (Wang et al., 2021).
Revised: However, most existing reading comprehension datasets are not oriented for the logical reasoning (e.g., SQuAD and HotpotQA), with the exception of LogiQA (Liu et al., 2020), ReClor (Yu et al., 2020) and LR-LSAT (Wang et al., 2021).

Original: Above datasets (LogiQA, ReClor and LR-LSAT) are limited and inadequate to train and evaluate logical reasoning capability.
Revised: Above datasets (LogiQA, ReClor and LR-LSAT) are limited and inadequate to train and evaluate logical reasoning capability.

Original: The reason is that all of the three datasets involve diverse types of logical reasoning, such as drawing an alternate conclusion to the argument, ﬁnding necessary/sufﬁcient assumptions, whether statements strengthen/weaken the argument or explain/resolve the situation.
Revised: The reason is that all of the three datasets involve diverse types of logical reasoning, such as drawing an alternate conclusion to the argument, ﬁnding necessary/sufﬁcient assumptions, whether statements strengthen/weaken the argument or explain/resolve the situation.

Original: Mixing multiple types of logical reasoning may pose the following challenges.
Revised: Mixing multiple types of logical reasoning may pose the following challenges.

Original: a).
Revised: a).

Original: From the perspective of human cognition, different types of logical reasoning correspond to different problem-solving ideas.
Revised: From the perspective of human cognition, different types of logical reasoning correspond to different problem-solving ideas.

Original: But in practice we usually train a model on the whole dataset with the same idea, which makes it more limited for models to learn different logical reasoning capability.
Revised: But in practice we usually train a model on the whole dataset with the same idea, which makes it more limited for models to learn different logical reasoning capability.

Original: b).
Revised: b).

Original: From the perspective of machine learning, if there are many reasoning types mixed in a dataset, then there will be less data for each reasoning type, which is inadequate to train and evaluate logical reasoning capability (demonstrated in our experiments).
Revised: From the perspective of machine learning, if there are many reasoning types mixed in a dataset, then there will be less data for each reasoning type, which is inadequate to train and evaluate logical reasoning capability (demonstrated in our experiments).

Original: Furthermore, when the model does not work, it is difﬁcult to determine which reasoning type is the bottleneck (no reasoning type annotation in the dataset), which may hinder the design of better models.
Revised: Furthermore, when the model does not work, it is difﬁcult to determine which reasoning type is the bottleneck (no reasoning type annotation in the dataset), which may hinder the design of better models.

Original: To tackle the challenges, we focus on a more ﬁne-grained type of logical reasoning, named na¨ıve logical reasoning (Section 2), which is to infer the logical conclusion from statements that describe triples (subject, predicate, object) and the relationships among them.
Revised: To tackle the challenges, we focus on a more ﬁne-grained type of logical reasoning, named na¨ıve logical reasoning (Section 2), which is to infer the logical conclusion from statements that describe triples (subject, predicate, object) and the relationships among them.

Original: A typical example of na ¨ ıve
Revised: A typical example of na ¨ ıve

Original: [Table]
Revised: [Table]

Original: Color of clothes
Revised: Color of clothes

Original: [Table]
Revised: [Table]

Original: A t o l d t h e t r u t h
Revised: A t o l d t h e t r u t h

Original: B t o l d t h e t r u t h
Revised: B t o l d t h e t r u t h

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: </latexit
Revised: </latexit

Original: [Table]
Revised: [Table]

Original: Color of clothes
Revised: Color of clothes

Original: Y
Revised: Y

Original: G
Revised: G

Original: B
Revised: B

Original: R
Revised: R

Original: A
Revised: A

Original: B
Revised: B

Original: C
Revised: C

Original: D
Revised: D

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: logical reasoning is shown in Figure 1.
Revised: logical reasoning is shown in Figure 1.

Original: To answer this query, we need to iteratively derive conclusions according to the conditions
Revised: To answer this query, we need to iteratively derive conclusions according to the conditions

Original: [Figure]
Revised: [Figure]

Original: and stop the searching branch if a conﬂict occurs.
Revised: and stop the searching branch if a conﬂict occurs.

Original: Speciﬁcally, in the initial table, we only know that “D is not
Revised: Speciﬁcally, in the initial table, we only know that “D is not

Original: Inspired by the datasets extracted from standardized examinations (Lai et al., 2017; Clark et al., 2018;
Revised: Inspired by the datasets extracted from standardized examinations (Lai et al., 2017; Clark et al., 2018;

Original: Liu et al., 2020), we build a new large-scale benchmark, N AIL , by selecting na ¨ ıve logical reasoning examples from standardized exams such as the National Civil Servants Examination of China and
Revised: Liu et al., 2020), we build a new large-scale benchmark, N AIL , by selecting na ¨ ıve logical reasoning examples from standardized exams such as the National Civil Servants Examination of China and

Original: Law School Admission Test.
Revised: Law School Admission Test.

Original: However, such examples are limited in their number, as it takes efforts for human experts to design these questions.
Revised: However, such examples are limited in their number, as it takes efforts for human experts to design these questions.

Original: To collect more data, we propose to imitate examples of standardized exams rather than designing them from scratch.
Revised: To collect more data, we propose to imitate examples of standardized exams rather than designing them from scratch.

Original: Unlike simple data augmentation (e.g., substitution, paraphrasing), human imitation aims at getting more diverse examples while maintaining the underlying logic (Figure 1).
Revised: Unlike simple data augmentation (e.g., substitution, paraphrasing), human imitation aims at getting more diverse examples while maintaining the underlying logic (Figure 1).

Original: N AIL is available in both Chinese and English, containing a total of 10 , 296 ∗ 2 instances.
Revised: N AIL is available in both Chinese and English, containing a total of 10 , 296 ∗ 2 instances.

Original: Empirical results show that state-of-the-art neural models struggle on N AIL with very poor accuracy (the best result is 30.1% for N AIL and 36.2% for Chinese N AIL ), while human experts can perform 100% accuracy.
Revised: Empirical results show that state-of-the-art neural models struggle on N AIL with very poor accuracy (the best result is 30.1% for N AIL and 36.2% for Chinese N AIL ), while human experts can perform 100% accuracy.

Original: Further results indicate that human imitations can signiﬁcantly help models learn na¨ıve logical ability from natural text.
Revised: Further results indicate that human imitations can signiﬁcantly help models learn na¨ıve logical ability from natural text.

Original:  NA ¨ IVE LOGICAL REASONING
Revised:  NA ¨ IVE LOGICAL REASONING

Original: The na¨ıve logical reasoning is a more ﬁne-grained type of logical reasoning.
Revised: The na¨ıve logical reasoning is a more ﬁne-grained type of logical reasoning.

Original: Formally, we give the deﬁnition as follows.
Revised: Formally, we give the deﬁnition as follows.

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: The subject is a described resource, usually an entity, such as person or location.
Revised: The subject is a described resource, usually an entity, such as person or location.

Original: The predicate indicates an attribute of the subject or indicates some kind of relationship between the subject and the object.
Revised: The predicate indicates an attribute of the subject or indicates some kind of relationship between the subject and the object.

Original: When denoting an attribute, the object is the attribute value, usually a literal value, e.g., (Jacket, Color, Red), otherwise the object is an entity, e.g., (Beijing, Capital, China).
Revised: When denoting an attribute, the object is the attribute value, usually a literal value, e.g., (Jacket, Color, Red), otherwise the object is an entity, e.g., (Beijing, Capital, China).

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: Assuming the set of subjects S , the set of predicates P , the set of objects O and several statements, each statement describes a triple (subject, predicate, object) or some logical relationship between triples.
Revised: Assuming the set of subjects S , the set of predicates P , the set of objects O and several statements, each statement describes a triple (subject, predicate, object) or some logical relationship between triples.

Original: The naive logical reasoning is the process of reasoning from these statements to reach a logical conclusion.
Revised: The naive logical reasoning is the process of reasoning from these statements to reach a logical conclusion.

Original: In this work, we explore na¨ıve logical reasoning in the form of reading comprehension.
Revised: In this work, we explore na¨ıve logical reasoning in the form of reading comprehension.

Original: A typical example and detailed reasoning process is shown in Figure 1.
Revised: A typical example and detailed reasoning process is shown in Figure 1.

Original: Similar to the format of multiple-choice reading comprehension, it contains a context, a query and four options with only one correct answer.
Revised: Similar to the format of multiple-choice reading comprehension, it contains a context, a query and four options with only one correct answer.

Original: To solve the problem, the model needs to understand the logical connections between the subjects, predicates and objects, and then derive a valid option.
Revised: To solve the problem, the model needs to understand the logical connections between the subjects, predicates and objects, and then derive a valid option.

Original:  N AIL : D ATA C OLLECTION AND A NALYSIS
Revised:  N AIL : D ATA C OLLECTION AND A NALYSIS

Original: N AIL is a carefully designed benchmark for na¨ıve logical reasoning similar to the format of multiplechoice reading comprehension.
Revised: N AIL is a carefully designed benchmark for na¨ıve logical reasoning similar to the format of multiplechoice reading comprehension.

Original: Inspired by the datasets extracted from standardized examinations (Lai et al., 2017; Clark et al., 2018; Liu et al., 2020), we ﬁrst collect a small amount of examples from examinations, denoted as N AIL -E (Section 3.1).
Revised: Inspired by the datasets extracted from standardized examinations (Lai et al., 2017; Clark et al., 2018; Liu et al., 2020), we ﬁrst collect a small amount of examples from examinations, denoted as N AIL -E (Section 3.1).

Original: Then we propose to imitate examples of N AIL -E to collect more data, denoted as N AIL -I (Section 3.2).
Revised: Then we propose to imitate examples of N AIL -E to collect more data, denoted as N AIL -I (Section 3.2).

Original: Finally we provide a detailed analysis of the proposed N AIL (Section 3.3).
Revised: Finally we provide a detailed analysis of the proposed N AIL (Section 3.3).

Original: 3. C OLLECTION FROM E XAMINATION
Revised: 3. C OLLECTION FROM E XAMINATION

Original: We searched for such examples widely from two different types of public examinations: Chinese
Revised: We searched for such examples widely from two different types of public examinations: Chinese

Original: National Civil Servants Examination (CNCSE) and Law School Admission Test (LSAT).
Revised: National Civil Servants Examination (CNCSE) and Law School Admission Test (LSAT).

Original: 2 CNCSE is a once-a-year competitive examination in China, and there are overall 120-140 examples per exam per year.
Revised: 2 CNCSE is a once-a-year competitive examination in China, and there are overall 120-140 examples per exam per year.

Original: But only 1-4 examples belong to the scope of na ¨ ıve logical reasoning, which is also the most difﬁcult type of problems for candidates within the given 120 minutes.
Revised: But only 1-4 examples belong to the scope of na ¨ ıve logical reasoning, which is also the most difﬁcult type of problems for candidates within the given 120 minutes.

Original: And the LSAT is a standardized test for prospective law school candidates in the United States, Canada, and a growing number of other countries.
Revised: And the LSAT is a standardized test for prospective law school candidates in the United States, Canada, and a growing number of other countries.

Original: Logical Reasoning is a multiple-choice section of LSAT, containing 24-26 questions under the limitation of 35 minutes, where about 2-4 problems fall into na¨ıve logical reasoning category.
Revised: Logical Reasoning is a multiple-choice section of LSAT, containing 24-26 questions under the limitation of 35 minutes, where about 2-4 problems fall into na¨ıve logical reasoning category.

Original: We artiﬁcially selected examples that belong to the na ¨ ıve logical reasoning category from the above two examinations.
Revised: We artiﬁcially selected examples that belong to the na ¨ ıve logical reasoning category from the above two examinations.

Original: Finally we obtained 488 examples from the last 25 years of CNCSE and LSAT in the last 30 years, denoted as N AIL -E. These two exams are from countries with different native languages.
Revised: Finally we obtained 488 examples from the last 25 years of CNCSE and LSAT in the last 30 years, denoted as N AIL -E. These two exams are from countries with different native languages.

Original: The former is expressed in Chinese, and the latter is expressed in English.
Revised: The former is expressed in Chinese, and the latter is expressed in English.

Original: And note that there are slight difference in language style between them.
Revised: And note that there are slight difference in language style between them.

Original: And for diversity and fairness, in later step we get all examples in both Chinese and English through translation.
Revised: And for diversity and fairness, in later step we get all examples in both Chinese and English through translation.

Original: 3. C OLLECTION FROM I MITATION
Revised: 3. C OLLECTION FROM I MITATION

Original: After collecting a small amount of N AIL -E, we expect to expand the number of the dataset.
Revised: After collecting a small amount of N AIL -E, we expect to expand the number of the dataset.

Original: Designing examples from scratch requires a huge effort from human experts.
Revised: Designing examples from scratch requires a huge effort from human experts.

Original: One simple solution is data augmentation, which artiﬁcially scales up data by creating modiﬁed data from existing data, such as word/sentence shufﬂing, word replacement and syntactic variation.
Revised: One simple solution is data augmentation, which artiﬁcially scales up data by creating modiﬁed data from existing data, such as word/sentence shufﬂing, word replacement and syntactic variation.

Original: However, the data augmented in this way is highly correlated with the original data, and the model easily captures these semantic surface correlations, which makes it limited to train and evaluate logical reasoning capability.
Revised: However, the data augmented in this way is highly correlated with the original data, and the model easily captures these semantic surface correlations, which makes it limited to train and evaluate logical reasoning capability.

Original: To alleviate this limitation, we propose to imitate examples of N AIL -E, to create more examples with a diverse semantic surface while keeping the underlying logic of the original example.
Revised: To alleviate this limitation, we propose to imitate examples of N AIL -E, to create more examples with a diverse semantic surface while keeping the underlying logic of the original example.

Original: Furthermore, in the process of human imitation, we design strict strategies to control the quality of imitation.
Revised: Furthermore, in the process of human imitation, we design strict strategies to control the quality of imitation.

Original: Imitation Example See Figure 2, the context of an example from N AIL -E consists of 6 sentences and a query (split by blank lines), each of which can be represented as a logical template (see “backbone template”).
Revised: Imitation Example See Figure 2, the context of an example from N AIL -E consists of 6 sentences and a query (split by blank lines), each of which can be represented as a logical template (see “backbone template”).

Original: The example focus on the description of a scenario: picking Prince
Revised: The example focus on the description of a scenario: picking Prince

Original: Charming ( m 1 ), which involves ﬁve entities: Li Na ( s 1 ), Wang Wei ( p 1 ), Wu Gang ( p 2 ),
Revised: Charming ( m 1 ), which involves ﬁve entities: Li Na ( s 1 ), Wang Wei ( p 1 ), Wu Gang ( p 2 ),

Original: Li Qiang ( p 3 ), Liu Dawei ( p 4 ), and three noun/adjective properties: tall ( a 1 ), handsome ( a 2 ), a PhD ( a 3 ).
Revised: Li Qiang ( p 3 ), Liu Dawei ( p 4 ), and three noun/adjective properties: tall ( a 1 ), handsome ( a 2 ), a PhD ( a 3 ).

Original: An accepted imitation needs to keep original underlying logic but have semantically different groundings, i.e., to describe a completely different scenario.
Revised: An accepted imitation needs to keep original underlying logic but have semantically different groundings, i.e., to describe a completely different scenario.

Original: Imitation 1 and
Revised: Imitation 1 and

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: are unqualiﬁed imitations.
Revised: are unqualiﬁed imitations.

Original: Since Imitation 1 only conducts subject -level imitations (S imitations), i.e., only superﬁcially substitute the ﬁve entities ( s 1 : Li Na → Sun Yi , p 1 :
Revised: Since Imitation 1 only conducts subject -level imitations (S imitations), i.e., only superﬁcially substitute the ﬁve entities ( s 1 : Li Na → Sun Yi , p 1 :

Original: Wang Wei → Qu Heng , p 2 : Wu Gang → Mei Zhen , p 3 : Li Qiang → Ji Fan , p 4 : Liu Dawei → Zhang Lei ).
Revised: Wang Wei → Qu Heng , p 2 : Wu Gang → Mei Zhen , p 3 : Li Qiang → Ji Fan , p 4 : Liu Dawei → Zhang Lei ).

Original: Further, Imitation 2 conducts subject-and-object-level imitations (SO imitations), i.e., not only substituting the entities, but also altering the corresponding objects (property values).
Revised: Further, Imitation 2 conducts subject-and-object-level imitations (SO imitations), i.e., not only substituting the entities, but also altering the corresponding objects (property values).

Original: ( a 1 : tall → rich , a 2 : handsome → having a house , a 3 : a PhD → having a car ).
Revised: ( a 1 : tall → rich , a 2 : handsome → having a house , a 3 : a PhD → having a car ).

Original: The difference between Imitation 2 and the original example is much greater than that between Imitation 1 and and the raw problem, however, Imitation 2 is still not an ideal imitation.
Revised: The difference between Imitation 2 and the original example is much greater than that between Imitation 1 and and the raw problem, however, Imitation 2 is still not an ideal imitation.

Original: Furthermore, Imitation 3 changes the scenario m 1 into picking ideal gift .
Revised: Furthermore, Imitation 3 changes the scenario m 1 into picking ideal gift .

Original: Imitation 3 is an expected imitative writing, which conducts subject-and-predicate-and-object-level imitations (SPO imitations).
Revised: Imitation 3 is an expected imitative writing, which conducts subject-and-predicate-and-object-level imitations (SPO imitations).

Original: Note that all of the three imitations share the same logic templates.
Revised: Note that all of the three imitations share the same logic templates.

Original: And p 2 is the answer for all of the original example and above three imitations.
Revised: And p 2 is the answer for all of the original example and above three imitations.

Original: Imitation Process We ﬁrst select a group of people from a variety of occupations: professional editors, legal practitioners, in-service civil servants and college students (from different majors).
Revised: Imitation Process We ﬁrst select a group of people from a variety of occupations: professional editors, legal practitioners, in-service civil servants and college students (from different majors).

Original: Empirically, people in these occupations have strong logical and verbal skills.
Revised: Empirically, people in these occupations have strong logical and verbal skills.

Original: These people are asked to conduct SPO imitations based on original examples from N AIL -E, and reasonable imitations should meet two requirements: logic invariance and semantic diversity .
Revised: These people are asked to conduct SPO imitations based on original examples from N AIL -E, and reasonable imitations should meet two requirements: logic invariance and semantic diversity .

Original: We trained these candidate people how to conduct SPO imitations such as Figure 2.
Revised: We trained these candidate people how to conduct SPO imitations such as Figure 2.

Original: We then conducted a trial phase before the ofﬁcial imitation phase, in which process we eliminated some people of poor quality.
Revised: We then conducted a trial phase before the ofﬁcial imitation phase, in which process we eliminated some people of poor quality.

Original: Finally we employed 82 qualiﬁed people 3 to imitatively construct problems based on N AIL -E, and they are paid
Revised: Finally we employed 82 qualiﬁed people 3 to imitatively construct problems based on N AIL -E, and they are paid

Original: RMB¥2.8 per imitation 4 .
Revised: RMB¥2.8 per imitation 4 .

Original: Averagely, it costs a trained person about 3-4 minutes to ﬁnish an imitation: starting from coming up a scenario, then replacing the subjects, predicates, and objects of the raw problem with those scenario-related while keeping invariant logic, and ﬁnally smoothing the new sentence with some transition words if necessary.
Revised: Averagely, it costs a trained person about 3-4 minutes to ﬁnish an imitation: starting from coming up a scenario, then replacing the subjects, predicates, and objects of the raw problem with those scenario-related while keeping invariant logic, and ﬁnally smoothing the new sentence with some transition words if necessary.

Original: For each original example in N AIL -E, we expect at least 20 imitations (except for extremely difﬁcult cases).
Revised: For each original example in N AIL -E, we expect at least 20 imitations (except for extremely difﬁcult cases).

Original: Overall we use 813 paid work hours in total to build the N AIL -I.
Revised: Overall we use 813 paid work hours in total to build the N AIL -I.

Original: Imitation Quality Control We adopt following strategies to ensure the quality of imitations: 1.
Revised: Imitation Quality Control We adopt following strategies to ensure the quality of imitations: 1.

Original: As mentioned above, we conducted a trial phase before the ofﬁcial imitation phase.
Revised: As mentioned above, we conducted a trial phase before the ofﬁcial imitation phase.

Original: In this phase, we asked them to imitate a small number of problems.
Revised: In this phase, we asked them to imitate a small number of problems.

Original: Although we do not necessarily need them to write the backbone template, we will check the logic and give feedback to help them understand the task.
Revised: Although we do not necessarily need them to write the backbone template, we will check the logic and give feedback to help them understand the task.

Original: This process was iterated for three rounds.
Revised: This process was iterated for three rounds.

Original: Only those who passed the trial phase could participate in the ofﬁcial imitation.
Revised: Only those who passed the trial phase could participate in the ofﬁcial imitation.

Original: During the ofﬁcial imitation, we set up an online chat room to communicate with employees and answer their questions timely.
Revised: During the ofﬁcial imitation, we set up an online chat room to communicate with employees and answer their questions timely.

Original: To embrace semantic diversity , each original example is shown to at least
Revised: To embrace semantic diversity , each original example is shown to at least

Original: people
Revised: people

Original: , that is, one can only conduct 4 imitations based
Revised: , that is, one can only conduct 4 imitations based

Original: on one original example.
Revised: on one original example.

Original: People who are assigned to the same example imitate independently without interference from each other, to ensure varied inspiration for imitation.
Revised: People who are assigned to the same example imitate independently without interference from each other, to ensure varied inspiration for imitation.

Original: To ensure logic invariance , we adopt a double-checking strategy: • Cross Checking :
Revised: To ensure logic invariance , we adopt a double-checking strategy: • Cross Checking :

Original: Everyday, for each employee, we sample 5 imitations from all of his/her daily imitations.
Revised: Everyday, for each employee, we sample 5 imitations from all of his/her daily imitations.

Original: And the sampled imitation is assigned to other 2 employees for cross-checking.
Revised: And the sampled imitation is assigned to other 2 employees for cross-checking.

Original: The imitation will be qualiﬁed only if they both approved, and the criterion for approval is that the imitation share the same logics with the original example.
Revised: The imitation will be qualiﬁed only if they both approved, and the criterion for approval is that the imitation share the same logics with the original example.

Original: If any one of the 5 imitations produced by one employee fail, then all imitations of that employee for that day will be returned to re-check and repair.
Revised: If any one of the 5 imitations produced by one employee fail, then all imitations of that employee for that day will be returned to re-check and repair.

Original: • Post Checking :
Revised: • Post Checking :

Original: To further ensure that the underlying logic do not deviate during imitation, we introduce another team of experts to solve the imitative examples.
Revised: To further ensure that the underlying logic do not deviate during imitation, we introduce another team of experts to solve the imitative examples.

Original: The team consists ofexperienced experts.
Revised: The team consists ofexperienced experts.

Original: 10 of them speaks Chinese as native language and have passed CNCSE, while the other 10 speaks English and have passed LSAT.
Revised: 10 of them speaks Chinese as native language and have passed CNCSE, while the other 10 speaks English and have passed LSAT.

Original: Each imitative example was presented to 3 experts randomly, who are allowed to select one and only
Revised: Each imitative example was presented to 3 experts randomly, who are allowed to select one and only

Original: choice from “ A ”, “ B ”, “ C ”, “ D ”, otherwise, “ UNABLE TO ANSWER ” if bugs exist in the example, causing no correct choice or multiple correct choices.
Revised: choice from “ A ”, “ B ”, “ C ”, “ D ”, otherwise, “ UNABLE TO ANSWER ” if bugs exist in the example, causing no correct choice or multiple correct choices.

Original: As long as one of the 3 experts pointed out “ UNABLE TO ANSWER ”, then the imitator of this problem should recheck the logic, until each of these 3 experts could give a choice from “ A ”, “ B ”, “ C ”, “ D ”.
Revised: As long as one of the 3 experts pointed out “ UNABLE TO ANSWER ”, then the imitator of this problem should recheck the logic, until each of these 3 experts could give a choice from “ A ”, “ B ”, “ C ”, “ D ”.

Original: Note that in the post checking process, we broke up the imitations together and shufﬂe randomly, otherwise if a person is faced with imitations from same original example, he/she is prone to give a shortcut option with speculation.
Revised: Note that in the post checking process, we broke up the imitations together and shufﬂe randomly, otherwise if a person is faced with imitations from same original example, he/she is prone to give a shortcut option with speculation.

Original: Translation Quality Control After collecting high-quality mono-collections, we ﬁrst adopted
Revised: Translation Quality Control After collecting high-quality mono-collections, we ﬁrst adopted

Original: Google Translation to translate Chinese/English collections into another language, and then employed 10 professional bilingual experts in Chinese and English for manual correction.
Revised: Google Translation to translate Chinese/English collections into another language, and then employed 10 professional bilingual experts in Chinese and English for manual correction.

Original: Bilingual experts were asked to pay attention to logic-invariance and faithfulness during translation.
Revised: Bilingual experts were asked to pay attention to logic-invariance and faithfulness during translation.

Original: Next, to ensure translation quality, we also adopted the post checking strategy.
Revised: Next, to ensure translation quality, we also adopted the post checking strategy.

Original: That is, we asked the 20 human experts mentioned above to solve the translated examples.
Revised: That is, we asked the 20 human experts mentioned above to solve the translated examples.

Original: Each translated example was presented to 3 experts randomly.
Revised: Each translated example was presented to 3 experts randomly.

Original: Since human experts excel in solving na¨ıve logical reasoning problems, (i.e. achieve 100% accuracy on mono-collections), if any expert made a mistake on a translated sample or pointed out “ UNABLE TO ANSWER ”, the translated instance is sent back to the bilingual experts for revision.
Revised: Since human experts excel in solving na¨ıve logical reasoning problems, (i.e. achieve 100% accuracy on mono-collections), if any expert made a mistake on a translated sample or pointed out “ UNABLE TO ANSWER ”, the translated instance is sent back to the bilingual experts for revision.

Original: Finally, we asked 50 native speakers to read through all paragraphs of the translation parts in NAIL and mark “0”/“1” for each, where “1” stands for a translated sample is idiomatic, and “0” otherwise.
Revised: Finally, we asked 50 native speakers to read through all paragraphs of the translation parts in NAIL and mark “0”/“1” for each, where “1” stands for a translated sample is idiomatic, and “0” otherwise.

Original: Then for all samples marked with “0” (about 20%), the bilingual experts and native speakers will work together to polish them and conform to the target language norms.
Revised: Then for all samples marked with “0” (about 20%), the bilingual experts and native speakers will work together to polish them and conform to the target language norms.

Original: Human Evaluation As mentioned above, an imitation is ﬁnally regarded as qualiﬁed only if the sampled 3 experts could all solve the example.
Revised: Human Evaluation As mentioned above, an imitation is ﬁnally regarded as qualiﬁed only if the sampled 3 experts could all solve the example.

Original: For any original example in N AIL -E, we also ask 3 experts in the team to solve it.
Revised: For any original example in N AIL -E, we also ask 3 experts in the team to solve it.

Original: Since the gold answers to examples in N AIL -E are provided in public by the examination committee, and corresponding imitations in N AIL -I share the same answer with the original example.
Revised: Since the gold answers to examples in N AIL -E are provided in public by the examination committee, and corresponding imitations in N AIL -I share the same answer with the original example.

Original: Therefore, we calculated the mean accuracy of these three submissions on the overall N AIL , denoted as the performance of human experts .
Revised: Therefore, we calculated the mean accuracy of these three submissions on the overall N AIL , denoted as the performance of human experts .

Original: To better demonstrate the
Revised: To better demonstrate the

Original: [Table]
Revised: [Table]

Original: 一年级学生是农民工子弟，而农民工子弟的户籍都不在本市。
Revised: 一年级学生是农民工子弟，而农民工子弟的户籍都不在本市。

Original: (An elementary school in City S has complicated Hukou (a systemof household registration in China) issues.
Revised: (An elementary school in City S has complicated Hukou (a systemof household registration in China) issues.

Original: All students in grade 3 are registered in the City S.
Revised: All students in grade 3 are registered in the City S.

Original: Some of the students in grade 2 are registered in the City S.
Revised: Some of the students in grade 2 are registered in the City S.

Original: The parents of some students in grade 1 are migrant workers, who are not registered in thecityS. ) Q : 据此，可以推出： (Whichof thefollowingstatementscanbeinferredfromthematerial?)
Revised: The parents of some students in grade 1 are migrant workers, who are not registered in thecityS. ) Q : 据此，可以推出： (Whichof thefollowingstatementscanbeinferredfromthematerial?)

Original: A. 所有二年级学生都不是农民工子弟 (all studentsingrade2arenot childrenof migrant workers)
Revised: A. 所有二年级学生都不是农民工子弟 (all studentsingrade2arenot childrenof migrant workers)

Original: B. 有些农民工子弟是三年级学生 (somemigrant workers' childrenareingrade3) C. 有些户籍在本市的学生是三年级学生 (somestudentsregisteredinthiscityareingrade3)
Revised: B. 有些农民工子弟是三年级学生 (somemigrant workers' childrenareingrade3) C. 有些户籍在本市的学生是三年级学生 (somestudentsregisteredinthiscityareingrade3)

Original: D. 有些一年级学生不是农民工子弟 (somestudentsingrade1arenot childrenof migrant workers)
Revised: D. 有些一年级学生不是农民工子弟 (somestudentsingrade1arenot childrenof migrant workers)

Original: SetOperation (14.31%) difﬁculty of N AIL , we also selected another team consisting of 20 ﬁrst-year college students.
Revised: SetOperation (14.31%) difﬁculty of N AIL , we also selected another team consisting of 20 ﬁrst-year college students.

Original: Same as above, an example is shown to 3 students randomly and they have to give the answer independently.
Revised: Same as above, an example is shown to 3 students randomly and they have to give the answer independently.

Original: We calculated the overall mean accuracy as the performance of human baseline .The
Revised: We calculated the overall mean accuracy as the performance of human baseline .The

Original: evaluation of human baseline is paid separately.
Revised: evaluation of human baseline is paid separately.

Original: Each person can receive RMB¥1.5 for answering each example, which generally cost the person 2-3 minutes.
Revised: Each person can receive RMB¥1.5 for answering each example, which generally cost the person 2-3 minutes.

Original: Since we hire part-time people to write imitatively rather than using existing crowd-sourcing platforms, we build our own website, where the quality and overall progress can be viewed at any time.
Revised: Since we hire part-time people to write imitatively rather than using existing crowd-sourcing platforms, we build our own website, where the quality and overall progress can be viewed at any time.

Original: 3. D ATA A NALYSIS
Revised: 3. D ATA A NALYSIS

Original: As mentioned above, N AIL -E and N AIL -I together compose N AIL .
Revised: As mentioned above, N AIL -E and N AIL -I together compose N AIL .

Original: N AIL -E, N AIL -I and N AIL are divided into training set, validation set and test set respectively.
Revised: N AIL -E, N AIL -I and N AIL are divided into training set, validation set and test set respectively.

Original: The overall statistics of N AIL are summarized in Table 1.
Revised: The overall statistics of N AIL are summarized in Table 1.

Original: It is worth noting that an original example in N AIL -E and its corresponding imitations will only be in the same data split.
Revised: It is worth noting that an original example in N AIL -E and its corresponding imitations will only be in the same data split.

Original: We analyze and manually annotate the ﬁne-grained types of examples in the N AIL -E and group them into 4 categories including real/fake mixture, ordering, matching and set operation, whose percentages and representative examples are shown in
Revised: We analyze and manually annotate the ﬁne-grained types of examples in the N AIL -E and group them into 4 categories including real/fake mixture, ordering, matching and set operation, whose percentages and representative examples are shown in

Original: Figure 3.
Revised: Figure 3.

Original: Each of these types of examples requires na¨ıve logical reasoning.
Revised: Each of these types of examples requires na¨ıve logical reasoning.

Original: [Table]
Revised: [Table]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We adopt the accuracy as the evaluation metric.
Revised: We adopt the accuracy as the evaluation metric.

Original: Simple rule-based methods and strong neuralbased methods are included as our baseline.
Revised: Simple rule-based methods and strong neuralbased methods are included as our baseline.

Original: Rule-based methods involve text matching and sliding window .
Revised: Rule-based methods involve text matching and sliding window .

Original: Neural-based methods include BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019).
Revised: Neural-based methods include BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019).

Original: Detailed descriptions (e
Revised: Detailed descriptions (e

Original: [Table]
Revised: [Table]

Original: g., hyper-parameters) of the baseline models and the results on the validation set are listed in the Appendix A.
Revised: g., hyper-parameters) of the baseline models and the results on the validation set are listed in the Appendix A.

Original: 4. R ESULTS AND A NALYSIS
Revised: 4. R ESULTS AND A NALYSIS

Original: [Table]
Revised: [Table]

Original: The performance of all baselines on the N AIL is presented in Table 2.
Revised: The performance of all baselines on the N AIL is presented in Table 2.

Original: In particular, the human baseline is 71.
Revised: In particular, the human baseline is 71.

Original: and 76.4 on the test set of N AIL and
Revised: and 76.4 on the test set of N AIL and

Original: Chinese N AIL separately, while the human expert is 100.0 since ambiguous examples are not included in the dataset.
Revised: Chinese N AIL separately, while the human expert is 100.0 since ambiguous examples are not included in the dataset.

Original: In comparison, all of the baselines perform much worse than humans, demonstrating that these methods are very limited in na ¨ ıve logical reasoning.
Revised: In comparison, all of the baselines perform much worse than humans, demonstrating that these methods are very limited in na ¨ ıve logical reasoning.

Original: In addition, the results are relatively similar on the English and Chinese datasets.
Revised: In addition, the results are relatively similar on the English and Chinese datasets.

Original: The rule-based approaches achieve an accuracy of 25.8 and 24.2, which is similar to random guess, indicating that word correlation can not be used to help improve performance.
Revised: The rule-based approaches achieve an accuracy of 25.8 and 24.2, which is similar to random guess, indicating that word correlation can not be used to help improve performance.

Original: Pre-trained models have relatively good performance with about 4-10 points improvement compared to the rule-based approaches, showing that pre-trained models have a certain degree of commonsense and logical reasoning capabilities (Huang et al., 2019b).
Revised: Pre-trained models have relatively good performance with about 4-10 points improvement compared to the rule-based approaches, showing that pre-trained models have a certain degree of commonsense and logical reasoning capabilities (Huang et al., 2019b).

Original: However, the best result by RoBERTa LARGE is 36.2 on the testing data of Chinese N AIL , which is still far below the human performance.
Revised: However, the best result by RoBERTa LARGE is 36.2 on the testing data of Chinese N AIL , which is still far below the human performance.

Original: This indicates that the knowledge in the pre-trained model is quite weak in logical reasoning.
Revised: This indicates that the knowledge in the pre-trained model is quite weak in logical reasoning.

Original: Different Input Settings We conduct experiments with different input settings.
Revised: Different Input Settings We conduct experiments with different input settings.

Original: The setting of questions and answer options (Q, A) does not lead to signiﬁcant improvements compared to the input setting of only answer options (A).
Revised: The setting of questions and answer options (Q, A) does not lead to signiﬁcant improvements compared to the input setting of only answer options (A).

Original: One likely reason is that the queries usually do not provide much information, e.g.,
Revised: One likely reason is that the queries usually do not provide much information, e.g.,

Original: According to this, it can be deduced that?
Revised: According to this, it can be deduced that?

Original: Further adding context yields a noticeable boost, showing that the context is highly informative.
Revised: Further adding context yields a noticeable boost, showing that the context is highly informative.

Original: Transfer Learning We conduct a set of transfer learning experiments to understand the degree of overlap in terms of necessary knowledge for solving problems in our dataset and existing datasets.
Revised: Transfer Learning We conduct a set of transfer learning experiments to understand the degree of overlap in terms of necessary knowledge for solving problems in our dataset and existing datasets.

Original: What are the results if pre-trained model is ﬁrst trained on existing reading comprehension datasets, and then ﬁne-tuned on N AIL ?
Revised: What are the results if pre-trained model is ﬁrst trained on existing reading comprehension datasets, and then ﬁne-tuned on N AIL ?

Original: Table 3 shows the results where LogiQA, ReClor and RACE are adopted.
Revised: Table 3 shows the results where LogiQA, ReClor and RACE are adopted.

Original: 7 Overall, we observe that when LogiQA, ReClor or RACE is regarded as extra training resource for RoBERTa, the performance on N AIL will increase (30.1 → 34.5/33.4/31.0), since models can learn some degree of general reasoning ability when learning other comprehension tasks.
Revised: 7 Overall, we observe that when LogiQA, ReClor or RACE is regarded as extra training resource for RoBERTa, the performance on N AIL will increase (30.1 → 34.5/33.4/31.0), since models can learn some degree of general reasoning ability when learning other comprehension tasks.

Original: While interestingly, if models are trained only on LogiQA, ReClor, or RACE, and then zero-shot to directly test on NAIL, the performance is poor and close to random guess.
Revised: While interestingly, if models are trained only on LogiQA, ReClor, or RACE, and then zero-shot to directly test on NAIL, the performance is poor and close to random guess.

Original: And as for RACE, the zero-shot performance on N AIL (22.0) is even far below than that of RoBERTa (23.4).
Revised: And as for RACE, the zero-shot performance on N AIL (22.0) is even far below than that of RoBERTa (23.4).

Original: We attribute this to the
Revised: We attribute this to the

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: fact that, na¨ıve logical reasoning is a typical category of reasoning which should be signiﬁcantly distinguished from RACE.
Revised: fact that, na¨ıve logical reasoning is a typical category of reasoning which should be signiﬁcantly distinguished from RACE.

Original: Fine-tuning on RACE makes the parameters ﬁt the RACE data, which can lead to side effects on N AIL .
Revised: Fine-tuning on RACE makes the parameters ﬁt the RACE data, which can lead to side effects on N AIL .

Original: We believe that completely different categories of reasoning deserve to be explored separately since they may have different forms or strategies of reasoning, especially when training data is insufﬁcient.
Revised: We believe that completely different categories of reasoning deserve to be explored separately since they may have different forms or strategies of reasoning, especially when training data is insufﬁcient.

Original: What are the results if N AIL is used as extra training resource for existing reading comprehension tasks?
Revised: What are the results if N AIL is used as extra training resource for existing reading comprehension tasks?

Original: Figure 4 shows answers to this question.
Revised: Figure 4 shows answers to this question.

Original: Generally, we can draw the conclusion that using
Revised: Generally, we can draw the conclusion that using

Original: N AIL as a pre-training step can signiﬁcantly improve the supervised-learning performance for other tasks, such as LogiQA (35.3 → 36.9 for test), ReClor (62.6 → 64.4), 8 and RACE (83.2 → 85.2).
Revised: N AIL as a pre-training step can signiﬁcantly improve the supervised-learning performance for other tasks, such as LogiQA (35.3 → 36.9 for test), ReClor (62.6 → 64.4), 8 and RACE (83.2 → 85.2).

Original: This indicates that N AIL can bring na¨ıve logical reasoning ability to the model, which is a basic reasoning ability and can be reﬂected into other comprehension tasks.
Revised: This indicates that N AIL can bring na¨ıve logical reasoning ability to the model, which is a basic reasoning ability and can be reﬂected into other comprehension tasks.

Original: An interesting thing is that for zero-shot evaluating on RACE, N AIL seems to have a side effect in the pretraining process (22.6 < 27.8).
Revised: An interesting thing is that for zero-shot evaluating on RACE, N AIL seems to have a side effect in the pretraining process (22.6 < 27.8).

Original: This is because of a similar reason as mentioned above, that N AIL and RACE consist of completely different categories of examples.
Revised: This is because of a similar reason as mentioned above, that N AIL and RACE consist of completely different categories of examples.

Original: Results of more datasets are shown in Appendix D.2.
Revised: Results of more datasets are shown in Appendix D.2.

Original: Fine-grained Types We further analyze the model performance with respect to different types of na ¨ ıve logical reasoning (Figure 5).
Revised: Fine-grained Types We further analyze the model performance with respect to different types of na ¨ ıve logical reasoning (Figure 5).

Original: We ﬁnd that language models perform well on set operation problems, while struggle on matching and ordering.
Revised: We ﬁnd that language models perform well on set operation problems, while struggle on matching and ordering.

Original: We think that language models can provide good representation of set object, even if models do not really reason derived from the context.
Revised: We think that language models can provide good representation of set object, even if models do not really reason derived from the context.

Original: [Figure]
Revised: [Figure]

Original: RoBERTa
Revised: RoBERTa

Original: Error Analysis We further perform detailed analysis of human-baseline errors and model errors, while from Table 2, we can observe that human baseline on NAIL (around 70%) is much lower than that of human expert (100%), and RoBERTa performs much worse (above 30%).
Revised: Error Analysis We further perform detailed analysis of human-baseline errors and model errors, while from Table 2, we can observe that human baseline on NAIL (around 70%) is much lower than that of human expert (100%), and RoBERTa performs much worse (above 30%).

Original: We measure the accuracy against several factors on 
Revised: We measure the accuracy against several factors on context of

Original: See Figure 6(a), as the number of sentences in the backbone template decreases, the accuracy rate of human baseline increases signiﬁcantly, and when reduced to 2 and below, human baseline does
Revised: See Figure 6(a), as the number of sentences in the backbone template decreases, the accuracy rate of human baseline increases signiﬁcantly, and when reduced to 2 and below, human baseline does

Original: [Figure]
Revised: [Figure]

Original: not make any mistakes, while performance of RoBERTa does not show any trends, even when the number equals to 1, the model still makes mistakes in a large percentage of cases.
Revised: not make any mistakes, while performance of RoBERTa does not show any trends, even when the number equals to 1, the model still makes mistakes in a large percentage of cases.

Original: See Figure 6(b), as the number of possible worlds in the context increases, the accuracy of human baseline tends to decrease signiﬁcantly.
Revised: See Figure 6(b), as the number of possible worlds in the context increases, the accuracy of human baseline tends to decrease signiﬁcantly.

Original: This is because the more possible worlds that need to be considered, the more judgments humans need to perform, and humans tend to overlook certain conﬂicts, which leads to wrong decisions for problems.
Revised: This is because the more possible worlds that need to be considered, the more judgments humans need to perform, and humans tend to overlook certain conﬂicts, which leads to wrong decisions for problems.

Original: However there is no signiﬁcant correlation between the performance of the model and number of possible worlds.
Revised: However there is no signiﬁcant correlation between the performance of the model and number of possible worlds.

Original: This indicates that the model does not really ﬁlter and judge the possible worlds.
Revised: This indicates that the model does not really ﬁlter and judge the possible worlds.

Original: See Figure 6(c), the model makes mistakes at a variety of context lengths, while humans perform perfectly for problems with context length ≤ 73.
Revised: See Figure 6(c), the model makes mistakes at a variety of context lengths, while humans perform perfectly for problems with context length ≤ 73.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: There are abundant datasets in reading comprehension, which could facilitate the development of the ﬁeld.
Revised: There are abundant datasets in reading comprehension, which could facilitate the development of the ﬁeld.

Original: MCTest (Richardson et al., 2013) is a multiple-choice reading comprehension dataset that contains 500 ﬁctional stories and 2k questions.
Revised: MCTest (Richardson et al., 2013) is a multiple-choice reading comprehension dataset that contains 500 ﬁctional stories and 2k questions.

Original: Rajpurkar et al.
Revised: Rajpurkar et al.

Original: (2016) proposes the ﬁrst large-scale reading comprehension dataset SQuAD (100k+ questions), where the answer to each question is a span of text from the passage.
Revised: (2016) proposes the ﬁrst large-scale reading comprehension dataset SQuAD (100k+ questions), where the answer to each question is a span of text from the passage.

Original: Recently more datasets requiring more complicated reasoning types are introduced, such as multi-document (Joshi et al., 2017; Dunn et al., 2017), multi-hop reasoning (Yang et al., 2018;
Revised: Recently more datasets requiring more complicated reasoning types are introduced, such as multi-document (Joshi et al., 2017; Dunn et al., 2017), multi-hop reasoning (Yang et al., 2018;

Original: Welbl et al., 2018; Talmor & Berant, 2018), numerical discrete reasoning (Dua et al., 2019) and commonsense reasoning (Mihaylov et al., 2018; Zhang et al., 2018; Huang et al., 2019a).
Revised: Welbl et al., 2018; Talmor & Berant, 2018), numerical discrete reasoning (Dua et al., 2019) and commonsense reasoning (Mihaylov et al., 2018; Zhang et al., 2018; Huang et al., 2019a).

Original: However, these datasets cannot test the logical reasoning ability of the models.
Revised: However, these datasets cannot test the logical reasoning ability of the models.

Original: To ﬁll this gap,
Revised: To ﬁll this gap,

Original: LogiQA (Liu et al., 2020), ReClor (Yu et al., 2020) and LR-LSAT (Wang et al., 2021) was proposed.
Revised: LogiQA (Liu et al., 2020), ReClor (Yu et al., 2020) and LR-LSAT (Wang et al., 2021) was proposed.

Original: LogiQA is collected from National Civil Servants Examination.
Revised: LogiQA is collected from National Civil Servants Examination.

Original: ReClor and LR-LSAT are extracted from Law School Admission Test.
Revised: ReClor and LR-LSAT are extracted from Law School Admission Test.

Original: These datasets require logical reasoning to answer the questions.
Revised: These datasets require logical reasoning to answer the questions.

Original: However, from human experience, there are different forms of reasoning strategies in answering different logical questions.
Revised: However, from human experience, there are different forms of reasoning strategies in answering different logical questions.

Original: We believe that completely different categories of logical reasoning deserve to be explored separately (Rudinger et al., 2020).
Revised: We believe that completely different categories of logical reasoning deserve to be explored separately (Rudinger et al., 2020).

Original: Different from these datasets, we inductively deﬁne a typical class of logical reasoning, named na¨ıve logical reasoning, and then create a new benchmark targeting the task, named N AIL .
Revised: Different from these datasets, we inductively deﬁne a typical class of logical reasoning, named na¨ıve logical reasoning, and then create a new benchmark targeting the task, named N AIL .

Original: Compared with LogiQA, Reclor and LR-LSAT, N AIL focus on the more ﬁne-grained logical reasoning type (na¨ıve logical reasoning).
Revised: Compared with LogiQA, Reclor and LR-LSAT, N AIL focus on the more ﬁne-grained logical reasoning type (na¨ıve logical reasoning).

Original: There have been many datasets extracted from human examinations, such as LogiQA and ReClor mentioned above.
Revised: There have been many datasets extracted from human examinations, such as LogiQA and ReClor mentioned above.

Original: Besides, RACE dataset (Lai et al., 2017) is collected from English exams for middle and high school Chinese students.
Revised: Besides, RACE dataset (Lai et al., 2017) is collected from English exams for middle and high school Chinese students.

Original: ARC dataset (Clark et al., 2018) consists of 7,787 science exam questions drawn from a variety of sources.
Revised: ARC dataset (Clark et al., 2018) consists of 7,787 science exam questions drawn from a variety of sources.

Original: DREAM (Sun et al., 2019) is dialogue-based multiplechoice reading comprehension dataset collected from English as a Foreign Language examinations which contains 10,197 questions for 6,444 dialogues.
Revised: DREAM (Sun et al., 2019) is dialogue-based multiplechoice reading comprehension dataset collected from English as a Foreign Language examinations which contains 10,197 questions for 6,444 dialogues.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We introduce a more ﬁne-grained logical reasoning, na¨ıve logical reasoning, and We propose a new large-scale benchmark, N AIL , aiming to help models learn and evaluate na¨ıve logical reasoning capability.
Revised: We introduce a more ﬁne-grained logical reasoning, na¨ıve logical reasoning, and We propose a new large-scale benchmark, N AIL , aiming to help models learn and evaluate na¨ıve logical reasoning capability.

Original: N AIL is sourced from standardized exams and human imitation.
Revised: N AIL is sourced from standardized exams and human imitation.

Original: Preliminary results show that there is still a long way to go to equip deep models with true logical reasoning capability.
Revised: Preliminary results show that there is still a long way to go to equip deep models with true logical reasoning capability.

Original: A I MPLEMENTATION D ETAILS OF B ASELINES
Revised: A I MPLEMENTATION D ETAILS OF B ASELINES

Original: Rule-based Methods Following LogiQA (Liu et al., 2020)
Revised: Rule-based Methods Following LogiQA (Liu et al., 2020)

Original: , we adopt two simple rule-based methods based on text matching.
Revised: , we adopt two simple rule-based methods based on text matching.

Original: Speciﬁcally, word matching is to measure the degree of unigram overlap between the candidate answer and the given paragraph-query pair; sliding window takes into account the n-gram when calculating the matching score.
Revised: Speciﬁcally, word matching is to measure the degree of unigram overlap between the candidate answer and the given paragraph-query pair; sliding window takes into account the n-gram when calculating the matching score.

Original: Neural-based Methods Pre-trained neural models, such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), have achieved impressive performance on reading comprehension.
Revised: Neural-based Methods Pre-trained neural models, such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), have achieved impressive performance on reading comprehension.

Original: The input to the pre-trained model is the concatenation of the paragraph, the query and the candidate answer, separated by [SEP] tokens, denoted as: [CLS] , paragraph, [SEP] , query, [SEP] , answer.
Revised: The input to the pre-trained model is the concatenation of the paragraph, the query and the candidate answer, separated by [SEP] tokens, denoted as: [CLS] , paragraph, [SEP] , query, [SEP] , answer.

Original: After the encoding of the pre-trained model, the representation of [CLS] token followed by an multiple layer perceptron (MLP) is used for scoring.
Revised: After the encoding of the pre-trained model, the representation of [CLS] token followed by an multiple layer perceptron (MLP) is used for scoring.

Original: We re-implement the rule-based methods following LogiQA (Liu et al., 2020).
Revised: We re-implement the rule-based methods following LogiQA (Liu et al., 2020).

Original: For pre-trained models, we modify the code of Transformers of HuggingFace (Wolf et al., 2020) to implement them on N AIL .
Revised: For pre-trained models, we modify the code of Transformers of HuggingFace (Wolf et al., 2020) to implement them on N AIL .

Original: We take the off-the-shelf model BERT and RoBERTa for N AIL , and Chinese BERT and
Revised: We take the off-the-shelf model BERT and RoBERTa for N AIL , and Chinese BERT and

Original: Chinese RoBERTa (Cui et al., 2019) for Chinese N AIL .
Revised: Chinese RoBERTa (Cui et al., 2019) for Chinese N AIL .

Original: All hyper-parameters are selected by the model performance on the development sets.
Revised: All hyper-parameters are selected by the model performance on the development sets.

Original: B M ORE S TRONG B ASELINES
Revised: B M ORE S TRONG B ASELINES

Original: We also try seq2seq-based pre-trained models (i.e. T5 (Raffel et al., 2019)) and well-designed models for other related tasks (i.e. DAGN (Huang et al., 2021) for LogiQA and Reclor, HGN (Fang et al., 2020) for HotpotQA (Yang et al., 2018), and QDGAT (Chen et al., 2020) for DROP (Dua et al., 2019)) on our English version of N AIL .
Revised: We also try seq2seq-based pre-trained models (i.e. T5 (Raffel et al., 2019)) and well-designed models for other related tasks (i.e. DAGN (Huang et al., 2021) for LogiQA and Reclor, HGN (Fang et al., 2020) for HotpotQA (Yang et al., 2018), and QDGAT (Chen et al., 2020) for DROP (Dua et al., 2019)) on our English version of N AIL .

Original: A problem is regarded as four samples to T5, the input of a sample is: context+query+one of the four choices.
Revised: A problem is regarded as four samples to T5 during training. The input of a sample is: context+query+one of the four choices.

Original: And we let the modelto generate  label for the sample: if the choice is the correct answer, then generate “True”, otherwise generate “False”.
Revised: And we let the model generate the label for the sample: if the choice is the correct answer, then generate “True”, otherwise generate “False”.

Original: And during inference, for a problem, we choose the option with the highest probability of generating “True” among the four options as the prediction.
Revised: And during inference, for a problem, we choose the option with the highest probability of generating “True” among the four options as the prediction.

Original: We set input max len = 512 and ﬁnetuned the off-the-shelf T5-base model on the training set of N AIL , results are shown in Table 4.
Revised: We set input max len = 512 and ﬁnetuned the off-the-shelf T5-base model on the training set of N AIL , results are shown in Table 4.

Original: LogiQA and ReClor are two logical benchmarks introduced in the text of the paper to test models’ various reasoning abilities.
Revised: LogiQA and ReClor are two logical benchmarks introduced in the text of the paper to test models’ various reasoning abilities.

Original: HotpotQA is a multiple-choice QA dataset featuring natural, multi-hop questions, with strong supervision for supporting facts to test models’ multi-hop factual reasoning ability.
Revised: HotpotQA is a multiple-choice QA dataset featuring natural, multi-hop questions, with strong supervision for supporting facts to test models’ multi-hop factual reasoning ability.

Original: DROP is a 96k-question benchmark to test models’ numerical discrete reasoning ability over paragraphs (such as addition, counting, or sorting).
Revised: DROP is a 96k-question benchmark to test models’ numerical discrete reasoning ability over paragraphs (such as addition, counting, or sorting).

Original: We choose three state-of-the-art and open-sourced models designed for the above benchmarks: DAGN for LogiQA and Reclor, HGN for HotpotQA, and QDGAT for DROP respectively.
Revised: We choose three state-of-the-art and open-sourced models designed for the above benchmarks: DAGN for LogiQA and Reclor, HGN for HotpotQA, and QDGAT for DROP respectively.

Original: We reproduced these three models with the code released by authors 10 and N AIL as training data.
Revised: We reproduced these three models with the code released by authors 10 and N AIL as training data.

Original: Results on N AIL are shown as Table 4.
Revised: Results on N AIL are shown as Table 4.

Original: [Table]
Revised: [Table]

Original: We observe that DAGN achieves the best results on N AIL , since ReClor and LogiQA are much closer to N AIL , the well-designed discourse-aware graph network DAGN is more applicable to
Revised: We observe that DAGN achieves the best results on N AIL , since ReClor and LogiQA are much closer to N AIL , the well-designed discourse-aware graph network DAGN is more applicable to

Original: N AIL .
Revised: N AIL .

Original: However there is still very much space for models’ improvement compared to humans.
Revised: However there is still very much space for models’ improvement compared to humans.

Original: HGN is a hierarchical graph network for multi-hop question answering, which is designed to aggregate heterogeneous information, and QDGAT is a question directed graph attention network, which is dedicated to identifying numbers and the computation between them, which are both of minimal help to N AIL .
Revised: HGN is a hierarchical graph network for multi-hop question answering, which is designed to aggregate heterogeneous information, and QDGAT is a question directed graph attention network, which is dedicated to identifying numbers and the computation between them, which are both of minimal help to N AIL .

Original: T5 is a very powerful generative model, but it seems that T5 lacks the na¨ıve reasoning capability as well.
Revised: T5 is a very powerful generative model, but it seems that T5 lacks the na¨ıve reasoning capability as well.

Original: Due to computational resource issues, we did not try a T5 model with larger number of parameters.
Revised: Due to computational resource limitation, we did not try a T5 model with larger number of parameters.

Original: C T HE EFFECT OF N AIL , N AIL -E AND N AIL -I
Revised: C T HE EFFECT OF N AIL , N AIL -E AND N AIL -I

Original: To investigate the effect of N AIL , N AIL -E and N AIL -I, we train a RoBERTa LARGE model on N AIL ,
Revised: To investigate the effect of N AIL , N AIL -E and N AIL -I, we train a RoBERTa LARGE model on N AIL ,

Original: N AIL -E and N AIL -I separately and evaluate on the corresponding set of all tests (Table 5).
Revised: N AIL -E and N AIL -I separately and evaluate on the corresponding set of all tests (Table 5).

Original: When trained on N AIL -I and evaluated on N AIL -E, the model achieves the best accuracy 38.5%, showing that human imitations can signiﬁcantly help models learn logic from natural text.
Revised: When trained on N AIL -I and evaluated on N AIL -E, the model achieves the best accuracy 38.5%, showing that human imitations can signiﬁcantly help models learn logic from natural text.

Original: When trained on
Revised: When trained on

Original: N AIL -E and evaluated on N AIL -E, the model achieves the worst accuracy 22.9%.
Revised: N AIL -E and evaluated on N AIL -E, the model achieves the worst accuracy 22.9%.

Original: The possible reason is that the amount of data of N AIL -E is too small to train a deep model.
Revised: The possible reason is that the amount of data of N AIL -E is too small to train a deep model.

Original: Corresponding results on the validation set are shown in Table 6.
Revised: Corresponding results on the validation set are shown in Table 6.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: D T RANSFER L EARNING D. O THER D ATASETS AS E XTRA T RAINING R ESOURCE
Revised: D T RANSFER L EARNING D. O THER D ATASETS AS E XTRA T RAINING R ESOURCE

Original: What are the results if pre-trained model is ﬁrst trained on existing reading comprehension datasets, and then ﬁne-tuned on N AIL ?
Revised: What are the results if pre-trained model is ﬁrst trained on existing reading comprehension datasets, and then ﬁne-tuned on N AIL ?

Original: Table 3 shows the results on the test set of N AIL ,N AIL -E, N AIL -I where LogiQA, ReClor 11 and RACE are adopted.
Revised: Table 3 shows the results on the test set of N AIL ,N AIL -E, N AIL -I where LogiQA, ReClor 11 and RACE are adopted.

Original: And here in Table 7 we give the results on the validation splits of N AIL ,N AIL -E, N AIL -I.
Revised: And here in Table 7 we give the results on the validation splits of N AIL ,N AIL -E, N AIL -I.

Original: D. N AIL AS E XTRA T RAINING R ESOURCE
Revised: D. N AIL AS E XTRA T RAINING R ESOURCE

Original: What are the results if N AIL is used as extra training resource for existing reading comprehension tasks?
Revised: What are the results if N AIL is used as extra training resource for existing reading comprehension tasks?

Original: Using NAIL as extra training resource, we conduct plenty of experiments on existing benchmarks.
Revised: Using NAIL as extra training resource, we conduct plenty of experiments on existing benchmarks.

Original: Besides LogiQA, ReClor and RACE (see in Figure 4), more results are shown in
Revised: Besides LogiQA, ReClor and RACE (see in Figure 4), more results are shown in

Original: [Table]
Revised: [Table]

Original: Table 8, 9.
Revised: Table 8, 9.

Original: We adopted MathQA (Amini et al., 2019), and HotpotQA (Yang et al., 2018).
Revised: We adopted MathQA (Amini et al., 2019), and HotpotQA (Yang et al., 2018).

Original: For
Revised: For

Original: MathQA, we include a prediction head for multiple choice based on the RoBERTa-large model, and for HotpotQA, 12 we include a prediction head for question answering.
Revised: MathQA, we include a prediction head for multiple choice based on the RoBERTa-large model, and for HotpotQA, 12 we include a prediction head for question answering.

Original: For MathQA, we concat the problem and annotated formula as input.
Revised: For MathQA, we concat the problem and annotated formula as input.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Experimental results show that, if we use N AIL as extra training resource, the supervised-learning results on all of these six datasets: LogiQA, ReClor, RACE, MathQA, HotpotQA will improve.
Revised: Experimental results show that, if we use N AIL as extra training resource, the supervised-learning results on all of these six datasets: LogiQA, ReClor, RACE, MathQA, HotpotQA will improve.

Original: This indicates that equipping models with na¨ıve logical reasoning ability can help solve math word problems (MathQA), improve multi-hop factual reasoning skills (HotpotQA), solve various logical reasoning problems (LogiQA and ReClor), and enhance general understanding and reading comprehension capability (RACE).
Revised: This indicates that equipping models with na¨ıve logical reasoning ability can help solve math word problems (MathQA), improve multi-hop factual reasoning skills (HotpotQA), solve various logical reasoning problems (LogiQA and ReClor), and enhance general understanding and reading comprehension capability (RACE).

Original: In the future, we will test on more known datasets, to verify that na¨ıve logical reasoning is a basic capability and is helpful for other tasks.
Revised: In the future, we will test on more known datasets, to verify that na¨ıve logical reasoning is a basic capability and is helpful for other tasks.

Original: E A NALYSIS OF F INE - GRAINED T YPES
Revised: E A NALYSIS OF F INE - GRAINED T YPES

Original: In Section 4, we analyze the model performance with respect to different types of na¨ıve logical reasoning (Figure 5).
Revised: In Section 4, we analyze the model performance with respect to different types of na¨ıve logical reasoning (Figure 5).

Original: Figure 8 shows the accuracy against ﬁne-grained reasoning types on the test set of N AIL , N AIL -E, N AIL -I when training on N AIL , N AIL -E, N AIL -I respectively.
Revised: Figure 8 shows the accuracy against ﬁne-grained reasoning types on the test set of N AIL , N AIL -E, N AIL -I when training on N AIL , N AIL -E, N AIL -I respectively.

Original: And Figure 7 shows the corresponding results on the validation set.
Revised: And Figure 7 shows the corresponding results on the validation set.

Original: From above ﬁgures we can ﬁnd that language models perform well on set operation problems, while struggle on matching and ordering.
Revised: From above ﬁgures we can ﬁnd that language models perform well on set operation problems, while struggle on matching and ordering.

Original: We think that language models can provide good representation of set object, even if models do not really reason derived from the context.
Revised: We think that language models can provide good representation of set object, even if models do not really reason derived from the context.

Original: F D ISCUSSIONS ABOUT F UTURE D IRECTIONS
Revised: F D ISCUSSIONS ABOUT F UTURE D IRECTIONS

Original: We have some ideas for future directions for models to solve N AIL .
Revised: We have some ideas for future directions for models to solve N AIL .

Original: The ﬁrst point is to identify deterministic information or information with low uncertainty.
Revised: The ﬁrst point is to identify deterministic information or information with low uncertainty.

Original: Generally, we need to detect the atomic information that can be used directly without reasoning, which is the key to solving the problem.
Revised: Generally, we need to detect the atomic information that can be used directly without reasoning, which is the key to solving the problem.

Original: Different conditions should not be considered with equal priority,
Revised: Different conditions should not be considered with equal priority,

Original: R e a l / F a k e
Revised: R e a l / F a k e

Original: O r d e r i n g
Revised: O r d e r i n g

Original: M a t c h i n g
Revised: M a t c h i n g

Original: S e t o p e r a t i o n
Revised: S e t o p e r a t i o n

Original: A cc u r a c y
Revised: A cc u r a c y

Original: [Table]
Revised: [Table]

Original: BERTRoBERTa
Revised: BERTRoBERTa

Original: R e a l / F a
Revised: R e a l / F a

Original: [Table]
Revised: [Table]

Original: O r d e r i n g
Revised: O r d e r i n g

Original: M
Revised: M

Original: [Table]
Revised: [Table]

Original: S
Revised: S

Original: [Table]
Revised: [Table]

Original: p
Revised: p

Original: [Table]
Revised: [Table]

Original: R e a l / F a k e
Revised: R e a l / F a k e

Original: O r d e r i n g
Revised: O r d e r i n g

Original: M a t c h i n
Revised: M a t c h i n

Original: [Table]
Revised: [Table]

Original: e t
Revised: e t

Original: [Table]
Revised: [Table]

Original: p
Revised: p

Original: [Table]
Revised: [Table]

Original: t i o
Revised: t i o

Original: [Table]
Revised: [Table]

Original: A cc u r a c y
Revised: A cc u r a c y

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: R e a l / F a k e
Revised: R e a l / F a k e

Original: O r d e r i n g
Revised: O r d e r i n g

Original: M a t c h i n g
Revised: M a t c h i n g

Original: S e t o p e r a t i o n
Revised: S e t o p e r a t i o n

Original: [Figure]
Revised: [Figure]

Original: A
Revised: A

Original: [Figure]
Revised: [Figure]

Original: u
Revised: u

Original: [Figure]
Revised: [Figure]

Original: R e a l / F a k e
Revised: R e a l / F a k e

Original: O r d e r i n g
Revised: O r d e r i n g

Original: M a t c h i n g
Revised: M a t c h i n g

Original: S e t o p e r a t i o n
Revised: S e t o p e r a t i o n

Original: [Figure]
Revised: [Figure]

Original: e
Revised: e

Original: [Figure]
Revised: [Figure]

Original: l
Revised: l

Original: [Figure]
Revised: [Figure]

Original: a
Revised: a

Original: [Figure]
Revised: [Figure]

Original: O r d e r i n g
Revised: O r d e r i n g

Original: [Figure]
Revised: [Figure]

Original: a t c h
Revised: a t c h

Original: [Figure]
Revised: [Figure]

Original: n
Revised: n

Original: [Figure]
Revised: [Figure]

Original: e t o p e r a
Revised: e t o p e r a

Original: [Figure]
Revised: [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: n
Revised: n

Original: [Figure]
Revised: [Figure]

Original: t
Revised: t

Original: [Figure]
Revised: [Figure]

Original: n
Revised: n

Original: [Figure]
Revised: [Figure]

Original: S e t o p e r
Revised: S e t o p e r

Original: [Figure]
Revised: [Figure]

Original: i.e., the more certain a condition we start from, the fewer possible worlds that the condition yields.
Revised: i.e., the more certain a condition we start from, the fewer possible worlds that the condition yields.

Original: Speciﬁcally, for real/fake mixture problems, we also need to promptly identify two atomic statements that yield a contradictory, that is, the two statements cannot be both true (there must be one false) or both false (there must be one true), which is often the breakthrough in solving real/fake mixture problems.
Revised: Speciﬁcally, for real/fake mixture problems, we also need to promptly identify two atomic statements that yield a contradictory, that is, the two statements cannot be both true (there must be one false) or both false (there must be one true), which is often the breakthrough in solving real/fake mixture problems.

Original: The second is to detect informative subjects/objects.
Revised: The second is to detect informative subjects/objects.

Original: Generally, the more frequently a subject/object is mentioned in context, the more relevant information it carries.
Revised: Generally, the more frequently a subject/object is mentioned in context, the more relevant information it carries.

Original: Tables and bi(/tri)partite diagrams can help to clarify the correspondence between given subjects and objects.
Revised: Tables and bi(/tri)partite diagrams can help to clarify the correspondence between given subjects and objects.

Original: For example, in the case of Figure 1, tables play an effective role in solving the problems.
Revised: For example, in the case of Figure 1, tables play an effective role in solving the problems.

Original: And as for the third case in Figure 3, which is categorized as a matching problem, a tri-partite diagram could help, that is, three disjoint and independent sets U, V and W represent { A,B,C } , { Beijing, Nanjing, Chengdu } , { a doctor, an actor, and a teacher } respectively, an edge connecting a vertex in one set to one in another set denotes the “is a” predicate.
Revised: And as for the third case in Figure 3, which is categorized as a matching problem, a tri-partite diagram could help, that is, three disjoint and independent sets U, V and W represent { A,B,C } , { Beijing, Nanjing, Chengdu } , { a doctor, an actor, and a teacher } respectively, an edge connecting a vertex in one set to one in another set denotes the “is a” predicate.

Original: The third direction is allowing models to better utilize elimination.
Revised: The third direction is allowing models to better utilize elimination.

Original: On the one hand, models can perform forward elimination, i.e., according to the context, each time we draw a deﬁnite conclusion, we can retain options that are logically consistent and exclude those that do not ﬁt the derived conclusion.
Revised: On the one hand, models can perform forward elimination, i.e., according to the context, each time we draw a deﬁnite conclusion, we can retain options that are logically consistent and exclude those that do not ﬁt the derived conclusion.

Original: On the other hand, models can also perform backward elimination.
Revised: On the other hand, models can also perform backward elimination.

Original: For example, when sometimes it is not easy to draw exact inferences directly from the context, then we can substitute the options into the context.
Revised: For example, when sometimes it is not easy to draw exact inferences directly from the context, then we can substitute the options into the context.

Original: If substituting an option creates a contradiction within the context, then the option should be excluded.
Revised: If substituting an option creates a contradiction within the context, then the option should be excluded.

Original: Forward and backward elimination facilitate the model in arriving at the correct answer.
Revised: Forward and backward elimination facilitate the model in arriving at the correct answer.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: R e a l / F a k e
Revised: R e a l / F a k e

Original: O r d e r i n g
Revised: O r d e r i n g

Original: M a t c h i n g
Revised: M a t c h i n g

Original: S e t o p e r a t i o n
Revised: S e t o p e r a t i o n

Original: [Figure]
Revised: [Figure]

Original: A
Revised: A

Original: [Figure]
Revised: [Figure]

Original: u r a c y
Revised: u r a c y

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: r
Revised: r

Original: [Table]
Revised: [Table]

Original: n
Revised: n

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: R e a l
Revised: R e a l

Original: [Figure]
Revised: [Figure]

Original: F a k e
Revised: F a k e

Original: O r d e r i n g
Revised: O r d e r i n g

Original: M a t c h i n g
Revised: M a t c h i n g

Original: S e t o
Revised: S e t o

Original: [Figure]
Revised: [Figure]

Original: e r a t i
Revised: e r a t i

Original: [Figure]
Revised: [Figure]

Original: n
Revised: n

Original: [Figure]
Revised: [Figure]

Original: r
Revised: r

Original: [Figure]
Revised: [Figure]