Original: A BDUCTIVE C OMMONSENSE R EASONING
Revised: A BDUCTIVE C OMMONSENSE R EASONING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Abductive reasoning is inference to the most plausible explanation .
Revised: Abductive reasoning is inference to the most plausible explanation .

Original: For example, if Jenny ﬁnds her house in a mess when she returns from work, and remembers that she left a window open, she can hypothesize that a thief broke into her house and caused the mess, as the most plausible explanation.
Revised: For example, if Jenny ﬁnds her house in a mess when she returns from work, and remembers that she left a window open, she can hypothesize that a thief broke into her house and caused the mess, as the most plausible explanation.

Original: While abduction has long been considered to be at the core of how people interpret and read between the lines in natural language (Hobbs et al., 1988), there has been relatively little research in support of abductive natural language inference and generation.
Revised: While abduction has long been considered to be at the core of how people interpret and read between the lines in natural language (Hobbs et al., 1988), there has been relatively little research in support of abductive natural language inference and generation.

Original: We present the ﬁrst study that investigates the viability of language-based abductive reasoning.
Revised: We present the ﬁrst study that investigates the viability of language-based abductive reasoning.

Original: We introduce a challenge dataset, ART , that consists of over 20k commonsense narrative contexts and 200k explanations.
Revised: We introduce a challenge dataset, ART , that consists of over 20k commonsense narrative contexts and 200k explanations.

Original: Based on this dataset, we conceptualize two new tasks – (i)
Revised: Based on this dataset, we conceptualize two new tasks – (i)

Original: Abductive NLI : a multiple-choice question answering task for choosing the more likely explanation, and (ii)
Revised: Abductive NLI : a multiple-choice question answering task for choosing the more likely explanation, and (ii)

Original: Abductive NLG : a conditional generation task for explaining given observations in natural language.
Revised: Abductive NLG : a conditional generation task for explaining given observations in natural language.

Original: On Abductive NLI , the best model achieves 68.9% accuracy, well below human performance of 91.4%.
Revised: On Abductive NLI , the best model achieves 68.9% accuracy, well below human performance of 91.4%.

Original: On Abductive NLG , the current best language generators struggle even more, as they lack reasoning capabilities that are trivial for humans.
Revised: On Abductive NLG , the current best language generators struggle even more, as they lack reasoning capabilities that are trivial for humans.

Original: Our analysis leads to new insights into the types of reasoning that deep pre-trained language models fail to perform—despite their strong performance on the related but more narrowly deﬁned task of entailment NLI—pointing to interesting avenues for future research.
Revised: Our analysis leads to new insights into the types of reasoning that deep pre-trained language models fail to perform—despite their strong performance on the related but more narrowly deﬁned task of entailment NLI—pointing to interesting avenues for future research.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The brain is an abduction machine, continuously trying to prove abductively that the ob- servables in its environment constitute a coherent situation.
Revised: The brain is an abduction machine, continuously trying to prove abductively that the ob- servables in its environment constitute a coherent situation.

Original: – Jerry Hobbs, ACL 2013 Lifetime Achievement Award
Revised: – Jerry Hobbs, ACL 2013 Lifetime Achievement Award

Original: Abductive reasoning is inference to the most plausible explanation for incomplete observations
Revised: Abductive reasoning is inference to the most plausible explanation for incomplete observations

Original: Peirce (1965a).
Revised: Peirce (1965a).

Original: Figure 1 illustrates an example.
Revised: Figure 1 illustrates an example.

Original: Given the incomplete observations about the world that O 1 : “Jenny cleaned her house and went to work, leaving the window just a crack open.”
Revised: Given the incomplete observations about the world that O 1 : “Jenny cleaned her house and went to work, leaving the window just a crack open.”

Original: and sometime later O 2 : “When Jenny returned home, she saw her house was a mess.”
Revised: and sometime later O 2 : “When Jenny returned home, she saw her house was a mess.”

Original: , we can hypothesize different potential explanations and reason about which is the most likely.
Revised: , we can hypothesize different potential explanations and reason about which is the most likely.

Original: We can readily rule out H 3 since it fails to justify the observation O 2 .
Revised: We can readily rule out H 3 since it fails to justify the observation O 2 .

Original: While H 1 and H 2 are both plausible, the most likely explanation based on commonsense is H 1 as H 2 is somewhat implausible given O 1 .
Revised: While H 1 and H 2 are both plausible, the most likely explanation based on commonsense is H 1 as H 2 is somewhat implausible given O 1 .

Original: One crucial point Peirce makes about abductive reasoning is that abduction is “the only logical operation which introduces any new ideas” , which contrasts with other types of inference such as entailment, that focuses on inferring only such information that is already provided in the premise.
Revised: One crucial point Peirce makes about abductive reasoning is that abduction is “the only logical operation which introduces any new ideas” , which contrasts with other types of inference such as entailment, that focuses on inferring only such information that is already provided in the premise.

Original: Abductive reasoning has long been considered to be at the core of understanding narratives (Hobbs et al., 1988), reading between the lines (Norvig, 1987; Charniak & Shimony, 1990), reasoning about everyday situations (Peirce, 1965b; Andersen, 1973), and counterfactual reasoning (Pearl, 2002;
Revised: Abductive reasoning has long been considered to be at the core of understanding narratives (Hobbs et al., 1988), reading between the lines (Norvig, 1987; Charniak & Shimony, 1990), reasoning about everyday situations (Peirce, 1965b; Andersen, 1973), and counterfactual reasoning (Pearl, 2002;

Original: Pearl & Mackenzie, 2018).
Revised: Pearl & Mackenzie, 2018).

Original: Despite the broad recognition of its importance, however, the study of abductive reasoning in narrative text has very rarely appeared in the NLP literature, in large part because most previous work on abductive reasoning has focused on formal logic, which has proven to be too rigid to generalize to the full complexity of natural language.
Revised: Despite the broad recognition of its importance, however, the study of abductive reasoning in narrative text has very rarely appeared in the NLP literature, in large part because most previous work on abductive reasoning has focused on formal logic, which has proven to be too rigid to generalize to the full complexity of natural language.

Original: A thief broke into the house by pulling open the window .
Revised: A thief broke into the house by pulling open the window .

Original: The bird got stuck inside the house, flew around while trying to escape, and made a mess .
Revised: The bird got stuck inside the house, flew around while trying to escape, and made a mess .

Original: O
Revised: O

Original: O
Revised: O

Original: At work , she opened her window and the wind blew her papers everywhere .
Revised: At work , she opened her window and the wind blew her papers everywhere .

Original: H H
Revised: H H

Original: Jenny left an insecure opening to her house.
Revised: Jenny left an insecure opening to her house.

Original: Fails to Justify to O 2 .
Revised: Fails to Justify to O 2 .

Original: Although wind caused a mess, the event happened at Jenny’s workplace .
Revised: Although wind caused a mess, the event happened at Jenny’s workplace .

Original: The thief got into the house through the window and rifled through Jenny’s things, which made a mess.
Revised: The thief got into the house through the window and rifled through Jenny’s things, which made a mess.

Original: When Jenny returned home she saw that her house was a mess !
Revised: When Jenny returned home she saw that her house was a mess !

Original: Jenny cleaned her house and went to work, leaving the window just a crack open .
Revised: Jenny cleaned her house and went to work, leaving the window just a crack open .

Original: It was a breezy day and a large bird flew into the house.
Revised: It was a breezy day and a large bird flew into the house.

Original: t n tt
Revised: t n tt

Original: i
Revised: i

Original: H
Revised: H

Original: Likely to follow O 1 .
Revised: Likely to follow O 1 .

Original: Somewhat Unlikely.
Revised: Somewhat Unlikely.

Original: If the window was just a crack open , a large bird is unlikely to get in?
Revised: If the window was just a crack open , a large bird is unlikely to get in?

Original: In this paper, we present the ﬁrst study to investigate the viability of language-based abductive reasoning.
Revised: In this paper, we present the ﬁrst study to investigate the viability of language-based abductive reasoning.

Original: This shift from logic -based to language -based reasoning draws inspirations from a signiﬁcant body of work on language-based entailment (Bowman et al., 2015; Williams et al., 2018b), language-based logic (Lakoff, 1970; MacCartney & Manning, 2007), and language-based commonsense reasoning (Mostafazadeh et al., 2016;
Revised: This shift from logic -based to language -based reasoning draws inspirations from a signiﬁcant body of work on language-based entailment (Bowman et al., 2015; Williams et al., 2018b), language-based logic (Lakoff, 1970; MacCartney & Manning, 2007), and language-based commonsense reasoning (Mostafazadeh et al., 2016;

Original: Zellers et al., 2018).
Revised: Zellers et al., 2018).

Original: In particular, we investigate the use of natural language as the representation medium, and probe deep neural models on language-based abductive reasoning.
Revised: In particular, we investigate the use of natural language as the representation medium, and probe deep neural models on language-based abductive reasoning.

Original: More concretely, we propose Abductive Natural Language Inference ( α NLI) 2 and Abductive Natural
Revised: More concretely, we propose Abductive Natural Language Inference ( α NLI) 2 and Abductive Natural

Original: Language Generation ( α NLG) as novel reasoning tasks in narrative contexts.
Revised: Language Generation ( α NLG) as novel reasoning tasks in narrative contexts.

Original: We formulate α NLI as a multiple-choice task to support easy and reliable automatic evaluation: given a context, the task is to choose the more likely explanation from a given pair of hypotheses.
Revised: We formulate α NLI as a multiple-choice task to support easy and reliable automatic evaluation: given a context, the task is to choose the more likely explanation from a given pair of hypotheses.

Original: We also introduce a new challenge dataset, ART , that consists of 20K narratives accompanied by over 200K explanatory hypothesis.
Revised: We also introduce a new challenge dataset, ART , that consists of 20K narratives accompanied by over 200K explanatory hypothesis.

Original: We then establish comprehensive baseline performance based on state-of-the-art NLI and language models.
Revised: We then establish comprehensive baseline performance based on state-of-the-art NLI and language models.

Original: The best baseline for α NLI based on BERT achieves 68.9% accuracy, with a considerable gap compared to human performance of 91.4%( § 5.2).
Revised: The best baseline for α NLI based on BERT achieves 68.9% accuracy, with a considerable gap compared to human performance of 91.4%( § 5.2).

Original: The best generative model, based on GPT2, performs well below human performance on the α NLG task ( § 5.2).
Revised: The best generative model, based on GPT2, performs well below human performance on the α NLG task ( § 5.2).

Original: Our analysis leads to insights into the types of reasoning that deep pre-trained language models fail to perform — despite their strong performance on the closely related but different task of entailment NLI — pointing to future research directions.
Revised: Our analysis leads to insights into the types of reasoning that deep pre-trained language models fail to perform — despite their strong performance on the closely related but different task of entailment NLI — pointing to future research directions.

Original:  T ASK D EFINITION
Revised:  T ASK D EFINITION

Original: Abductive Natural Language Inference We formulate α NLI as multiple choice problems consisting of a pair of observations as context and a pair of hypothesis choices.
Revised: Abductive Natural Language Inference We formulate α NLI as multiple choice problems consisting of a pair of observations as context and a pair of hypothesis choices.

Original: Each instance in ART is deﬁned as follows:
Revised: Each instance in ART is deﬁned as follows:

Original: • O 1 : The observation at time t 1 .
Revised: • O 1 : The observation at time t 1 .

Original: • O 2 : The observation at time t 2 > t 1 .
Revised: • O 2 : The observation at time t 2 > t 1 .

Original: • h
Revised: • h

Original: + : A plausible hypothesis that explains the two observations O 1 and O 2 .
Revised: + : A plausible hypothesis that explains the two observations O 1 and O 2 .

Original: • h − : An implausible (or less plausible) hypothesis for observations O 

Revised: • h − : An implausible (or less plausible) hypothesis for observations O

Original: and O 2 .
Revised: and O 2 .

Original: Given the observations and a pair of hypotheses, the α NLI task is to select the most plausible explanation (hypothesis).
Revised: Given the observations and a pair of hypotheses, the α NLI task is to select the most plausible explanation (hypothesis).

Original: Abductive Natural Language Generation α NLG is the task of generating a valid hypothesis h + given the two observations
Revised: Abductive Natural Language Generation α NLG is the task of generating a valid hypothesis h + given the two observations

Original: O 1 and O 2 .
Revised: O 1 and O 2 .

Original: Formally, the task requires to maximize P ( h + | O 1 , O 2 ) .
Revised: Formally, the task requires to maximize P ( h + | O 1 , O 2 ) .

Original: 2 Pronounced as alpha -NLI
Revised: 2 Pronounced as alpha -NLI

Original: ART :
Revised: ART :

Original: A bductive R easoning in narrative T ext.
Revised: A bductive R easoning in narrative T ext.

Original:  M ODELS FOR A BDUCTIVE C OMMONSENSE R EASONING 3. A BDUCTIVE N ATURAL L ANGUAGE I NFERENCE
Revised:  M ODELS FOR A BDUCTIVE C OMMONSENSE R EASONING 3. A BDUCTIVE N ATURAL L ANGUAGE I NFERENCE

Original: A Probabilistic Framework for α NLI: A distinct feature of the α NLI task is that it requires jointly considering all available observations and their commonsense implications, to identify the correct hypothesis.
Revised: A Probabilistic Framework for α NLI: A distinct feature of the α NLI task is that it requires jointly considering all available observations and their commonsense implications, to identify the correct hypothesis.

Original: Formally, the α NLI task is to select the hypothesis h ∗ that is most probable given the observations.
Revised: Formally, the α NLI task is to select the hypothesis h ∗ that is most probable given the observations.

Original: [Equation]
Revised: [Equation]

Original: Rewriting the objective using Bayes Rule, we have:
Revised: Rewriting the objective using Bayes Rule, we have:

Original: [Equation]
Revised: [Equation]

Original: We formulate a set of probabilistic models for α NLI that make various independence assumptions on Equation 2 – starting from a simple baseline that ignores the observations entirely, and building up to a fully joint model.
Revised: We formulate a set of probabilistic models for α NLI that make various independence assumptions on Equation 2 – starting from a simple baseline that ignores the observations entirely, and building up to a fully joint model.

Original: These models are depicted as Bayesian Networks in Figure 2.
Revised: These models are depicted as Bayesian Networks in Figure 2.

Original: [Figure]
Revised: [Figure]

Original: Hypothesis Only: Our simplest model makes the strong assumption that the hypothesis is entirely independent of both observations, i.e. ( H ⊥ O 1 , O 2 ) , in which case we simply aim to maximize the marginal P ( H ) .
Revised: Hypothesis Only: Our simplest model makes the strong assumption that the hypothesis is entirely independent of both observations, i.e. ( H ⊥ O 1 , O 2 ) , in which case we simply aim to maximize the marginal P ( H ) .

Original: First (or Second) Observation Only: Our next two models make weaker assumptions: that the hypothesis depends on only one of the ﬁrst O 1 or second O 2 observation.
Revised: First (or Second) Observation Only: Our next two models make weaker assumptions: that the hypothesis depends on only one of the ﬁrst O 1 or second O 2 observation.

Original: Linear Chain: Our next model uses both observations, but considers each observation’s inﬂuence on the hypothesis independently , i.e. it does not combine information across the observations.
Revised: Linear Chain: Our next model uses both observations, but considers each observation’s inﬂuence on the hypothesis independently , i.e. it does not combine information across the observations.

Original: Formally, the model assumes that the three variables (cid:104) O 1 , H, O 2 (cid:105) form a linear Markov chain, where the second observation is conditionally independent of the ﬁrst, given the hypothesis (i.e.
Revised: Formally, the model assumes that the three variables (cid:104) O 1 , H, O 2 (cid:105) form a linear Markov chain, where the second observation is conditionally independent of the ﬁrst, given the hypothesis (i.e.

Original: ( O 1 ⊥ O 2 | H ) ).
Revised: ( O 1 ⊥ O 2 | H ) ).

Original: Under this assumption, we aim to maximize a somewhat simpler objective than
Revised: Under this assumption, we aim to maximize a somewhat simpler objective than

Original: Equation 2:
Revised: Equation 2:

Original: [Equation]
Revised: [Equation]

Original: Fully Connected: Finally, our most sophisticated model jointly models all three random variables as in Equation 2, and can in principle combine information across both observations to choose the correct hypothesis.
Revised: Fully Connected: Finally, our most sophisticated model jointly models all three random variables as in Equation 2, and can in principle combine information across both observations to choose the correct hypothesis.

Original: To help illustrate the subtle distinction between how the Linear Chain and Fully Connected models consider both observations, consider the following example.
Revised: To help illustrate the subtle distinction between how the Linear Chain and Fully Connected models consider both observations, consider the following example.

Original: Let observation O 1 : “Carl went to the store desperately searching for ﬂour tortillas for a recipe.”
Revised: Let observation O 1 : “Carl went to the store desperately searching for ﬂour tortillas for a recipe.”

Original: and O 2 : “Carl left the store very frustrated.”.
Revised: and O 2 : “Carl left the store very frustrated.”.

Original: Then consider two distinct hypotheses, an incorrect h 1 : “The cashier was rude” and the correct h 2 : “The store had corn tortillas, but not ﬂour ones.”.
Revised: Then consider two distinct hypotheses, an incorrect h 1 : “The cashier was rude” and the correct h 2 : “The store had corn tortillas, but not ﬂour ones.”.

Original: For this example, a Linear Chain model could arrive at the wrong answer, because it reasons about the observations separately—taking Oin isolation, both h 1 and h 2 seem plausible next events, albeit each a priori unlikely.
Revised: For this example, a Linear Chain model could arrive at the wrong answer, because it reasons about the observations separately—taking Oin isolation, both h 1 and h 2 seem plausible next events, albeit each a priori unlikely.

Original: And for Oin isolation—i.e. in the absence of O 1 , as for a randomly drawn shopper—the h 1 explanation of a rude cashier seems a much more plausible explanation of Carl’s frustration than are the details of the store’s tortilla selection.
Revised: And for Oin isolation—i.e. in the absence of O 1 , as for a randomly drawn shopper—the h 1 explanation of a rude cashier seems a much more plausible explanation of Carl’s frustration than are the details of the store’s tortilla selection.

Original: Combining these two separate factors leads the Linear Chain to select h 1 as the more plausible explanation.
Revised: Combining these two separate factors leads the Linear Chain to select h 1 as the more plausible explanation.

Original: It is only by reasoning about Carl’s goal in O 1 jointly with
Revised: It is only by reasoning about Carl’s goal in O 1 jointly with

Original: < ο 1> </ ο 1> ω ## ω $# … < ο 2> < /ο 2> ω #& ω $& …
Revised: < ο 1> </ ο 1> ω ## ω $# … < ο 2> < /ο 2> ω #& ω $& …

Original: Transformer … … e (# … e ## … e $# … e $)## … … … …
Revised: Transformer … … e (# … e ## … e $# … e $)## … … … …

Original: e #& e $& e $)#& … e (&
Revised: e #& e $& e $)#& … e (&

Original: Commonsense+Contextualized Word Representations
Revised: Commonsense+Contextualized Word Representations

Original: Word Embeddings (e.g. COMeTEmbeddings) … … … … … …
Revised: Word Embeddings (e.g. COMeTEmbeddings) … … … … … …

Original: c ## c +# c #& c +& COMeT Commonsense Transformers < ο 1> </ ο 1> ω
Revised: c ## c +# c #& c +& COMeT Commonsense Transformers < ο 1> </ ο 1> ω

Original: ## ω $# … < ο 2> < /ο 2> ω #& ω $& … his frustration in O 2 , as in the Fully Connected model, that we arrive at the correct answer h 2 as the more plausible explanation.
Revised: ## ω $# … < ο 2> < /ο 2> ω #& ω $& … his frustration in O 2 , as in the Fully Connected model, that we arrive at the correct answer h 2 as the more plausible explanation.

Original: In our experiments, we encode the different independence assumptions in the best performing neural network model.
Revised: In our experiments, we encode the different independence assumptions in the best performing neural network model.

Original: For the hypothesis-only and single observation models, we can enforce the independencies by simply restricting the inputs of the model to only the relevant variables.
Revised: For the hypothesis-only and single observation models, we can enforce the independencies by simply restricting the inputs of the model to only the relevant variables.

Original: On the other hand, the Linear Chain model takes all three variables as input, but we restrict the form of the model to enforce the conditional independence.
Revised: On the other hand, the Linear Chain model takes all three variables as input, but we restrict the form of the model to enforce the conditional independence.

Original: Speciﬁcally, we learn a discriminative classiﬁer:
Revised: Speciﬁcally, we learn a discriminative classiﬁer:

Original: [Equation]
Revised: [Equation]

Original: where φ and φ (cid:48) are neural networks that produce scalar values.
Revised: where φ and φ (cid:48) are neural networks that produce scalar values.

Original: 3. A BDUCTIVE N ATURAL L ANGUAGE G ENERATION
Revised: 3. A BDUCTIVE N ATURAL L ANGUAGE G ENERATION

Original: Given h + = { w h 1 . . .
Revised: Given h + = { w h 1 . . .

Original: w hl } , O 1 = { w o 11 . . .
Revised: w hl } , O 1 = { w o 11 . . .

Original: w o 1 m } and O 2 = { w o 21 . . .
Revised: w o 1 m } and O 2 = { w o 21 . . .

Original: w o 2 n } as sequences of tokens, the α NLG task can be modeled as P ( h + | O 1 , O 2 ) =
Revised: w o 2 n } as sequences of tokens, the α NLG task can be modeled as P ( h + | O 1 , O 2 ) =

Original: (cid:81) P ( w hi | w h<i , w o 11 . . .
Revised: (cid:81) P ( w hi | w h<i , w o 11 . . .

Original: w o 1 m , w o 21 . . .
Revised: w o 1 m , w o 21 . . .

Original: w o 2 n ) Optionally, the model can also be conditioned on background knowledge K .
Revised: w o 2 n ) Optionally, the model can also be conditioned on background knowledge K .

Original: Parameterized models can then be trained to minimize the negative log-likelihood over instances in ART :
Revised: Parameterized models can then be trained to minimize the negative log-likelihood over instances in ART :

Original: [Equation]
Revised: [Equation]

Original:  ART D ATASET : A BDUCTIVE R EASONING IN N ARRATIVE T EXT
Revised:  ART D ATASET : A BDUCTIVE R EASONING IN N ARRATIVE T EXT

Original: ART is the ﬁrst large-scale benchmark dataset for studying abductive reasoning in narrative texts.
Revised: ART is the ﬁrst large-scale benchmark dataset for studying abductive reasoning in narrative texts.

Original: It consists of ∼ 20K narrative contexts (pairs of observations (cid:104)
Revised: It consists of ∼ 20K narrative contexts (pairs of observations (cid:104)

Original: O 1 , O 2 (cid:105) )
Revised: O 1 , O 2 (cid:105) )

Original: with over 200K explanatory hypotheses.
Revised: with over 200K explanatory hypotheses.

Original: Table 5 in the Appendix summarizes corpus-level statistics of the ART dataset.
Revised: Table 5 in the Appendix summarizes corpus-level statistics of the ART dataset.

Original: 4 Figure 4 shows some illustrative examples from ART (dev split).
Revised: 4 Figure 4 shows some illustrative examples from ART (dev split).

Original: The best model based on BERT fails to correctly predict the ﬁrst two dev examples.
Revised: The best model based on BERT fails to correctly predict the ﬁrst two dev examples.

Original: Collecting Observations: The pairs O 1 , O 2 in ART are drawn from the ROCStories dataset (Mostafazadeh et al., 2016).
Revised: Collecting Observations: The pairs O 1 , O 2 in ART are drawn from the ROCStories dataset (Mostafazadeh et al., 2016).

Original: ROCStories is a large collection of short, manually curated ﬁvesentence stories.
Revised: ROCStories is a large collection of short, manually curated ﬁvesentence stories.

Original: It was designed to have a clear beginning and ending for each story, which naturally map to the ﬁrst ( O 1 ) and second ( O 2 ) observations in ART .
Revised: It was designed to have a clear beginning and ending for each story, which naturally map to the ﬁrst ( O 1 ) and second ( O 2 ) observations in ART .

Original: Collecting Hypotheses Options: We crowdsourced the plausible and implausible hypotheses options on Amazon Mechanical Turk (AMT) in two separate tasks 5 : 1.
Revised: Collecting Hypotheses Options: We crowdsourced the plausible and implausible hypotheses options on Amazon Mechanical Turk (AMT) in two separate tasks 5 : 1.

Original: Plausible Hypothesis Options: We presented O 1 and O 2 as narrative context to crowdworkers who were prompted to ﬁll in “What happened in-between?”
Revised: Plausible Hypothesis Options: We presented O 1 and O 2 as narrative context to crowdworkers who were prompted to ﬁll in “What happened in-between?”

Original: in natural language.
Revised: in natural language.

Original: The design of
Revised: The design of

Original: Both crowdsourcing tasks are complex and require creative writing.
Revised: Both crowdsourcing tasks are complex and require creative writing.

Original: Along with the ART dataset, we will publicly release templates and the full set of instructions for all crowdsourcing tasks to facilitate future data collection and research in this direction.
Revised: Along with the ART dataset, we will publicly release templates and the full set of instructions for all crowdsourcing tasks to facilitate future data collection and research in this direction.

Original: The More Plausible Hypothesis is … Given These Partial Observations …
Revised: The More Plausible Hypothesis is … Given These Partial Observations …

Original: HO
Revised: HO

Original: O
Revised: O

Original: H +
Revised: H +

Original: Leslie went to the mall to look for a purse to match her new dress .
Revised: Leslie went to the mall to look for a purse to match her new dress .

Original: Leslie found a beautiful dress after looking for hours.
Revised: Leslie found a beautiful dress after looking for hours.

Original: She took it to the counter and paid right away, and went home happy.
Revised: She took it to the counter and paid right away, and went home happy.

Original: Leslie found one that was perfect.
Revised: Leslie found one that was perfect.

Original: O 

Revised: O

Original: It was a very hot summer day .
Revised: It was a very hot summer day .

Original: HO
Revised: HO

Original: H +
Revised: H +

Original: He decided to run in the heat .
Revised: He decided to run in the heat .

Original: He felt much better !
Revised: He felt much better !

Original: He drank a glass of ice cold water.
Revised: He drank a glass of ice cold water.

Original: O
Revised: O

Original: O 1 Chad loves Barry Bonds.
Revised: O 1 Chad loves Barry Bonds.

Original: Chad got to meet Barry Bonds online , chatting.
Revised: Chad got to meet Barry Bonds online , chatting.

Original: Chad ensured that he took a picture to remember the event.
Revised: Chad ensured that he took a picture to remember the event.

Original: Chad waited after a game and met Barry.
Revised: Chad waited after a game and met Barry.

Original: HH +
Revised: HH +

Original: DEV DEV
Revised: DEV DEV

Original: DEV the task motivates the use of abductive reasoning to hypothesize likely explanations for the two given observations.
Revised: DEV the task motivates the use of abductive reasoning to hypothesize likely explanations for the two given observations.

Original: Implausible Hypothesis Options: In this task, we presented workers with observations
Revised: Implausible Hypothesis Options: In this task, we presented workers with observations

Original: O 1 , O 2 and one plausible hypothesis option h + ∈ H + collected from the previous task.
Revised: O 1 , O 2 and one plausible hypothesis option h + ∈ H + collected from the previous task.

Original: Crowdworkers were instructed to make minimal edits (up to 5 words) to a given h + to create implausible hypothesis variations for each plausible hypothesis.
Revised: Crowdworkers were instructed to make minimal edits (up to 5 words) to a given h + to create implausible hypothesis variations for each plausible hypothesis.

Original: A signiﬁcant challenge in creating datasets is avoiding annotation artifacts – unintentional patterns in the data that leak information about the target label – that several recent studies (Gururangan et al., 2018; Poliak et al., 2018; Tsuchiya, 2018) have reported on crowdsourced datasets .
Revised: A signiﬁcant challenge in creating datasets is avoiding annotation artifacts – unintentional patterns in the data that leak information about the target label – that several recent studies (Gururangan et al., 2018; Poliak et al., 2018; Tsuchiya, 2018) have reported on crowdsourced datasets .

Original: To tackle this challenge, we collect multiple plausible and implausible hypotheses for each (cid:104)
Revised: To tackle this challenge, we collect multiple plausible and implausible hypotheses for each (cid:104)

Original: O 1 , O 2 (cid:105) pair (as described above)
Revised: O 1 , O 2 (cid:105) pair (as described above)

Original: and then apply an adversarial ﬁltering algorithm to retain one challenging pair of hypotheses that are hard to distinguish between.
Revised: and then apply an adversarial ﬁltering algorithm to retain one challenging pair of hypotheses that are hard to distinguish between.

Original: We describe our algorithm in detail in Appendix
Revised: We describe our algorithm in detail in Appendix

Original: A.5.
Revised: A.5.

Original: While our ﬁnal dataset uses BERT as the adversary, preliminary experiments that used GPT as an adversary resulted in similar drops in performance of all models, including all BERT variants.
Revised: While our ﬁnal dataset uses BERT as the adversary, preliminary experiments that used GPT as an adversary resulted in similar drops in performance of all models, including all BERT variants.

Original: We compare the results of the two adversaries in Table 1.
Revised: We compare the results of the two adversaries in Table 1.

Original:  E XPERIMENTS AND R ESULTS
Revised:  E XPERIMENTS AND R ESULTS

Original: We now present our evaluation of ﬁnetuned state-of-the-art pre-trained language models on the ART dataset, and several other baseline systems for both α NLI and α NLG.
Revised: We now present our evaluation of ﬁnetuned state-of-the-art pre-trained language models on the ART dataset, and several other baseline systems for both α NLI and α NLG.

Original: Since α NLI is framed as a binary classiﬁcation problem, we choose accuracy as our primary metric.
Revised: Since α NLI is framed as a binary classiﬁcation problem, we choose accuracy as our primary metric.

Original: For α NLG, we report performance on automated metrics such as BLEU (Papineni et al., 2002), CIDEr (Vedantam et al., 2015), METEOR (Banerjee & Lavie, 2005) and also report human evaluation results.
Revised: For α NLG, we report performance on automated metrics such as BLEU (Papineni et al., 2002), CIDEr (Vedantam et al., 2015), METEOR (Banerjee & Lavie, 2005) and also report human evaluation results.

Original: 5. A BDUCTIVE N ATURAL L ANGUAGE I NFERENCE
Revised: 5. A BDUCTIVE N ATURAL L ANGUAGE I NFERENCE

Original: Despite strong performance on several other NLP benchmark datasets, the best baseline model based on BERT achieves an accuracy of just 68.9% on ART compared to human performance of 91.4%.
Revised: Despite strong performance on several other NLP benchmark datasets, the best baseline model based on BERT achieves an accuracy of just 68.9% on ART compared to human performance of 91.4%.

Original: The large gap between human performance and that of the best system provides signiﬁcant scope for development of more sophisticated abductive reasoning models.
Revised: The large gap between human performance and that of the best system provides signiﬁcant scope for development of more sophisticated abductive reasoning models.

Original: Our experiments show that introducing the additional independence assumptions described in Section 3.1 over the fully connected model tends to degrade system performance (see Table 1) in general.
Revised: Our experiments show that introducing the additional independence assumptions described in Section 3.1 over the fully connected model tends to degrade system performance (see Table 1) in general.

Original: Human Performance We compute human performance using AMT.
Revised: Human Performance We compute human performance using AMT.

Original: Each instance (two observations and two hypothesis choices) is shown to three workers who were prompted to choose the more plausible hypothesis choice.
Revised: Each instance (two observations and two hypothesis choices) is shown to three workers who were prompted to choose the more plausible hypothesis choice.

Original: 6 We compute majority vote on the labels assigned which leads to a human accuracy of 91.4% on the ART test set.
Revised: 6 We compute majority vote on the labels assigned which leads to a human accuracy of 91.4% on the ART test set.

Original: Baselines We include baselines that rely on simple features to verify that ART is not trivially solvable due to noticeable annotation artifacts, observed in several crowdsourced datasets.
Revised: Baselines We include baselines that rely on simple features to verify that ART is not trivially solvable due to noticeable annotation artifacts, observed in several crowdsourced datasets.

Original: The accuracies of all simple baselines are close to chance-performance on the task – indicating that the dataset is free of simple annotation artifacts.
Revised: The accuracies of all simple baselines are close to chance-performance on the task – indicating that the dataset is free of simple annotation artifacts.

Original: A model for the related but distinct task of entailment NLI (e.g. SNLI) forms a natural baseline for α NLI.
Revised: A model for the related but distinct task of entailment NLI (e.g. SNLI) forms a natural baseline for α NLI.

Original: We re-train the ESIM+ELMo (Chen et al., 2017; Peters et al., 2018) model as its performance on entailment NLI ( 88 . 9% ) is close to state-of-the-art models (excluding pre-trained language models).
Revised: We re-train the ESIM+ELMo (Chen et al., 2017; Peters et al., 2018) model as its performance on entailment NLI ( 88 . 9% ) is close to state-of-the-art models (excluding pre-trained language models).

Original: This model only achieves an accuracy of 58 .
Revised: This model only achieves an accuracy of 58 .

Original: 8% highlighting that performing well on ART requires models to go far beyond the linguistic notion of entailment
Revised: 8% highlighting that performing well on ART requires models to go far beyond the linguistic notion of entailment

Original: [Table]
Revised: [Table]

Original: Test accuracy is reported as the mean of ﬁve models trained with random seeds, with the standard deviation in parenthesis.
Revised: Test accuracy is reported as the mean of ﬁve models trained with random seeds, with the standard deviation in parenthesis.

Original: Pre-trained Language Models BERT (Devlin et al., 2018) and GPT (Radford, 2018) have recently been shown to achieve state-of-the-art results on several NLP benchmarks (Wang et al., 2018).
Revised: Pre-trained Language Models BERT (Devlin et al., 2018) and GPT (Radford, 2018) have recently been shown to achieve state-of-the-art results on several NLP benchmarks (Wang et al., 2018).

Original: We ﬁnetune both BERT-Large and GPT as suggested in previous work and we present each instance in their natural narrative order.
Revised: We ﬁnetune both BERT-Large and GPT as suggested in previous work and we present each instance in their natural narrative order.

Original: BERT-ft (fully connected) is the best performing model achieving 68.9% accuracy, compared to GPT’s 63 .
Revised: BERT-ft (fully connected) is the best performing model achieving 68.9% accuracy, compared to GPT’s 63 .

Original: 7 Our AF approach was able to reduce BERT performance from over 88% by 20 points.
Revised: 7 Our AF approach was able to reduce BERT performance from over 88% by 20 points.

Original: Learning Curve and Dataset Size
Revised: Learning Curve and Dataset Size

Original: While there is enough scope for considerably scaling up the dataset based on ROCStories, the learning curve in
Revised: While there is enough scope for considerably scaling up the dataset based on ROCStories, the learning curve in

Original: Figure 5 shows that the performance of the best model plateaus after ∼ 10 ,instances.
Revised: Figure 5 shows that the performance of the best model plateaus after ∼ 10 ,instances.

Original: In addition, there is still a wide gap ( ∼ 23% ) between the performance of the best model and human performance.
Revised: In addition, there is still a wide gap ( ∼ 23% ) between the performance of the best model and human performance.

Original: [Figure]
Revised: [Figure]

Original: For each point on the x-axis, we ﬁne-tune BERT with ﬁve random seeds.
Revised: For each point on the x-axis, we ﬁne-tune BERT with ﬁve random seeds.

Original: Human performance is 91.4%.
Revised: Human performance is 91.4%.

Original: GPT Adversary Table 1 also includes results of our experiments where GPT was used as the adversary.
Revised: GPT Adversary Table 1 also includes results of our experiments where GPT was used as the adversary.

Original: Notably, in this case, adversarially ﬁltering the dataset brings down GPT performance under 53%.
Revised: Notably, in this case, adversarially ﬁltering the dataset brings down GPT performance under 53%.

Original: On the other hand, the best
Revised: On the other hand, the best

Original: BERT model, that encodes the fully connected bayesian network performs signiﬁcantly better than the BERT model that encodes the linear chain assumptions – 72% compared to 65%.
Revised: BERT model, that encodes the fully connected bayesian network performs signiﬁcantly better than the BERT model that encodes the linear chain assumptions – 72% compared to 65%.

Original: Therefore, we use the
Revised: Therefore, we use the

Original: BERT fully connected model as the adversary in ART .
Revised: BERT fully connected model as the adversary in ART .

Original: The gap between the linear chain and fully connected BERT models diminishes when BERT is used as an adversary – in spite of being a more powerful model – which indicates that adversarial ﬁltering disproportionately impacts the model used as the adversary.
Revised: The gap between the linear chain and fully connected BERT models diminishes when BERT is used as an adversary – in spite of being a more powerful model – which indicates that adversarial ﬁltering disproportionately impacts the model used as the adversary.

Original: However, the dataset also becomes more difﬁcult for the other models that were not used as adversaries.
Revised: However, the dataset also becomes more difﬁcult for the other models that were not used as adversaries.

Original: For example, before any ﬁltering, BERT scores 88% and
Revised: For example, before any ﬁltering, BERT scores 88% and

Original: OpenGPT gets 80% , which is much higher than either model achieves in Table 1 when the other
Revised: OpenGPT gets 80% , which is much higher than either model achieves in Table 1 when the other

Original: [Table]
Revised: [Table]

Original: model is used for ﬁltering.
Revised: model is used for ﬁltering.

Original: This result is a reasonable indicator, albeit not a guarantee, that ART will remain challenging for new models released in the future.
Revised: This result is a reasonable indicator, albeit not a guarantee, that ART will remain challenging for new models released in the future.

Original: 5. A BDUCTIVE N ATURAL L ANGUAGE G ENERATION
Revised: 5. A BDUCTIVE N ATURAL L ANGUAGE G ENERATION

Revised: [Figure]

Revised: For each point on x we ﬁne tune BERT ﬁve

Original: Generative Language Models As described in Equation 4, we train GPT2 conditioned on the tokens of the two observations O 1 and O 2 .
Revised: Generative Language Models As described in Equation 4, we train GPT2 conditioned on the tokens of the two observations O 1 and O 2 .

Original: Both observations are enclosed with ﬁeld-speciﬁc tags.
Revised: Both observations are enclosed with ﬁeld-speciﬁc tags.

Original: ATOMIC (Sap et al., 2019), a repository of inferential if-then knowledge is a natural source of background commonsense required to reason about narrative contexts in ART .
Revised: ATOMIC (Sap et al., 2019), a repository of inferential if-then knowledge is a natural source of background commonsense required to reason about narrative contexts in ART .

Original: Yet, there is no straightforward way to include such knowledge into a neural model as ATOMIC’s nodes are not canonicalized and are represented as short phrases of text.
Revised: Yet, there is no straightforward way to include such knowledge into a neural model as ATOMIC’s nodes are not canonicalized and are represented as short phrases of text.

Original: Thus, we rely on COMeT – a transformer model trained on ATOMIC that generates nine commonsense inferences of events in natural language.
Revised: Thus, we rely on COMeT – a transformer model trained on ATOMIC that generates nine commonsense inferences of events in natural language.

Original: Speciﬁcally, we experiment with two ways of integrating information from COMeT in GPT2: (i) as phrases , and (ii) as embeddings .
Revised: Speciﬁcally, we experiment with two ways of integrating information from COMeT in GPT2: (i) as phrases , and (ii) as embeddings .

Original: Figure 3 shows how we integrate COMeT representations.
Revised: Figure 3 shows how we integrate COMeT representations.

Original: Concretely, after the input tokens are embedded by the word-embedding layer, we append eighteen (nine relations for each observation) embeddings to the sequence before passing through the layers of the Transformer architecture.
Revised: Concretely, after the input tokens are embedded by the word-embedding layer, we append eighteen (nine relations for each observation) embeddings to the sequence before passing through the layers of the Transformer architecture.

Original: This allows the model to learn each token’s representation while attending to the COMeT embeddings – effectively integrating background commonsense knowledge into a language model.
Revised: This allows the model to learn each token’s representation while attending to the COMeT embeddings – effectively integrating background commonsense knowledge into a language model.

Original: Discussion Table 7 reports results on the α NLG task.
Revised: Discussion Table 2 reports results on the α NLG task.

Original: Among automatic metrics, we report BLEU4 (Papineni et al., 2002), METEOR (Banerjee & Lavie, 2005), ROUGE (Lin, 2004), CIDEr (Vedantam et al., 2015) and BERT-Score (Zhang et al., 2019) (with the bert-base-uncased model).
Revised: Among automatic metrics, we report BLEU4 (Papineni et al., 2002), METEOR (Banerjee & Lavie, 2005), ROUGE (Lin, 2004), CIDEr (Vedantam et al., 2015) and BERT-Score (Zhang et al., 2019) (with the bert-base-uncased model).

Original: We establish human performance through crowdsourcing on AMT.
Revised: We establish human performance through crowdsourcing on AMT.

Original: Crowdworkers are shown pairs of observations and a generated hypothesis and asked to label whether the hypothesis explains the given observations.
Revised: Crowdworkers are shown pairs of observations and a generated hypothesis and asked to label whether the hypothesis explains the given observations.

Original: The last column reports the human evaluation score.
Revised: The last column reports the human evaluation score.

Original: The last row reports the score of a held-out human-written hypothesis and serves as a ceiling for model performance.
Revised: The last row reports the score of a held-out human-written hypothesis and serves as a ceiling for model performance.

Original: Human-written hypotheses are found to be correct for 96% of instances, while our best generative models, even when enhanced with background commonsense knowledge, only achieve 45% – indicating that the α NLG generation task is especially challenging for current state-of-the-art text generators.
Revised: Human-written hypotheses are found to be correct for 96% of instances, while our best generative models, even when enhanced with background commonsense knowledge, only achieve 45% – indicating that the α NLG generation task is especially challenging for current state-of-the-art text generators.

Original:  A NALYSIS 6. α NLI
Revised:  A NALYSIS 6. α NLI

Original: Commonsense reasoning categories We investigate the categories of commonsense-based abductive reasoning that are challenging for current systems and the ones where the best model overperforms.
Revised: Commonsense reasoning categories We investigate the categories of commonsense-based abductive reasoning that are challenging for current systems and the ones where the best model overperforms.

Original: While there have been previous attempts to categorize commonsense knowledge required for entailment (LoBue & Yates, 2011; Clark et al., 2007), crowdsourcing this task at scale with high ﬁdelity and high agreement across annotators remains challenging.
Revised: While there have been previous attempts to categorize commonsense knowledge required for entailment (LoBue & Yates, 2011; Clark et al., 2007), crowdsourcing this task at scale with high ﬁdelity and high agreement across annotators remains challenging.

Original: Instead, we aim to probe the model with soft categories identiﬁed by matching lists of category-speciﬁc keywords to the hypothesis choices.
Revised: Instead, we aim to probe the model with soft categories identiﬁed by matching lists of category-speciﬁc keywords to the hypothesis choices.

Original: [Table]
Revised: [Table]

Original: Table 3 shows the accuracy of the best model (BERT-ft) across various categories of commonsense knowledge.
Revised: Table 3 shows the accuracy of the best model (BERT-ft) across various categories of commonsense knowledge.

Original: BERT-ft signiﬁcantly underperforms on instances involving Numerical ( 56 . 8% ) and
Revised: BERT-ft signiﬁcantly underperforms on instances involving Numerical ( 56 . 8% ) and

Original: Spatial ( 65 . 4% ) commonsense.
Revised: Spatial ( 65 . 4% ) commonsense.

Original: These two categories include reasoning about numerical quantities and the spatial location of agents and objects, and highlight some of the limitations of the language models.
Revised: These two categories include reasoning about numerical quantities and the spatial location of agents and objects, and highlight some of the limitations of the language models.

Original: In contrast, it signiﬁcantly overperforms on the Emotional category ( 72 . 6% ) where the hypotheses exhibit strong textual cues about emotions and sentiments.
Revised: In contrast, it signiﬁcantly overperforms on the Emotional category ( 72 . 6% ) where the hypotheses exhibit strong textual cues about emotions and sentiments.

Original: Implausible transitions A model for an instance of the ART dataset should discard implausible hypotheses in the context of the two given observations.
Revised: Implausible transitions A model for an instance of the ART dataset should discard implausible hypotheses in the context of the two given observations.

Original: In narrative contexts, there are three main reasons for an implausible hypothesis to be labeled as such:
Revised: In narrative contexts, there are three main reasons for an implausible hypothesis to be labeled as such:

Original: O 1 (cid:54)→ h − : h − is unlikely to follow after the ﬁrst observation O 1 . 2.
Revised: O 1 (cid:54)→ h − : h − is unlikely to follow after the ﬁrst observation O 1 . 2.

Original: h − (cid:54)→ O 2 : h − is plausible after O 1 but unlikely to precede the second observation O 2 . 3.
Revised: h − (cid:54)→ O 2 : h − is plausible after O 1 but unlikely to precede the second observation O 2 . 3.

Original: Plausible: (cid:104)
Revised: Plausible: (cid:104)

Original: O 1 , h − , O 2 (cid:105) is a coherent narrative and forms a plausible alternative, but it is less plausible than (cid:104)
Revised: O 1 , h − , O 2 (cid:105) is a coherent narrative and forms a plausible alternative, but it is less plausible than (cid:104)

Original: O 1 , h + , O 2 (cid:105) .
Revised: O 1 , h + , O 2 (cid:105) .

Original: We analyze the prevalence of each of these reasons in ART .
Revised: We analyze the prevalence of each of these reasons in ART .

Original: We design a crowdsourcing task in which we show the implausible option along with the narrative context (cid:104)
Revised: We design a crowdsourcing task in which we show the implausible option along with the narrative context (cid:104)

Original: O 1 , O 2 (cid:105) and get labels for which transition ( O 1 (cid:54)→ h − , h − (cid:54)→
Revised: O 1 , O 2 (cid:105) and get labels for which transition ( O 1 (cid:54)→ h − , h − (cid:54)→

Original: O 2 or neither) in the narrative chain is broken.
Revised: O 2 or neither) in the narrative chain is broken.

Original: Table 4 shows the proportion of each category from a subset of 1 , 000 instances from the test set.
Revised: Table 4 shows the proportion of each category from a subset of 1 , 000 instances from the test set.

Original: While h − (cid:54)→ Oaccounts for almost half of the implausible transitions in ART , all three categories are substantially present in the dataset.
Revised: While h − (cid:54)→ O 2 accounts for almost half of the implausible transitions in ART , all three categories are substantially present in the dataset.

Original: BERT performance on each of these categories indicates that the model ﬁnds it particularly hard when the narrative created by the incorrect hypothesis is plausible, but less plausible than the correct hypothesis.
Revised: BERT performance on each of these categories indicates that the model ﬁnds it particularly hard when the narrative created by the incorrect hypothesis is plausible, but less plausible than the correct hypothesis.

Original: On that subset of the test set, the fully connected model performs better than the linear chain model where it is important to consider both observations jointly to arrive at the more likely hypothesis.6. α NLG
Revised: On that subset of the test set, the fully connected model performs better than the linear chain model where it is important to consider both observations jointly to arrive at the more likely hypothesis.

Original: [Table]
Revised: [Table]

Original: Figure 6 shows some examples of generations from the trained models compared to humanwritten generations.
Revised: Figure 7 shows some examples of generations from the trained models compared to humanwritten generations.

Original: The example on the left is an example of an instance that only humans could get correct, while for the one on the right,
Revised: The example on the left is an example of an instance that only humans could get correct, while for the one on the right,

Original: COMeT-Emb+GPT2 also generates the correct explanation for the observations.
Revised: COMeT-Emb+GPT2 also generates the correct explanation for the observations.

Revised: [Figure]

Original:  R ELATED W ORK Cloze-Style Task vs. Abductive Reasoning
Revised:  R ELATED W ORK Cloze-Style Task vs. Abductive Reasoning

Original: Since abduction is fundamentally concerned with plausible chains of cause-and-effect, our work draws inspiration from previous works that deal with narratives such as script learning (Schank & Abelson, 1975) and the narrative cloze test (Chambers & Jurafsky, 2009; Jans et al., 2012; Pichotta &Mooney, 2014;
Revised: Since abduction is fundamentally concerned with plausible chains of cause-and-effect, our work draws inspiration from previous works that deal with narratives such as script learning (Schank & Abelson, 1975) and the narrative cloze test (Chambers & Jurafsky, 2009; Jans et al., 2012; Pichotta &

Original:  Rudinger et al., 2015).
Revised: Mooney, 2014; Rudinger et al., 2015).

Original: Rather than learning prototypical scripts or narrative chains, we instead reason about the most plausible events conditioned on observations.
Revised: Rather than learning prototypical scripts or narrative chains, we instead reason about the most plausible events conditioned on observations.

Original: We make use of the ROCStories dataset (Mostafazadeh et al., 2016), which was speciﬁcally designed for the narrative cloze task.
Revised: We make use of the ROCStories dataset (Mostafazadeh et al., 2016), which was speciﬁcally designed for the narrative cloze task.

Original: But, instead of reasoning about plausible event sequences, our task requires reasoning about plausible explanations for narrative omissions.
Revised: But, instead of reasoning about plausible event sequences, our task requires reasoning about plausible explanations for narrative omissions.

Original: Entailment vs. Abductive Reasoning
Revised: Entailment vs. Abductive Reasoning

Original: The formulation of α NLI is closely related to entailment
Revised: The formulation of α NLI is closely related to entailment

Original: NLI, but there are two critical distinctions that make abductive reasoning uniquely challenging.
Revised: NLI, but there are two critical distinctions that make abductive reasoning uniquely challenging.

Original: First,[Figure] abduction requires reasoning about commonsense implications of observations (e.g., if we observe that the “grass is wet”, a likely hypothesis is that “it rained earlier”) which go beyond the linguistic notion of entailment (also noted by Josephson (2000)).
Revised: First, abduction requires reasoning about commonsense implications of observations (e.g., if we observe that the “grass is wet”, a likely hypothesis is that “it rained earlier”) which go beyond the linguistic notion of entailment (also noted by Josephson (2000)).

Original: Second, abduction requires non-monotonic reasoning about a set of commonsense implications collectively, to check the potential contradictions against multiple observations and to compare the level of plausibility of different hypotheses.
Revised: Second, abduction requires non-monotonic reasoning about a set of commonsense implications collectively, to check the potential contradictions against multiple observations and to compare the level of plausibility of different hypotheses.

Original: This makes abductive reasoning distinctly challenging compared to other forms of reasoning such as induction and deduction (Shank, 1998).
Revised: This makes abductive reasoning distinctly challenging compared to other forms of reasoning such as induction and deduction (Shank, 1998).

Original: Perhaps more importantly, abduction is closely related to the kind of reasoning humans perform in everyday situations, where information is incomplete and deﬁnite inferences cannot be made.
Revised: Perhaps more importantly, abduction is closely related to the kind of reasoning humans perform in everyday situations, where information is incomplete and deﬁnite inferences cannot be made.

Original: Generative Language Modeling Recent advancements in the development of large-scale pretrained language models (Radford, 2018; Devlin et al., 2018; Radford et al., 2019) have improved the quality and coherence of generated language.
Revised: Generative Language Modeling Recent advancements in the development of large-scale pretrained language models (Radford, 2018; Devlin et al., 2018; Radford et al., 2019) have improved the quality and coherence of generated language.

Original: Although these models have shown to generate reasonably coherent text when condition on a sequence of text, our experiments highlight the limitations of these models to 1) generate language non-monotonically and 2) adhere to commonsense knowledge.
Revised: Although these models have shown to generate reasonably coherent text when condition on a sequence of text, our experiments highlight the limitations of these models to 1) generate language non-monotonically and 2) adhere to commonsense knowledge.

Original: We attempt to overcome these limitations with the incorporation of a generative commonsense model during hypothesis generation.
Revised: We attempt to overcome these limitations with the incorporation of a generative commonsense model during hypothesis generation.

Original: Related Datasets Our new resource ART complements ongoing efforts in building resources for natural language inference (Dagan et al., 2006; MacCartney & Manning, 2009;
Revised: Related Datasets Our new resource ART complements ongoing efforts in building resources for natural language inference (Dagan et al., 2006; MacCartney & Manning, 2009;

Original: Bowman et al., 2015; Williams et al., 2018a; Camburu et al., 2018).
Revised: Bowman et al., 2015; Williams et al., 2018a; Camburu et al., 2018).

Original: Existing datasets have mostly focused on textual entailment in a deductive reasoning set-up (Bowman et al., 2015; Williams et al., 2018a) and making inferences about plausible events (Maslan et al., 2015; Zhang et al., 2017).
Revised: Existing datasets have mostly focused on textual entailment in a deductive reasoning set-up (Bowman et al., 2015; Williams et al., 2018a) and making inferences about plausible events (Maslan et al., 2015; Zhang et al., 2017).

Original: In their typical setting, these datasets require a system to deduce the logically entailed consequences of a given premise .
Revised: In their typical setting, these datasets require a system to deduce the logically entailed consequences of a given premise .

Original: In contrast, the nature of abduction requires the use of commonsense reasoning capabilities, with less focus on lexical entailment.
Revised: In contrast, the nature of abduction requires the use of commonsense reasoning capabilities, with less focus on lexical entailment.

Original: While abductive reasoning has been applied to entailment datasets (Raina et al., 2005), they have been applied in a logical theorem-proving framework as an intermediate step to perform textual entailment – a fundamentally different task than α NLI.
Revised: While abductive reasoning has been applied to entailment datasets (Raina et al., 2005), they have been applied in a logical theorem-proving framework as an intermediate step to perform textual entailment – a fundamentally different task than α NLI.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We present the ﬁrst study that investigates the viability of language-based abductive reasoning.
Revised: We present the ﬁrst study that investigates the viability of language-based abductive reasoning.

Original: We conceptualize and introduce Abductive Natural Language Inference ( α NLI) – a novel task focused on abductive reasoning in narrative contexts.
Revised: We conceptualize and introduce Abductive Natural Language Inference ( α NLI) – a novel task focused on abductive reasoning in narrative contexts.

Original: The task is formulated as a multiple-choice questionanswering problem.
Revised: The task is formulated as a multiple-choice questionanswering problem.

Original: We also introduce Abductive Natural Language Generation ( α NLG) – a novel task that requires machines to generate plausible hypotheses for given observations.
Revised: We also introduce Abductive Natural Language Generation ( α NLG) – a novel task that requires machines to generate plausible hypotheses for given observations.

Original: To support these tasks, we create and introduce a new challenge dataset, ART , which consists of 20,000 commonsense narratives accompanied with over 200,000 explanatory hypotheses.
Revised: To support these tasks, we create and introduce a new challenge dataset, ART , which consists of 20,000 commonsense narratives accompanied with over 200,000 explanatory hypotheses.

Original: In our experiments, we establish comprehensive baseline performance on this new task based on state-of-the-art NLI and language models, which leads to 68.9% accuracy with a considerable gap with human performance (91.4%).
Revised: In our experiments, we establish comprehensive baseline performance on this new task based on state-of-the-art NLI and language models, which leads to 68.9% accuracy with a considerable gap with human performance (91.4%).

Original: The α NLG task is signiﬁcantly harder – while humans can write a valid explanation 96% of times, the best generator models can only achieve 45%.
Revised: The α NLG task is signiﬁcantly harder – while humans can write a valid explanation 96% of times, the best generator models can only achieve 45%.

Original: Our analysis leads to new insights into the types of reasoning that deep pre-trained language models fail to perform – despite their strong performance on the closely related but different task of entailment NLI – pointing to interesting avenues for future research.
Revised: Our analysis leads to new insights into the types of reasoning that deep pre-trained language models fail to perform – despite their strong performance on the closely related but different task of entailment NLI – pointing to interesting avenues for future research.

Original: We hope that ART will serve as a challenging benchmark for future research in languagebased abductive reasoning and the α NLI and α NLG tasks will encourage representation learning that enables complex reasoning capabilities in AI systems
Revised: We hope that ART will serve as a challenging benchmark for future research in languagebased abductive reasoning and the α NLI and α NLG tasks will encourage representation learning that enables complex reasoning capabilities in AI systems.

Original: A A PPENDICES A. D ATA C OLLECTION D ETAILS
Revised: A A PPENDICES A. D ATA C OLLECTION D ETAILS

Original: We describe the crowdsourcing details of our data collection method.
Revised: We describe the crowdsourcing details of our data collection method.

Original: Task 1 - Plausible Hypothesis Options
Revised: Task 1 - Plausible Hypothesis Options

Original: In this task, participants were presented an incomplete three-part story, which consisted of the ﬁrst observation ( O 1 ) and the second observation ( O 2 ) of the story.
Revised: In this task, participants were presented an incomplete three-part story, which consisted of the ﬁrst observation ( O 1 ) and the second observation ( O 2 ) of the story.

Original: They were then asked to complete the story by writing a probable middle sentence that explains why the second observation should follow after the ﬁrst one.
Revised: They were then asked to complete the story by writing a probable middle sentence that explains why the second observation should follow after the ﬁrst one.

Original: We instructed participants to make sure that the plausible middle sentence (1) is short (fewer than 10 words) and (2) simple as if narrating to a child, (3) avoids introducing any extraneous information, and (4) uses names instead of pronouns (e.g., he/she) wherever possible.
Revised: We instructed participants to make sure that the plausible middle sentence (1) is short (fewer than 10 words) and (2) simple as if narrating to a child, (3) avoids introducing any extraneous information, and (4) uses names instead of pronouns (e.g., he/she) wherever possible.

Original: All participants were required to meet the following qualiﬁcation requirements: (1) their location is in the US, (2) HIT approval rate is greater than 95(%), and (3) Number of HITs approved is greater than 5,000.
Revised: All participants were required to meet the following qualiﬁcation requirements: (1) their location is in the US, (2) HIT approval rate is greater than 95(%), and (3) Number of HITs approved is greater than 5,000.

Original: The reward of this task was set to be $0.07 per question ($14/hour in average), and each
Revised: The reward of this task was set to be $0.07 per question ($14/hour in average), and each

Original: HIT was assigned to ﬁve different workers (i.e., 5-way redundancy).
Revised: HIT was assigned to ﬁve different workers (i.e., 5-way redundancy).

Original: Task 2 - Implausible Hypothesis Options
Revised: Task 2 - Implausible Hypothesis Options

Original: In this task, participants were presented a three-part story, which consisted of the ﬁrst observation ( O 1 ), a middle sentence ( h + ) collected in Task 1, and the second observation ( O 2 ) of the story.
Revised: In this task, participants were presented a three-part story, which consisted of the ﬁrst observation ( O 1 ), a middle sentence ( h + ) collected in Task 1, and the second observation ( O 2 ) of the story.

Original: They were then asked to rewrite the middle sentence ( h + ) with minimal changes , so that the story becomes unlikely, implausible or inconsistent ( h − ).
Revised: They were then asked to rewrite the middle sentence ( h + ) with minimal changes , so that the story becomes unlikely, implausible or inconsistent ( h − ).

Original: We asked participants to add or remove at most four words to h + , while ensuring that the new middle sentence is grammatical.
Revised: We asked participants to add or remove at most four words to h + , while ensuring that the new middle sentence is grammatical.

Original: In addition, we asked them to stick to the context in the given story.
Revised: In addition, we asked them to stick to the context in the given story.

Original: For example, if the story talks about “doctors”, they are welcome to talk about “health” or “diagnosis”, but not mention “aliens”.
Revised: For example, if the story talks about “doctors”, they are welcome to talk about “health” or “diagnosis”, but not mention “aliens”.

Original: Finally, we also asked workers to verify if the given middle ( h + ) makes a plausible story, in order to conﬁrm the plausibility of h + collected in Task 1.
Revised: Finally, we also asked workers to verify if the given middle ( h + ) makes a plausible story, in order to conﬁrm the plausibility of h + collected in Task 1.

Original: With respect to this task’s qualiﬁcation, participants were required to fulﬁll the following requirements: (1) their location is the US or Canada, (2) HIT approval rate is greater than or equal to 99(%), and (3) number of HITs approved is greater than or equal to 10 , 000 .
Revised: With respect to this task’s qualiﬁcation, participants were required to fulﬁll the following requirements: (1) their location is the US or Canada, (2) HIT approval rate is greater than or equal to 99(%), and (3) number of HITs approved is greater than or equal to 10 , 000 .

Original: Participants were paid $0.
Revised: Participants were paid $0.

Original: per question ($14/hour in average), and each HIT was assigned to three different participants (i.e., 3-way redundancy).
Revised: per question ($14/hour in average), and each HIT was assigned to three different participants (i.e., 3-way redundancy).

Original: Task 3 - α NLI Human Performance Human performance was evaluated by asking participants to answer the α NLI questions.
Revised: Task 3 - α NLI Human Performance Human performance was evaluated by asking participants to answer the α NLI questions.

Original: Given a narrative context (cid:104) O 1 , O 2 (cid:105) and two hypotheses, they were asked to choose the more plausible hypothesis.
Revised: Given a narrative context (cid:104) O 1 , O 2 (cid:105) and two hypotheses, they were asked to choose the more plausible hypothesis.

Original: They were also allowed to choose “None of the above“ when neither hypothesis was deemed plausible.
Revised: They were also allowed to choose “None of the above“ when neither hypothesis was deemed plausible.

Original: We asked each question to seven participants with the following qualiﬁcation requirements: (1) their location is either in the US, UK, or Canada, (2) HIT approval rate is greater than 98(%), (3) Number of HITs approved is greater than 10 , 000 .
Revised: We asked each question to seven participants with the following qualiﬁcation requirements: (1) their location is either in the US, UK, or Canada, (2) HIT approval rate is greater than 98(%), (3) Number of HITs approved is greater than 10 , 000 .

Original: The reward was set to $0.05 per HIT.
Revised: The reward was set to $0.05 per HIT.

Original: We took the majority vote among the seven participants for every question to compute human performance.
Revised: We took the majority vote among the seven participants for every question to compute human performance.

Original: A. ART D ATA S TATISTICS
Revised: A. ART D ATA S TATISTICS

Original: Table 5 shows some statistics of the ART dataset.
Revised: Table 5 shows some statistics of the ART dataset.

Original: A. F INE - TUNING BERT
Revised: A. F INE - TUNING BERT

Original: We ﬁne-tuned the BERT model using a grid search with the following set of hyper-parameters:
Revised: We ﬁne-tuned the BERT model using a grid search with the following set of hyper-parameters:

Original: • batch size: { 3 , 4 , 8 } • number of epochs: { 3 , 4 , 10 } • learning rate: { 1e-5, 2e-5, 3e-5, 5e-5 }
Revised: • batch size: { 3 , 4 , 8 } • number of epochs: { 3 , 4 , 10 } • learning rate: { 1e-5, 2e-5, 3e-5, 5e-5 }

Original: The warmup proportion was set to 0 .
Revised: The warmup proportion was set to 0 .

Original: 2 , and cross-entropy was used for computing the loss.
Revised: 2 , and cross-entropy was used for computing the loss.

Original: The best performance was obtained with a batch size of 4 , learning rate of 5e-5, and number of epochs equal to 10 .
Revised: The best performance was obtained with a batch size of 4 , learning rate of 5e-5, and number of epochs equal to 10 .

Original: Table 6 describes the input format for GPT and BERT (and its variants).
Revised: Table 6 describes the input format for GPT and BERT (and its variants).

Original: [Table]
Revised: [Table]

Original: A. B ASELINES
Revised: A. B ASELINES

Original: The SVM classiﬁer is trained on simple features like word length, overlap and sentiment features to select one of the two hypothesis choices.
Revised: The SVM classiﬁer is trained on simple features like word length, overlap and sentiment features to select one of the two hypothesis choices.

Original: The bag-of-words baseline computes the average of GloVe (Pennington et al., 2014) embeddings for words in each sentence to form sentence embeddings.
Revised: The bag-of-words baseline computes the average of GloVe (Pennington et al., 2014) embeddings for words in each sentence to form sentence embeddings.

Original: The sentence embeddings in a story (two observations and a hypothesis option) are concatenated and passed through fully-connected layers to produce a score for each hypothesis.
Revised: The sentence embeddings in a story (two observations and a hypothesis option) are concatenated and passed through fully-connected layers to produce a score for each hypothesis.

Original: The accuracies of both baselines are close to 50% (SVM: 50.6; BOW: 50.5).
Revised: The accuracies of both baselines are close to 50% (SVM: 50.6; BOW: 50.5).

Original: Speciﬁcally, we train an SVM classiﬁer and a bag-of-words model using GLoVE embeddings.
Revised: Speciﬁcally, we train an SVM classiﬁer and a bag-of-words model using GLoVE embeddings.

Original: Both models achieve accuracies close to 50%.
Revised: Both models achieve accuracies close to 50%.

Original: An Infersent (Conneau et al., 2017) baseline that uses sentences embedded by max-pooling over Bi-LSTM token representations achieves only 50.8% accuracy.
Revised: An Infersent (Conneau et al., 2017) baseline that uses sentences embedded by max-pooling over Bi-LSTM token representations achieves only 50.8% accuracy.

Original: [Table]
Revised: [Table]

Original: A. A DVERSARIAL F ILTERING OF H YPOTHESES C HOICES
Revised: A. A DVERSARIAL F ILTERING OF H YPOTHESES C HOICES

Original: Given an observation pair and sets of plausible and implausible hypotheses (cid:104)
Revised: Given an observation pair and sets of plausible and implausible hypotheses (cid:104)

Original: O
Revised: O

Original: [Table]
Revised: [Table]

Original: O
Revised: O

Original: [Table]
Revised: [Table]

Original: H +
Revised: H +

Original: [Table]
Revised: [Table]

Original: H − (cid:105)
Revised: H − (cid:105)

Original: [Table]
Revised: [Table]

Original: our adversarial ﬁltering algorithm selects one plausible and one implausible hypothesis (cid:104)
Revised: our adversarial ﬁltering algorithm selects one plausible and one implausible hypothesis (cid:104)

Original: O
Revised: O

Original: [Table]
Revised: [Table]

Original: O
Revised: O

Original: [Table]
Revised: [Table]

Original: h +
Revised: h +

Original: [Table]
Revised: [Table]

Original: h − (cid:105) such that h + and h − are hard to distinguish between.
Revised: h − (cid:105) such that h + and h − are hard to distinguish between.

Original: We make three key improvements over the previously proposed Adversarial Filtering (AF) approach in Zellers et al.
Revised: We make three key improvements over the previously proposed Adversarial Filtering (AF) approach in Zellers et al.

Original: First, Instead of a single positive sample
Revised: First, Instead of a single positive sample

Original: [Table]
Revised: [Table]

Original: we exploit a pool H + of positive samples to choose from (i
Revised: we exploit a pool H + of positive samples to choose from (i

Original: [Table]
Revised: [Table]

Original: e. plausible hypotheses).
Revised: e. plausible hypotheses).

Original: Second, Instead of machine generated distractors
Revised: Second, Instead of machine generated distractors

Original: [Table]
Revised: [Table]

Original: the pool H − of negative samples (i
Revised: the pool H − of negative samples (i

Original: [Table]
Revised: [Table]

Original: e. implausible hypotheses) is human-generated.
Revised: e. implausible hypotheses) is human-generated.

Original: Thus, the distractors share stylistic features of the positive samples as well as that of the context (i
Revised: Thus, the distractors share stylistic features of the positive samples as well as that of the context (i

Original: [Table]
Revised: [Table]

Original: e. observations O 1 and O 2 ) – making the negative samples harder to distinguish from positive samples.
Revised: e. observations O 1 and O 2 ) – making the negative samples harder to distinguish from positive samples.

Original: Finally
Revised: Finally

Original: [Table]
Revised: [Table]

Original: We use BERT (Devlin et al., 2018) as the adversary and introduce a temperature parameter that controls the maximum number of instances that can be modiﬁed in each iteration of AF.
Revised: We use BERT (Devlin et al., 2018) as the adversary and introduce a temperature parameter that controls the maximum number of instances that can be modiﬁed in each iteration of AF.

Original: In later iterations, fewer instances get modiﬁed resulting in a smoother convergence of the AF algorithm (described in more detail below).
Revised: In later iterations, fewer instances get modiﬁed resulting in a smoother convergence of the AF algorithm (described in more detail below).

Original: Algorithm 1 provides a formal description of our approach.
Revised: Algorithm 1 provides a formal description of our approach.

Original: In each iteration i , we train an adversarial model M i on a random subset T i of the data and update the validation set V i to make it more challenging for M i .
Revised: In each iteration i , we train an adversarial model M i on a random subset T i of the data and update the validation set V i to make it more challenging for M i .

Original: For a pair ( h + k , h − k ) of plausible and implausible hypotheses for an instance k , we denote δ =
Revised: For a pair ( h + k , h − k ) of plausible and implausible hypotheses for an instance k , we denote δ =

Original: ∆ M i ( h + k , h − k ) the difference in the model evaluation of h + k and h − k .
Revised: ∆ M i ( h + k , h − k ) the difference in the model evaluation of h + k and h − k .

Original: A positive value of δ indicates that the model M i favors the plausible hypothesis h + k over the implausible one h − k .
Revised: A positive value of δ indicates that the model M i favors the plausible hypothesis h + k over the implausible one h − k .

Original: With probability t i , we update instance k that M i gets correct with a pair ( h + , h −
Revised: With probability t i , we update instance k that M i gets correct with a pair ( h + , h −

Original: ) ∈ H + k ×
Revised: ) ∈ H + k ×

Original: H − k of hypotheses that reduces the value of δ ,
Revised: H − k of hypotheses that reduces the value of δ ,

Original: where H + k (resp.
Revised: where H + k (resp.

Original: H − k ) is the pool of plausible (resp. implausible) hypotheses for instance k .
Revised: H − k ) is the pool of plausible (resp. implausible) hypotheses for instance k .

Original: We ran AF for 50 iterations and the temperature t i follows a sigmoid function, parameterized by the iteration number, between t s = 1 . 0 and t e = 0 .
Revised: We ran AF for 50 iterations and the temperature t i follows a sigmoid function, parameterized by the iteration number, between t s = 1 . 0 and t e = 0 .

Original: Our ﬁnal dataset, ART , is generated using BERT as the adversary in Algorithm 1.
Revised: Our ﬁnal dataset, ART , is generated using BERT as the adversary in Algorithm 1.

Original: Algorithm input : dataset D 0 , plausible & implausible hypothesis sets ( H + , H − ) , number of iterations n , initial & ﬁnal temperatures ( t s , t e ) output: dataset D n 1 for iteration i : 0 ..n − 1 do t i =
Revised: Algorithm input : dataset D 0 , plausible & implausible hypothesis sets ( H + , H − ) , number of iterations n , initial & ﬁnal temperatures ( t s , t e ) output: dataset D n 1 for iteration i : 0 ..n − 1 do t i =

Original: t e + t s − t e 1+ e 0 .
Revised: t e + t s − t e 1+ e 0 .

Original: 3( i − 3 n 4 )
Revised: 3( i − 3 n 4 )

Original: Randomly partition D i into ( T i , V i ).
Revised: Randomly partition D i into ( T i , V i ).

Original: Train model M i on T i .
Revised: Train model M i on T i .

Original: S i = ∅ , the selected hypotheses for V i .
Revised: S i = ∅ , the selected hypotheses for V i .

Original: for ( h + k , h − k ) ∈ V i do
Revised: for ( h + k , h − k ) ∈ V i do

Original: Pick r uniformly at random in [0 , 1] .
Revised: Pick r uniformly at random in [0 , 1] .

Original: if r > t i or ∆ M i ( h + k , h − k ) < 0 then
Revised: if r > t i or ∆ M i ( h + k , h − k ) < 0 then

Original: Add ( h + k , h − k ) to S i .
Revised: Add ( h + k , h − k ) to S i .

Original: else
Revised: else

Original: Pick ( h + , h −
Revised: Pick ( h + , h −

Original: ) ∈ H + k ×
Revised: ) ∈ H + k ×

Original: H − k s.t.
Revised: H − k s.t.

Original: ∆ M i ( h + , h − ) <
Revised: ∆ M i ( h + , h − ) <

Original: ∆ M i ( h + k , h − k )
Revised: ∆ M i ( h + k , h − k )

Original: Add ( h + , h − ) to S i .
Revised: Add ( h + , h − ) to S i .

Original: end end
Revised: end end

Original: D i +

Revised: D i +

Original: =
Revised: =

Original: T i ∪ S
Revised: T i ∪ S

Original: i 16 end
Revised: i 16 end

Original: A. ATOMIC R ELATIONS
Revised: A. ATOMIC R ELATIONS

Original: ATOMIC (Sap et al., 2019) represents commonsense knowledge as a graph with events are nodes and the following nine relations as edges:
Revised: ATOMIC (Sap et al., 2019) represents commonsense knowledge as a graph with events are nodes and the following nine relations as edges:

Original: xIntent: Why does X cause an event?
Revised: xIntent: Why does X cause an event?

Original: xNeed: What does X need to do before the event?
Revised: xNeed: What does X need to do before the event?

Original: xAttr: How would X be described?
Revised: xAttr: How would X be described?

Original: xEffect:
Revised: xEffect:

Original: What effects does the event have on X? 5.
Revised: What effects does the event have on X? 5.

Original: xWant: What would X likely want to do after the event?
Revised: xWant: What would X likely want to do after the event?

Original: xReaction:
Revised: xReaction:

Original: How does X feel after the event?
Revised: How does X feel after the event?

Original: oReact: How do others’ feel after the event?
Revised: oReact: How do others’ feel after the event?

Original: oWant: What would others likely want to do after the event?
Revised: oWant: What would others likely want to do after the event?

Original: oEffect:
Revised: oEffect:

Original: What effects does the event have on others?
Revised: What effects does the event have on others?

Original: A. G ENERATION M ODELS I NPUT F ORMAT
Revised: A. G ENERATION M ODELS I NPUT F ORMAT

Original: [Table]
Revised: [Table]