Paper:
Abductive reasoning is inference to the most plausible explanation. For example, if Jenny finds her house in a mess when she returns from work, and remembers that she left a window open, she can hypothesize that a thief broke into her house and caused the mess, as the most plausible explanation. While abduction has long been considered to be at the core of how people interpret and read between the lines in natural language (Hobbs et al., 1988), there has been relatively little research in support of abductive natural language inference and generation. We present the first study that investigates the viability of language-based abductive reasoning. We introduce a challenge dataset, ART, that consists of over 20k commonsense narrative contexts and 200k explanations. Based on this dataset, we conceptualize two new tasks -(i) Abductive NLI: a multiple-choice question answering task for choosing the more likely explanation, and (ii) Abductive NLG: a conditional generation task for explaining given observations in natural language. On Abductive NLI, the best model achieves 68.9% accuracy, well below human performance of 91.4%. On Abductive NLG, the current best language generators struggle even more, as they lack reasoning capabilities that are trivial for humans. Our analysis leads to new insights into the types of reasoning that deep pre-trained language models fail to perform-despite their strong performance on the related but more narrowly defined task of entailment NLI-pointing to interesting avenues for future research.

Review:
This paper introduces two new natural language tasks in the area of commonsense reasoning: natural language abductive  inference and natural language abductive  generation. The paper also introduces a new dataset, ART, to support training and evaluating models for the introduced tasks. The paper describes the new language abductive tasks, contrasting it to the related, and recently established, natural language inference (entailment) task.  They go on to describe the construction of baseline models for these tasks. These models were primarily constructed to diagnose potential unwanted biases in the dataset (e.g., are the tasks partially solved by looking at parts of the input, do existing NLI models far well on the dataset, etc.), demonstrating a significant gap with respect to human performance.
The paper, and the dataset specifically, represent an important contribution to the area of natural language commonsense reasoning. It convincingly demonstrates that the proposed tasks, while highly related to natural language entailment, are not trivially addressed by existing state-of-the-art models. I expect that teaching models to perform well on this task can lead to improvements in other tasks, although empirical evidence of this hypothesis is currently absent from the paper.
Below are a set of more specific observations about the paper. Some of these comments aim to improve the presentation or content of the paper.
1. In Section “5.1 - Pre-trained Language Models” and attendant Table 1describe results of different baselines on the ART inference task. The results in the table confused me for quite some time, I’d appreciate some clarifications. With respect to the differences with columns 1 (GPT AF) and 2 (ART, also BERT AF) I would like the comparison to be made more clear. As far as I understand it, there are 2 parts of the dataset that can be varied: (1) the train+dev sets and (2) the test set. Furthermore, it seems that it makes sense to vary each of these at a time, if we are to compare results with variants. For example: we can fix the test set, and vary how we generate training and dev examples. If a model does better with the same test set, we can assume the train+dev examples were better for the model (for whatever reasons, closer distribution to test, harder or more 
informative training examples, etc). We can also keep the train+set constant, and vary the test set. This allows us to evaluate which test set is harder with respect to the training examples. The caption of Table 1 implies that both columns are evaluations based on the “ART” test set. If that is correct, then the train+dev set generated from the GPT adversarial examples is of better quality, generating a BERT-ft (fully connected) model that is 3% better. But the overall analysis seems to indicate that this is not what was done in the experiment. Rather, it seems that *both* the train+dev _and_ the test sets were modified concurrently. If that is the case, I would emphasize that the text needs to make this distinction clear. Furthermore, I would say that varying both train and test sets concurrently is sub-optimal, and makes it a bit harder to draw the conclusion that BERT adversarial filtering leads to a stronger overall dataset.
2. Along the lines of the argument in (1), above, I would urge the authors to publish the *entire* set of generated hypotheses (plausible and implausible) instead of only releasing the adversarially filtered pairs. Our group’s experience with training inference models is that it is often beneficial to train using “easy” examples, not only hard examples. I suspect the adversarially filtered set will focus on hard examples only. While this is fine to do in the test set, I think if the full set of annotated/generated hypotheses are released, model designers can experiment with combining pairs of hypothesis in different ways.
3. Furthering the argument of (2): in certain few-shot classification tasks, one is typically asked to identify similarity between one test example and different class representatives. Experience shows that it is often beneficial to train the model on a larger number of distractor classes than what the model is eventually evaluated on (e.g., https://papers.nips.cc/paper/6996-prototypical-networks-for-few-shot-learning). In the alpha-NLI setting, have you experimented with training using multiple distractors, instead of only 1, during training (even if you end up evaluating over 2 hypotheses)?
4. One potential argument for introducing a new natural language task is of transfer learning: learning to perform a complex natural language task should lead to better natural language models more generally, or, for some other related tasks. This paper does not really touch on this aspect of the work. But, potentially, one investigation that could be conducted is through a reversal of the paper’s existing NLI entailment experiment. The paper shows that NLI entailment models do not perform well on alpha-NLI. But it would be interesting to see if a model trained on alpha-NLI, and fine-tuned or multi-tasked on NLI entailment, does better on NLI entailment (i.e., is there transfer from alpha-NLI to entailment NLI?).
5. Another option is to evaluate whether alpha-NLI helps with other commonsense tasks. One other example is Winograd Schema Challenge, which current systems also perform well below human performance. It also seems that the Winograd Schema Challenge questions are not too far from abductive inference.
6. In the abstract of the paper, before the paper defines the abductive inference/generation task specifically, the claim that abductive reasoning has had much attention in research seemed awkward. Informally, most commonsense reasoning (including NLI entailment) could be cast as abductive reasoning.
7. In at least one occasion, I found an acronym which was hard to find the definition for (“AF” used in Section “5.1 - Pre-trained Language Models”; I assumed it was “adversarial filtering”.)
8. In Section “5.1 - Pre-trained Language Models” it seems that the text quotes an accuracy for BERT-ft (fully connected) of 68.9%, but Table 1 indicates 69.6%.
9. In Section “5.1 - Learning Curve and Dataset Size”, there is a claim that the performance of the model plateaus at ~10,000 instances. This does not seem supported by Figure 5. There appears to be over 5-7% accuracy (absolute) improvements from 10k to 100k examples. Maybe the graph needs to be enhanced for legibility?
10. It is great that the paper includes human numbers for both tasks, including all the metrics for generation.
11. Period missing in footnote 8.
12. The analysis section is interesting, it is useful to have in the paper. However, Table 3 is a bit disappointing in that ~26% of the sampled examples fit into one of the categories. It would be great if the authors could comment on the remaining ~74% of the sampled dataset.