Paper:
Recent powerful pre-trained language models have achieved remarkable performance on most of the popular datasets for reading comprehension. It is time to introduce more challenging datasets to push the development of this field towards more comprehensive reasoning of text. In this paper, we introduce a new Reading Comprehension dataset requiring logical reasoning (ReClor) extracted from standardized graduate admission examinations. As earlier studies suggest, human-annotated datasets usually contain biases, which are often exploited by models to achieve high accuracy without truly understanding the text. In order to comprehensively evaluate the logical reasoning ability of models on ReClor, we propose to identify biased data points and separate them into EASY set while the rest as HARD set. Empirical results show that state-of-the-art models have an outstanding ability to capture biases contained in the dataset with high accuracy on EASY set. However, they struggle on HARD set with poor performance near that of random guess, indicating more research is needed to essentially enhance the logical reasoning ability of current models.

Review:
This paper presents a small multiple choice reading comprehension dataset drawn from LSAT and GMAT exams.  I like the idea of using these standardized tests as benchmarks for machine reading, and I think this will be a valuable resource for the community.
The two major concerns I have with this paper are with its presentation and with the quality of the baselines.  These concerns leave me at a weak accept, instead of a strong accept, which I would be if these issues were fixed.
Presentation:
The main contribution here is in collecting a new dataset drawn from the LSAT and GMAT.  (I assume that is all that is used, though the text actually says "such as".  Or is this from practice exams and not the actual exams?)  The job of this paper is to convince me, a researcher focusing on reading comprehension, that I should use this dataset as a target of my research.  Out of 8 pages, however, at most 2 are devoted to actually describing this contribution and why it's a good target for reading comprehension.  Much of the interesting description of the phenomena in the dataset is relegated to the appendix, which I did not really look at because it far exceeds the page limit of an ICLR submission.  I have a lot of questions about this dataset that could have been answered in the main text had more of the paper been given to actually describing the data.  Such as:
- What are some actual examples of the different kinds of reasoning in table 2?
- How did you decide the proportions of questions listed in table 2?  Was that manual?
- Where did the questions actually come from?  Real exams?  Practice exams?  Which ones?
- What is the reasoning process that a person would actually use to answer some of these questions?  Preferably answered for several of the question types that you listed.  You have 8 pages; there's a lot of space that could be given to this.
- What kinds of things do you see in the distractors that make this hard?
I'm recommending a weak accept for this paper, but only because of my background knowledge about the LSAT and GMAT, and my prior understanding of reading comprehension and why this would be a really interesting dataset.  I think the paper itself could do a *much* better job convincing a reader about the dataset than what is done here.
What should be cut to make room for this?
I think far too much space is given to the discussion of easy vs. hard (~2.5 pages).  Given that the best performance is at ~54% on the full test set, there isn't a lot of need for this.  The argument about four flips of a 25% coin giving a 0.39% chance of guessing right every time is only true if the coin flips are independently drawn; training the same pretrained model on the same data with a different random seed is clearly not independent, so this argument rings hollow.  It's not really clear what to conclude from the easy vs. hard split, other than the models that you used to create it expectedly do well on the easy split and hard on the test split.  You'd want to have separate models that create the split than what you're evaluating them with, but even then, you're training them on the same data, so it's still not really clear what to conclude.  It's sufficient in a case like this to just evaluate as a baseline a model that shouldn't have access to enough information to solve the task, and measure its performance on the test set.  That would have taken a few sentences to describe, instead of 2.5 pages.
You could also recover a lot of space by evaluating fewer baselines.  Describing six baseline models and including all of their performance numbers takes up a lot of space, and we really don't need to know what all of those numbers are.  One or two, along with question-only and option-only baselines, would be plenty.  If you really want to include all of them, that's information that should go into the appendix, instead of something that's core to your paper's contribution, like describing the dataset that you're releasing.
Quality of baselines:
With a dataset this small, you definitely want to pretrain BERT / RoBERTa on a larger QA dataset before training on ReClor.  I would guess that performance would be significantly higher if you used a RACE-trained model as a starting point, instead of raw BERT / RoBERTa.  And if it doesn't end up helping, then you've made a stronger argument about how your dataset is different from what has come before, and a good target for future research.  What I would recommend for table 6:
- chance
- option-only
- question-only
- raw XLNet or RoBERTa
- XLNet or RoBERTa pretrained on RACE, or SQuAD, or similar
And I would pick either XLNet or RoBERTa based on which one did the best after pre-training.  That way, related to my first point, you can remove the description of all other models, and table 4, and free up a bunch of space for more interesting things.
As an aside, I think it's good that the dataset is relatively small, as it gives these over-parameterized models less opportunity to overfit, and it makes us come up with other ways of obtaining supervision.  But you've got to be sure to use the best baselines available, and with small datasets, that means more pretraining on other things.
EDIT Nov. 19, 2019: The authors' revision has satisfied my concerns.  I think that this is a good paper and it should be accepted.