Paper:
It has been argued that current machine learning models do not have common sense, and therefore must be hard-coded with prior knowledge (Marcus, 2018). Here we show surprising evidence that language models can already learn to capture certain common sense knowledge. Our key observation is that a language model can compute the probability of any statement, and this probability can be used to evaluate the truthfulness of that statement. On the Winograd Schema Challenge (Levesque et al., 2011), language models are 11% higher in accuracy than previous state-of-the-art supervised methods. Language models can also be fine-tuned for the task of Mining Commonsense Knowledge on ConceptNet to achieve an F1 score of 0.912 and 0.824, outperforming previous best results (Jastrzebski et al., 2018). Further analysis demonstrates that language models can discover unique features of Winograd Schema contexts that decide the correct answers without explicit supervision.

Review:
This paper evaluates language models for tasks that involve "commonsense knowledge" such as the Winograd Schema Challenge (WSC), Pronoun Disambiguation Problems (PDP), and commonsense knowledge base completion (KBC). 
Pros:
The approach is relatively simple in that it boils down to just applying language models. 
The results outperform prior work, in some cases by pretty large margins. 
The language models are quite large and it appears that this is the first time that large-scale language models have been applied seriously to the Winograd Schema Challenge (rather than, say, to the NLI version of it in GLUE, to which it is hard to compare these results). 
Some of the additional and ablation experiments are interesting. 
Cons:
While this paper has some nice results, there are some aspects of it that concern me, specifically related to hyperparameter tuning and experimental rigor:
There are three methods given for using an LM to make a prediction: full, full-normalized, and partial. For PDP, full (or perhaps full-normalized?) works best, while for WSC, partial works best. The differences among methods, at least for WSC, are quite large: from 2% to 10% based on Figure 3. I don't see a numerical comparison for PDP, so I'm not sure how these methods compare on it. Since the datasets are so small, there is no train/dev/test split, so how were these decisions made? They seem to be oracle decisions. This is concerning to me, as there is not much explanation given for why one method is better than another method. 
My guess is that the reason why partial works better than full for WSC is because the WSC sentences were constructed such that the words up to and including the ambiguous pronoun were written such that it would be difficult to identify the antecedent of the pronoun. The rest of the sentence would be needed to identify the antecedent. I'll assume for this discussion that the sentence can be divided into three parts x, y, and z, where x is the part before the pronoun, y is the phrase that replaces the pronoun, and z is the part after the pronoun. Then p(z|xy), which is partial scoring, corresponds to p(xyz)/p(xy), which can be viewed as "discounting" or "normalizing for" the probability of putting y in place of the pronoun given the context x. For WSC, I think one of the goals in writing the instances is to make the "true" p(xy) approximately equal for both values of y. The language model will not naturally have this be the case (i.e., that p(xy) is the same for both antecedents), so dividing by p(xy) causes the resulting partial score to account for the natural differences in p(xy) for different antecedents. This could be explored empirically. For example, the authors could compute p(xy) for both alternatives for all PDP and WSC instances and see if the difference (|p(xy_1) - p(xy_2)|, where y_1 and y_2 are the two alternatives) is systematically different between WSC and PDP. Or one could see if p(xy) is greater for the antecedent that is closer to the pronoun position or if it is triggered by some other effects. It could be the case that the PDP instances are not as carefully controlled as the WSC instances and therefore some of the PDP instances may exhibit the situation where the prediction can be made partially based on p(xy). The paper does not give an explanation for why full scoring works better for PDP and chalks it up to noise from the small size of PDP, but I wonder if there could be a good reason for the difference.
The results on KBC are positive, but not super convincing. The method involves fine-tuning pretrained LMs on the KBC training data, the same training data used by prior work. The new result is better than prior work (compared to the "Factorized", the finetuned LM is 2.1% better on the full test set, and 0.3% better on the novelty-based test set), but also uses a lot more unlabeled data than the prior work (if I understand the prior work correctly). It would be more impressive if the LM could use far fewer than the 100K examples for fine-tuning. Also, when discussing that task, the paper says: "During evaluation, a threshold is used to classify low-perplexity and high-perlexity instances as fact and non-fact." How was this threshold chosen?
I also have a concern about the framing of the overall significance of the results. While the results show roughly a 9% absolute improvement on WSC, the accuracies are still far from human performance on the WSC task. The accuracy for the best pretrained ensemble of LMs in this paper is 61.5%, and when training on WSC-oriented training data, it goes up to nearly 64%. But humans get at least 92% on this task. This doesn't mean that the results shouldn't be taken seriously, but it does suggest that we still have a long way to go and that language models may only be learning a fraction of what is needed to solve this task. This, along with my concerns about the experimental rigor expressed above, limits the potential impact of the paper.
Minor issues/questions:
In Sec. 3.1: Why refer to the full scoring strategy as "naive"? Is there some non-empirical reason to choose partial over full?
The use of SQuAD for language modeling data was surprising to me. Why SQuAD? It's only 536 articles from Wikipedia. Why not use all of Wikipedia? Or, if you're concerned about some of the overly-specific language in more domain-specific Wikipedia articles, then you could restrict the dataset to be the 100K most frequently-visited Wikipedia articles or something like that. 
I think it would be helpful to give an example from PDP-60.
Sec. 5.1: How is F_1(n) defined?  I also don't see how a perfect score is 1.0, but maybe it's because I don't understand how F_1(n) is defined.
Sec. 6.1: Why would t range from 1 to n for full scoring? Positions before k are unchanged, right? So q_1 through q_{k-1} would be the same for both, right?
In the final example in Figure 2, I don't understand why "yelled at" is the keyword, rather than "upset". Who determined the special keywords?
I was confused about the keyword detection/retrieval evaluation. How are multi-word keywords handled, like the final example in Figure 2? The caption of Table 5 mentions "retrieving top-2 tokens". But after getting the top 2 tokens, how is the evaluation done?
Sec. 6.3 says: "This normalization indeed fixes full scoring in 9 out of 10 tested LMs on PDP-60." Are those results reported somewhere in the paper? Was that normalization used for the results in Table 2?
Sec. 6.3 says: "On WSC-273, the observation is again confirmed as partial scoring, which ignores c [the candidate] altogether, strongly outperforms the other two scorings in all cases" -- What is meant by "which ignores c altogether"?  c is still being conditioned on and it must not be ignored or else partial scoring would be meaningless (because c is the only part that differs between the two options). 
Typos and minor issues:
Be consistent about "common sense" vs. "commonsense".
Be consistent about "Deepnet" vs. "DeepNet" (Tables 2-3).
Sec. 1:
"even best" --> "even the best"
"such as Winograd" --> "such as the Winograd"
"a few hundreds" --> "a few hundred"
"this type of questions" --> "this type of question"
"does not present" --> "is not present"
"non-facts tuples" --> "non-fact tuples"
Sec. 2:
"solving Winograd" --> "solving the Winograd"
"Store Cloze" --> "Story Cloze"
"constructed by human" --> "constructed by humans"
Sec. 4:
What is "LM-1-Billion"?
Why SQuAD?
"Another test set in included" --> "Another test set is included"
Sec. 5.2:
Check margin in loss_new
"high-perlexity" --> "high-perplexity"
Sec. 6:
Figure 2 caption: "keyword appear" --> "keyword appears"
Sec. 6.2:
"for correct answer" --> "for the correct answer"
Appendix A:
"acitvation" --> "activation"
Appendix B:
Figure 4 caption: "is of" --> "is"
The right part of Figure 4 has some odd spacing and hyphenation.