Paper:
Work on the problem of contextualized word representation-the development of reusable neural network components for sentence understanding-has recently seen a surge of progress centered on the unsupervised pretraining task of language modeling with methods like ELMo . This paper contributes the first large-scale systematic study comparing different pretraining tasks in this context, both as complements to language modeling and as potential alternatives. The primary results of the study support the use of language modeling as a pretraining task and set a new state of the art among comparable models using multitask learning with language models. However, a closer look at these results reveals worryingly strong baselines and strikingly varied results across target tasks, suggesting that the widely-used paradigm of pretraining and freezing sentence encoders may not be an ideal platform for further work.

Review:
The work presented in this paper relates to the impact of the dataset on the performance of contextual embedding (namely ELMO in this paper) on many downstream tasks, including GLUE tasks, but also alternative NLP tasks.
The work is focused on experiments, and draws several conclusions that are interesting, mostly around the amount of gain one can expect and the fact that the choice of the dataset is task-dependent. 
One of the issue is that the authors if seems to believe that ELMO is the best contextual language model. The field is moving so quickly that the experiments might become invalid pretty soon (e.g. see BERT model referenced below).
Finally, the analysis is mostly descriptive and there is few insight by the author about what should be the future work, apart from "we need a better understanding".
Minor details:
Page 1: "can yield very strong performance on NLP tasks" is a very busy way to express the fact that Sentence Encoders work well in practice. 
The field evolves quickly and ELMO has now a competitive models called BERT (arXiv.org > cs > arXiv:1810.04805). I understand that the results of the current papers would hove to be re-run on all these tasks, but I'm afraid the current paper will have a limited impact if it does not use the most effective method at the date of publication...