Paper:
We propose a general-purpose approach to discovering active learning (AL) strategies from data. These strategies are transferable from one domain to another and can be used in conjunction with many machine learning models. To this end, we formalize the annotation process as a Markov decision process, design universal state and action spaces and introduce a new reward function that precisely model the AL objective of minimizing the annotation cost We seek to find an optimal (non-myopic) AL strategy using reinforcement learning. We evaluate the learned strategies on multiple unrelated domains and show that they consistently outperform state-of-the-art baselines.

Review:
Summary:
This paper presents an RL approach to active learning that is generic across ML model being learned, and across dataset being used. The paper formulates the standard active learning problem as an MDP with the objective of minimizing the number of annotated labels required to meet a pre-specified prediction quality. 
The MDP state proposed by this paper is the current performance score on each sample in a hold-out set. The actions are specified by selecting a datapoint from the set of all un-annotated datapoints. The action feature vector consists of the current performance score of the model on the datapoint, and the average distance of that datapoint from every datapoint in the labeled set and every datapoint in the unlabeled set.
Review:
I do not recommend this paper for publication in ICLR because I believe:
1) the work is too incremental
2) the comparison to baseline and competing methods is incomplete
3) some design decisions of the proposed method are not well motivated.
I appreciated the clarity of the writting, and the paper organization. I also believe that the proposed method is quite intuitive, and is a good addition to the field. Finally, I appreciate that sufficient experimental details are available within the paper to be able to easily reproduce the results.
Details:
My points (1) and (2) are highly related, so I will discuss both simultaneously. I find that this paper makes only incremental forward progress from the Pang 2018 paper and the Konyushkova 2017 paper. The methodology here looks very similar to the SingleRL method, which Pang 2018 notes can be considered a special case of Konyushkova 2017's method. I think that the work in this paper would be sufficient to stand on its own if it performed a convincing comparison to SingleRL and/or MLP-GAL from Pang 2018. I recognize that this paper references why no such comparison currently exists, but I think this comparison would be extremely valuable to the paper.
A further comment on my point (2), I do not find the comparisons to baseline methods to be entirely convincing. Of note, only the average performance for each method is reported. I'm curious of the variance---and more specifically the standard error and number of independent runs---of each of the reported results. On many of the datasets, the performance difference between the proposed method and uncertainty sampling is quite small in table 1.
A final comment on point (2): I would have liked to see more exploration of different models. I think table 2 is quite informative, showing notable differences between simple baseline AL methods. I would have liked to see table 2 with more classifiers and with more competing AL methods. Because logistic regression is a simple model, the differences between AL methods may be more subtle. Perhaps a more complex model (say a single hidden layer NN) would show more notable differences.
For point (3), I would have liked to see either an exploration of other design decisions or an explanation of given design decisions. For instance, why only use 30 hold-out samples for the state? I imagine the proposed method would be fairly sensitive to this choice.  Another unexplained design decision was using a maximum budget of 100 datapoints. Table 2 shows some extremely interesting interactions with this budget in its comparison between LogReg-100 and LogReg-200, and further explanation would have been useful. Finally, I would have liked to see some motivation for choice of stopping condition. Using the stopping condition of 98% of maximum performance may have some biasing effect of each method, and it would helpful to have some motivation behind this choice.
Questions:
 - Why did uncertainty sampling have such limited benefits on LogReg-200 in table 2? This was a surprising result to me, as uncertainty sampling consistently outperformed most other methods.
 - Why is there a disparity between the results for the SVM in table 2 and the discussion in the first paragraph of section 4.3?
 - How does choice of final performance metric affect all methods? Choosing final performance to be 98% of maximum performance could have a major effect on each method. Because the proposed method is non-myopic, I would expect that it performs well when this value is large but would perform poorly with a smaller percentage of maximum performance.
 - Is the proposed method sensitive to number of samples used to compute the state?
 - What does figure 1 show? Are the same 30 samples used for all three subfigures? Perhaps this would more interpretable if, instead of showing the predicted class, this figure showed the prediction error.
Minor nitpicks (did not influence decision):
 - The datasets are 1-based indexed sometimes and 0-based indexed sometimes, even with disparities within a single paragraph.
 - Figure 1 appears a long time before it is discussed, which made it difficult to understand what was going on.