Paper:
In contextual bandits, an algorithm must choose actions given observed contexts, learning from a reward signal that is observed only for the action chosen. This leads to an exploration/exploitation trade-off: the algorithm must balance taking actions it already believes are good with taking new actions to potentially discover better choices. We develop a meta-learning algorithm, MÊLÉE, that learns an exploration policy based on simulated, synthetic contextual bandit tasks. MÊLÉE uses imitation learning against these simulations to train an exploration policy that can be applied to true contextual bandit tasks at test time. We evaluate on both a natural contextual bandit problem derived from a learning to rank dataset as well as hundreds of simulated contextual bandit problems derived from classification tasks. MÊLÉE outperforms seven strong baselines on most of these datasets by leveraging a rich feature representation for learning an exploration strategy.

Review:
•	Summary
    This paper introduced a meta-learning algorithm for the contextual bandit problem, MELEE, which learns an exploration policy based on simulated and synthetic contextual bandit tasks. The training is mainly divided into two steps. In step one, they proposed to train a policy optimizer, which maps features and actions to rewards. This policy optimizer could be used to reveal the most valuable action to take according to the modeled reward. All possible actions and their corresponding values are revealed to the policy optimizer because of the existing ground-truth labels in the synthetic dataset. The policy optimizer would then suggest which action to take. The algorithm takes the action in an  greedy fashion, i.e. with probability  it will follow the suggestion and with probability  it will sample it uniformly at random. The policy optimizer, historical actions and the taken actions are appended to the training set for training the exploration policy  in the next step. The procedure in step one is proposed to be done in  rounds. In step two, the training set is used for training an exploration policy  . During testing, the contexts are drawn from the real world, the policy optimizer will first evaluate the whole history and the exploration policy will generate actions with the input from the policy optimizer and the context. The algorithm suggests the action to explore in an  greedy fashion. The proposed algorithm is evaluated on a dataset for learning to rank, and 300 synthetic datasets. It shows better performances in most cases.
    I am critical about the paper  because 1) the experiments show that all more recently published exploration methods are worse than weak baselines, for which it lacks enough justification and convincing explanations. 2) the experiments are difficult to understand with only citation to publications. Detailed information of the datasets, tasks, procedures is missing.
•	Main arguments
    The paper is in general hard to follow because of too many citations of the previous works without simple explanations. It refers to the imitation algorithm, AggreVate, which is an instantiate of meta-learning for contextual bandits. Meanwhile, they failed to clarify the difference between the proposed algorithm and the existing one, making the training algorithm part confusing. The major concern lies in the experiment section, I can not see big performance difference between the proposed method and the  greedy based methods in Figure 1 (left) as 1) the variances are large 2) there are overlapping . It is surprising to see all recently published method are worse than the classical  greedy method. These results may require deeper investigations. In addiction, the used datasets in experiments are mentioned without any details and task definitions, which makes the experiments part unclear and the result not that convincing. Below are some other inconsistencies in the paper:
1.	I had a hard time to understand what the function  means. In Section 2,  is used to map user feature to predicted rewards for actions, i.e.  is a function with user features and actions as input and reward as output. However, in equation 1, it only takes user feature as input. In Section 5.1,  is called a classifier (I regard it as a variant of function  ). Both are not consistent with the mapping definition when first formally defined. This point confuses me so that I cannot fully understand what the POLOPT does as the function  is the output of it.
2.	By the end of section 2, the paper claims that they used direct method for it simplicity and unbiased property. However, as verified in [1], the direct method is biased with low variance whereas IPS is unbiased with high variance.
[1] Dudık, Miroslav, John Langford, and Lihong Li. “Doubly Robust Policy Evaluation and Learning.”
•	Things to improve that did not impact the score
a.	clear definition of the introduced notations, including the ones in algorithms 1.
•	Questions:
a.	What is  ?
b.	What is POLOPT?
c.	Why do you choose AggreVate?
d.	Will the performance change if using any other methods instead of direct method for policy optimizer?
e.	What is roll-out policy in line 8 of Algorithm 1? (In text only the roll-out value is defined.)
f.	What will happen if the roll-in action is different from the behavior in test time?
g.	How is exploration policy trained?