Paper:
Preference Based Adaptation for Learning Objectives
In many real-world learning tasks, it is hard to directly optimize the true performance measures, meanwhile choosing the right surrogate objectives is also difficult. Under this situation, it is desirable to incorporate an optimization of objective process into the learning loop based on weak modeling of the relationship between the true measure and the objective. In this work, we discuss the task of objective adaptation, in which the learner iteratively adapts the learning objective to the underlying true objective based on the preference feedback from an oracle. We show that when the objective can be linearly parameterized, this preference based learning problem can be solved by utilizing the dueling bandit model. A novel sampling based algorithm DL 2 M is proposed to learn the optimal parameter, which enjoys strong theoretical guarantees and efficient empirical performance. To avoid learning a hypothesis from scratch after each objective function update, a boosting based hypothesis adaptation approach is proposed to efficiently adapt any pre-learned element hypotheses to the current objective. We apply the overall approach to multi-label learning, and show that the proposed approach achieves significant performance under various multi-label performance measures.

Review:
The paper discusses a problem of adapting an optimization objective to a complex test objective. A complex test objective can either be hard to define (revenue, attrition, etc.) or hard to optimize directly (e.g. ranking losses), and it may not be clear which optimization objective (continuous, possibly convex) should be used. The paper proposes an approach that jointly optimizes the parameter according to an optimization objective and adapts the optimization objective to the complex test objective.  The main idea is to consider multiple optimization objectives and find the best linear combination of them. The solution has two main components. The first assumes black box access to a solver of the optimization task and finds the linear combination of the different objectives best suited to the test objective. The second solves each of the optimization objectives efficiently given the previous solutions. The first component assumes oracle access to the test loss, and build a solution via the multi-armed bandit technology. The second component adapts methods from multi-task learning to solve the particular problem at hand.  Novelty / Impact: The problem proposed is very common, and I am surprised it did not receive more attention. The authors bring an educated solution to this problem, and this type of approach is likely to be proven useful in many real-life problems.   The solution proposed seems somewhat raw. The main idea of having two components for (1) tuning the optimization objective (2) efficiently solving each optimization problem by utilizing previous solutions feels like the right thing to do. The details probably have room for improvement (see details below). That being said, the problem doesn’t have to be completely solved by a single paper, and I think the impact of the paper is still significant based on the importance of the problem and the clean high level solution  Writing / Clarity: There are some parts of the paper that could be improved. There are many definitions that are not here, but can only be obtained by reading the cited papers. The motivation, at least for algorithm 2 can be better explained, especially given that the algorithm does not come with any guarantees. Furthermore, if one would like to implement algorithm 2 based on the paper, they would have a very hard time doing so. The authors should make a considerable effort to explain exactly how the algorithm works.  Experiments: The experiments contain one experiment measuring the bandit algorithm (component 1) and another for the full system.   In the section discussing the full system, the authors use the test set for measuring the test objective in the bandit algorithm. Thought claiming it does not leak information, it does, exactly as in the case of hyper parameter optimization. A cleaner experiment could have been done with a validation set used to tune the optimization objective.   Another part that seems to be missing is an experiment measuring the effectiveness of the 2nd component without the first. For example, given the solutions to weights w_1,..,w_n, apply algorithm 2 to solve a new objective, and compare its performance to a solution obtained directly. Given that algorithm 2 doesn’t come with guarantees, that experiment would be more useful than experiments showing that algorithm 1 works.  Overall the experiments paint an OK picture but can be made much more convincing.   Summary: The paper tackles an important problem, its high level solution is clean and elegant, but the details leave room for improvement. The experiments are probably sufficient but not much more. Still, everything considered the fixes are doable within the time for the camera ready version and the advantages of the paper out-weigh its disadvantages.  Comments: * For algorithm 1, the guarantee is for the regret, but the objective is clearly simple regret, i.e. identifying the best linear function. The authors should consider either presenting a different proof, or even adapting the algorithm with that in mind. In particular, the discussion about the UCB approach not working is not surprising as UCB may not work well for best arm identification.   * In line 151 there is an explanation of how to adapt from the unit ball to positive numbers. A naive implementation of what is written may take time exponential in the dimension (drawing a random vector until all its components are positive). Please provide a more detailed explanation of how this can be done efficiently. Generally the explanation in the paragraph about the implementation is very vague and short. Perhaps add a detailed section in the appendix.  * line 163. The parameter l^i seems to come out of nowhere * line 214: the parameter \kappa in alg. 1 is a matrix, but in the experiments it’s fixed as a constant number. Please fix this issue. Also, compare the method to a parameter kappa that is set according to what the algorithm dictates, and not just a constant. The choice of setting it as a constant is not clear, and is even shown to be wrong by the experiments.  * Equation (13): Please define the the paper all the optimization objectives * Table 1: Please bold the best experiments in each setting