Paper:
Preference Completion from Partial Rankings
We propose a novel and efficient algorithm for the collaborative preference completion problem, which involves jointly estimating individualized rankings for a set of entities over a shared set of items, based on a limited number of observed affinity values. Our approach exploits the observation that while preferences are often recorded as numerical scores, the predictive quantity of interest is the underlying rankings. Thus, attempts to closely match the recorded scores may lead to overfitting and impair generalization performance. Instead, we propose an estimator that directly fits the underlying preference order, combined with nuclear norm constraints to encourage low-rank parameters. Besides (approximate) correctness of the ranking order, the proposed estimator makes no generative assumption on the numerical scores of the observations. One consequence is that the proposed estimator can fit any consistent partial ranking over a subset of the items represented as a directed acyclic graph (DAG), generalizing standard techniques that can only fit preference scores. Despite this generality, for supervision representing total or blockwise total orders, the computational complexity of our algorithm is within a log factor of the standard algorithms for nuclear norm regularization based estimates for matrix completion. We further show promising empirical results for a novel and challenging application of collaboratively ranking of the associations between brain-regions and cognitive neuroscience terms.

Review:
The submission proposes a new approach for preference completion. Frequently used methods are based on low rank matrix completion or the `learning to rank' formulation, both of which rely heavily on the scores provided by the user. In contrast, the proposed approach does not directly rely on scores but rather on rankings between pairs of elements. Hence scores can be transformed in an arbitrary but monotonic way. More generally, the proposed approach is also directly applicable to given partial rankings, without any transformation to scores.  To this end, a set of vectors is defined which fulfill the provided ordering with at least an epsilon-margin. The estimator for the preference matrix is then obtained by minimizing its nuclear norm and its deviation from the entries in the set of vectors fulfilling the provided ordering.  A proximal gradient descent algorithm is suggested which results in two steps, (i) a singular value thresholding when optimizing w.r.t. the preference matrix, and (ii) a projection onto the feasible set of valid rankings. The latter projection is equivalent to isotonic regression, hence it can be solved using the `Pool Adjacent Violators' algorithm if we consider strict or blockwise total ordering of preferences. A more complex algorithm has to be used in case we are given an arbitrary (consistent) ordering.  The algorithmic complexity of the proposed algorithm is composed out of two parts, (i) singular value thresholding, which is also required in standard matrix completion techniques, and (ii) projection onto the feasible set, which is feasible in simple ordering cases but expensive generally.  The proposed approach is evaluated on the Movielens dataset and a neurosynth dataset and shown to outperform simple baselines such as standard low-rank matrix completion and Cofi-Rank. 
Review summary: -------- The proposed technique does not directly take into account the provided preference scores which is a clear advantage since scores may often be noisy. However the proposed approach is generally very expensive (an experimental comparison hasn't been shown) and it might not handle inconsistent ordering very well. The set of feasible vectors may indeed be empty and no mechanism is provided for the algorithm to fail gracefully.  Technical quality: the feasible set might be empty if confusing rankings are provided as input, which hasn't been analyzed Novelty: an interesting technique for ranking Impact: a comparison with recent methods is missing Clarity: well described generally  Review details: -------- 1. The defined set of vectors fulfilling the provided ordering may be empty or contain only a few elements if the provided groundtruth ordering is not consistent. The proposed approach does not seem to fail gracefully in this setting, i.e., a confusing ranking may actually remove good solutions. Did the authors investigate the properties of the proposed algorithm in this setting?  2. While Lemma 2 is interesting, it's important to relate it back to the original problem. As a reader I'm therefore wondering whether the proposed approach still optimizes the original formulation.  3. Some more intuition could according to my opinion be provided by shortining the intro. In particular the description of the generalization bound is very short and follows from existing literature. This could be stated more clearly. Was the symbol `d' defined?  4. In the experimental evaluation, classical baselines are employed. As a reader I'm wondering about more recent approaches and a comparison w.r.t. the proposed algorithm. Can the authors comment?  5. Ideally it should also be evaluated whether it's worth to spend the additional effort of projecting onto the feasible set. An evaluation of the computation time would be desirable.  Minor comments: -------------- - typo in Eq. (2)? Shouldn't there be a minimization over z? - possible typo in Prop. 1? It seems like the second \gamma in the last equation shouldn't be inverted. Didn't check completely though. - possible typo in Lemma 2: a minus sign may be missing in the characterization of the constraint set to make it consistent with the definition of the feasible set in line 218. - the writing should be corrected here and there