Paper:
Hybrid-MST: A Hybrid Active Sampling Strategy for Pairwise Preference Aggregation
In this paper we present a hybrid active sampling strategy for pairwise preference aggregation, which aims at recovering the underlying rating of the test candidates from sparse and noisy pairwise labelling. Our method employs Bayesian optimization framework and Bradley-Terry model to construct the utility function, then to obtain the Expected Information Gain (EIG) of each pair. For computational efficiency, Gaussian-Hermite quadrature is used for estimation of EIG. In this work, a hybrid active sampling strategy is proposed, either using Global Maximum (GM) EIG sampling or Minimum Spanning Tree (MST) sampling in each trial, which is determined by the test budget. The proposed method has been validated on both simulated and real-world datasets, where it shows higher preference aggregation ability than the state-of-the-art methods.

Review:
This paper proposed a novel hybrid active strategy for collecting pair-wise samples for BT model estimation. The estimation is based on minimum spanning tree and several other insights. The proposed method shows reliable and more precise performance on simulated and real-world datasets. The paper is overall well organized.   - missing some prior arts in the active learning and BLT model fields. Just to recall a few from top of my head. http://www.jmlr.org/papers/volume17/15-189/15-189.pdf http://proceedings.mlr.press/v28/wauthier13.pdf https://arxiv.org/abs/1109.3701 - There has been a lot of abbreviations in the paper which makes it difficult to read.  - In table 1, the author claimed that  the computation cost is comparable with Hodge based methods in large-scale crowd-sourcing experiments. It is not clear why this is a valid claim and is quite opposite from what stated in table 1.   [based on the author response and the re-organized/commented computation cost table, it is clear that how the proposed approach gain advantage in terms of computation cost. It would be good to update the caption as well as the discussion in the paper draft to make this confusing point clear. In addition, since there are some "navie" non-active baselines, it would be good to clarify in the caption of table saying these approaches are not appropriate to compare and judge efficiency. ]  Some additional comments: - In section 3.2, it would be better to clarify how the ground-truth labels/scores are assigned so it is easier to understand the RMSE/PLCC metrics.  Though there is a space limits, some high-level description might help reader to understand in these two problems, what are the target problem and why intuitively ranking them are challenging tasks.   - It seems a bit untypical that section 2.3.3 contains simulation while other experiments are in separate sections. Many of the simulation settings are discussed here which kind of complicated the reading process.   - it would be better not to use many abbreviations in the main algorithm. Indeed, in Alg 1 the main steps of estimating info-gain eq(6) is not what used in the end. It is explicitly eq(7) and its practical approximations that enables the efficient computation. Highly suggest to make Alg. 1 referring to the exact algorithms used.   -2.0 section is labeled as "notation" but actually contains key distribution assumptions.   - Figure 1 (a) is almost unrecognizable. Please update it and include the parameters used to simulate these contour plots.