Paper:
Faster Online Learning of Optimal Threshold for Consistent F-measure Optimization
In this paper, we consider online F-measure optimization (OFO). Unlike traditional performance metrics (e.g., classification error rate), F-measure is nondecomposable over training examples and is a non-convex function of model parameters, making it much more difficult to be optimized in an online fashion. Most existing results of OFO usually suffer from high memory/computational costs and/or lack statistical consistency guarantee for optimizing F-measure at the population level. To advance OFO, we propose an efficient online algorithm based on simultaneously learning a posterior probability of class and learning an optimal threshold by minimizing a stochastic strongly convex function with unknown strong convexity parameter. A key component of the proposed method is a novel stochastic algorithm with low memory and computational costs, which can enjoy a convergence rate of O(1/ √ n) for learning the optimal threshold under a mild condition on the convergence of the posterior probability, where n is the number of processed examples. It is provably faster than its predecessor based on a heuristic for updating the threshold. The experiments verify the efficiency of the proposed algorithm in comparison with state-of-the-art OFO algorithms.

Review:
The paper proposes a stochastic/online algorithm for optimizing the F-measure. Previous stochastic approaches for F-measure optimization either come with guarantees of rates of convergence to the optimal value of a 'surrogate' objective, or come with asymptotic consistency guarantees for the 'original' F-measure. In this paper, the authors build on ideas from Busa-Fekete et al. (2015) and develop an algorithm which has better empirical performance than previous methods, while enjoying (non-asymptotic) rates of convergence to the optimal F-measure value.  The proposed algorithm is based on the observation that the optimal classifier for the F-measure can be obtained by thresholding the posterior class probability at a suitable point, and performs simultaneous stochastic updates on the posterior class probability model and on the threshold. While Busa-Fekete et al. pose the threshold selection problem as a root finding problem, the present paper formulates this problem as a strongly convex optimization problem. This allows the authors to show that under very specific assumptions on the posterior class probability model and on the algorithm for learning the model, the proposed approach has a O(1/\sqrt{n}) convergence rate to the optimal F-measure value.   Pros: - The idea of posing the threshold selection problem as a strongly convex optimization problem is indeed interesting. - The proposed algorithm seems to empirically out-perform the state-of-the-art algorithms for F-measure, achieving higher or comparable F-measure values at a faster rate.  Cons: However, the convergence/consistency guarantee that the authors provide seems to be under strong assumptions on the posterior class probability distribution and on the algorithm used to learn the posterior probability model. In particular, the authors assume/state:  - The posterior class probability distribution can be parametrized as a generalized linear model  - The expected logistic loss computed on the posterior probability model is (globally) *strongly convex* - There exists an online algorithm that enjoys *high probability convergence rates* to the optimal parameters w^* of the posterior prob. model, i.e. can provides iterates w_1,\ldots, w_T such that for each ’t’, with high probability, ||w_t - w^*|| \leq C’ / t, for a problem-independent constant C’  (of these the second statement is not an assumption, but stated in ll. 217)  I find the above to be problematic:  Firstly, the assumption that the posterior class probability distribution takes a very specific parametrized form seems restrictive, especially because the analysis gives no room for model misspecification. I understand the analysis might also extend to slightly more general posterior probability distributions that can be written as a  sigmoid of a score function, but even this is a special case, and excludes a whole range of real-world distributions.  Secondly, the authors’ statement that the expected logistic loss is (globally) strongly convex is not true in general. The authors cite Agarwal et al. (2012) [1] , but even there it is only shown that the expected logistic loss is locally strongly convex (p.6) where the strong convexity parameter depends on the marginal distribution over ‘x’ (in particular on the min. Eigen value of the covariance matrix of ‘x’).  Finally, in the absence of (global) strong convexity, I do not think (as the authors claim) the algorithm of Agarwal et al. (2012) [15] can be applied to provide high probability convergence guarantees on individual model vectors. Indeed there have been some works (e.g. Bach et al., "Adaptivity of Averaged Stochastic Gradient Descent to Local Strong Convexity for Logistic Regression”, JMLR’14) that analyze SGD specifically for the logistic loss, but seem to provide weaker “expected” guarantees (and where C’ would be distribution-dependent)  (also please see pp. 2, end of paragraph 2 in Bach (2014) for a discussion on strong convexity of logistic loss)  It would be good if the authors are able to provide a reference to a specific algorithm that can satisfy the assumptions they make.  I do understand that the previous work of Busa-Fekete et al. also makes assumptions on the class probability learner, but their assumptions was on the class probability estimates \eta(x) and in particular on the expected absolute difference between the current class probability estimates and the true class probabilities (i.e. on E[|\eta(x) - \hat{\eta}_t(x)|]). The present paper makes stronger assumptions, namely a specific parametric form for the underlying class probability model, and high probability rates of convergence on the model parameters.  Having said this, I would also like to mention that I am sympathetic to this line of work and can see why it is challenging to provide non-asymptotic guarantees for F-measure optimization (i.e. on the original F-measure and not a surrogate). The strongly convex re-formulation that the authors propose for threshold search is definitely interesting, and their empirical results indicate that this formulation can be beneficial in practice. However, the theoretical guarantees provided seem to be under restrictive assumptions.   Other Comments: - Experiments: -- Please report F-measure performance as a function of running time rather than *iteration count*. Since an iteration may constitute different computations for different methods, it is only fair that you compare the methods against their running time. Indeed you do include run-time plots in the supplementary material, but please do have them in the main text. -- I see a large gap between FOFO and OFO for some of the data sets. Do you find that when you run for enough iterations eventually, both FOFO and OFO converge to the same threshold? How does the learned threshold compare with an offline/brute-force threshold tuning procedure? -- I see that the STAMP method you compare against has an epoch size parameter. How do you tune this parameter? Is there a specific reason you chose the number of classifiers in OMCSL to 10?  - Remark on line 232: You mention that Theorem 2 (convergence of inner SFO) can be used to derive a O(1/\sqrt{n}) convergence result for FIFO. I don’t see immediately how you would get this result (are you referring to having only one run of SFO with T = n?). In particular, Theorem 2 holds under the assumption that |\theta_1 - \theta^*| \leq R. This is true for the first invocation of SFO, but for subsequent invocations (k > 2), where the radius R is multiplicatively reduced, how do you make sure that this condition continues to hold?  - Same sample used for learning probability model and threshold: It seems to me that in Algorithm 2, the same sample is used by for both updating the posterior model parameters w_t (line 15) and updating the threshold \theta_t (line 13). I wonder if this would introduce an additional probabilistic dependence in your analysis. My guess is that a dependence is avoided by first updating the threshold and then the posterior model.  Minor comments: - Definition 1: Might be good to explicitly mention that the probability is over random draw of n samples - Lemma 2: It seem from the proof in the supplementary material that c = 2/\pi for all \theta. May want to just mention this in the lemma. - Theorem 3: SFO -> FOFO, also should ‘m’ be a superscript on \theta (looking at Algorithm 1)  ------- POST REBUTTAL ------- ------------------------------------ I thank the authors for the rebuttal. Having read their response and heard the views of the other reviewers, I'm increasing my score to 7.  I would however recommend that the authors:  (1) restate assumption 1 in terms of a condition on the estimated class probabilities \hat{\eta}_t (rather than making a specific parameterized assumption on the posterior distribution) (2) give examples where this assumption holds (one of which is when the expected loss is strongly convex) (3) clarify up-front in the paper that their result holds under a ‘point-wise’ convergence assumption and that it differs from previous works (currently the authors just mention that they need a ‘mild’ condition)  If one were to just assume that the estimated probabilities converge in a weaker expected sense (e.g. the assumption in [3[), would your algorithm at least have an asymptotic convergence guarantee like [3]? If so, I would appreciate it if you could add a discussion on this.