From: NIPS.cc/2018/Workshop/IRASL
title: Acceptance Decision
decision: Accept

From: ~Tzeviya_Sylvia_Fuchs1
title: Author response 
comment: Thank you very much for your reviews, these are all fair comments.

(1) Thank you for your comment. In Lemma 1 in [9], we show that Acc_theta <= AUC, so PA-ACC’s performances are indeed bounded by PA-AUC. Additionally, please note that the model used on WSJ was trained on TIMIT, which could explain the mild improvement in AUC and ACC_theta values.

(2) This is a valid point. However, given that the weights were initialized randomly, we believe that the max pooling scalars and the ReLU scalar are not necessarily positive. Therefore, this case can be viewed as non-linear as well.

(3) The test set accuracy is indeed higher than the validation set accuracy. This could occur if the test set is much smaller and happens to be lest noisy than the validation set. Additionally, it could happen due to the algorithm’s regularization factor, and shows that the algorithm generalized well.

From: ~Tzeviya_Sylvia_Fuchs1
title: Author response
comment: Thank you very much for your reviews, these are all fair points.

(1) A surrogate loss function is a convex substitute to the original loss function, and is used when the
original loss function is hard to optimize.

(2) The siamese network has been implemented as in Kamper et al. [14], and yielded state of the art results: 1-D convolution with 96 filters over 9 frames; ReLU (Rectified Linear Unit) ; max pooling over 3 units; 1-D convolution with
96 filters over 8 units; ReLU; max pooling over 3 units; 2048-unit fully-connected ReLU; 1024-unit
fully-connected linear layer.

(3) Thank you for your comment. We agree that a graph would be clearer, but specifically, Table 4’s
main purpose is to show that Acc_theta loss gets higher results than the ranking loss for all epochs, which we believe is clear from the table.

From: ~Tzeviya_Sylvia_Fuchs1
title: Author response 
comment: Thank you very much for your reviews, these are all fair comments.

(1) The threshold we use is for all terms, not necessarily those in the training set. For example, in Table
1 we show experiments of PA-ACC on TIMIT, where the test set is composed of terms that are distinct
from the terms used in the training and validation sets.

(2) Please note that the threshold we use is not learned; rather, it is set. The algorithm learns to assign the
examples’ scores around this set threshold; therefore, by definition, the set threshold is the optimal
threshold.

(3) Thank you for your comment,  it would indeed be interesting to check our algorithm
against other datasets.

From: NIPS.cc/2018/Workshop/IRASL/Paper28/AnonReviewer3
title: Trainable thresholds help with the deployment of KWS systems
review: This is an excellent read and there is not much to add to the previous reviews.

I might be wrong about this: Is the threshold learned for a *fixed* set of keywords (ie. for a number of detectors and given a tuning dataset)? If so, it would be great if the authors could comment on how to deal with unknown search terms and potential OOV strategies.

Another interesting detail would be: how close to the optimal (cheating) threshold is the learned solution? Are there specific observations that go with that?

Also, it would be great to see those results on BABEL data, for comparison to other ATWV tuning strategies.
rating: 4: Top 50% of accepted papers, clear accept
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/IRASL/Paper28/AnonReviewer1
title: Well written paper with promising experimental results, but with some technical details.
review: In this paper, authors propose an approach for setting an absolute detection threshold for all terms in the spoken term detection task, by introducing a new calibrated loss function. Using this loss function, they explore two models for the spoken term detection: First, using a the structured prediction setting, and then, on a deep neural Siamese network in a weakly supervised setting.

Authors claim that the advantage of minimizing this loss function is that it aims at maximizing not only the relative ranking scores, but also adjusts the system to use a fixed threshold and thus enhances system robustness. Experiments on different datasets show the benefit of using the presented loss function.

Overall, the paper is well written and easy to read. Just a few comment that I consider important to clarify:
1) Experimental results shown in table 3 do not seem to give a significant improvement compared to other techniques. You are showing an improvement of 0.001 in AUC and 0.012 in Acc_theta respect the model PAC-AUC. An explanation of this small improvement can be useful. Is PAC-ACC converging to PAC-AUC? Both methods are equivalent using large data sets?
2) Introduction of parameter $\gamma$ in second model.  Authors say “the parameter $\gamma$ is not set to 1, since the function $f_u$ is not a linear function and hence is not scale invariant to the margin”. However, this is not totally true for the architecture presented in the experiments. In fact, this architecture is a concatenation of convolutions (which are linear), max pooling (which is scale invariant if the scalar is positive), relu (which is scale invariant if the scalar is positive (max(0,lambda x) = lambda max(0,x)), and linear layers.  Understanding that statement is true for general cases, it is not clear the utility in this particular case.
3) Results in table 1 look odd. In general, we expect that the validation accuracy is larger than the testing accuracy, under the assumption that both data sets come from the same distribution. A comment about this can help.

rating: 4: Top 50% of accepted papers, clear accept
confidence: 3: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature

From: NIPS.cc/2018/Workshop/IRASL/Paper28/AnonReviewer2
title: Very good paper. I like to congratulate the authors. 
review: >Define what is surrogate loss
Which implementation did the authors use for the Siamese network?
Table 4 can be presented as graph. Will be easier to follow.
Good read
rating: 4: Top 50% of accepted papers, clear accept
confidence: 1: The reviewer's evaluation is an educated guess