AB / BA analysis: A framework for estimating keyword spotting recall improvement while maintaining audio privacy (2022)

Abstract: Evaluation of keyword spotting (KWS) systems that detect keywords in speech is a challenging task under realistic privacy constraints. The KWS is designed to only collect data when the keyword is present, limiting the availability of hard samples that may contain false negatives, and preventing direct estimation of model recall from production data. Alternatively, complementary data collected from other sources may not be fully representative of the real application. In this work, we propose an evaluation technique which we call AB/BA analysis. Our framework evaluates a candidate KWS model B against a baseline model A, using cross-dataset offline decoding for relative recall estimation, without requiring negative examples. Moreover, we propose a formulation with assumptions that allow estimation of relative false positive rate between models with low variance even when the number of false positives is small. Finally, we propose to leverage machine-generated soft labels, in a technique we call Semi-Supervised AB/BA analysis, that improves the analysis time, privacy, and cost. Experiments with both simulation and real data show that AB/BA analysis is successful at measuring recall improvement in conjunction with the trade-off in relative false positive rate.

## introduction keyword spotting (kws) is the task of identifying if one of a set of keywords, also called wakewords, is present in a speech segment. it is the gatekeeper component that enables hands-free interaction with many smart assistants using voice-enabled smart devices, such as amazon echo, google home, and apple homepod. extensive work has been done to develop and improve kws performance, including improvements in architecture @xcite @xcite @xcite * corresponding authors @xcite , training efficiency @xcite , as well as audio front-end (afe) algorithms @xcite . with many lines of research aiming to improve the quality of the kws, there is also growing interest in techniques to measure if such new technologies are able to improve the customer experience, but less research attention has been given to this evaluation topic. one challenge is that kws systems are designed to maximize user privacy by only collecting data when the keyword is identified. as a result, evaluations done using this biased dataset do not allow direct measurement of gains in recall metrics to determine if a new model is better than a baseline. alternatively, evaluations can also make use of datasets not collected by the kws, such as media recordings, background noise, and environmental sounds. however, these datasets may not be fully representative of the user experience from the evaluation point of view. prior research, such as from @xcite , has made use of datasets with positive and negative labels in order to evaluate, respectively, gain invariant kws models and cnn for small-footprint kws. however, the negative data essentially consists of either negative labels obtained from data accepted by the previous models, or datasets composed of just background noise and noise from environment. in a different application context, @xcite estimates the recall and the derivative of the precision with respect to the recall by modeling the unseen data distribution according to underlying assumptions. this distribution, however, is not clearly defined in the context of kws when a variety of sounds, noisy conditions, and reverberation can occur. in this paper, we present a new evaluation framework called ab/ba analysis. to the best of our knowledge, our work is the first to explore the problem of estimating recall improvement when only accepted data is available, such as in a kws system with rigorous privacy settings, in conjunction with the trade-off with false positive rate, without underlying assumptions on the data distribution. we show that the ab/ba analysis is a cross-model offline evaluation framework. in this framework, data is collected from two kws models, say a baseline, a , and a candidate, b. by running offline model a through data collected by model b, and model b through data collected by model a, we are able to calculate relative metrics without the need for data not seen by individual models. we also present assumptions that can be applied to the relative metrics calculation that result in a lower variance estimator, even if the number of false positive is small. we additionally describe a semi-supervised formulation of ab/ba analysis which provides improvements in the analysis time, cost, and data privacy by utilizing soft machine-generated labels instead of human annotations from the models being evaluated. we present experiments using simulation data, real data, and also experiments comparing the performance of ab/ba and semi-supervised ab/ba. ## methods in this section, we describe the proposed method for estimation of recall improvement. basic metrics concepts can be found in appendix a.1. ## experiments in this section we present experiments to show the performance of ab/ba and ssab/ba on simulation and real data. ## conclusion in this paper, we presented a new framework called ab/ba analysis for recall improvement estimation of kws under high privacy settings. we have shown that by running a candidate model offline on data collected by a baseline, and the baseline model offline on data collected by the candidate model, we were able to compute the relative recall and relative fpr ratios using only utterances accepted by both models and use it to indicate if a candidate model is better than the baseline. we have also shown that reasonable assumptions can be used to construct an estimator with low variance, even when the number of fps is small. finally, we saw that a semi-supervised formulation of ab/ba can be used with machine-generated labels representing the probability of a true accept. this techniques brings further improvements to ab/ba analysis, especially regarding privacy on the audio collected, which does not need to be listened to and annotated. in the past few years, much improvement has been made on the kws systems. evaluation metrics, however, are typically based on previously collected data from similar kws models or data from other sources, resulting in evaluations that do not necessarily translate to customer experience, as it fails to show improvements in data never collected due to privacy constraints. given that not much attention has been paid to this research topic, we believe that ab/ba analysis is a valuable contribution, and we hope it helps bringing interest in this line of research.