Robust Spoken Term Detection Automatically Adjusted for a Given Threshold
Abstract
Spoken term detection (STD) is the task of determining whether and where a given word or phrase appears in a given segment of speech.
Algorithms for STD are often aimed at maximizing the gap between the scores of positive and negative examples.
As such they are focused on ensuring that utterances where the term appears are ranked higher than utterances where the term does not appear.
However, they do not determine a detection threshold between the two.
In this paper, we propose a new approach for setting an absolute detection threshold for all terms by introducing a new calibrated loss function.
The advantage of minimizing this loss function during training is that it aims at maximizing not only the relative ranking scores, but also adjusts the system to use a ﬁxed threshold and thus enhances system robustness and maximizes the detection accuracy rates.
We use the new loss function in the structured prediction setting and extend the discriminative keyword spotting algorithm for learning the spoken term detector with a single threshold for all terms.
We further demonstrate the effectiveness of the new loss function by applying it on a deep neural Siamese network in a weakly supervised setting for template-based spoken term detection, again with a single ﬁxed threshold.
Experiments with the
TIMIT, WSJ and Switchboard corpora showed that our approach not only improved the accuracy rates when a ﬁxed threshold was used but also obtained higher Area
Under Curve (AUC).
Introduction
Spoken term detection (STD) refers to the proper detection of any occurrence of a given word or phrase in a speech signal.
Typically, any such system assigns a conﬁdence score to every term it presumably detects.
A speech signal is called positive or negative , depending on whether or not it contains the desired term.
Ideally, an STD system assigns a positive speech input with a score higher than the score it assigns to a negative speech input.
During inference, a detection threshold is chosen to determine the point from which a score would be considered positive or negative.
The choice of the threshold represents a trade-off between different operational settings, as a high value of the threshold could cause an excessive amount of false negatives (instances incorrectly classiﬁed as negative), whereas a low value of the threshold could cause additional false positives (instances incorrectly classiﬁed as positive).
The performance of STD systems can be measured by the Receiver Operation Characteristics (ROC) curve, that is, a plot of the true positive (spotting a term correctly) rate as a function of the false positive (mis-spotting a term) rate.
Every point on the graph corresponds to a speciﬁc threshold value.
The area under the ROC curve (AUC) is the expected performance of the system for all threshold values.
A common practice for ﬁnding the threshold is to empirically select the desired value using a cross validation procedure.
In [3], the threshold was selected using the ROC curve.
Similarly, in [8, 17] and the references therein, the threshold was chosen such that the system maximized the Actual Term
Weighted Value (ATWV) score [15].
Additionally, [19] claims that a global threshold that was chosen for all terms was inferior to using a term speciﬁc threshold [18].
In this paper we propose a new method to embed an automatic adjustment of the detection threshold within a learning algorithm, so that it is ﬁxed and known for all terms.
We present two algorithmic implementations of our method: the ﬁrst is a structured prediction model that is a variant of the discriminative keyword spotting algorithm proposed by [16, 21, 22], and the second implementation extends the approach used for the structured prediction model on a variant of whole-word Siamese deep network models [10, 2, 14].
Both of these approaches in their original form aim to assign positive speech inputs with higher scores than those assigned to negative speech inputs, and were shown to have good results on several datasets.
However, maximizing the gap between the scores of the positive and negative examples only ensures the correct relative order between those examples, and does not ﬁx a threshold between them; therefore it cannot guarantee a correct detection for a global threshold.
Our goal is to train a system adjusted to use a global threshold valid for all terms.
In this work, we set the threshold to be a ﬁxed value, and adjust the decoding function accordingly.
To do so, we propose a new loss function that trains the ranking function to separate the positive and negative instances; that is, instead of merely assigning a higher score to the positive examples, it rather ﬁxes the threshold to be a certain constant, and assigns the positive examples with scores greater than the threshold, and the negative examples with scores less than the threshold.
Additionally, this loss function is a surrogate loss function which extends the hinge loss to penalize misdetected instances, thus enhancing the system’s robustness.
The new loss function is an upper bound to the ranking loss function, hence minimizing the new loss function can lead to minimization of ranking errors, or equivalently to the maximization of the AUC.
Problem Setting
In the STD task, we are provided with a speech utterance and a term and the goal is to decide whether or not the term is uttered.
The term can be provided as a sequence of phonemes or by an acoustic representation given by a speech segment in which the term is known to be uttered.
Throughout the paper, scalars are denoted using lower case Latin letters, e.g., x , and vectors using bold face letters, e.g., x .
A sequence of elements is denoted with a bar ( ¯ x ) and its length is written as |
¯ x | .
Formally, the speech signal is represented by a sequence of acoustic feature vectors ¯ x = ( x 1 , . . .
, x T ) , where each feature vector is d dimensional x t ∈ R d for all 1 ≤ t ≤ T .
Note that in our setting the number of frames T is not ﬁxed.
We denote by X
=
( R d ) ∗ the set of all ﬁnite length sequences over
R d .
A sequence of L r phonemes of a term r is denoted as ¯ p r =
( p 1 , . .
, p L r ) , where p l ∈ P for all 1 ≤ l ≤ L r and P is the set of phoneme symbols.
We denote by P ∗ the set of all ﬁnite length sequences over P .
A term is a word or a short phrase and is presented to the system either as a sequence of phonemes in the strongly supervised setting or as an acoustic segment containing the term in the weakly supervised setting.
We denote the abstract domain of the term representations (as either a phoneme sequence or an acoustic segment) by R .
Our goal is to ﬁnd a spoken term detector , which takes as input a speech segment and a term and returns a binary output indicating whether the term was pronounced in the acoustic segment or not.
Most often the spoken term detector is a function that returns a real value expressing the conﬁdence that the target term has been uttered.
The conﬁdence score outputted by this function is compared to a threshold , and if the score is above the threshold the term is declared to have been pronounced in the speech segment.
Formally, the detector is a function f from X × R to R .
The detection threshold is denoted by the scalar θ ∈ R .
Usually there is no single threshold for all terms, and it needs to be adjusted after decoding.
Our goal in this work is to propose a new method to learn the spoken term detector from a training set of examples, so that the model is adjusted to use a ﬁxed given threshold for all terms.
The function f is found from a training set of examples, where each example is composed of two speech segments and a representation of a term.
Although the training set contains many different terms, the function f should be able to detect any term, not only those already seen in the training phase.
Loss function for detection with a ﬁxed threshold
In this section we describe our main idea, whereas in the next sections we propose two implementations: one with a structured prediction model where the training data is fully supervised and the term is given as a phoneme sequence, and the other with a deep learning model where the training data is weakly supervised and the term is given using a segment of speech.
Recall that during inference the input to the detector is a speech segment and a term and the output is a conﬁdence that the term was pronounced in the speech segment, which is compared to a threshold.
Since the detection threshold is typically not ﬁxed and does depend on the input term, it is often desired to learn the function f such that the conﬁdence of a speech segment that contains the term is higher than the conﬁdence of a speech segment that does not contain the term.
Formally, let us consider two sets of speech segments.
Denote by X r + a set of speech segments in which the term r is articulated.
Similarly, denote by X r − a set of speech segments in which the term r is not articulated.
We assume that term r , and two instances ¯ x + ∈ X r + and ¯ x − ∈
X r − are drawn from a ﬁxed but unknown probability distribution, and we denote by P { π } and E [ π ] the probability and the expectation of an event π under this distribution.
The probability that the conﬁdence of ¯ x + is higher than the conﬁdence of ¯ x − is the area under the ROC curve (AUC) [11, 1]:
[Equation]
Instead of keeping a threshold for each term, we adjust f so that the detection threshold will be ﬁxed, and set to a predeﬁned value.
Assume that the predeﬁned threshold is θ , then the accuracy in the prediction can be measured by
[Equation]
where ∧ is the logical conjunction symbol.
Hence our goal is to ﬁnd the parameters of the function f
so as to maximize the accuracy Acc θ for a given threshold θ .
Equivalently we ﬁnd the parameters of function f that minimize the error deﬁned as
[Equation]
where ∨ is the logical disjunction symbol, and I { π } is the indicator function, that equals 1 if the predicate π holds true and 0 otherwise.
Unfortunately, we cannot minimize the error function (5) directly, since it is a combinatorial quantity.
A common practice is to replace the error function with a surrogate loss function which is easy to minimize.
We suggest to minimize a convex upper-bound to the error function.
Speciﬁcally, we replace the last term with the hinge upper bound,
[Equation]
where [ π ] + = max { π, 0 } .
The last upper bound holds true since I { π < 0 } ≤ [1 − π ] + .
Adding the margin of 1 means that the function f faces a harder problem: not only does it need to have a conﬁdence greater than θ for a positive speech segment and a conﬁdence lower than θ for a negative speech segment – the conﬁdence value must be at least θ + 1 and at most θ − 1 for positive and negative speech segments, respectively.
We now turn to present two algorithmic implementations that are aimed at minimizing the loss function derived from (6), namely,
[Equation]
Hopefully the minimization of this loss function will lead to the minimization of Err θ in (6).
Structured prediction model
Our ﬁrst construction is based on previous work on discriminative keyword spotting and spoken term detection [16, 21, 22], where the goal was to maximize the AUC.
In this setting we assume that the term is expressed as a sequence of phonemes denoted ¯ p r ∈ P ∗ .
In this fully-supervised setting we deﬁne the alignment between a phoneme sequence and a speech signal.
We denote by y l ∈ N the start time of phoneme p l (in frame units), and by e l = y l +1 −the end time of phoneme p l , except for the phoneme p L , where the end frame is e L .
The alignment sequence ¯ y r corresponding to the phonemes sequence ¯ p r is a sequence of start-times and an end-time, ¯ y r =
( y 1 , . . .
, y L , e L ) , where y l is the start-time of phoneme p l and e L is the end-time of the last phoneme p L .
Similar to previous work [16, 21, 22], our detection function f is composed of a predeﬁned set of n feature functions , { φ j } nj =1 , each of the form φ j : X ∗ × P ∗ × N ∗ → R .
Each feature function takes as input an acoustic representation of a speech utterance ¯ x ∈ X ∗ , together with the term phoneme sequence ¯ p r ∈ P ∗ , and a candidate alignment sequence ¯ y r ∈ N ∗ , and returns a scalar in R which represents the conﬁdence in the suggested alignment sequence given the term r .
For example, one element of the feature function can sum the number of times phoneme p comes after phoneme p (cid:48) , while other elements of the feature function may extract properties of each acoustic feature vector x t provided that phoneme p is pronounced at time t .
Our basic set of feature functions is the same as the set used in [16].
We believe that the threshold value for each term depends on the term’s phonetic content and its relative duration.
In order to allow f to learn these subtle differences from the data we introduced an additional set of 4 feature functions: a feature function representing a bias; a feature function that counts the number of occurrences of a phoneme in a term, i.e., |{ q | q ∈ ¯ p r }| ; a feature function holding the number of phonemes in the term, i.e., | ¯ p r | ; and a feature function holding the average length of the phonemes in a term, i.e., 1 L r (cid:80) L r i =
( y i +1 − y i ) .
As mentioned above, our goal is to learn a spoken term detector f , which takes as input a sequence of acoustic features ¯ x , a term ¯ p r , and returns a conﬁdence value in R .
The form of the function f we use is
[Equation]
where w ∈ R
n is a vector of importance weights that should be learned and φ ∈ R n is a vector function composed out of the feature functions φ j .
In other words, f returns a conﬁdence prediction about the existence of the term in the utterance by maximizing a weighted sum of the scores returned by the feature function elements over all possible alignment sequences.
If the conﬁdence of the function f is above the threshold θ then we predict that the term is pronounced in the signal and located in the time span deﬁned by the alignment sequence ¯ y that maximizes (8):
[Equation]
where the search for the best sequence is practically performed using the Viterbi algorithm as described in [16].
Speciﬁcally, the algorithm ﬁnds the optimal time segment for the keyword r in the speech signal ¯ x , and then aligns the phoneme sequence ¯ p r within the chosen time segment.
The parameters of the model w are found by minimizing the loss function deﬁned in (7).
In the fully supervised case we use a slightly modiﬁed version of it, which is deﬁned as
[Equation]
This is a convex function in the vector of the parameters w .
We use the Passive-Aggressive (PA) algorithm
[6, 16] to ﬁnd the parameters w .
The algorithm receives as input a set of training examples
S =
(¯ p r i , ¯ x + i , ¯ x − i , ¯ y r i ) } mi =1 and examines each of them sequentially.
Initially, we set w = 0 .
At each iteration i , the algorithm updates w according to the current example (¯ p r i , ¯ x + i , ¯ x − i , ¯ y r i ) as we now describe.
Denote by w i − 1 the value of the weight vector before the i th iteration.
We set the next weight vector w i to be the minimizer of the following optimization problem,
[Equation]
where C serves as a complexity-accuracy trade-off parameter and ξ is a non-negative slack variable, which indicates the loss of the i th example.
Intuitively, we would like to minimize the loss of the current example, i.e., the slack variable ξ , while keeping the weight vector w as close as possible to our previous weight vector w i − 1 .
The constraint makes the projection of the utterance in which the term is uttered onto w greater than θ + 1 , and the projection of the utterance in which the term is not uttered onto w less than θ − 1 .
The closed form solution to the above optimization problem can be derived using the Karush-Kuhn-Tucker conditions in the same lines of [6, App.
A].
The loss in (10) is composed of two hinge functions and therefore introduces a more elaborate solution than the one derived for the ranking loss of [16].
We call this algorithm PA-ACC (Passive-Aggressive to maximize Accuracy).
Details about the implementation of the algorithm can be seen in [9].
The
PA-ACC algorithm is an online algorithm, and deals with drifting hypotheses; therefore, it is highly inﬂuenced by the recent examples.
Common methods to convert an online algorithm to a batch algorithm are either by taking the average over all the parameters { w i } , or by taking the best w i over a validation set [7, 4].
Deep network model
We turn to exemplify our idea in the weakly supervised setting using deep networks.
This implementation is based on recent work on whole-word segmental systems [10, 2, 14].
These works present a Siamese network model trained with a ranking loss function.
Siamese networks [5] are neural networks with a tied set of parameters which take as input a pair of speech segments and are trained to minimize or maximize the similarity between the segments depending on whether the same term has been pronounced in the pair of segments.
In this setting the term r is represented by two speech segments rather than a phoneme sequence: a speech segment in which the term r is pronounced, ¯ x + , and a speech segment, in which the term r is not pronounced, ¯ x − .
Similar to those works, we assume that each example in the training set is composed of the triplet (¯ x t , ¯ x + , ¯ x − ) , where ¯ x t , ¯ x + ∈ X r + and ¯ x − ∈ X r − .
The goal in training the network is that the similarity score between ¯ x t and ¯ x + should be above the similarity score between ¯ x t and ¯ x − .
Denote by g u : X ∗ → R d a deep network (the speciﬁc architecture is discussed in Section 6) with a set of parameters u , where d is the dimension of the output.
Denote by ρ : R d × R d → R a measure of similarity between two output vectors of size d .
The spoken term detector f u : X ∗ ×
X
∗
→
R is the composition of Siamese networks g u and the similarity function.
Hence an unknown speech segment ¯ x t can be compared to a positive or negative speech segment, as follows:
[Equation]
The tied parameters u of all the models were found in [10, 2, 14] using the minimization of the ranking loss function
[Equation] [Equation]
for different options of the similarity function ρ .
In this work we propose to minimize the loss function in (7), which is deﬁned for the weakly supervised case as follows:
[Equation]
when the margin of γ > 0 is used.
In this case, the parameter γ is not set to 1, since the function f u is not a linear function and hence is not scale invariant to the margin, as in the structured prediction case.
In the next section we present our empirical comparison for all the loss functions on different speech corpora.
Experiments
In this section we present experimental results that demonstrate the effectiveness of our proposed calibrated loss function (7).
We compared the proposed loss to the standard approach of maximizing
AUC using the ranking loss as in (12) where no ﬁxed threshold can be set.
The experiments on the structured prediction model were conducted using fully supervised training sets of read speech (TIMIT, WSJ).
The experiments on the deep network model performed on a weakly supervised data of conversational speech (Switchboard).
6.1 Structured prediction model
To validate the effectiveness of the proposed approach, we performed experiments with the TIMIT corpus.
The training and validation sets were taken from the TIMIT training set.
The training set was composed from 1,512 randomly chosen terms, corresponding to 11,139 pairs of positive and negative utterances (each term repeated more than once).
Similarly, the validation set was composed fromdifferent randomly chosen terms, corresponding to 2,892 pairs.
The validation set was used to tune the algorithm’s parameters.
The test set was composed of 80 terms that were suggested as a benchmark in [16], and are distinct from the terms used in the training and validation sets.
For each term, we randomly picked at most 20 utterances in which the term was uttered and at most 20 utterances in which it was not uttered.
The utterances were taken from the TIMIT test set.
The number of test utterances in which the term was uttered was not always 20, since some terms were uttered less than 20 times in the whole TIMIT test set.
We measure performance using the AUC deﬁned in (1) and using the accuracy of a ﬁxed threshold θ denoted Acc θ .
Speciﬁcally, we calculate AUC on the test set of m test examples according to
[Equation] [Equation]
and the accuracy by
[Equation]
We tested the PA-ACC algorithm using two options.
The ﬁrst was whether the ﬁnal weight vector was a result of averaging or was the best to perform on the validation set.
The second option was whether the new feature functions we introduced were normalized by the length of the phoneme sequence | ¯ p r | or not.
The AUC and Acc θ rates found on our validation and test set are presented in Table 1.
In training PA-ACC we chose arbitrarily θ = 0 .
[Table]
We can see from the table that since the TIMIT dataset is very clean the detection rates are very good and the AUC is almost always 1.
The results presented here are improved over the results presented in [16] due to the introduction of the new feature functions.
It is interesting to note that the best Accresults on the validation set were obtained when the additional features were not normalized and the ﬁnal weight vector was selected over the validation set, while the best Acc 0 results on the test set were obtained with the opposite conditions: when the ﬁnal weight vector was the average one and the additional feature functions were normalized.
Further research on feature functions should be conducted and extended to a larger dataset.
We now turn to compare the performance of our algorithm against two other algorithms.
The ﬁrst is the discriminative keyword spotting algorithm presented in [16], which is the Passive-Aggressive algorithm trained with the ranking loss to maximize the AUC.
It is denoted here as PA-AUC.
We introduce two versions of this algorithm: the original version and an extended version with the additional set of feature functions described in Sec. 4.
When using the extended version of PA-AUC, normalizing the features had no affect on our results.
Similarly, a comparison of using the ﬁnal weight vector versus the best weight vector yielded similar outcomes.
The second algorithm is an
HMM-based spoken term detection algorithm presented in [16].
For all the algorithms we report the AUC and Acc θ in Table 2.
For the two versions of PA-AUC we selected a single threshold θ that gave the best Acc θ on the validation set.
Similarly we selected the best threshold for the HMM algorithm.
For PA-ACC we arbitrarily selected θ = 0 .
[Table]
It is interesting to see that the AUC of PA-ACC is the same or even higher than that of the PA-AUC.
Since Acc θ is a lower bound to AUC, the AUC can be thought of as Acc θ with the best threshold selected for every term in the set.
Indeed from Table 2 we see that the Acc θ was very close to the
AUC but did not reach it.
We evaluate the model trained on TIMIT on the Wall Street Journal (WSJ) corpus [20].
This corpus corresponds to read articles of the Wall Street Journal, and hence presents a different linguistic context compared to TIMIT.
Both the discriminative system and the HMM-based system were trained on the TIMIT corpus as described above and evaluated on a different set of 80 keywords from the WSJ corpus.
For each keyword, we randomly picked at most 20 utterances in which the keyword was uttered and at most 20 utterances in which it was not uttered from the si_tr_s portion of the WSJ corpus.
We used the same setting as in [16].
As before we arbitrarily chose the threshold θ = 0 .
The results are presented in Table 3.
[Table]
Again we see from Table 3 that the model trained with the proposed loss function led to higher accuracy rates with similar AUC rates, meaning a better separation between the positive speech utterances and the negative speech utterances.
6.2 Deep network model
Our second set of experiments is focused on deep networks trained on weakly supervised data.
Our model is based on previous work on network training using the ranking loss [10, 2, 14].
We used the same experimental setup as Kamper et al.
The term weak supervision [14], refers to the fact that supervision is given in the form of known word pairs, rather than the exact location of the term and its phonetic content as in Subsection 6.1.
The data was taken from the Switchboard corpus of English conversational telephone speech.
Melfrequency cepstral coefﬁcients (MFCCs) with ﬁrst and second order derivatives features were extracted and cepstral mean and variance normalization (CMVN) was applied per conversation side.
The training set consisted of the set of about 10k word tokens from [12, 13]; it consisted of word segments of at least 5 characters and 0.5 seconds in duration extracted from a forced alignment of the transcriptions, and comprises about 105 minutes of speech.
For the Siamese convolutional neural networks (CNNs), this set results in about 100k word segment pairs.
For testing, we used the 11k-token set from [12, 13].
The architecture of each network was the same as Kamper et al.
(2016)[14]: 1-D convolution withﬁlters over 9 frames; ReLU (Rectiﬁed Linear Unit) ; max pooling over 3 units; 1-D convolution with 96 ﬁlters over 8 units; ReLU; max pooling over 3 units; 2048-unit fully-connected ReLU; 1024-unit fully-connected linear layer.
All weights were initialized randomly.
Models were trained using
ADADELTA [23].
We reproduced the results in [14] by training the Siamese network using the ranking loss in (12) with the cosine similarity as a similarity function ρ .
The cosine similarity of two vectors v 1 ∈ R d and v 2 ∈ R d is deﬁned as
[Equation]
where this function returns a number close to 1 if the two vectors are similar and a number close toif the two vectors are not.
We also train the network using the same similarity function using the Acc θ loss function with θ = 0 as in (13).
For the ranking loss we used γ = 0 .
15 while for the Acc θ loss we used γ = 0 .
10 , because the margin γ is counted twice for Acc θ loss.
These values were chosen by maximizing Acc θ on a validation set over 5 epochs.
The AUC and Acc θ values for training of 5 toepochs are given in Table 4.
Other training parameters and settings were exactly the same as in [14].
[Table]
We can see in the table that both AUC and Acc θ are higher when the system is trained with the calibrated ranking loss function.
The reason that the AUC was also improved is most likely because the calibrated ranking loss function is harder to optimize than the ranking loss.
Conclusions
In this work, we introduced a new loss function that can be used to train a spoken term detection system with a ﬁxed desired threshold for all terms.
We introduced a new discriminative structured prediction model that is based on the Passive-Aggressive algorithm.
We show that the new loss can be used in training weakly supervised deep network models as well.
Results suggest that our new loss function yields AUC and accuracy values that are better than previous works’ results
Acknowledgments
This work was supported by a grant from the MAGNET program of the Israeli Innovation Authority.
c (cid:13) 2017 IEEE.
Reprinted, with permission, from IEEE Journal of Selected Topics in Signal Processing,
Dec 2017, Volume 11, Issue 8, pp. 1-8.