S EQUENTIAL D ENSITY R ATIO E STIMATION FOR S IMULTANEOUS O PTIMIZATION OF S PEED AND A CCURACY
A BSTRACT
Classifying sequential data as early and as accurately as possible is a challenging yet critical problem, especially when a sampling cost is high.
One algorithm that achieves this goal is the sequential probability ratio test (SPRT), which is known as Bayes-optimal: it can keep the expected number of data samples as small as possible, given the desired error upper-bound.
However, the original SPRT makes two critical assumptions that limit its application in real-world scenarios: (i) samples are independently and identically distributed, and (ii) the likelihood of the data being derived from each class can be calculated precisely.
Here, we propose the SPRT-TANDEM, a deep neural network-based SPRT algorithm that overcomes the above two obstacles.
The SPRT-TANDEM sequentially estimates the loglikelihood ratio of two alternative hypotheses by leveraging a novel Loss function for Log-Likelihood Ratio estimation (LLLR) while allowing correlations up to N ( ∈ N ) preceding samples.
In tests on one original and two public video databases, Nosaic MNIST, UCF101, and SiW, the SPRT-TANDEM achieves statistically signiﬁcantly better classiﬁcation accuracy than other baseline classiﬁers, with a smaller number of data samples.
The code and Nosaic MNIST are publicly available at https://github.com/TaikiMiyagawa/SPRT-TANDEM .
I NTRODUCTION
The sequential probability ratio test, or SPRT, was originally invented by Abraham Wald, and an equivalent approach was also independently developed and used by Alan Turing in the 1940s (Good, 1979; Simpson, 2010; Wald, 1945).
SPRT calculates the log-likelihood ratio (LLR) of two competing hypotheses and updates the LLR every time a new sample is acquired until the LLR reaches one of the two thresholds for alternative hypotheses (Figure 1).
Wald and his colleagues proved that when sequential data are sampled from independently and identically distributed (i.i.d.) data, SPRT can minimize the required number of samples to achieve the desired upper-bounds of false positive and false negative rates comparably to the Neyman-Pearson test, known as the most powerful likelihood test (Wald & Wolfowitz, 1948) (see also Theorem (A.5) in Appendix A).
Note that Wald used the i.i.d.
assumption only for ensuring a ﬁnite decision time (i.e., LLR reaches a threshold within ﬁnite steps) and for facilitating LLR calculation: the non-i.i.d.
property does not affect other aspects of the SPRT including the error upper bounds (Wald, 1947).
More recently, Tartakovsky et al.
veriﬁed that the non-i.i.d.
SPRT is optimal or at least asymptotically optimal as the sample size increases (Tartakovsky et al., 2014), opening the possibility of potential applications of the SPRT to non-i.i.d.
data series.
About 70 years after Wald’s invention, neuroscientists found that neurons in the part of the primate brain called the lateral intraparietal cortex (LIP) showed neural activities reminiscent of the SPRT (Kira et al., 2015); when a monkey sequentially collects random pieces of evidence to make a binary choice, LIP neurons show activities proportional to the LLR.
Importantly, the time of the decision can be predicted from when the neural activity reaches a ﬁxed threshold, the same as the SPRT’s decision rule.
Thus, the SPRT, the optimal sequential decision strategy, was re-discovered to be an [Figure] algorithm explaining primate brains’ computing strategy.
It remains an open question, however, what algorithm will be used in the brain when the sequential evidence is correlated, non-i.i.d. series.
The SPRT is now used for several engineering applications (Cabri et al., 2018;
Chen et al., 2017;
Kulldorff et al., 2011).
However, its i.i.d.
assumption is too crude for it to be applied to other real-world scenarios, including time-series classiﬁcation, where data are highly correlated, and key dynamic features for classiﬁcation often extend across more than one data point, violating the i.i.d. assumption.
Moreover, the LLR of alternative hypotheses needs to be calculated as precisely as possible, which is infeasible in many practical applications.
In this paper, we overcome the above difﬁculties by using an SPRT-based algorithm that Treats data series As an N-th orDEr Markov process (SPRT-TANDEM), aided by a sequential probability density ratio estimation based on deep neural networks.
A novel Loss function for Log-Likelihood Ratio estimation (LLLR) efﬁciently estimates the density ratio that let the SPRT-TANDEM approach close to asymptotic Bayes-optimality (i.e., Appendix A.4).
In other words, LLLR optimizes classiﬁcation speed and accuracy at the same time.
The SPRT-TANDEM can classify non-i.i.d.
data series with user-deﬁned model complexity by changing N ( ∈ N ) , the order of approximation, to deﬁne the number of past samples on which the given sample depends.
By dynamically changing the number of samples used for classiﬁcation, the SPRT-TANDEM can maintain high classiﬁcation accuracy while minimizing the sample size as much as possible.
Moreover, the SPRT-TANDEM enables a user to ﬂexibly control the speed-accuracy tradeoff without additional training, making it applicable to various practical applications.
We test the SPRT-TANDEM on our new database, Nosaic MNIST (NMNIST), in addition to the publicly available UCF101 action recognition database (Soomro et al., 2012) and Spooﬁng in the
Wild (SiW) database (Liu et al., 2018).
Two-way analysis of variance (ANOVA, (Fisher, 1925)) followed by a Tukey-Kramer multi-comparison test (Tukey, 1949; Kramer, 1956) shows that our proposed SPRT-TANDEM provides statistically signiﬁcantly higher accuracy than other ﬁxed-length and variable-length classiﬁers at a smaller number of data samples, making Wald’s SPRT applicable even to non-i.i.d.
data series.
Our contribution is ﬁvefold:
We invented a deep neural network-based algorithm, SPRT-TANDEM, which enables Wald’s SPRT on arbitrary sequential data without knowing the true LLR.
The SPRT-TANDEM extends the SPRT to non-i.i
d. data series without knowing the true LLR.
With a novel loss, LLLR, the SPRT-TANDEM sequentially estimates LLR to optimize speed and accuracy simultaneously.
The SPRT-TANDEM can control the speed-accuracy tradeoff without additional training.
We introduce Nosaic MNIST, a novel early-classiﬁcation database.
R ELATED WORK
The SPRT-TANDEM has multiple interdisciplinary intersections with other ﬁelds of research: Wald’s classical SPRT, probability density estimation, neurophysiological decision making, and time-series classiﬁcation.
The comprehensive review is left to Appendix B, while in the following, we introduce the SPRT, probability density estimation algorithms, and early classiﬁcation of the time series.
Sequential Probability Ratio Test (SPRT).
The SPRT, denoted by δ ∗ , is deﬁned as the tuple of a decision rule and a stopping rule (Tartakovsky et al., 2014; Wald, 1947):
Deﬁnition 2.1.
Sequential Probability Ratio Test (SPRT).
Let λ t as the LLR at time t , and X (1 ,T ) as a sequential data X (1 ,T ) : = { x ( t ) } Tt =1 .
Given the absolute values of lower and upper decision threshold, a 0 ≥ 0 and a 1 ≥ 0 , SPRT, δ ∗ , is deﬁned as
[Equation]
where the decision rule d ∗ and stopping time τ ∗ are
[Equation]
We review the proof of optimality in Appendix A.4, while Figure 1 shows an intuitive explanation.
Probability density ratio estimation.
Instead of estimating numerator and denominator of a density ratio separately, the probability density ratio estimation algorithms estimate the ratio as a whole, reducing the degree of freedom for more precise estimation (Sugiyama et al., 2010; 2012).
Two of the probability density ratio estimation algorithms that closely related to our work are the probabilistic classiﬁcation (Bickel et al., 2007; Cheng & Chu, 2004; Qin, 1998) and density ﬁtting approach (Sugiyama et al., 2008; Tsuboi et al., 2009) algorithms.
As we show in Section 4 and
Appendix E, the SPRT-TANDEM sequentially estimates the LLR by combining the two algorithms.
Early classiﬁcation of time series.
To make decision time as short as possible, algorithms for early classiﬁcation of time series can handle variable length of data (Mori et al., 2018;
Mori et al., 2016;
Xing et al., 2009; 2012) to minimize high sampling costs (e.g., medical diagnostics (Evans et al., 2015;
Grifﬁn & Moorman, 2001), or stock crisis identiﬁcation (Ghalwash et al., 2014)).
Leveraging deep neural networks is no exception in the early classiﬁcation of time series (Dennis et al., 2018; Suzuki et al., 2018).
Long short-term memory (LSTM) variants LSTM-s/LSTM-m impose monotonicity on classiﬁcation score and inter-class margin, respectively, to speed up action detection (Ma et al., 2016).
Early and Adaptive Recurrent Label ESTimator (EARLIEST) combines reinforcement learning and a recurrent neural network to decide when to classify and assign a class label (Hartvigsen et al., 2019).
P ROPOSED ALGORITHM : SPRT-TANDEM
In this section, we propose the TANDEM formula, which provides the N -th order approximation of the LLR with respect to posterior probabilities.
The i.i.d. assumption of Wald’s SPRT greatly simpliﬁes the LLR calculation at the expense of the precise temporal relationship between data samples.
On the other hand, incorporating a long correlation among multiple data may improve the LLR estimation; however, calculating too long a correlation may potentially be detrimental in the following cases.
First, if a class signature is signiﬁcantly shorter than the correlation length in consideration, uninformative data samples are included in calculating LLR, resulting in a late or wrong decision (Campos et al., 2018).
Second, long correlations require calculating a long-range of backpropagation, prone to vanishing gradient problem (Hochreiter et al., 2001).
Thus, we relax the i.i.d.
assumption by keeping only up to the N -th order correlation to calculate the LLR.
The TANDEM formula.
Here, we introduce the TANDEM formula, which computes the approximated LLR, the decision value of the SPRT-TANDEM algorithm.
The data series is approximated as an N -th order Markov process.
For the complete derivation of the 0th (i.i.d.), 1st, and N -th order
TANDEM formula, see Appendix C. Given a maximum timestamp T ∈ N , let X (1 ,T ) and y be a sequential data X (1 ,T ) :
= { x ( t ) }
Tt =1 and a class label y ∈ { 1 , 0 } , respectively, where x ( t ) ∈
R
d x and d x ∈ N .
By using Bayes’ rule with the N -th order Markov assumption, the joint LLR of data at a timestamp t is written as follows:
[Equation]
(see Equation (84) and (85) in Appendix C for the full formula).
Hereafter we use terms k -let or multiplet to indicate the posterior probabilities, p ( y | x (1) , ..., x ( k ) )
=
p ( y | X (1 ,k ) ) that consider correlation across k data points.
The ﬁrst two terms of the TANDEM formula (Equation (4)), N + 1let and N -let, have the opposite signs working in “tandem” adjusting each other to compute the LLR.
The third term is a prior (bias) term.
In the experiment, we assume a ﬂat prior or zero bias term, but a user may impose a non-ﬂat prior to handling the biased distribution of a dataset.
The TANDEM formula can be interpreted as a realization of the probability matching approach of the probability density estimation, under an N -th order Markov assumption of data series.
Neural network that calculates the SPRT-TANDEM formula.
The SPRT-TANDEM is designed to explicitly calculate the N -th order TANDEM formula to realize sequential density ratio estimation, which is the critical difference between our SPRT-TANDEM network and other architecture based on convolutional neural networks (CNNs) and recurrent neural networks (RNN).
Figure 2 illustrates a conceptual diagram explaining a generalized neural network structure, in accordance with the 1st-order TANDEM formula for simplicity.
The network consists of a feature extractor and a temporal integrator (highlighted by red and blue boxes, respectively).
They are arbitrary networks that a user can choose depending on classiﬁcation problems or available computational resources.
The feature extractor and temporal integrator are separately trained because we ﬁnd that this achieves better performance than the end-to-end approach (also see Appendix D).
The feature extractor outputs single-frame features (e.g., outputs from a global average pooling layer), which are the input vectors of the temporal integrator.
The output vectors from the temporal integrator are transformed with a fully-connected layer into two-dimensional logits, which are then input to the softmax layer to obtain posterior probabilities.
They are used to compute the LLR to run the SPRT (Equation (2)).
Note that during the training phase of the feature extractor, the global average pooling layer is followed by a fully-connected layer for binary classiﬁcation.
How to choose the hyperparameter N ?
By tuning the hyperparameter N , a user can efﬁciently boost the model performance depending on databases; in Section 5, we change N to visualize the model performance as a function of N .
Here, we provide two ways to choose N .
One is to choose
N based on the speciﬁc time scale, a concept introduced in Appendix D, where we describe in detail how to guess on the best N depending on databases.
The other is to use a hyperparameter tuning algorithm, such as Optuna, (Akiba et al., 2019) to choose N objectively.
Optuna has multiple hyperparameter searching algorithms, the default of which is the Tree-structured Parzen Estimator (Bergstra et al., 2011).
Note that tuning N is not computationally expensive, because N is only related to the temporal integrator, not the feature extractor.
In fact, the temporal integrator’s training speed is much faster than that of the feature extractor: 9 mins/epoch vs. 10 hrs/epoch ( N = 49 ,
NVIDIA RTX2080Ti, SiW database).
LLLR AND MULTIPLET CROSS - ENTROPY LOSS
Given a maximum timestamp T ∈ N and dataset size M ∈ N , let S :
= { ( X (1 ,T ) i , y i ) } Mi
=1 be a sequential dataset.
Training our network to calculate the TANDEM formula involves the following loss functions in combination: (i) the Loss for Log Likelihood Ratio estimation (LLLR), L LLR , and (ii) multiplet cross-entropy loss, L multiplet .
The total loss, L total is deﬁned as
[Equation]
[Figure]
4. L OSS FOR L OG -L IKELIHOOD R ATIO ESTIMATION (LLLR).
The SPRT is Bayes-optimal as long as the true LLR is available; however, the true LLR is often inaccessible under real-world scenarios.
To empirically estimate the LLR with the TANDEM formula, we propose the LLLR
[Equation]
(6) where σ is the sigmoid function.
We use ˆ p to highlight a probability density estimated by a neural network.
The LLLR minimizes the Kullback-Leibler divergence (Kullback & Leibler, 1951) between the estimated and the true densities, as we brieﬂy discuss below.
The full discussion is given in
Appendix E due to page limit.
Density ﬁtting.
First, we introduce KLIEP (Kullback-Leibler Importance Estimation Procedure, Sugiyama et al. (2008)), a density ﬁtting approach of the density ratio estimation Sugiyama et al.
KLIEP is an optimization problem of the Kullback-Leibler divergence between p ( X | y = 1) and ˆ r ( X ) p ( X | y = 0) with constraint conditions, where X and y are random variables corresponding to X (1 ,t ) i and y i , and ˆ r ( X ) :
= ˆ p ( X | y = 1) /
ˆ p ( X | y = 0) is the estimated density ratio.
Formally, argmin ˆ r
[Equation]
with the constraints 0 ≤ ˆ r ( X ) and (cid:82) dX ˆ r ( X ) p ( X | y = 0) = 1 .
The ﬁrst constraint ensures the positivity of the estimated density ˆ r ( X ) p ( X | y = 0) , while the second one is the normalization condition.
Applying the empirical approximation, we obtain the ﬁnal optimization problem: argmin
[Equation]
(cid:88) i ∈ I− log ˆ r ( X (1 ,t ) i ) (cid:35) , with ˆ r
( X (1 ,t ) i ) ≥ 0 and
M(cid:88)
i ∈
Iˆ r
( X (1 ,t ) i )
= 1 , (8) where I 1 :=
{ i ∈ [ M ] | y i = 1 } , I 0 :
= { i ∈ [ M ] | y i = 0 } , M 1 :=
| I 1 | , and M 0 :
=
| I 0 | .
Stabilization.
The original KLIEP (8), however, is asymmetric with respect to p ( X | y = 1) and p ( X | y = 0) .
To recover the symmetry, we add 1 M 0 (cid:80) i ∈ I
− log(ˆ r
( X (1 ,t ) i ) − 1 ) to the objective and impose an additional constraint 1 M 1 (cid:80) i ∈
I
ˆ r
( X (1 ,t ) i ) − 1 = 1 .
Besides, the symmetrized objective still has unbounded gradients, which cause instability in the training.
Therefore, we normalize the
LLRs with the sigmoid function, obtaining the LLLR (6).
We can also show that the constraints are effectively satisﬁed due to the sigmoid funciton.
See Appendix E for the details.
In summary, we have shown that the LLLR minimizes the Kullback-Leibler divergence of the true and the estimated density and further stabilizes the training by restricting the value of LLR.
Here we emphasize the contributions of the LLLR again.
The LLLR enables us to conduct the stable LLR estimation and thus to perform the SPRT, the algorithm optimizing two objectives: stopping time and accuracy.
In previous works (Mori et al., 2018; Hartvigsen et al., 2020), on the other hand, these two objectives are achieved with separate loss sub-functions.
Compared to KLIEP, the proposed LLLR statistically signiﬁcantly boosts the performance of the
SPRT-TANDEM (Appendix E.4).
Besides, experiment on multivariate Gaussian with a simple toy-model also shows that the LLLR minimize errors between the estimated and the true density ratio (Appendix F).
4. M ULTIPLET CROSS - ENTROPY LOSS .
To further facilitate training the neural network, we add binary cross-entropy losses, though the LLLR sufﬁces to estimate LLR.
We call them multiplet cross-entropy loss here, and deﬁned as:
[Equation]
) where
L k
[Equation]
[Equation]
[Equation]
Minimizing the multiplet cross-entropy loss is equivalent to minimizing the Kullback-Leibler divergence of the estimated posterior k -let ˆ p
( y i | x ( t − k +1)
i , ..., x ( t ) i ) and the true posterior p ( y i | x ( t − k +1) i , ..., x ( t ) i ) (shown in Appendix G), which is a consistent objective with the LLLR and thus the multiplet loss accelerates the training.
Note also that the multiplet loss optimizes all the logits output from the temporal integrator, unlike the LLLR.
E XPERIMENTS AND RESULTS
In the following experiments, we use two quantities as evaluation criteria:
(i) balanced accuracy, the arithmetic mean of the true positive and true negative rates, and (ii) mean hitting time, the average number of data samples used for classiﬁcation.
Note that the balanced accuracy is robust to class imbalance (Luque et al., 2019), and is equal to accuracy on balanced datasets.
Evaluated public databases are NMNIST, UCF, and SiW. Training, validation, and test datasets are split and ﬁxed throughout the experiment.
We selected three early-classiﬁcation models (LSTM-s (Ma et al., 2016), LSTM-m (Ma et al., 2016), and EARLIEST (Hartvigsen et al., 2019)) and one ﬁxed-length classiﬁer (3DResNet (Hara et al., 2017)), as baseline models.
All the early-classiﬁcation models share the same feature extractor as that of the SPRT-TANDEM for a fair comparison.
Hyperparameters of all the models are optimized with Optuna unless otherwise noted so that no models are disadvantaged by choice of hyperparameters.
See Appendix H for the search spaces and ﬁxed ﬁnal parameters.
After ﬁxing hyperparameters, experiments are repeated with different random seeds to obtain statistics.
In each of the training runs, we evaluate the validation set after each training epoch and then save the weight parameters if the balanced accuracy on the validation set updates the largest value.
The last saved weights are used as the model of that run.
The model evaluation is performed on the test dataset.
During the test stage of the SPRT-TANDEM, we used various values of the SPRT thresholds to obtain a range of balanced accuracy-mean hitting time combinations to plot a speed-accuracy tradeoff (SAT) curve.
If all the samples in a video are used up, the thresholds are collapsed to a 1 = a 0 = 0 to force a decision.
[Figure] To objectively compare all the models with various trial numbers, we conducted the two-way ANOVA followed by the Tukey-Kramer multi-comparison test to compute statistical signiﬁcance.
For the details of the statistical test, see Appendix I.
We show our experimental results below.
Due to space limitations, we can only show representative results.
For the full details, see Appendix J. For our computing infrastructure, see Appendix K.
Nosaic MNIST (Noise + mosaic MNIST) database.
We introduce a novel dataset, NMNIST, whose video is buried with noise at the ﬁrst frame, and gradually denoised toward the last, 20th frame (see Appendix L for example data).
The motivation to create NMNIST instead of using a preexisting time-series database is as follows: for simple video databases such as Moving MNIST (MMNIST, (Srivastava et al., 2015)), each data sample contains too much information so that well-trained classiﬁers can correctly classify a video only with one or two frames (see Appendix M for the results of the SPRT-TANDEM and LSTM-m on MMNIST).
We design a parity classiﬁcation task, classifying 0 − 9 digits into an odd or even class.
The training, validation, and test datasets contain 50,000, 10,000, and 10,000 videos with frames of size 28 × 28 ×(gray scale).
Each pixel value is divided by 127 . 5 , before subtracted by 1 .
The feature extractor of the
SPRT-TANDEM is ResNet-
(He et al., 2016a), with the ﬁnal output reduced to 128 channels.
The temporal integrator is a peephole-LSTM (Gers & Schmidhuber, 2000; Hochreiter & Schmidhuber, 1997), with hidden layers of 128 units.
The total numbers of trainable parameters on the feature extractor and temporal integrator are 6.9M and 0.1M, respectively.
We train 0th, 1st, 2nd, 3rd, 4th, 5th, 10th, and 19th order SPRT-TANDEM networks.
LSTM-s / LSTM-m and EARLIEST use peephole-LSTM and LSTM, respectively, both with hidden layers of 128 units.
3DResNet haslayers with 128 ﬁnal output channels so that the total number of trainable parameters is in the same order (7.7M) as that of the SPRT-TANDEM.
Figure 3a and Table 1 shows representative results of the experiment.
Figure 3d shows example LLR trajectories calculated with the 10th order SPRT-TANDEM.
The SPRT-TANDEM outperforms other baseline algorithms by large margins at all mean hitting times.
The best performing model is the 10th order TANDEM, which achieves statistically signiﬁcantly higher balanced accuracy than the other algorithms ( p -value < 0 . 001 ).
Is the proposed algorithm’s superiority because the SPRT-TANDEM successfully estimates the true LLR to approach asymptotic Bayes optimality?
We discuss potential interpretations of the experimental results in the Appendix D. [Table]
UCF101 action recognition database.
To create a more challenging task [Table] we selected two classes, handstand-pushups and handstand-walking, from the 101 classes in the UCF database.
At a glimpse of one frame, the two classes are hard to distinguish.
Thus, to correctly classify these classes, temporal information must be properly used.
We resize each video’s duration as multiples offrames and sample every 50 frames with 25 frames of stride as one data.
Training, validation, and test datasets contain 1026, 106, and 105 videos with frames of size 224 × 224 × 3 , randomly cropped to 200 × 200 × 3 at training.
The mean and variance of a frame are normalized to zero and one, respectively.
The feature extractor of the SPRT-TANDEM is ResNet-50 (He et al., 2016b), with the ﬁnal output reduced to 64 channels.
The temporal integrator is a peephole-LSTM, with hidden layers of 64 units.
The total numbers of trainable parameters in the feature extractor and temporal integrator are 26K and 33K, respectively.
We train 0th, 1st, 2nd, 3rd, 5th, 10th, 19th, 24th, and 49th-order
SPRT-TANDEM.
LSTM-s / LSTM-m and EARLIEST use peephole-LSTM and LSTM, respectively, both with hidden layers of 64 units.
3DResNet has 50 layers with 64 ﬁnal output channels so that the total number of trainable parameters (52K) is on the same order as that of the SPRT-TANDEM.
Figure 3b and Table 2 shows representative results of the experiment.
The best performing model is the 10th order TANDEM, which achieves statistically signiﬁcantly higher balanced accuracy than other models ( p -value < 0 . 001 ).
The superiority of the higher-order TANDEM indicates that a classiﬁer needs to integrate longer temporal information in order to distinguish the two classes (also see Appendix D). [Table]
Spooﬁng in the Wild (SiW) database.
To test the SPRT-TANDEM in a more practical situation, we conducted experiments on the SiW database.
We use a sliding window of 50 frames-length andframes-stride to sample data, which yields training, validation, and test datasets of 46,729, 4,968, and 43,878 videos of live or spooﬁng face.
Each frame is resized to 256 × 256 × 3 pixels and randomly cropped to 244 × 244 × 3 at training.
The mean and variance of a frame are normalized to zero and one, respectively.
The feature extractor of the SPRT-TANDEM is ResNet-152, with the ﬁnal output reduced to 512 channels.
The temporal integrator is a peephole-LSTM, with hidden layers of 512 units.
The total number of trainable parameters in the feature extractor and temporal integrator is 3.7M and 2.1M, respectively.
We train 0th, 1st, 2nd, 3rd, 5th, 10th, 19th, 24th, and 49th-order
SPRT-TANDEM networks.
LSTM-s / LSTM-m and EARLIEST use peephole-LSTM and LSTM, respectively, both with hidden layers of 512 units.
3DResNet has 101 layers with 512 ﬁnal output channels so that the total number of trainable parameters (5.3M) is in the same order as that of the
SPRT-TANDEM.
Optuna is not applied due to the large database and network size.
Figure 3c and Table 3 shows representative results of the experiment.
The best performing model is the 10th order TANDEM, which achieves statistically signiﬁcantly higher balanced accuracy than other models ( p -value < 0 . 001 ).
The superiority of the lower-order TANDEM indicates that each video frame contains a high amount of information necessary for the classiﬁcation, imposing less need to collect a large number of frames (also see Appendix D) [Table] [Table]
Ablation study.
To understand contributions of the L LLR and L multiplet to the SAT curve, we conduct an ablation study.
The 1st-order SPRT-TANDEM is trained with L LLR only, L multiplet only, and both L LLR and L multiplet .
The hyperparameters of the three models are independently optimized using Optuna (see Appendix H).
The evaluated database and model are NMNIST and the 1st-order
SPRT-TANDEM, respectively.
Figure 3e shows the three SAT curves.
The result shows that L LLR leads to higher classiﬁcation accuracy, whereas L multiplet enables faster classiﬁcation.
The best performance is obtained by using both L LLR and L multiplet .
We also conﬁrmed this tendency with the 19th order SPRT-TANDEM, as shown in Appendix N.
SPRT vs. Neyman-Pearson test.
As we discuss in Appendix A, the Neyman-Person test is the optimal likelihood ratio test with a ﬁxed number of samples.
On the other hand, the SPRT takes a ﬂexible number of samples for an earlier decisions.
To experimentally test this prediction, we compare the SPRT-TANDEM and the corresponding Neyman-Pearson test.
The Neyman-Pearson test classiﬁes the entire data into two classes at each number of frames, using the estimated LLRs with threshold λ = 0 .
Results support the theoretical prediction, as shown in Figure 3f: the Neyman-Pearson test needs a larger number of samples than the SPRT-TANDEM.
C ONCLUSION
We presented the SPRT-TANDEM, a novel algorithm making Wald’s SPRT applicable to arbitrary data series without knowing the true LLR.
Leveraging deep neural networks and the novel loss function, LLLR, the SPRT-TANDEM minimizes the distance of the true LLR and the LLR sequentially estimated with the TANDEM formula, enabling simultaneous optimization of speed and accuracy.
Tested on the three publicly available databases, the SPRT-TANDEM achieves statistically significantly higher accuracy over other existing algorithms with a smaller number of data points.
The
SPRT-TANDEM enables a user to control the speed-accuracy tradeoff without additional training, opening up various potential applications where either high-accuracy or high-speed is required.
A CKNOWLEDGEMENTS
The authors thank anonymous reviewers for their careful reading to improve the manuscript.
We would also like to thank Hirofumi Nakayama and Yuka Fujii for insightful discussions.
Special thanks to Yuka for naming the proposed algorithm.
A UTHOR CONTRIBUTIONS
A.F.E. conceived the study.
A.F.E. and T.M. constructed the theory, conducted the experiments, and wrote the paper.
T. M. organized python codes to be ready for the release.
K.S. and H.I. supervised the study.
C ONTENTS 1 Introduction
2 Related work 3 Proposed algorithm: SPRT-TANDEM 4 LLLR and multiplet cross-entropy loss 4.
Loss for Log-Likelihood Ratio estimation (LLLR). . . . . . . . . . . . . . . . . .
Multiplet cross-entropy loss. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5 Experiments and results 6 Conclusion
A Theoretical aspects of the sequential probability ratio test
A.1 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.2 Deﬁnition and the tradeoff of false alarms and stopping time . . . . . . . . . . . .
A.3 The Neyman-Pearson test and the SPRT . . . . . . . . . . . . . . . . . . . . . . .
A.4 The Optimality of the SPRT . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B Supplementary
review of
the related work
C Derivation of the TANDEM formula
D Supplementary discussion
E Loss for Log-Likelihood Ratio estimation (LLLR)
E.1 Density ratio estimation and KLIEP . .
E.2 The symmetrized KLIEP loss . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.
The LLLR and density ratio estimation . . . . . . . . . . . . . . . . . . . . . . . .
E.4 Preparatory experiment testing the effectiveness of the LLLR . . . . . . . . . . . .
F Probability density ratio estimation with the LLLR
F.
Experimental settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
F.
Density estimation results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
G Multiplet cross-entropy loss
H Hyperparameter optimization
H.1 Nosaic MNIST (NMNIST) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
H.2 UCF101 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
H.3 SiW . .
I Statistical test details
J Details of the experiments in Section
K Computing infrastructure
L An example video of the Nosaic MNIST database.
M Supplementary experiment on Moving MNIST database
N Supplementary ablation experiment
A PPENDIX A T HEORETICAL ASPECTS OF THE SEQUENTIAL PROBABILITY RATIO TEST
In this section, we review the mathematical background of the SPRT following the discussion in
Tartakovsky et al.
First, we deﬁne the SPRT based on the measure theory and introduce
Stein’s lemma, which assures the termination of the SPRT.
To deﬁne the optimality of the SPRT, we introduce two performance metrics that measure the false alarm rate and the expected stopping time, and discuss their tradeoff — the SPRT solves it.
Through this analysis, we utilize two important approximations, the asymptotic approximation, and the no-overshoot approximation, which play essential roles to simplify our analysis.
The asymptotic approximation assumes the upper and lower thresholds are inﬁnitely far away from the origin, being equivalent to making the most careful decision to reduce the error rate, at the expense of the stopping time.
On the other hand, the no-overshoot approximation assumes that we can neglect the threshold overshoots of the likelihood ratio.
Next, we show the superiority of the SPRT to the Neyman-Pearson test, using a simple Gaussian model.
The Neyman-Pearson test is known to be optimal in the two-hypothesis testing problem and is often compared with the SPRT.
Finally, we introduce several types of optimal conditions of the SPRT.
A. P RELIMINARIES
Notations.
Let (Ω , F , P ) be a probability space; Ω is a sample space, F ⊂ P Ω is a sigma-algebra of Ω , where P A denotes the power set of a set A , and P is a probability measure.
Intuitively, Ω represents the set of all the elementary events under consideration, e.g., Ω = { all the possible elementary events such that "a human is walking through a gate." } .
F is deﬁned as a set of the subsets of Ω , and stands for all the possible combinations of the elementary events; e.g., F (cid:51) { "Akinori is walking through the gate at the speed of 80 m/min," "Taiki is walking through the gate at the speed of 77 m/min," or "Nothing happened."
} P : F → [0 , 1] is a probability measure, a function that is normalized and countably additive; i.e., P measures the probability that the event A ∈ F occurs.
A random variable X is deﬁned as the measurable function from Ω to a measurable space, practically R d ( d ∈ N ); e.g., if ω ( ∈ Ω) is "Taiki is walking through the gate with a big smile," then X ( ω ) may be 100 frames of the color images with 128 × 128 pixels ( d = 128 × 128 × 3 × 100 ), i.e., a video recorded with a camera attached at the top of the gate.
The probability that a random variable X takes a set of values S ∈ R d is deﬁned as P ( X ∈ S ) :
= P ( X − 1 ( S )) , where X − 1 is the preimage of X .
By deﬁnition of the measurable function, X − 1 ( S ) ∈ F for all S ∈ R d .
Let {F t } t ≥ 0 be a ﬁltration.
By deﬁnition, {F t } t ≥ 0 is a non-decreasing sequence of sub-sigma-algebras of F ; i.e., F s ⊂ F t ⊂ F for all s and t such that 0 < s < t .
Each element of ﬁltration can be interpreted as the available information at a given point t .
(Ω , F , {F t } t ≥ 0 , P ) is called a ﬁltered probability space.
As in the main manuscript, let X (1 ,T ) :
= { x ( t ) } Tt =1 be a sequential data point sampled from the density p , where T ∈ N ∪ {∞} .
For each t ∈ [ T ] , x ( t ) ∈ R d x , where d x ∈ N is the dimensionality of the input data.
In the i.i.d.
case, p ( X (1 ,T ) )
= (cid:81)
Tt =1 f ( x ( t ) ) , where f is the density of x (1) .
For each time-series data X (1 ,T ) , the associated label y takes the value 1 or 0; we focus on the binary classiﬁcation, or equivalently the two-hypothesis testing throughout this paper.
When y is a class label, p ( X (1 ,T ) | θ ) is the likelihood density function.
Note that X (1 ,T ) with label y is sampled according to density p ( X (1 ,T ) | y ) .
Our goal is, given a sequence X
(1 ,T ) , to identify which one of the two densities p 1 or p
the sequence
X (1 ,T ) is sampled from; formally, to test two hypotheses H 1 : y = 1 and H 0 :
y = 0 given X (1 ,T ) .
The decision function or test of a stochastic process X (1 ,T ) is denoted by d ( X (1 ,T ) ) :
Ω → { 1 , 0 } .
We can identify this deﬁnition with, for each realization of X (1 ,T ) , d : R d x × T → { 1 , 0 } , i.e.,
X (1 ,T ) (cid:55)→ y , where y ∈ { 1 , 0 } .
Thus we write d instead of d ( X (1 ,T ) ) , for simplicity.
The stopping time of X (1 ,T ) with respect to a ﬁltration {F t } t ≥ 0 is deﬁned as τ :=
τ ( X (1 ,T ) ) :
Ω → R ≥ 0 such that { ω ∈ Ω | τ ( ω ) ≤ t } ∈ F t .
Accordingly, for ﬁxed T ∈ N ∪ {∞} and y ∈ { 1 , 0 } , { d = y } means the set of time-series data such that the decision function accepts the hypothesis H i with a ﬁnite stopping time; more speciﬁcally, { d = y } = { ω ∈ Ω | d ( X (1 ,T ) )( ω ) =
y, τ ( X (1 ,T ) )( ω ) < ∞} .
The decision rule δ is deﬁned as the doublet ( d, τ ) .
Let Λ T :=
Λ( X (1 ,T ) ) :
= p ( X (1 ,T ) | y =1) p ( X (1 ,T ) | y =0) and λ T := log Λ T be the likelihood ratio and the log-likelihood ratio of X (1 ,T ) .
In the i.i.d. case, Λ T = (cid:81) Tt =1 p ( x ( t ) | y =1) p
x ( t ) | y =0)
= (cid:81)
Tt =1 Z ( t ) , where p ( X (1 ,T ) | y )
= (cid:81)
Tt =1 p ( x ( t ) | y )
( y ∈ { 1 , 0 } ) and
Z ( t ) :
= p ( x ( t ) | y =1) p ( x ( t ) | y =0) .
A. D EFINITION AND THE TRADEOFF OF FALSE ALARMS AND STOPPING TIME
Let us overview the theoretical structure of the SPRT.
In the following, we assume that the time-series data points are i.i.d.
until otherwise stated.
Deﬁnition of the SPRT.
The sequential probability ratio test (SPRT), denoted by δ ∗ is deﬁned as the doublet of the decision function and the stopping time.
Deﬁnition A.1.
Sequential probability ratio test (SPRT)
Let a 0 = − log A 0 ≥ 0 and a 1 = log A 1 ≥ 0 be (the absolute values of) a lower and an upper threshold respectively.
[Equation]
[Equation] [Equation]
Note that d ∗ and τ ∗ implicitly depend on a stochastic process X (1 ,T ) .
In general, a doublet δ of a terminal decision function and a stopping time is called a decision rule or a hypothesis test.
Termination.
The i.i.d.-SPRT terminates with probability one and all the moments of the stopping time are ﬁnite, provided that the two hypotheses are distinguishable:
Lemma A.1.
Stein’s lemma
Let (Ω , F , P ) be a probability space and { Y ( t ) } t ≥ 1 be a sequence of i.i.d.
random variables under
P .
Deﬁne τ := inf { T ≥ 1 | (cid:80)
Tt =1 Y ( t ) / ∈ ( − a 0 , a 1 ) } .
If P ( Y (1) ) (cid:54) = 1 , the stopping time τ is exponentially bounded; i.e., there exist constants C > 0 and 0 < ρ < 1 such that P ( τ > T ) ≤ Cρ T for all T ≥ 1 .
Therefore, P ( τ < ∞ )
= 1 and E [ τ k ] < ∞ for all k > 0 .
Two performance metrics.
Considering the two-hypothesis testing, we employ two kinds of performance metrics to evaluate the efﬁciency of decision rules from complementary points of view: the false alarm rate and the stopping time.
The ﬁrst kind of metrics is the operation characteristic, denoted by β ( δ, y ) , and its related metrics.
The operation characteristic is the probability of the decision being 0 when the true label is y =
y ; formally,
Deﬁnition A.2. Operation characteristic
The operation characteristic is the probability of accepting the hypothesis H 0 as a function of y :
[Equation]
Using the operation characteristic, we can deﬁne four statistical measures based on the confusion matrix; namely, False Positive Rate (FPR), False Negative Rate (FNR), True Negative Rate (TNR), and True Positive Rate (TPR).
[Equation]
[Equation] [Equation]
[Equation]
Note that balanced accuracy is denoted by (1 + β ( δ, 0) − β ( δ, 1)) /
2 according to this notation.
The second kind of metrics is the mean hitting time, and is deﬁned as the expected stopping time of the decision rule:
Deﬁnition A.3.
Mean hitting time The mean hitting time is the expected number of time-series data points that are necessary for testing a hypothesis when the true parameter value is y :
E y τ = (cid:82) Ω τdP ( ·| y ) .
The mean hitting time is also referred to as the expected sample size of the average sample number.
There is a tradeoff between the false alarm rate and the mean hitting time.
For example, the quickness may be sacriﬁced, if we use a decision rule δ that makes careful decisions, i.e., with the false alarm rate 1 − β ( δ, 0) less than some small constant.
On the other hand, if we use δ that makes quick decisions, then δ may make careless decisions, i.e., raise lots of false alarms because the amount of evidences is insufﬁcient.
At the end of this section, we show that the SPRT is optimal in the sense of this tradeoff.
The tradeoff of false alarms and stopping times for both i.i.d.
and non-i.i.d.
We formulate the tradeoff of the false alarm rate and the stopping time.
We can derive the fundamental relation of the threshold to the operation characteristic in both i.i.d.
and non-i.i.d.
cases (Tartakovsky et al.
[Equation]
(19) where we deﬁned α ∗ y :
= α y ( δ ∗ ) ( y ∈ { 1 , 0 } ).
These inequalities essentially represent the tradeoff of the false alarm rate and the stopping time.
For example, as the thresholds a y ( y ∈ { 1 , 0 } ) increase, the false alarm rate and the false rejection rate decrease, as (19) suggests, but the stopping time is likely to be larger, because more observations are needed to accumulate log-likelihood ratios to hit the larger thresholds.
The asymptotic approximation and the no-overshoot approximation.
Equation 19 is an example of the tradeoff of the false alarm rate and the stopping time; further, we can derive another example in terms of the mean hitting time.
Before that, we introduce two types of approximations that simplify our analysis.
The ﬁrst one is the no-overshoot approximation.
It assumes to ignore the threshold overshoots of the log-likelihood ratio at the decision time.
This approximation is valid when the log-likelihood ratio of a single frame is sufﬁciently small compared to the gap of the thresholds, at least around the decision time.
On the other hand, the second one is the asymptotic approximation, which assumes a 0 , a 1 → ∞ , being equivalent to sufﬁciently low false alarm rates and false rejection rates at the expense of the stopping time.
These approximations drastically facilitate the theoretical analysis; in fact, the no-overshoot approximation alters (19) as follows (see Tartakovsky et al.
[Equation]
which is equivalent to
[Equation]
where β ∗ ( y ) :
= β ( δ ∗ , y ) ( y ∈ { 1 , 0 } ).
Further assuming the asymptotic approximation, we obtain
[Equation]
Therefore, as the threshold gap increases, the false alarm rate and the false rejection rate decrease exponentially, while the decision making becomes slow, as is shown in the following.
Mean hitting time without overshoots.
Let I y :=
E y
[ Z (1) ] ( y ∈ { 1 , 0 } ) be the Kullback-Leibler divergence of f 1 and f 0 .
I y is larger if the two densities are more distinguishable.
Note that
I y (cid:13)since P y ( Z (1) = 0) (cid:12) 1 , and thus the mean hitting times of the SPRT without overshoots are expressed as
[Equation]
In Tartakovsky et al.
Introducing the function
[Equation]
we can simplify (25-26):
[Equation]
(25-26) shows the tradeoff as we mentioned above: the mean hitting time of positive (negative) data diverges if we are to set the false alarm (rejection) rate to be zero.
The tradeoff with overshoots.
Introducing the overshoots explicitly, we can obtain the equality, instead of the inequality such as (19), that connects the the error rates and the thresholds.
We ﬁrst deﬁne the overshoots of the thresholds a 0 and a 1 at the stopping time as
[Equation]
We further deﬁne the expectation of the exponentiated overshoots as
[Equation]
Then we can relate the thresholds to the error rates (without the no-overshoots approximation,
Tartakovsky (1991)):
[Equation]
To obtain more speciﬁc dependence on the thresholds a y ( y ∈ { 1 , 0 } ), we adopt the asymptotic approximation.
Let T 0 ( a 0 ) and T 1 ( a 1 ) be the one-sided stopping times, i.e., T 0 ( a 0 ) :
= inf { T ≥ 1 | λ T ≤ − a 0 } and T 1 ( a 1 ) :
= inf { T ≥ 1 | λ T ≥ a 1 } .
We then deﬁne the associated overshoots as
[Equation]
[Equation]
According to Lotov (1988), we can show that
[Equation]
(37) under the asymptotic approximation.
Note that
[Equation]
(38) have no dependence on the thresholds a y ( y ∈ { 1 , 0 } ).
Therefore we have obtained more precise dependence of the error rates on the thresholds than (24):
Theorem A.1.
The Asymptotic tradeoff with overshoots Assume that 0 < I y < ∞ ( y ∈ { 1 , 0 } ).
Let ζ y be given in (38).
Then
[Equation]
Mean hitting time with overshoots.
A more general form of the mean hitting time is provided in
Tartakovsky (1991).
We can show that
[Equation]
The mean hitting times (40-41) explicitly depend on the overshoots, compared with (25-26).
Let
[Equation]
be the limiting average overshoots in the one-sided tests.
Note that χ y have no dependence on a y ( y ∈ { 1 , 0 } ).
The asymptotic mean hitting times with overshoots are
[Equation]
As expressed in Tartakovsky et al.
Therefore, they have an asymptotically linear dependence on the thresholds.
A. T HE N EYMAN -P EARSON TEST AND THE SPRT
So far, we have discussed the tradeoff of the false alarm rate and the mean hitting time and several properties of the operation characteristic and the mean hitting time.
Next, we compare the SPRT with the Neyman-Pearson test, which is well-known to be optimal in the classiﬁcation of time-series with ﬁxed sample lengths; in contrast, the SPRT is optimal in the early classiﬁcation of time-series with indeﬁnite sample lengths, as we show in the next section.
We show that the Neyman-Pearson test is optimal in the two-hypothesis testing problem or the binary classiﬁcation of time-series.
Nevertheless, we show that in the i.i.d.
Gaussian model, the SPRT terminates earlier than the Neyman-Pearson test despite the same error rates.
Preliminaries.
Before deﬁning the Neyman-Pearson test, we specify what the "best" test should be.
There are three criteria, namely the most powerful test, Bayes test, and minimax test.
To explain them in detail, we have to deﬁne the size and the power of the test.
The signiﬁcance level, or simply the size of test d is deﬁned as
[Equation]
It is also known as the false positive rate, the false alarm rate, or the false acceptance rate of the test.
On the other hand, the power of the test d is given by
[Equation]
γ is also called the true positive rate, the true acceptance rate.
the recall, or the sensitivity.
β is known as the false negative rate or the false rejection rate.
Now, we can deﬁne the three criteria mentioned above.
Deﬁnition A.4.
Most powerful test
The most powerful test d of signiﬁcance level α ( > 0) is deﬁned as the test that for every other test d (cid:48) of signiﬁcance level α , the power of d is greater than or equal to that of d (cid:48) :
[Equation]
[Equation]
Deﬁnition A.5.
Bayes test
Let π 0 :
= P ( y = 0) and π 1 := P ( y = 1)
= 1 − π 0 be the prior probabilities of hypotheses H 0 and
H 1 , and ¯ α ( d ) be the average probability of error:
[Equation]
where α i ( d ) :
= P ( d (cid:54) = i | y
= i ) is the false negative rate of the class i ∈ { 1 , 0 } .
A Bayes test, denoted by d B , for the priors is deﬁned as the test that minimizes the average probability of error:
[Equation]
48) where the inﬁmum is taken over all ﬁxed-sample-size decision rules.
Deﬁnition A.6.
Minimax test
Let α max ( d ) be the maximum error probability:
[Equation]
A minimax test, denoted by d M , is deﬁned as the test that minimizes the maximum error probability:
[Equation]
where the inﬁmum is taken over all ﬁxed-sample-size tests.
Note that a ﬁxed-sample-size decision rule or non-sequential rule is the decision rule with a ﬁxed stopping time T = N with probability one.
Deﬁnition and the optimality of the Neyman-Pearson test.
Based on the above notions, we state the deﬁnition and the optimality of the Neyman-Pearson test.
We see the most powerful test for the two-hypothesis testing problem is the Neyman-Pearson test; the theorem below is also the deﬁnition of the Neyman-Pearson test.
Theorem A.2.
Neyman-Pearson lemma Consider the two-hypothesis testing problem, i.e., the problem of testing two hypotheses
H o : P = P 0 and H 1 : P 1 , where P 0 and P 1 are two probability distributions with densities p 0 and p 1 with respect to some probability measure.
The most powerful test is given by
[Equation]
where Λ( X (1 ,T ) )
=
p 1 ( X (1 ,T ) )
p
( X (1 ,T ) ) is the likelihood ratio and the threshold h ( α ) is deﬁned as
[Equation]
to ensure for the false positive rate to be the user-deﬁned value α ( > 0) .
d NP is referred to as the Neyman-Pearson test and is also optimal with respect to the Bayes and minimax criteria:
Theorem A.3.
Neyman-Pearson test is Bayes optimal Consider the two-hypothesis testing problem.
Given a prior distribution π i ( i ∈ { 1 , 0 } ) the Bayes test d B , which minimizes the average error probability ¯ α ( d ) =
π
[Equation] [Equation]
π [Equation] 1 ( d ) , is given by
[Equation]
That is, the Bayesian test is given by the Neyman-Pearson test with the threshold π 0 /π 1 .
Theorem A.4.
Neyman-Pearson test is minimax optimal Consider the two-hypothesis testing problem.
the minimax test d M , which minimizes the maximal error probability α max ( d ) = max i ∈{ 1 , 0 } { α i ( d ) } , is the Neyman-Pearson test with the threshold such that α 0 ( d M )
=
α 1 ( d M ) .
The proofs are given in Borovkov (1998) and Lehmann & Romano (2006).
The SPRT is more efﬁcient.
We have shown that the Neyman-Pearson test is optimal in the twohypothesis testing problem, in the sense that the Neyman-Pearson test is the most powerful, Bayes, and minimax test; nevertheless, we can show that the SPRT terminates faster than the Neyman-Pearson test even when these two show the same error rate.
Consider the two-hypothesis testing problem for the i.i.d.
Gaussian model:
[Equation]
where N (0 , σ 2 ) denotes the Gaussian distribution with mean 0 and variance σ 2 .
The Neyman-Pearson test has the form
[Equation]
The sequence length n = n ( α 0 , α 1 ) and the threshold h
= h ( α 0 , α 1 ) are deﬁned so as for the false positie rate and the false negative rate to be equal to α 0 and α 1 respectively; i.e.,
[Equation]
We can solve them for the i.i.d.
Gaussian model (Tartakovsky et al.
To see the efﬁciency of the SPRT to the Neyman-Pearson test, we deﬁne
[Equation]
Assuming the overshoots are negligible, we obtain the following asymptotic efﬁciency (Tartakovsky et al.
[Equation]
In other words, under the no-overshoot and the asymptotic assumptions, the SPRT terminates four times earlier than the Neyman-Pearson test in expectation, despite the same false positive and negative rates.
A. T HE O PTIMALITY OF THE SPRT
Optimality in i.i.d.
cases.
The theorem below shows that the SPRT minimizes the expected hitting times in the class of decision rules that have bounded false positive and negative rates.
Consider the two-hypothesis testing problem.
We deﬁne the class of decision rules as
[Equation]
Then the optimality theorem states:
Theorem A.5.
I.I.D. Optimality (Tartakovsky et al.
Let the time-series data points x ( t ) , t = 1 , 2 , ... be i.i.d.
with density f 0 under H 0 and with density f 1 under H 1 , where f 0 (cid:54)≡ f 1 .
Let α 0 > 0 and α 1 > 0 be ﬁxed constants such that α
+ α 1 < 1 .
If the thresholds − a o and a 1 satisﬁes α ∗ 0 ( a 0 , a 1 ) =
α
and α ∗ 1 ( a 0 , a 1 )
=
α 1 , then the SPRT δ ∗ =
( d ∗ , τ ∗ ) satisﬁes inf δ =( d,τ ) ∈ C ( α 0 ,α 1 ) (cid:26) E [ τ | H 0 ] (cid:27) =
E [ τ ∗ | H 0 ] and inf δ =( d,τ ) ∈ C ( α 0 ,α 1 ) (cid:26) E
[Equation]
A similar optimality holds for continuous-time processes (Irle & Schmitz (1984)).
Therefore the
SPRT terminates at the earliest stopping time in expectation of any other decision rules achieving the same or less error rates — the SPRT is optimal.
Theorem A.5 tells us that given user-deﬁned thresholds, the SPRT attains the optimal mean hitting time.
Also, remember that the thresholds determine the error rates (e.g., Equation (24)).
Therefore, the SPRT can minimize the required number of samples and achieve the desired upper-bounds of false positive and false negative rates.
Asymptotic optimality in general non-i.i.d.
cases.
In most of the discussion above, we have assumed the time-series samples are i.i.d.
For general non-i.i.d.
distributions, we have the asymptotic optimality; i.e., the SPRT asymptotically minimizes the moments of the stopping time distribution (Tartakovsky et al.
Before stating the theorem, we ﬁrst deﬁne a type of convergence of random variables.
Deﬁnition A.7.
r-quick convergence Let { x ( t ) } t ≥ 1 be a stochastic process.
Let T (cid:15) ( { x ( t ) } t ≥ 1 ) be the last entry time of the stochastic process { x ( t ) } t ≥ 1 in the region ( (cid:15), ∞ ) ∪ ( −∞ , − (cid:15) ) , i.e.,
[Equation]
Then, we say that the stochastic process { x ( t ) } t ≥ 1 converges to zero r-quickly, or
[Equation]
) for some r > 0 , if E
[Equation]
[Equation]
(65) r -quick convergence ensures that the last entry time in the large-deviation region ( T (cid:15) ( { x ( t ) } t ≥ 1 ) ) is ﬁnite almost surely.
The asymptotic optimality theorem is:
Theorem A.6.
Non-i.i.d.
asymptotic optimality
If there exist positive constants I
and I 1 and an increasing non-negative function ψ ( t ) such that
[Equation]
where λ t is deﬁned in section A.1, then
[Equation]
a
[Equation]
Moreover, if the thresholds a 0 and a 1 are chosen to satisfy (19), a 0 → log(1 /α
∗ 1 ) , and a 1 → log(1 /α ∗ 0 ) ( a i → ∞ ), then for all 0 < m ≤ r ,
[Equation]
[Equation] [Equation]
as max { α 0 , α 1 } −→ 0 with | log α 0 / log α 1 | −→ c , where c ∈ (0 , ∞ ) .
B S UPPLEMENTARY REVIEW OF THE RELATED WORK
Primate’s decision making and parietal cortical neurons.
The process of decision making involves multiple steps, such as evidence accumulation, reward prediction, risk evaluation, and action selection.
We give a brief overview regarding mainly to neural activities of primate parietal lobe and their relationship to the evidence accumulation, instead of providing a comprehensive review of the decision making literature.
Interested readers may refer to review articles, such as Doya (2008);
Gallivan et al.
; Gold & Shadlen (2007).
In order to study neural correlates of decision making, Roitman & Shadlen (2002) used a random dot motion (RDM) task on non-human primates.
They found that the neurons in the cortical area lateral intraparietal cortex, or LIP, gradually accumulated sensory evidence represented as increasing ﬁring rate, toward one of the two thresholds corresponding to the two-alternative choices.
Moreover, while a steeper increase of ﬁring rates leads to an early decision of the animal, the ﬁnal ﬁring rates at the decision time is almost constant regardless of reaction time.
Thus, at least in population-level LIP neurons are representing information very similar to that of the LLR in the SPRT algorithm
(It is under active discussion whether the ramping activity is seen only in averaged population ﬁring rate or both in population and single neuron level.
See Latimer et al.
Shadlen et al. (2016)).
But also see Okazawa et al.
(2021) for a recent ﬁnding that evidence accumulation is represented in a high-dimensional manifold of neural population.
In any case, LIP neurons seem to represent accumulated evidence as their activity patterns.
To test whether the ramping activity is explained by Wald’s SPRT, Kira et al. (2015) used visual stimuli associated with reward likelihood: each stimulus indicates the answer of the binary choice task with a certain probability (e.g., if stimulus ’A’ is presented, choice 1 is the correct answer with 30% probability).
LIP neurons’ activities in response to these randomly presented stimuli are proportional to LLR calculated from the associated likelihood of the stimuli, letting authors concluded that the activity of LIP neurons are best explained by SPRT than other alternative models.
It remains unclear, however, what algorithm is used in the brain when stimuli are not randomly presented but temporary dependent.
More complex decision making involving risk evaluation such as ”delayed, large reward V.S. immediate, small reward” is thought to be guided by other regions including orbitofrontal cortex, dorsal striatum or dorsal prefrontal cortex (McClure et al.
Rudebeck et al.
; Tanaka et al.
Application of SPRT.
Ever since Wald’s formulation, the sequential hypothesis testing was applied to study decision making and its reaction time (Stone (1960); Edwards (1965); Ashby (1983)).
Several extensions to more general problem settings were also proposed.
In order to test more than two hypotheses, multi-hypothesis SPRT (MSPRT) was introduced (Armitage (1950); Baum & Veeravalli (1994)), and shown to be asymptotically optimal (Dragalin et al.
(1999; 2000); Veeravalli & Baum (1995)).
The SPRT was also generalized for non-i.i.d.
data (Lai (1981); Tartakovsky (1999)), and theoretically shown to be asymptotically optimal, given the known LLR (Dragalin et al. (1999; 2000)).
Tartakovsky et al.
(2014) provided a comprehensive review of these theoretical analyses, a part of whose reasoning we also follow to show optimality in Appendix A.
The SPRT, and closely related, generalized LLR test, applied to solve several problems includes drug safety surveillance (Kulldorff et al.
(2011)), exoplanet detection (Hu et al.
, and the LLR test out of weak classiﬁers (WaldBoost, Sochman & Matas (2005)), to name a few.
On an A/B test, Johari et al.
(2017) tackled an important problem of inﬂating error rates at the sequential hypothesis testing.
Ju et al.
(2019) proposed an inputed Girshick test to determine a better variant.
Time-series classiﬁcation.
Here, we use the term “Time-series” interchangeably to mention both continuous data or discrete data such as video frames.
One of the traditional approaches to univariate or multivariate time series classiﬁcation is distancebased methods, such as dynamic time warping (Bagnall (2014); Jeong et al.
(2011); Kate (2015)) or k-nearest neighbors (Dau et al.
; Wei & Keogh (2006); Yang & Shahabi (2007)).
More recently,
Collective Of Transformation-based Ensembles (COTE) and its variant, COTE with Hierarchical
Vote system (HIVE-COTE) showed high classiﬁcation performance at the expense of their high computational cost (Bagnall et al.
; Lines et al.
Word Extraction for time series classiﬁcation (WEASEL) and its variant, WEASEL + MUSE take a bag-of-pattern approach to utilize carefully designed feature vectors (Schäfer & Leser (2017)).
The advent of deep learning allows researchers to classify not only univariate/multivariate data, but also large-size, video data using convolutional neural networks (Hara et al.
; Carreira &
Zisserman (2017); Karim et al.
; Wang et al.
Thanks to the increasing computation power and memory of modern processing units, each video data in a minibatch are designed to be sufﬁciently long in the time domain such that class signature can be contained.
Video length of the training, validation, and test data are often assumed to be ﬁxed; however, ensuring sufﬁcient length for all data may compromise the classiﬁcation speed (i.e., number of samples that used for classiﬁcation).
We extensively test this issue in Section 5.
C D ERIVATION OF THE TANDEM FORMULA
The derivations of the important formulas in Section 3 are provided below.
The 0th order (i.i.d.)
TANDEM formula.
We use the following probability ratio to identify if the input sequence { x ( s ) } ts =1 is derived from either hypothesis H 1 : y = 1 or H 0 : y = 0 .
[Equation]
We can rewrite it with the posterior.
First, by repeatedly using the Bayes rule, we obtain p
[Equation]
×
[Equation]
x ( t
[Equation]
) | x ( t
[Equation] [Equation]
x ( t −
[Equation]
[Equation]
x (1)
[Equation]
y )
[Equation]
x ( t −
[Equation] [Equation]
x ( t
[Equation] [Equation]
[Equation]
x
[Equation]
y )
[Equation]
[Equation]
[Equation]
We use this formula hereafter.
Let us assume that the process { x ( s ) } ts =1 is conditionally-independently and identically distributed (hereafter simply noted as i.i.d.), namely
[Equation]
[Equation]
= t (cid:89) s =
[Equation]
p (
[Equation]
[Equation]
(72) which yields the following LLR representation ("0-th order Markov process"):
[Equation]
Then p ( x (1) , x (2) , ..., x ( t ) | y ) =
p ( x ( t ) | y ) p ( x ( t − 1) | y ) . . .
p ( x (2) | y ) p ( x (1) | y )
=
t (cid:89) s = (cid:104) p
( x ( s ) | y ) (cid:105)
=
t (cid:89) s = (cid:20)
p
( y | x ( s ) )
p ( x ( s ) ) p ( y ) (cid:21) .
Hence p ( x (1) , x (2) , ..., x ( t ) | y = 1) p ( x (1) , x (2) , ..., x ( t ) | y = 0) =
t (cid:89) s = (cid:20)
p
( y = 1 | x ( s ) )
p ( y =
| x ( s ) )
(cid:21) (cid:18) p
( y = 0) p ( y = 1) (cid:19) t , (74) or log (cid:18) p ( x (1) , x (2) , ..., x ( t ) | y
= 1) p ( x (1) , x (2) , ..., x ( t ) | y = 0) (cid:19) =
t (cid:88) s = log (cid:18)
p ( y = 1 | x ( s ) )
p ( y =
| x ( s ) )
(cid:19) − t log (cid:18) p (
[Equation]
The 1st-order TANDEM formula.
So far, we have utilized the i.i.d.
assumption (73) or (72).
Now let us derive the probability ratio of the ﬁrst-order Markov process , which assumes
[Equation]
Applying (76) to (71), we obtain p [Equation]
=
[Equation]
( s [Equation]
[Equation] x (
[Equation]
[Equation]
[Equation] [Equation] (77) for t ≥ 2 .
Hence p
[Equation]
=
[-( y = 1 | x ( s ) , x ( s − 1) ) p ( y = 
 | x ( s ) , x ( s − 1) ) (cid:21) t (cid:89) s = (cid:20)-][Equation]
p
[-( y = 
-]
[Equation]
s
[Equation]
or log
[Equation]
For t = 1 and t = 2 , the natural extensions are log
[Equation]
The N -th order TANDEM formula.
Finally we extend the 1st order TANDEM formula so that it can calculate the general N -th order log-likelihood ratio.
The N -th order Markov process is deﬁned as
[Equation]
[Equation]
Therefore, for t ≥ N +
[Equation]
.., x ( s − N ) ) (cid:21) p ( y | x ( N ) , ..., x (1) ) p ( x ( N ) , ..., x (1) ) p ( y ) .
Hence p ( x (1) , x (2) , ..., x ( t ) | y = 1) p ( x (1) , x (2) , ..., x ( t ) | y = 0) =
t (cid:89) s = N + (cid:20)
[Equation]
( y =
[Equation]
| x ( s ) , ..., x ( s − N ) ) p ( y =
[Equation]
x ( s ) , ..., x ( s − N ) ) (cid:21) t (cid:89)
[Equation]
N
[Equation]
(cid:20)
[Equation]
( y
[Equation]
| x ( s − 1) , ..., x ( s − N ) ) p ( y
[Equation]
x ( s
[Equation]
1) , ..., x ( s
[Equation]
N ) )
(cid:21) p ( y = 0) p ( y = 1) , (83) or log
[Equation]
[Equation]
For t < N + 2 , we obtain log
[Equation]
D S UPPLEMENTARY DISCUSSION
Why is the SPRT-TANDEM superior to other baselines?
The potential drawbacks common to the LSTM-s/m and EARLIEST is that they incorporate long temporal correlation: it may lead to (1) the class signature length problem and (2) vanishing gradient problem, as we described in
Section 3.
(1) If a class signature is signiﬁcantly shorter than the correlation length in consideration, uninformative data samples are included in calculating the log-likelihood ratio, resulting in a late or wrong decision.
(2) long correlations require calculating a long-range of backpropagation, prone to the vanishing gradient problem.
An LSTM-s/m-speciﬁc drawback is similar to that of Neyman-Pearson test, in the sense that it ﬁxes the number of samples before performance evaluations.
On the other hand, the SPRT, and the SPRT-TANDEM, classify various lengths of samples: thus, the SPRT-TANDEM can achieve a smaller sampling number with high accuracy on average.
Another potential drawback of LSTM-s/m is that their loss function explicitly imposes monotonicity to the scores.
While the monotonicity is advantageous for quick decisions, it may sacriﬁce ﬂexibility: the LSTM-s/m can hardly change its mind during a classiﬁcation.
EARLIEST, the reinforcement-learning based classiﬁer, decides on the various length of samples.
A potential EARLIEST-speciﬁc drawback is that deep reinforcement learning is known to be unstable (Nikishin et al.
; Kumar et al.).
How optimal is the SPRT-TANDEM?
In practice, it is difﬁcult to strictly satisfy the necessary conditions for the SPRT’s optimality (Theorem A.5 and A.6) because of experimental limitations.
One of our primary interests is to apply the SPRT, the provably optimal algorithm, to real-world datasets.
A major concern about extending Wald’s SPRT is that we need to know the true likelihood ratio a priori to implement the SPRT.
Thus we propose the SPRT-TANDEM with the help of machine learning and density ratio estimation to remove the concern, if not completely.
However, some technical limitations still exist.
Let us introduce two properties of the SPRT that can prevent the
SPRT-TANDEM from approaching exact optimality.
Firstly, the SPRT is assumed to terminate for all the LLR trajectories under consideration with probability one.
The corresponding equation stating this assumption is Equation (61) and (66) under the i.i.d.
and non-i.i.d.
condition, respectively.
Given that this assumption (and the other minor technical conditions in Theorem A.5 and A.6) is satisﬁed, the more precisely we estimate the LLRs, the more we approach the genuine SPRT implementation and thus its asymptotic Bayes optimality.
Secondly, the non-i.i.d.
SPRT is asymptotically optimal when the maximum number of samples allowed is not ﬁxed ( inﬁnite horizon ).
On the other hand, our experiment truncates the SPRT ( ﬁnite horizon ) at the maximum timestamp, which depends on the datasets.
Under the truncation, gradually collapsing thresholds are proven to give the optimal stopping (Tartakovsky et al.
; however, the collapsing thresholds are obtained via backward induction (Bingham et al.
(2006)), which is possible only after observing the full sequence.
Thus, under the truncation, ﬁnding the optimal solutions in a strict sense critically limits practical applicability.
The truncation is just an experimental requirement and is not an essential assumption for the SPRTTANDEM.
Under the inﬁnite horizon settings, the LLRs is assumed to increase or decrease toward the thresholds (Theorem A.6) in order to ensure the asymptotic optimality.
However, we observed that the estimated LLRs tend to be asymptotically ﬂat, especially when N is large (Figure 10, 11, and 12); the estimated LLRs can violate the assumption of Theorem A.6.
One potential reason for the ﬂat LLRs is the TANDEM formula: the ﬁrst and second term of the formula has a different sign.
Thus, the resulting log-likelihood ratio will be updated only when the difference between the two terms are non-zero.
Because the ﬁrst and second term depends on N +and N inputs, respectively, it is expected that the contribution of one input becomes relatively small as N is enlarged.
We are aware of this issue and already started working on it as future work.
Nevertheless, the ﬂat LLRs at least do not spoil the practical efﬁciency of the SPRT-TANDEM, as our experiment shows.
In fact, because we cannot know the true LLR of real-world datasets, it is not easy to discuss whether the assumption of the increasing LLRs is valid on the three databases (NMNIST,
UCF, and SiW) we tested.
Numerical simulation may be possible, but it is out of our scope because our primary interest is to implement a practically usable SPRT under real-world scenarios.
The best order N of the SPRT-TANDEM.
The order N is a hyperparamer, as we mentioned in
Section 3 and thus needs to be tuned to attain the best performance.
However, each dataset has its own temporal structure, and thus it is challenging to acquire the best order a priori.
In the following, we provide a rough estimation of the best order, which may give dramatic beneﬁt to the users and may lead to exciting future works.
Let us introduce a concept, speciﬁc time scale , which is used in physics to analyze qualitative behavior of a physical system.
Here, we deﬁne the speciﬁc time scale of a physical system as a temporal interval in which the physical system develops dramatically.
For example, suppose that a physical system under consideration is a small segment of spacetime in which an unstable particle, ortho-positronium (o-Ps), exists.
In this case, a speciﬁc time scale can be deﬁned as the lifetime of o-Ps, 0 .
14 µs (Czarnecki (1999)), because the o-Ps is likely to vanish in 0 .
14 × O (1) µs — the physical system has changed completely.
Note that the deﬁnition of speciﬁc time scale is not unique for one physical system; it depends on the phenomena the researcher focuses on.
Speciﬁc (time) scale are often found in fundamental equations that describes physical systems.
In the example above, the decay equation N ( t ) =
A exp( − t/τ ) has the lifetime τ ∈ R in itself.
Here N ( t ) ∈
R is the expected number of o-Ps’ at time t ∈ R , and A ∈ R is a constant.
Let us borrow the concept of the speciﬁc time scale to estimate the best order of the SPRT-TANDEM before training neural networks, though there is a gap in scale.
In this case, we deﬁne the speciﬁc time scale of a dataset as the number of frames after which a typical video in the dataset shows completely different scene.
As is discussed below, we claim that the speciﬁc time scale of a dataset is a good estimation of the best order of the SPRT-TANDEM , because the correlations shorter than the speciﬁc time scale are insufﬁcient to distinguish each class, while the longer correlations may be contaminated with noise and keep redundant information.
First, we consider Nosaic MNSIT (NMNIST).
The speciﬁc time scale of NMNIST can be deﬁned as the half-life 2 of the noise, i.e., the necessary temporal interval for half of the noise to disappear.
It isframes by deﬁnition of NMNIST, and approximately matches the best order of the SPRT-TANDEM: in Figure 3, our experiment shows that the 10th order SPRT-TANDEM (with 11-frames correlation) outperforms the other orders in the latter timestamps, though we did not perform experiments with all the possible orders.
A potential underlying mechanism is: Too long correlations keep noisy information in earlier timestamps, causing degradation, while too short correlations do not fully utilize the past information.
Next, we discuss the two classes in UCF101 action recognition database, handstand pushups and handstand walking, which are used in our experiment.
A speciﬁc time scale is ∼ 10 frames because of the following reasons.
The ﬁrst class, handstand pushups, has a speciﬁc time scale of one cycle of raising and lowering one’s body ∼ 50 frames (according to the shortest video in the class).
The second class, handstand walking, has a speciﬁc time scale of one cycle of walking, i.e., two steps, ∼ 10 frames (according to the longest video in the class).
Therefore the speciﬁc time scale of UCF is ∼ 10 , the smaller one, since we can see whether there is a class signature in a video within at most ∼ 10 frames.
The speciﬁc time scale matches the best order of the SPRT-TANDEM according to
Figure 3.
Finally, a speciﬁc time scale of SiW is ∼ 1 frame, because a single image sufﬁces to distinguish a real person and a spooﬁng image, because of the reﬂection of the display, texture of the photo, or the movement speciﬁc to a live person 3 .
The best order in Figure 3 is ∼ 1 , matching the speciﬁc time scale.
We make comments on two potential future works related to estimation of the best order of the
SPRT-TANDEM.
First, as our experiments include only short videos, it is an interesting future work to estimate the best order of the SPRT-TANDEM in super-long video classiﬁcation, where gradient vanishing becomes a problem and likelihood estimation becomes more challenging.
Second, it is an exciting future work to analyse the relation of the speciﬁc time scale to the best order when there are multiple time scales.
For example, recall the discussion of locality above in this Appendix D:
Applying the SPRT-TANDEM to a dataset with distributed class signatures is challenging.
Distributed class signatures may have two speciﬁc time scales: e.g., one is the mean length of the signatures, and the other is the mean interval between the signatures.
The best threshold λ τ ∗ of the SPRT-TANDEM.
In practice, a user can change the thresholds after deploying the SPRT-TANDEM algorithm once and control the speed-accuracy tradeoff.
Computing the speed-accuracy-tradeoff curve is not expensive, and importantly computable without re-training.
According to the speed-accuracy-tradeoff curve, a user can choose the desired accuracy and speed.
Note that this ﬂexible property is missing in most other deep neural networks: controlling speed usually means changing the network structures and training it all over again.
End-to-end v.s. separate training.
The design of SPRT-TANDEM does not hamper an end-to-end training of neural networks; the feature extractor and temporal integrator can be readily connected for thorough backpropagation calculation.
However, in Section 5, we trained the feature integrator and temporal integrator separately: after training the feature integrator, its trainable parameters are ﬁxed to start training the temporal integrator.
We decided to train the two networks separately because we found that it achieves better balanced accuracy and mean hitting time.
Originally we trained the network using NMNIST database with an end-to-end manner, but the accuracy was far lower than the result reported in Section 5.
We observed the same phenomenon when we trained the SPRT-TANDEM on our private video database containing 1-channel infrared videos.
These observations might indicate that while the separate training may lose necessary information for classiﬁcation compared to the end-to-end approach, it helps the training of the temporal integrator by ﬁxing information at each data point.
It will be interesting to study if this is a common problem in early-classiﬁcation algorithms
and
ﬁnd the right balance between the end-to-end and separate training to beneﬁt both approaches.
Feedback to the ﬁeld of neuroscience.
Kira et al.
(2015) experimentally showed that the SPRT could explain neural activities in the area LIP at the macaque parietal lobe.
They randomly presented a sequence of visual objects with associated reward probability.
A natural question arises from here: what if the presented sequence is not random, but a time-dependent visual sequence?
Will the neural activity be explained by our SPRT-TANDEM, or will the neurons utilize a completely different algorithm?
Our research provides one driving hypothesis to lead the neuroscience community to a deeper understanding of the brain’s decision-making system.
Usage of statistical tests.
As of writing this manuscript, not all of the computer science papers use statistical tests to evaluate their experiments.
However, in order to provide an objective comparison across proposed and existing models, running multiple validation trials with random seeds followed by a statistical test is helpful.
Thus, the authors hope that our paper stimulates the ﬁeld of computer science to utilize statistical tests more actively.
Ethical concern.
The proposed method, SPRT-TANDEM, is a general algorithm applicable to a broad range of serial data, such as auditory signals or video frames.
Thus, any ethical concerns entirely depend on the application and training database, not on our algorithm per se .
For example, if
SPRT-TANDEM is applied to a face spooﬁng detection, using faces of people of one particular racial or ethnic group as training data may lead to a bias toward or against people of other groups.
However, this bias is a concern in machine learning in general, not speciﬁc to the SPRT-TANDEM.
Is the SPRT-TANDEM “too local”?
In our experiments in Section 5, the SPRT-TANDEM with maximum correlation allowed (i.e., 19th, 49th, and 49th on NMNIST, UCF, and SiW databases, respectively) does not necessarily reach the highest accuracy with a larger number of frames.
Instead, depending on the database, the lower order of approximation, such as 10th order TANDEM, outperforms the other orders.
In the SiW database, this observation is especially prominent: the model records the highest balanced accuracy is the 2nd order SPRT-TANDEM.
While this may indicate our
TANDEM formula with the “dropping correlation” strategy works well as we expected, a remaining concern is the SPRT may integrate too local information.
What if class signatures are far separated in time?
In such a case, the SPRT-TANDEM may fail to integrate the distributed class signatures for correct classiﬁcation.
On the other hand, the SPRT-TANDEM may be able to add the useful information of the class signatures to the LLR only when encountering the signatures (in other words, do not add non-zero values to the LLR without seeing class signatures).
The SPRT-TANDEM may be able to skip unnecessary data points without modiﬁcation, or with a modiﬁcation similar to SkipRNN (Campos et al.
(2018)), which actively achieve this goal: by learning unnecessary data point, the SkipRNN skips updating the internal state of RNN to attend just to informative data.
Similarly, we can modify the SPRT-TANDEM so that it learns to skip updating LLR upon encountering uninformative data.
It will be exciting future work, and the authors are looking forward to testing the SPRT-TANDEM on a challenging database with distributed class signatures.
A more challenging dataset: Nosaic MNIST-Hard (NMNIST-H).
In the main text, we see the accuracy of the SPRT-TANDEM saturates within a few timestamps.
Therefore, it is worth testing the models on a dataset that require more samples for reaching good performance.
We create a more challenging dataset, Nosaic MNIST-Hard : The MNIST handwritten digits are buried with heavier noise than the oriﬁnal NMNIST (only 10 pixels/frame are revealed, while it is 40 pixels/frame for the original NMNIST).
The resulting speed-accuracy tradeoff curves below show that the SPRTTANDEM outperforms LSTM-s/m more than the error-bar range, even on the more challenging dataset requiring more timestamps to attain the accuracy saturation. [Table]
[Table]
E L OSS FOR L OG -L IKELIHOOD R ATIO ESTIMATION (LLLR)
In this section, we discuss a deep connection of the novel loss function, LLLR
[Equation]
to density ratio estimation (Sugiyama et al.
Here, X i := { x ( t ) i ∈
R d x } Tt =
and y i ∈ { 1 , 0 } ( i ∈ I :
= I 1 ∪ I 0 , T ∈ N , d x ∈ N ) are a sequence of samples and a label, respectively, where I , I 1 , and I 0 are the index sets of the whole dataset, class 1, and class 0, respectively.
ˆ r
( X i ) ( i ∈ I ) is the likelihood ratio of X i .
The hatted notation ( ˆ · ) means that the quantity is an estimation with, e.g., a neural network on the training dataset { ( X i , y i ) } i ∈ I .
Note that we do not necessarily have to compute ˆ p ( X i | y = 1) and ˆ p ( X i | y = 0) separately to obtain the likelihood ratio ˆ r ( X i )
=
ˆ p ( X | y =1) ˆ p ( X | y =0) ; we can estimate ˆ r directly, as is explained in the following subsections.
In the following, we ﬁrst introduce KLIEP (Kullback-Leibler Importance Estimation Procedure,
Sugiyama et al.
(2008)), which underlies the theoretical aspects of the LLLR.
KLIEP was originally invented to estimate density ratio without directly estimating the densities.
The idea is to minimize the
Kullback-Leibler divergence of the true density p ( X | y = 1) and the estimated density ˆ r ( X ) p ( X | y = 0) , where ( X, y ) is a sequential data-label pair deﬁned on the same space as ( X i , y i ) ’
s. Next, we introduce the symmetrized KLIEP, which cares about not only p ( X | y = 1) and ˆ r ( X ) p
( X | y = 0) , but p ( X | y = 0) and ˆ r − 1 ( X ) p ( X | y = 1) to remove the asymmetry inherent in the Kullback-Leibler divergence.
Finally, we show the equivalence of the symmetrized KLIEP to the LLLR; speciﬁcally, we show that the LLLR minimizes the Kullback-Leibler divergence of the true and the estimated density, and further stabilizes the training by restricting the value of likelihood ratio.
E. D ENSITY RATIO ESTIMATION AND KLIEP
In this section, we brieﬂy review density ratio estimation and introduce KLIEP.
Density estimation is the construction of underlying probability densities based on observed datasets.
Taking their ratio, we can naively estimate the density ratio; however, division by an estimated quantity is likely to enhance the estimation error (Sugiyama et al.
Density ratio estimation has been developed to circumvent this problem.
We can categorize the methods to the following four: probabilistic classiﬁcation, moment matching, density ratio ﬁtting, and density ﬁtting.
Probabilistic classiﬁcation.
The idea of the probabilistic classiﬁcation is that the posterior density p ( Y | X ) is easier to estimate than the likelihood p ( X | Y ) .
Notice that
[Equation]
where M 1 and M 0 denote the number of the training data points with label 1 and 0 respectively.
Thus we can estimate the likelihood ratio from the estimated posterior ratio.
The multiplet cross-entropy loss conducts the density ratio estimation in this way.
Moment matching.
The moment matching approach aims to match the moments of p ( X | y = 1) and ˆ r ( X ) p ( X | y = 0) , according to the fact that two distributions are identical if and only if all moments agree with each other.
Density ratio ﬁtting.
Without knowing the true densities, we can directly minimize the difference between the true and estimated ratio as follows:
[Equation]
Here, we applied the empirical approximation.
In addition, we restrict the value of ˆ r ( X ) : ˆ r ( X ) ≥ 0 .
Since (90) is not bounded below, we must add other terms or put more constraints, as is done in the original paper (Kanamori et al.
This formulation of density ratio estimation is referred to as least-squares importance ﬁtting (LSIF, Kanamori et al.
Density ﬁtting.
Instead of the squared expectation, KLIEP minimizes the Kullback-Leibler divergence:
[Equation]
[Equation]
We need to restrict ˆ r :
[Equation] [Equation]
The ﬁrst inequality ensures the positivity of the probability ratio, while the second equation is the normalization condition.
Applying the empirical approximation, we obtain the ﬁnal objective and the constraints:
[Equation]
Several papers implement the algorithms mentioned above using deep neural networks.
In Nam &
Sugiyama (2015), LSIF is applied to outlier detection with the deep neural network implementation, whereas in Khan et al.
(2019), KLIEP and its variant are applied to changepoint detection.
E. T HE SYMMETRIZED KLIEP LOSS
As shown above, KLIEP minimizes the Kullback-Leibler divergence; however, its asymmetry can cause instability of the training, and thus we introduce the symmetrized KLIEP loss .
A similar idea was proposed in Khan et al.
independently of our analysis.
First, notice that
[Equation]
The constant term is independent of the weight parameters of the network and thus negligible in the following discussion.
Similarly,
[Equation]
We need to restrict the value of ˆ r in order for p ( X | y = 1) and p ( X | y = 0) to be probability densities:
[Equation]
and
[Equation]
Therefore
[Equation]
we deﬁne the symmetrized KLIEP loss as
[Equation]
with the constraints (102)-(105).
The estimated ratio function argmin ˆ r ( X ) L KLIEP with the constraints minimizes KL( p ( X | y = 1)
||
ˆ r ( X ) p
( X | y = 0))
+ KL( p ( X | y = 0)
||
ˆ r − 1 p ( X | y = 1))) .
According to the empirical approximation, they reduce to
[Equation]
E. T HE LLLR AND DENSITY RATIO ESTIMATION
Let us investigate the LLLR in connection with the symmetrized KLIEP loss
[Equation]
Divergence terms.
First, we focus on the divergence terms in (107):
[Equation]
As shown above, decreasing (111) and (112) leads to minimizing the Kullback-Leibler divergence of p ( X | y = 1) and ˆ rp ( X | y = 0) and that of p ( X | y = 0) and ˆ r − 1 p ( X | y = 1) respectively.
The counterparts in the LLLR are
[Equation]
because, on one hand, both terms in (113) ensures the likelihood ratio ˆ r to be large for class 1, and, on the other hand, both terms in (114) ensures ˆ r to be small for class 0.
Therefore, minimizing
L LLR is equivalent to decreasing both (111) and (112) and therefore to minimizing (108), i.e., the
Kullback-Leibler divergences of the true and estimated densities.
Again, we emphasize that the LLLR is more stable, since L LLR is lower-bounded unlike the KLIEP loss.
Constraints.
Next, we show that the LLLR implicitly keeps ˆ r not too large nor too small; speciﬁcally, with increasing ˆ R
( X i ) :
= | log ˆ r
( X i ) | , the gradient converges to zero before ˆ R ( X i ) enters the region, e.g., ˆ R ( X i ) (cid:38) 1 .
Therefore the gradient descent converges before ˆ r
( X i ) becomes too large or small.
To show this, we ﬁrst write the gradients explicitly:
[Equation]
where W is the weight and σ is the sigmoid function.
We see that with increasing ˆ R ( X i ) =
| log ˆ r
( X i ) | , the factor
[Equation]
converges to zero, because (116) ∼ 0 for too large or small ˆ r ( X i ) , e.g., for around ˆ R ( X i ) (cid:38) 1 .
Thus the gradient (115) vanishes before ˆ r
( X i ) becomes too large or small, i.e., keeping ˆ r
( X i ) moderate.
In conclusion, the LLLR minimizes the difference between the true ( p ( X | y = 1) and p ( X | y = 0) ) and the estimated ( ˆ r − 1 ( X ) p ( X | y = 1) and ˆ r ( X ) p
( X | y = 0) )
densities in the sense of the KullbackLeibler divergence, including the effective constraints.
T HE TRIVIAL SOLUTION IN E QUATION (115).
We show that vanishing
[Equation]
in (115) corresponds to a trivial solution ; i.e., we show that (117) =
0 forces the bottleneck feature vectors to be zero.
Let us follow the notations in Table 4 and Figure 5.
We particularly focus on the last components in the gradient ∇ W log
ˆ r
( X i ) , i.e., ∇ W ( L ) ab log ˆ r ( X i ) ( a ∈ [ d L − 1 ] and b ∈ [ d L ] = { 0 , 1 } ).
Speciﬁcally,
[Equation]
| y = 0)
[Equation]
Since ∂ log ˆ p y (cid:48) /∂g
( L ) y
=
δ yy (cid:48) − ˆ p y , where y, y (cid:48) ∈ { 1 , 0 } and δ y,y (cid:48) is the Kronecker delta, we see [Equation] log ˆ
[Equation]
Thus (117) =
= ⇒ (118) =
⇐⇒ f
( L − 1) a = 0 ( ∀ a ∈ [ d L − 1 ] ), which is a trivial solution, because the bottleneck feature vector collapses to zero at convergence.
Our experiments, however, show that our model does not tend to such a trivial solution; otherwise, the SPRT-TANDEM cannot attain such a high performance.
[Equation]
vector L : number of layers t i ∈ R 2 : one-hot label vector σ : activation function
E. P REPARATORY EXPERIMENT TESTING THE EFFECTIVENESS OF THE LLLR
To test if the proposed L LLR could effectively train a neural network, we ran two preliminary experiments before the main manuscript.
First, we compared training the proposed network architecture
L multiplet , with and without L LLR .
Next, we compared training the network using L multiplet with L LLR , and training with L multiplet with the KLIEP loss, L KLIEP , whose numerator and denominator were carefully bounded so that the L KLIEP did not diverge.
[Figure]
We tested the effectiveness of L LLR on a 3D-ness detection task on a depth-from-motion (DfM) dataset 4 .
The DfM dataset was a small dataset containing 2320 and 2609 3D- and 2D- facial videos, respectively, each of which consisted of 10 frames.
In each of the video, a face was ﬁlmed from various angles (Figure 6a), so that the dynamics of the facial features could be used to determine whether the face appearing in a video is ﬂat, 2D face, or had a 3D structure.
The recording device was an iPhone7.
Here, the feature extractor f w FE ( x ( t ) ) was the LBP-AdaBoost algorithm (Viola & Jones (2001))
combined with the supervised descent method (Xiong & De la Torre (2013)), which took the t -th frame of the given video as input x ( t ) and output facial feature points (Figure 6b).
The feature points were output as a vector of 152 lengths, consisted of vertical and horizontal pixel positions of 76 facial feature points.
The temporal integrator g w TI ( x ( t ) ) is an LSTM, whose number of hidden units was the same as that of feature points.
The 1st-order SPRT-TANDEM was evaluated on two hypotheses, y = 1 : 2D face, and y = 0 : 3D face.
We assumed a ﬂat prior, p ( y = 1)
=
p ( y = 0) .
The validation and test data were 10% of the entire data randomly selected at the beginning of training, respectively.
We conducted a 10-fold cross-validation test to evaluate the effect of L LLR .
T e s t
EE R 0.
*(p=0.012) ***(p=9.56E-10) ***(p=9.56E-10)
We compared the classiﬁcation performance of the SPRT-TANDEM network using both L LLR and
L multiplet , and using L multiplet only.
We also compare L LLR + L multiplet and L KLIEP + L multiplet .
To use L KLIEP without making a loss diverge, we set the upper and lower bound of the numerator and denominator of ˆ r as 10 5 and 10 − 5 , respectively.
Out of 100 training epochs,
the results of the lastepochs were used to calculate test equal error rates (EERs).
Two-way ANOVA with factors “loss type” and “epoch” were conducted to see if the difference in loss function caused statistically signiﬁcantly different EERs.
We included the epoch as a factor in order to see if the value of EER reached a plateau in the last 80 epochs (i.e., statistically NOT signiﬁcant).
As we expected, EER values across training epochs were not signiﬁcantly different ( p = 0 . 17 ).
On the other hand, the loss type caused statistically signiﬁcant differences between the loss groups (i.e., L LLR + L multiplet , L KLIEP + L multiplet , and L multiplet .
p < 0 .
Following Tukey-Kramer multi-comparison test showed that training with
L LLR loss statistically signiﬁcantly reduced the EER, compared to both L KLIEP ( p = 9 .
56 ∗ 10 − 10 ) and the L LLR -ablated loss ( p = 9 .
56 ∗ 10 − 10 ).
The result is plotted in Figure 7.
F P ROBABILITY DENSITY RATIO ESTIMATION WITH THE LLLR
Below we test whether the proposed LLLR can help a neural network estimating the true probability density ratio.
Providing the ground-truth probability density ratio was difﬁcult in the three databases used in the main text, because it was prohibitive to ﬁnd the true probability distribution out of the public databases containing real-world scenes.
Thus, we create a toy-model estimating the probability density ratio of the two multivariate Gaussian distributions.
Experimental results show that a multilayer perceptron (MLP) trained with the proposed LLLR achieves smaller estimation error than an
MLP with crossentropy (CE)-loss.
F. E XPERIMENTAL SETTINGS
Following Sugiyama et al.
Sugiyama et al.
(2008), let p
( x ) be the d -dimensional Gaussian density with mean (2 , 0 , 0 , ..., 0) and covariance identity, and p 1 ( x ) be the d -dimensional Gaussian density with mean (0 , 2 , 0 , ..., 0) and covariance identity.
The task for the neural network is to estimate the density ratio:
[Equation]
Here, x is sampled from one of the two Gaussian distributions, p 0 or p 1 , and is associated with class label y
= 0 or y = 1 , respectively.
We compared the two loss functions, CE-loss and LLLR:
[Equation]
where σ is the sigmoid function.
A simple Neural network consists of 3-layer fully-connected network with nonlinear activation (ReLU) is used for estimating ˆ r ( x ) .
Evaluation metric is normalized mean squared error (NMSE, Sugiyama et al.
[Equation]
F. D ENSITY ESTIMATION RESULTS
To calculate statistics, the MLP was trained either with the LLLR or CE-loss, repeated 40 times with different random initial vairables.
Figure 8 shows the mean NMSE with the shading shows standard error of the mean.
Although the training with LLLR does not decrease NMSE well at the ﬁrst few thousands of trials, the NMSE reaches as low as 10 − 5 around 14000 iterations.
In contrast, the training with CE shows a steep decrease of NMSE in the ﬁrst 2000 iterations, but saturates after that.
Thus, the proposed LLLR not only facilitates the sequential binary hypothesis testing, but also facilitates the estimation of true density ratio.
G M ULTIPLET CROSS - ENTROPY LOSS
In this section, we show that estimation of the true posterior is realized by minimizing the multiplet cross-entropy loss deﬁned in Section 4 on the basis of the principle of maximum likelihood estimation.
First, let us consider the 1st order for simplicity.
The multiplet cross-entropy loss ensures for the posterior ˆ p ( y | x ( t ) ) estimated by the network to be close to the true posterior p ( y | x ( t ) ) .
Consider the Kullback-Leibler divergence of ˆ p ( y | x ( t ) ) and p ( y | x ( t ) ) for some x ( t ) ∈
R d x ( t ∈ N ), where y ∈ { 0 , 1 } :
[Equation]
Thus, the last line shows the smaller singlet loss leads to the smaller Kullback-Leibler divergence; in other words, we can estimate the true posterior density by minimizing the multiplet loss, which is necessary to run the SPRT algorithm.
Similarly, we adopt the doublet cross-entropy to estimate the true posterior p ( y | x ( t ) , x ( t +1) ) :
[Equation]
The crucial difference from the singlet loss is that the doublet loss involves the temporal correlation between x ( t ) and x ( t +1) , being necessary to implement the SPRT-TANDEM.
Similar statements hold for other orders.
H H YPERPARAMETER OPTIMIZATION
We used Optuna, the optimization software, to determine hyperparameters.
Hyperparameter search trials are followed by performance evaluation trials of the ﬁxed hyperparameter conﬁguration.
The evaluation criteria used by Optuna to ﬁnd the best parameter combination is balanced accuracy.
For models that produce multiple balanced accuracies / mean hitting time combinations, we use the average of balanced accuracy at every natural number of the mean hitting time (e.g., one frame, two frames).
[Equation]
SPRT-TANDEM: feature extractor.
ResNet version 1 with 110 layers and 128 ﬁnal output chan- nels (total trainable parameters: 6.9M) is used.
Hyperparameters are searched within the following space:
learning rate ∈ { 10 − 2 , 10 − 3 } optimizer ∈ { Adam , Momentum , RMSprop } weight decay ∈ { 10 − 3 , 10 − 4 , 10 − 5 } .
Where Adam, Momentum, and RMSprop are based on (Kingma & Ba (2014), Rumelhart et al.
(1986), and Graves (2013)), respectively.
Numbers of batch size and training epoch are ﬁxed to 64 and 50, respectively.
The best hyperparameter combination is summarized in Table 5.
[Table]
One search trial takes approximately 5 hours on our computing infrastructure (see Appendix K).
SPRT-TANDEM: temporal integrator.
Peephole-LSTM with a hidden layer of size 128 (total trainable parameters: 0.1M) is used.
Hyperparameters are searched within the following space:
learning rate ∈ { 10 − 2 , 10 − 3 , 10 − 4 , 10 − 5 } batch size ∈ { 256 , 512 , 1024 } optimizer ∈ { Adam , Momentum , Adagrad , RMSprop } weight decay ∈ { 10 − 3 , 10 − 4 , 10 − 5 } .
dropout ∈
Where Adagrad is based on (Duchi et al.
Number of training epochs is ﬁxed to 50.
The number of search trials and resulting best hyperparameter combination are summarized in Table 6.
[Table]
One search trial takes approximately 3 hours on our computing infrastructure (see Appendix K).
LSTM-m / LSTM-s. Peephole-LSTM with a hidden layer of size 128 (total trainable parameters: 0.1M) is used.
Hyperparameters are searched within the following space:
[Equation]
dropout
[Equation]
[Equation]
[Equation]
[Equation] [Equation] [Equation] [Equation]
[Equation]
100 } where the lambda is a speciﬁc parameter of LSTM-m / LSTM-s. Batch size and number of training epochs are ﬁxed to 1024 and 100, respectively.
The number of search trials and resulting best hyperparameter combination are summarized in Table 7.
[Table]
One search trial takes approximately 3 hours on our computing infrastructure (see Appendix K).
EARLIEST.
LSTM with a hidden layer of size 128 (total trainable parameters: 0.1M) is used.
Hyperparameters are searched within the following space:
[Equation]
Batch size and number of training epochs are ﬁxed to 1 5 and 2, respectively.
The number of search trials and resulting best hyperparameter combination are summarized in Table 8.
[Table]
One search trial takes approximately 48 hours on our computing infrastructure (see Appendix K) 3DResNet.
3DResNet with 101 layers and 128 ﬁnal output channels (total trainable parameters: 7.7M) is used.
Hyperparameters are searched within the following space:
[Equation]
Optimizer and number of training epochs are ﬁxed to Adam and 50, respectively.
The number of search trials and resulting best hyperparameter combination are summarized in Table 9.
[Table]
Ablation experiment.
Peephole-LSTM with a hidden layer of size 128 (total trainable parameters: 0.1M) is used.
Hyperparameters are searched within the following space: learning rate ∈ { 10 − 1 , 10 − 2 , 10 − 3 , 10 − 4 , 10 − 5 } optimizer ∈ { Adam , Momentum , Adagrad , RMSprop } weight decay ∈ { 10 − 3 , 10 − 4 , 10 − 5 } .
dropout ∈ { 0 , 0 .
Batch size and number of training epochs are ﬁxed to 1024 and 100, respectively.
The number of search trials and resulting best hyperparameter combination are summarized in Table 10.
[Table]
H. UCF
SPRT-TANDEM: feature extractor.
ResNet version 2 with 50 layers and 64 ﬁnal output channels (total trainable parameters: 26K) is used.
Hyperparameters are searched within the following space:
[Equation]
Numbers of batch size, optimizer, and training epochs are ﬁxed to 512, Adam, and 100, respectively.
The best hyperparameter combination is summarized in Table 11.
[Table]
SPRT-TANDEM: temporal integrator.
Peephole-LSTM with a hidden layer of size 64 (total trainable parameters: 33K) is used.
Hyperparameters are searched within the following space: learning rate ∈ { 10 − 4 , 10 − 5 , 10 − 6 , 10 − 7 } batch size ∈ { 57 , 114 , 171 , 342 } optimizer ∈ { Adam , RMSprop } dropout ∈ { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 }
Numbers of weight decay and training epochs are ﬁxed to 10 − 4 and 100, respectively.
The number of search trials and the best hyperparameter combination are summarized in Table 12.
[Table]
LSTM-m / LSTM-s. Peephole-LSTM with a hidden layer of size 64 (total trainable parameters: 33K) is used.
Hyperparameters are searched within the following space: learning rate ∈ { 10 − 4 , 10 − 5 , 10 − 6 , 10 − 7 } batch size ∈ { 57 , 114 , 171 , 342 } optimizer ∈ { Adam , Momentum , Adagrad , RMSprop } weight decay ∈ { 10 − 3 , 10 − 4 , 10 − 5 } .
dropout ∈ { 0 . 1 , 0 . 2 , 0 .
3 , 0 . 4 } lambda ∈ { 0 . 01 , 0 .
The number of training epochs is ﬁxed to 100.
The number of search trials and resulting best hyperparameter combination are summarized in Table 13.
[Table]
EARLIEST.
LSTM with a hidden layer of size 64 (total trainable parameters: 33K) is used.
Hyperparameters are searched within the following space: learning rate ∈ { 10 − 1 , 10 − 2 , 10 − 3 , 10 − 4 , 10 − 5 } optimizer ∈ { Adam , Momentum , Adagrad , RMSprop } weight decay ∈ { 10 − 3 , 10 − 4 , 10 − 5 } .
dropout ∈ { 0 , 0 .
Batch size and number of training epochs are ﬁxed to 1 and 30, respectively.
The number of search trials and resulting best hyperparameter combination are summarized in Table 14.
[Table]
[Figure]
3DResNet.
3DResNet with 50 layers and 64 ﬁnal output channels (total trainable parameters: 52K) is used.
Hyperparameters are searched within the following space:
[Equation]
Batch size, optimizer and number of training epochs are ﬁxed to 19, Adam, and 50, respectively.
The number of search trials and resulting best hyperparameter combination are summarized in Table 15.
[Table]
[Figure]
[Table]
[Figure]
H. S I W
The large database and network size prevent us to run multiple parameter search trials on SiW database.
Thus, we manually selected hyperparameters as follows.
SPRT-TANDEM: feature extractor.
ResNet version 2 with 152 layers and 512 ﬁnal output channels (total trainable parameters: 3.7M) is used.
Table 16 shows the ﬁxed parameter combination.
[Table]
SPRT-TANDEM: temporal integrator.
Peephole-LSTM with a hidden layer of size 512s (total trainable parameters: 2.1M) is used.
Table 17 shows the ﬁxed parameter combination.
[Table]
LSTM-m / LSTM-s. Peephole-LSTM with a hidden layer of size 512s (total trainable parameters: 2.1M) is used.
Table 18 shows the ﬁxed parameter combination.
[Table]
EARLIEST.
LSTM with a hidden layer of size 512s (total trainable parameters: 2.1M) is used.
Table 19 shows the ﬁxed parameter combination.
[Table]
3DResNet.
3DResNet with 101 layers and 512 ﬁnal output channels (total trainable parameters: 5.3M) is used.
Table 20 shows the ﬁxed parameter combination.
[Figure]
I S TATISTICAL TEST DETAILS
The models we compared in the experiment have various numbers of trials due to the difference in training time; some models were prohibitively expensive for multiple runs (for example, 3DResNet takes 20 hrs/epoch on SiW database with NVIDIA RTX2080Ti.)
In order to have an objective comparison of these models, we conducted statistical tests, Two-way ANOVA 6 followed by TukeyKramer multi-comparison test.
In the tests, small numbers of trials lead to reduced test statistics, making it difﬁcult to claim signiﬁcance, because the test statistic of Tukey-Kramer method is proportional to 1 / (cid:112) (1 /n + 1 /m ) , where n and m are trial numbers of two models to be compared.
Nevertheless, the SPRT-TANDEM is statistically signiﬁcantly better than other baselines.
One intuitive interpretation of this result is that “the SPRT-TANDEM achieved accuracy high enough so that only a few trials of baselines were needed to claim the signiﬁcance.”
These statistical tests are standard practice in some research ﬁelds such as biological science, in which variable trial numbers are inevitable in experiments.
All the statistical tests are executed with a customized MATLAB (2017) script.
Here, the two factors for ANOVA are (1) the model factor contains four members: the SPRT-TANDEM with the best performing order on the given database, LSTM-m, EARLIEST, and 3DResNet, and (2) the phase factor contains two or three members: early phase and late phase (NMNIST, UCF), or early, mid, and the late phase (SiW).
The early, mid, and late phases are deﬁned based on the number of frames used for classiﬁcation.
The actual number of frames is chosen so that the compared models can use as similar data samples as possible and thus depends on the database.
The SPRT-TANDEM, LSTM-m, and 3DResNet can be compared with the same number of samples used.
However, EARLIEST cannot ﬂexibly change the average number of samples (i.e., mean hitting time); thus, we include the results of EARLIEST to groups with the closest number of data possible.
For NMNIST, ﬁve frames and ten frames are used to calculate the statistics of the early and late phases, respectively, except EARLIEST uses 4.37 and 19.66 frames on average in each phase.
For UCF,frames and 25 frames are used to calculate the statistics of the early and late phases, respectively, except EARLIEST uses 2.01 and 2.09 frames on average in each phase 7 .
For SiW, 5, 15, andframes are used to calculate the early, mid, and late phases, respectively, except EARLIEST uses 1.19, 8.
[Table]
and 32.06 frames.
The p-values are summarized in the Tables 21-24.
P-values with asterisks are statistically signiﬁcant: one, two and three asterisks show p < 0 .
05 , p < 0 .
01 , and p < 0 .
001 , respectively.
[Table]
[Table] [Table] [Table]
[Table]
J D ETAILS OF THE EXPERIMENTS IN S ECTION
Here we present the details of the experiments in Section 5.
Figure 9 shows the SAT curves of all the models we use in the experiment.
Figure 10, 11, and 12 show example LLR trajectories calculated using NMNIST, UCF, and SiW database, respectively.
Tables 25 to 40 shows average balanced accuracy and standard error of the mean (SEM) at the corresponding number of frames that used for classiﬁcation.
[Figure]
[Figure]
[Figure] [Figure]
[Table] [Table]
[Table] [Table]
[Table] [Table]
1.0399e96. 1.2424e96. 1.1033e96.
[Table] [Table]
1.0603e96.
1.0399e96. 1.2424e96.
1.1033e96.
8.3202e-00496. 1.0603e96.
[Table] [Table]
[Table] [Table]
[Table] [Table]
1033e-
[Table] [Table]
{+8.3202e-
+}
[Table] [Table]
0603e-
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
1033e-
[Table] [Table]
[Table] [Table]
1.0603e-
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
1033e-
[Table] [Table]
1.0603e96.
1.0399e-
[Table] [Table]
[Table] [Table]
[Table]
1033e96. 8.3202e-00496. 1.0603e96.
[Table]
1.0399e96. 1.2424e96.
[Table]
1.1033e96.
8.3202e-
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
K C OMPUTING INFRASTRUCTURE
All the experiments are conducted with custom python scripts running on NVIDIA GeForce RTX 2080 Ti, GTX 1080 Ti, or GTX 1080 graphics card.
Numpy (Harris et al.
and Scipy (Virtanen et al.
(2020)) are used for mathematical computations.
We use Tensorﬂow 2.0.0 (Abadi et al.
as a machine learning framework except when running baseline algorithms that are implemented with
PyTorch (Paszke et al.
L A N EXAMPLE VIDEO OF THE N OSAIC MNIST DATABASE .
As we described in Section 5, the Nosaic (Noise + mOSAIC) MNIST, NMNIST for short, contains videos with 20 frames of MNIST handwritten digits, buried with noise at the ﬁrst frame, gradually denoised toward the last frame.
The ﬁrst frame has all 255-valued pixels (white) except onlymasks of pixels that are randomly selected to reveal the original image.
Another forty pixels are randomly selected at each of the next timestamps, ﬁnally revealing the original image at the last, 20th frame.
An example video is shown in Figure 13.
M S UPPLEMENTARY EXPERIMENT ON M OVING MNIST DATABASE
Prior to the experiment on Nosaic MNIST, we conducted a preliminary experiment on the Moving
MNIST (MMNIST) database.
1st, 2nd, 3rd, and 5th-order SPRT-TANDEM were compared to the LSTM-m. Hyperparameters of each model were independently optimized with Optuna.
The result plotted in Figure 14 showed that the balanced accuracy of the SPRT-TANDEM peaked and reached the plateau phase only after two or three frames.
This indicated that each of the frames in
MMNIST contained too much information so that a well-trained classiﬁer could classify a video easily.
Thus, although our SPRT-TANDEM outperformed LSTM-m with a large margin, we decided to design the original database, Nosaic MNNIST (NMNIST) for the early-classiﬁcation task.
NMNIST contains videos with noise-buried handwritten digits, gradually denoised towards the end of the videos, increasing mean hitting time compared to the MMNIST.
N S UPPLEMENTARY ABLATION EXPERIMENT
In addition to the ablation experiment presented in Figure 3e, which is calculated with 1st-order
SPRT-TANDEM, we also conduct an experiment with 19th-order SPRT-TANDEM.
The result shown in Figure 15 is qualitatively in line with Figure 3e: the L multiplet has an advantage at the early phase with a few data samples, while the L LLR leads to the higher ﬁnal balanced accuracy at the late phase, and using both loss functions the best SAT curves can be obtained.