AutoMS: Automatic Model Selection for Novelty Detection with Error Rate Control
Abstract
Given an unsupervised novelty detection task on a new dataset, how can we automatically select a “best” detection model while simultaneously controlling the error rate of the best model?
For novelty detection analysis, numerous detectors have been proposed to detect outliers on a new unseen dataset based on a score function trained on available clean data.
However, due to the absence of labeled data for model evaluation and comparison, there is a lack of systematic approaches that are able to select the “best” model/detector (i.e., the algorithm as well as its hyperparameters) and achieve certain error rate control simultaneously.
In this paper, we introduce a unified data-driven procedure to address this issue.
The key idea is to maximize the number of detected outliers while controlling the false discovery rate (FDR) with the help of Jackknife prediction.
We establish non-asymptotic bounds for the false discovery proportions and show that the proposed procedure yields valid FDR control under some mild conditions.
Numerical experiments on both synthetic and real data validate the theoretical results and demonstrate the effectiveness of our proposed AutoMS method.
The code is available at: https://github.com/ZhangYifan1996/AutoMS .
Introduction
With the development of big data, novelty detection has become critical for many machine learning applications, such as network intrusion detection [1], fraud detection in finance [2, 3], event detection in the earth sciences [4], and so on.
For these applications, there are a very large number of “normal” instances (i.e., inlier) but a lack of “abnormal” data (i.e., novelty, or outlier), because collecting abnormal data is much more expensive or difficult than collecting normal data [5].
This leads us to a situation where the available historical dataset contains only normal observations, and the basic goal is to detect outliers on a new unseen test dataset.
Novelty detection has achieved remarkable success in many application domains with numerous methods/algorithms being developed in machine learning literature [6–8], however, two problems need to be addressed in the current novelty detection models, (1) lack of error rate control in detection algorithms, and (2) ad-hoc model selection of algorithms/hyperparameters.
The most common way to solve the novelty detection problem is to use a detection algorithm to learn a plausible score function that assigns a score to each point in the test dataset.
Generally, data with large scores are more likely to be detected as outliers.
A decision threshold is given to determine whether a data point is an outlier or not.
There is an obvious trade-off between precision and recall: a small threshold would detect more outliers but make more false discoveries as well.
Therefore, a more viable way is to reformulate the novelty detection problem as a multiple hypothesis testing problem, and adopt some error rate control procedure as suitable criteria to choose the threshold [9].
Model selection (including algorithm selection or hyperparameter tuning) is a well-known feature of these novelty detection algorithms and arises naturally since none of these methods can universally outperform others on all tasks as noted in Wolpert and Macready [10], Zhao et al.
Due to the special setting of novelty detection, model evaluation and model comparison are not feasible when only “clean” data are provided as training dataset.
Those strategies are usually ad-hoc [11, 12], which may greatly hamper their applicability to novelty detection in general cases.
Furthermore, these methods only target the recall through model selection without considering the false discoveries.
Therefore, it is desirable to develop a unified framework to select the “best” detector with certain error rate control.
We see that designing a data-driven procedure to select the “best” model/detector while simultaneously achieving some error rate control remains an open challenge.
In this paper, we propose a datadriven procedure, AutoMS, to implement automatic model selection for novelty detection with error rate control.
We apply the Jackknife technique [13] to fully explore the clean data and avoid the randomness caused by data-splitting.
With the help of Jackknife prediction, we select from a pool of diversified models/detectors the “best” detector that maximizes the number of detected outliers while controlling the false discovery rate.
The theoretical guarantees for the FDR control of the selected “best” detector requires further investigation because the error distribution of the “best” detector is different from that of a given detector.
We address this issue by carefully studying the bound of false discovery proportion (FDP) and FDR which is calculated by taking expectation of FDP.
To our best knowledge, this is the first work to systematically select the “best” model while achieving certain error rate control.
The main contributions of this work are summarized as follows.
(1) The proposed AutoMS can select the best model and simultaneously control the error rate of the best model.
Notably, AutoMS is a unified data-driven framework, and it does not rely on any labeled anomalous data for model selection.
(2) To our best knowledge, it is the first effort to select the “best” model/detector with theoretical guarantees in the view of FDR control.
We establish non-asymptotic bounds for the FDP and show that the proposed AutoMS yields valid FDR control.
The AutoMS can be easily coupled with commonly used novelty detection algorithms.
Extensive numerical experiments indicate that AutoMS outperforms other methods significantly, with respect both error rate control and detection power.
Related Work
FDR Control.
Reformulating the outlier detection problem as a multiple testing problem, a commonly-used error rate in this setting is the false discovery rate (FDR): the expected proportion of true inliers among the test points reported as outliers.
The FDR control [14–16] has been fully studied in the statistical literature and is a particularly useful tool to maintain the ability to reliably detect true alternatives without excessive false discovery results when large-scale hypotheses are simultaneously tested.
Along this line, Bates et al.
[17] pioneered a method to split the clean dataset into two independent parts, one for training the score function and one for validating the empirical cumulative distribution function of the score function which will be used for constructing p-values for the new observations.
They showed that the FDR control can be achieved with theoretical finite sample guarantees.
It involves, however, a one-time random split of the clean dataset, which leads to a “p-value lottery” problem.
That is, those p-values based on sample-splitting are sensitive and difficult to be reproduced in practice.
Moreover, the procedure in Bates et al.
[17] can only control the FDR for a given novelty detection model, and no model efficiency is considered in their paper.
Model Selection in Novelty Detection.
As described in Sect. 1, various detection algorithms have been developed for different problems or different types of data, such as LOF [18], one-class
SVM [19], iForest [20] and so on.
More discussions can be found in Pimentel et al.
[6], Aggarwal [21].
However, no universal learning model performs well on all problem instances Wolpert and
Macready [10].
Therefore, it raises the problem of model selection, including algorithm selection and hyperparameter tuning.
For example, AutoOD [12] proposed an automated outlier detection framework focusing on automatic neural architecture search.
Recent work METAOD [11] built a measure for task similarity and use meta-learning to select the most similar task compared to the historical ones.
Most existing work is usually based on sophisticated black-box machine learning algorithms to learn the space of the normal data.
While generally effective in practice, they rarely offer a clear guarantee about the output.
We note that some statistical criteria can be used for model selection in novelty detection; e.g. using mass-volume and excess-mass [22] or some clustering validation metrics [23] to measure the quality of a score function, and using IREOS [24] to assess the separability.
However, Ma et al.
[25] indicated that those criteria are not useful in practice and cannot lead to a universal detector for general cases.
Our proposed AutoMS is not limited to one specific algorithm or model but can select the “best” one among any set of models/detectors.
Methodology: Automatic Model Selection
In a novelty detection problem, we are given a clean training dataset of inliers D = { ˜ Z j } mj = containing m independently and identically distributed (
i.i.d.) observations ˜
Z j ∼ P 0 .
The goal of novelty detection problem is to identify the outlier set
O =
{ Z i ∈
U :
Z i is not drawn from P 0 } among a new unseen test dataset U =
{ Z i } ni
=1 .
One of the most common approaches in addressing novelty detection is to employ a detection algorithm M to learn a plausible score function S M ( · ) with the clean dataset D and use the score
S M ( Z i ) to assess whether Z
i ∈
U is an outlier.
Generally, it is assumed that the larger the score
S M ( Z i ) , the more likely Z i is an outlier.
Hence, the decision set (detected outlier set) is selected as: (cid:98) O M =
{ Z i ∈
U : S M ( Z i ) ≥ L M } , where L M is a threshold associated with detector M .
Denote the subset of U containing all inliers as O c =
{ Z i ∈
U :
Z i is drawn from P 0 } with size n 0 , and
O (cid:83) O c
= U .
Let |A| denote the number of the elements in set A .
Denote a ∨ b = max { a, b } .
Taking a multiple testing perspective, the null hypothesis is that a new observation is an inlier, and the alternative asserts it is an outlier.
That is, the novelty detection problem is translated into a multiple testing problem as follows,
[Equation]
3.1 Problem Definition
Problem (Model Selection of Unsupervised Novelty Detection)
The model selection problem for unsupervised novelty detection problem can be stated as follows.
Given a training dataset of inliers D = { ˜ Z j } mj =1 , a new unlabeled dataset U =
{ Z i } ni =1 which may contain outliers, and all models G =
(cid:83) M ;
Select the best model M ∗ ∈ G , such that the FDR of M ∗ is controlled.
The false discovery proportion (FDP) and true discovery proportion (TDP) are defined as follows
[Equation]
The false discovery rate (FDR) is the expectation of FDP , i.e., FDR =
E (FDP) .
The true detection rate (TDR) is defined as the expectation of TDP , i.e., TDR = E (TDP) .
A detector is superior to another if the former has a larger TDR for the same FDR level.
3.2 Construction of AutoMS
We begin with the FDR control for one given detector M and then give the AutoMS of how to select the “best” detector from G .
For one detector M , we learn a score function S M ( · ) based on the historical clean dataset D and evaluate the score S M ( Z i ) at the new data point Z i ∈
U .
The canonical approach to FDR control in multiple testing problem is to apply the BenjaminiHochberg (BH) procedure [14–16] to the p-values of scores S M ( Z i ) for any Z
i ∈
U .
That is, the data-dependent threshold L is selected as follows,
[Equation]
where G M ( t ) =
P H
( S M ( Z i ) ≥ t |D ) is the p-value of S M ( Z i ) for Z i ∈ U when H 0 holds.
However, it is quite a challenge to estimate reliable p-values G M ( t ) since P 0 is unknown and the score function S M ( · ) may be derived from a sophisticated black-box machine learning algorithm.
Bates et al.
[17] suggested randomly splitting D into two disjoint parts with equal sample size, one for training the score function S M ( · ) and the other for estimating G M ( t ) with the help of conformal inference.
Whereas, this single-random-splitting based method suffers from, (1) the estimation of the p-values depends on the random split; (2) different split ratios also lead to different results.
To yield more stable estimates of p-values in practice, we propose to learn the score function S M ( · ) on the whole clean dataset D , where our key idea stems from the Jackknife estimate .
Denote
D [ − j ] =
D\{ ˜ Z j } as the subset of D with the j -th observation removed.
Define S [ − j ] M ( · ) as the score function trained on D [ − j ] .
Then we can evaluate m scores { S [ − j ] M ( ˜ Z j ) } mj =1 on the whole clean dataset
D .
The p-value function can be estimated by
[Equation]
The Jackknife technique provides an approximation of the score, S [ − j ] M ( ˜ Z j ) , for any ˜ Z j ∈ D .
Under the null H 0 that Z i ∈ U is from P 0 , S [ − j ] M ( ˜ Z j ) can be used for estimating the empirical distribution of S M ( Z i ) .
And the corresponding threshold L M is determined via
[Equation]
where ¯ t m, M is the upper bound for t and its choice will be discussed in Sect.
4 in detail, and S M ( · ) is the same score function trained on D .
Then, the detected set is identified as
[Equation]
Further, we select the “best” detector M ∗ via
[Equation] [Equation]
We refer to our procedure as AutoMS short for Automatic Model Selection and it is summarized in Algorithm 1.
In fact, our AutoMS consists of two phases: offline training in lines 2-7 and online prediction in lines 8-9 of pseudo-code in Algorithm 1.
Note that the running time of offline training is not critical.
Meanwhile, the additional running time is the detector selection in line 11, which should incur small run-time overhead due to only maximizing a finite set.
Note that the pool of detectors G =
(cid:83) M can be a set of different algorithms or a set of hyperparameters for one typical algorithm, or both, which fits the definition of model selection including algorithm selection and hyperparameter tuning.
Note that the size of G , |G| , can be infinite, and |G| is allowed to increase as the size of D .
More details will be discussed in Sect.
Rationale.
Here is the intuition why the selected detector obtained by AutoMS is the “best” one.
The general idea is as follows.
A good detector needs to achieve a large TDR while controlling the false discoveries under the target FDR level α .
With FDR control , the selected detector which has the largest number of discoveries is roughly the one with the largest TDR.
Otherwise, simply selecting the largest number of discoveries without FDR control , is not even a correct criterion, because in this case, most detected discoveries may be inliers and true outliers may not be detected.
For every detector M ∈ G , with FDR control, its FDP can be approximately controlled around the target FDR level α , as the proposed AutoMS resets the corresponding novelty threshold for every M .
Therefore, there are roughly | (cid:98) O M | (1 − α ) true outliers in the detected set (cid:98) O M , with the TDR being approximated by | (cid:98) O M | (1 − α ) |O| .
Since the number of true outliers |O| and the target FDR level α are fixed in given novelty detection task, selecting the largest | (cid:98) O M∗ | (1 − α ) |O| , namely largest | (cid:98) O M ∗ | , leads to the “best” detector M ∗ .
Why Jackknife not Jackknife+.
Barber et al.
[26] introduced the Jackknife+ as a new method for constructing predictive confidence intervals.
They provide non-asymptotic coverage guarantees under no assumptions beyond the training and test dataset being exchangeable and also study the theoretical guarantees for the original Jackknife with stability assumptions (see Section 5 of their paper).
Jackknife+ needs to calculate m leave-one-out predictions for one test point while the original
Jackknife only calculates one predicted value [26].
The model evaluation cost (counts the number of times evaluating a fitted model on a new point) of Jackknife+ is mn whereas the original Jackknife is of cost n , indicating that Jackknife+ has a much higher computation complexity compared to the original.
Thus, since most novelty detection algorithms satisfy the stability assumption, we are more inclined to apply the original Jackknife technique rather than Jackknife+ to detectors that meet stability assumption, and we also give theoretical guarantees to achieve the non-asymptotic bounds of FDP and maintain FDR control.
Statistical Performance Guarantees
In this section, we will theoretically establish NON-asymptotic bounds for FDP of the selected detector M ∗ by AutoMS and further show that it can control the FDR under the following mild assumptions.
Define the number of detectors ϖ m = |G| and let A n = o ( n ) be a pre-specified sequence.
Assumption 1 (Density) .
For any detector M ∈ G and sufficiently large m , the conditional density of S M ( Z ) given training dataset D is uniformly bounded above by a universal constant c f > 0 , where Z ∈ U is a new point.
Assumption 2 (Learning stability) .
For any fixed M , the score S M satisfies: for large enough m ,
[Equation]
with some sequences satisfying κ m = o (1) and ζ m =
o (1) as m → ∞ , and some function ˜ µ M .
Assumption 3 (Rates) .
Denote B m = 2 c f κ m + 3 ζ m .
Suppose that B m =
o (1) , nϖ m B m =
o (1) and mϖ m /A 2 / 3 n = o (1) .
Our main theoretical result on the validity of the AutoMS method for both FDP and FDR control is given by the next theorem.
Theorem 1 (FDR control) .
Suppose Assumptions 1–3 hold.
Let 0 < δ < 1 and 0 < α < 1 .
There exist universal constants C 1 > 0 and C 2 > 0 so that the FDP of the proposed method satisfies
[Equation]
with probability at least 1 − 2 δ , and
[Equation]
where W mn =
[Equation]
It is worth mentioning that even if all detectors control the FDR, we cannot assert without theoretical proof that the optimal detector M ∗ can still guarantee the FDR control.
And Bates et al.
[17] only gives the expectation version that FDR can be controlled for a given detector.
Briefly, we first prove that the FDP of each detector can be bounded uniformly through the uniform convergence of the
Jackknife estimator.
Hence, the selected “best” detector M ∗ with the largest number of discoveries will have the same non-asymptotic bounds for FDP.
Then we give the asymptotic FDR control for the optimal detector.
Detailed proofs for theoretical results are provided in Appendix E .
In Eq.
(4), we specify a data-driven upper bound ¯ t m, M when determining L M .
We define ¯ t m, M =
G − m, M ( A n /n ) where G m, M ( t ) is defined in Eq.
(3) and G − m, M ( y ) =
inf { t ≥ 0 : G m, M ( t ) ≤ y } with 0 ≤ y ≤ 1 .
The upper bound ¯ t m, M for threshold L M can facilitate the derivation of nonasymptotic bounds for FDP but is actually not required in practice.
In other words, this makes an implicit assumption that a non-trivial (not ∞ ) threshold can be attained within the interval (0 , ¯ t m, M ] .
By Theorem 1, we can see that a large A n would yield faster convergence of FDP.
However, too large
A n may lead to obtaining a trivial threshold.
In fact, how large A n could be depended on the number of identifiable outliers in U [27], which is certainly unknown to us.
Alternatively, by examining the requirements in Assumption 3, we can set A n ∼ min { ( mϖ m ) 2 , n 1 / 2 } for simplicity.
Remark 1 (Stability) .
Since we use the Jackknife to estimate FDP, certain stability conditions seem to be necessary.
In Assumption 2, ˜ µ needs not to be close to an optimal population detector.
We only need the S M to concentrate around ˜ µ .
This is just a stability assumption rather than consistency assumption.
When the sampling stability fails to hold, the S [ − j ] j, M may have a substantially different distribution than G n, M ( t ) .
In that case, we may resort to the sample-splitting procedure instead.
Remark 2 (Rates) .
In general, Assumption 3 is used to ensure the uniform consistency of the estimated p-values, which performs an important role in the convergence of the FDR control.
Specifically,
A n = o ( n ) is a pre-specified sequence which gives a non-trivial lower bound of the true p-value function G M ( t ) with G M ( t ) ≥
A n n = o (1) .
The B m is a quantity used to measure the difference magnitude of two p-values estimated in different ways, as in Lemma C.3 and Lemma C.2.
And
B m =
o (1) indicates two p-values should be close enough which is a very mild condition, and it can be expected that the p-values will be close if the training sample sizes of inliers m and test sample sizes of inliers n 0 are large enough in practice.
In the proposed AutoMS, the number of detectors ϖ m = |G| can go to infinity with some certain rate related to the training sample size m and test sample size n .
The assumptions nϖ m B m =
o (1) and mϖ m /A 2 / 3 n = o (1) indicate the required rate of ϖ m to ensure the uniform consistency among detectors.
Remark 3 (FDP bound) .
Although the FDP bound might be slightly larger than the target FDR level α in finite samples, the FDP bound converges asymptotically to the target
FDR level α .
Actually, in novelty detection tasks, it is not necessary to obtain procedures of finite FDR control with additional assumptions, instead, a powerful and stable procedure which controls FDR asymptotically is good enough for most cases.
Experiments and Evaluation 5.1 Experiment Setting
In this section, we evaluate the performance of the proposed AutoMS in terms of FDR control and detection power (TDP) through extensive numerical studies on both synthetic and real data.
We report the results based on 100 replication experiments throughout this paper.
We construct a detector set
G with 83 candidate models by combining 6 novelty detection algorithms and their corresponding hyperparameters (See Table S1 in Appendix A for the complete list).
These models/detectors are available in the Python “ PyOD ” [28] package.
Note that the size of detector set G can be infinite and our AutoMS can still be used.
Performance Measures.
Since we focus on both error rate control and detection power, we use
FDR =
E (FDP) and TDR = E (TDP) as performance measures for each model in this paper.
According to the definitions in Eq.
(2) of Sect.
3, we have FDP = 1 − Precision , and TDP = Recall .
Correspondingly, the empirical FDR and empirical TDR are calculated by averaging of FDPs and
TDPs over 100 replications, respectively.
We say a detection procedure is superior to its competitor if it has a larger TDR for the same target FDR level α , that is, a superior detector has a larger empirical
TDR whose empirical FDR is below or around the target FDR level α .
The preset target FDR level α should be an acceptable level, which means all detectors that control the FDR at this level α should be accepted.
As long as the FDR of the selected detector is below a preset level α , we have achieved our goal on FDR control and we should focus on improving TDR.
[Table]
Baseline methods.
To evaluate the performance of the proposed AutoMS method, we compare the proposed
AutoMS with two different types of baselines: (i) model selection methods (METAOD [11]) and (ii) errorrate-controlled methods (SRS-based methods [17]).
And we also consider two versions of the proposed AutoMS:
AutoMS-JK and AutoMS-SRS , where AutoMS-JK uses Jackknife technique and AutoMS-SRS replaces the Jackknife technique with single-random-splitting (SRS) technique for estimating score.
We explore the performances of the four methods with respect to FDR and TDR.
It is worth mentioning that SRS is guaranteed with FDR control for any given detector, but there is no model selection considered in SRS-based methods.
METAOD is a model selection method using meta-learning without considering error rate control, while the AutoMS can do both model selection and error rate control, as shown in Table 1.
For more details on the baseline methods, Refer to Appendix B .
5.2 Evaluation of Jackknife
[Figure]
Firstly, We would like to show the advantage of Jackknife prediction with a toy example using synthetic data.
We generate a training dataset D and a test dataset U from a Gaussian mixture model, where their sample sizes are m = n = 2000 respectively and U contains 10% outliers.
More details of the model setting can be found in Sect.
In Fig.
1, we compare the Jackknife prediction to the single-random-splitting (SRS) method in Bates et al.
[17] with different split ratios when the detector
M is the OCSVM with its default hyperparameter in PyOD package [28].
Focusing on one training set D , we randomly split D 100 times for each split ratio of SRS method while Jackknife technique only has one prediction result.
We can see from Fig.
1 that the Jackknife provides a quite tight FDP control and achieves a high TDP while finding more true outliers.
As discussed above, the SRS-based methods depend on the random split and the split ratio and are usually unstable due to the randomness of the segmentation, making it hard to reproduce.
The Jackknife estimate helps improve the stability and the accuracy of the scores due to the usage of more samples to train the score function.
Meanwhile, the Jackknife procedure will require a bit more computation compared to one-time data splitting, if m is not too small.
As the Jackknife procedure for S [ − j ] M ( · ) is trained offline, one can always replace Jackknife with cross-validation to reduce the computing burden to a certain extent.
Instead of omitting one observation, we can randomly partition
D into k equal-sized folds and remove one fold from D as the subsample to train the score function.
In the following of this section, we use k = 100 folds in the AutoMS-JK procedure.
5.3 Synthetic Data
Data Description.
We consider the same synthetic setting as in Bates et al.
We will compare the performance of different methods (the proposed AutoMS-JK, AutoMS-SRS, SRS-based methods and METAOD) when the target FDR level is set to α = 0 .
Then we will show the results under different target FDR levels.
The data are generated by sampling each data point Z i ∈ R 50 from a multivariate Gaussian mixture model, Z i =
W i + √ aV i , where W i is one of 50 cluster center sampled independently from the uniform distribution on [ − 3 , 3] 50 and V i ∈ R 50 are i.i.d from N (0 , 1) .
The difference between inliers and outliers is the constant a with H 0 : a = 1 v.s. H 1 : a = 2 .
We generate a clean dataset D with m = 2000 observations and a test dataset U with n = 2000 observations in which 90% are sampled as inliers.
We aim to identify the remaining 10% of outliers in U .
[Figure]
Results.
With the given target FDR level α = 0 . 1 , the AutoMS (either AutoMS-JK or AutoMS-SRS) outperforms all baseline methods as shown in Fig.
(a) All the SRS-based methods can perfectly control the FDR with FDPs scattering around the given target FDR level α .
But in terms of detection power, the TDP performance of SRS-based methods vary greatly with different hyperparameters and different algorithms.
Some SRS-based methods, such as SRS-LODA, SRS-HOBOS, SRS-iForest, even lose detection power with almost zero TDP/recall.
This necessitates model selection among numerous novelty detection methods.
(b) On the other hand, METAOD gives large FDPs, which means that most of the reported points are not true outliers.
This phenomenon can be explained as
METAOD does not employ any FDR control procedure.
And the TDPs of METAOD do not perform well because the model selection of METAOD is based on historical data which may not be suitable for the data we use.
(c) AutoMS-JK and AutoMS-SRS are superior to all SRS-based methods and
METAOD with respect to the average TDR, with the average FDR slightly above the target FDR level α , as it is more difficult to control the FDR of the selected detector than given detector.
Additionally,
AutoMS-JK gives a higher TDR than AutoMS-SRS, which indicates that using the Jackknife method instead of SRS enhances detection power.
Although both AutoMS and SRS-based methods can control the FDR, AutoMS has better detection power with a larger TDR compared to SRS-based methods.
(d) The reason why the FDR control of AutoMS is more difficult than SRS-based methods is that the FDP distribution of our selected detector is different from a given detector.
Take a simple example: there are 10 standard Gaussian variables, each with a mean 0.
The largest variable among those 10 does not have a mean 0.
Therefore, the selected detector is more difficult to control FDR.
Fig.
3 shows the performance of AutoMS-JK, AutoMS-SRS and SRS-based-best methods under different target FDR levels.
In particular, SRS-LOF-best is the best SRS-LOF detector with the largest
TDR among all its candidate hyperparameters.
Although all SRS-based-best methods can control the
FDR under any target FDR level, the average TDPs of AutoMS-JK/AutoMS-SRS are higher than all the SRS-based-best methods at every target FDR level.
The average FDP of AutoMS-JK/AutoMSSRS is slightly larger than the given target FDR level α , but their difference FDP − α does not
[Figure]
increase as α becomes larger.
The FDPs of AutoMS-JK has a smaller deviation above the target α than AutoMS-SRS.
Actually, in novelty detection tasks, it is not necessary to obtain procedures with finite FDR control, instead, a powerful and stable procedure that controls FDR asymptotically around the target FDR level α is good enough for most cases.
That is what AutoMS does.
AutoMS can uniformly offer much higher TDPs than SRS-based-best methods under different target FDR levels.
5.4 Real Data
We compare the proposed AutoMS with two different types of baselines: (i) model selection methods (METAOD [11]) and (ii) error-rate-controlled methods (SRS-based methods [17]), as in Table 1.
Datasets.
Four real-world datasets for outlier detection are considered, and the details of the datasets are summarized in Table 2.
We use empirical TDR (Recall) and empirical FDR (1-Precision) as performance measures.
For real data implementation, we construct the training dataset D by randomly sampling m = min { 20000 , N 0 / 3 } inlier samples from the total N 0 inliers.
Then we focus on a test dataset generating mechanism under which 90% of the n = min { 2000 , ( N 0 + N 1 ) / 6 } observations in each test dataset U are sampled from N 0 inliers, and we seek to identify the remaining 10% of outliers (i.e., 10% outlier ratio).
For results on 11 more real-world datasets [29] as used in Zhao et al.
[11], refer to Appendix D .
[Table]
Results.
Fig.
4 shows the performance of different methods on real-world datasets, and Tablegives the empirical FDR and empirical TDR of different methods, where SRS-best is the best SRS detector with the largest TDR among all detectors.
From Fig.
4 and Table 3, we can observe that
AutoMS can maintain the FDPs well around the target FDR level α = 0 .
1 , while METAOD has inflated the empirical FDP to values around or above 0 .
5 on all real-world datasets, indicating that in the detected set (cid:98) O obtained by METAOD, around 50% of the detected outliers are not true outliers.
It is worth mentioning that, all SRS-based methods can maintain the FDR level around 0 .
1 as well as expected from their theoretical guarantees.
As for true detection rate (TDR), AutoMS has reasonable empirical TDR on all datasets, while SRS-based methods sometimes perform too conservatively, have no detected outliers and their performance varies greatly from dataset to dataset.
In addition, the performances of SRS-LODA-best and SRS-LOF-best are rather different, particularly on the “Covertype” dataset and “Credit Card” dataset, which means SRS-based-best methods can not give a consistent good result on different tasks and its performance can be severely affected by the base detector the SRS used.
On the other hand, AutoMS keeps accurate and stable FDR levels on all datasets and has reasonable TDR for different tasks.
Overall, AutoMS can uniformly offer much
AutoMS−JK
AutoMS−SRS
SRS−LODA−best
SRS−LOF−best
METAOD 0.
Covertype Credit Card Satellite Shuttle
F D P ( 1 − P rec i s i o n ) 0.
Covertype Credit Card Satellite Shuttle
T D P ( R ec a ll ) higher TDRs than SRS-based-best methods across all settings and AutoMS can maintain the FDR level with a comparable TDR compared with METAOD.
Table 4 shows results under different target
FDR α levels (refer to Appendix C for more results and details).
[Table] [Table]
Conclusion
The detector selection problem in novelty detection analysis is a challenging problem and it has not been thoroughly investigated in the literature, especially in terms of theoretical guarantees.
In this paper, we propose an effective data-driven procedure, named AutoMS
[Table]
that can select the “best” detector which offers high TDR and controls FDR simultaneously.
The AutoMS is fully automatic without any supervision and is easily implemented.
We give theoretical results showing that the proposed AutoMS can achieve the non-asymptotic bounds of FDP and maintain FDR control under mild conditions.
Extensive experiments on synthetic and real data show AutoMS significantly improves the detection power and yields accurate FDR control.
Acknowledgments and Funding Disclosure
We thank anonymous area chair and reviewers for their helpful comments.
Changliang Zou’s research was supported by the National Natural Science Foundation of China Grants (Nos. 11925106, 12231011, 11931001 and 11971247).
Haojie Ren’s research was supported by the National Natural
Science Foundation of China Grants (No.12101398) and the Shanghai Sailing Program.
This work was partially supported by the Technology Innovation Institute and MBZUAI joint project (NO.
TII/ARRC/2073/2021): Energy-based Probing for Spiking Neural Networks.
A AutoMS Detector Set
We give an example of the detector set G .
We use 6 well-known novelty detection methods: HistogramBased Outlier Score (HBOS), isolation Forest (iForest), K Nearest Neighbors (KNN), Lightweight
On-line Detector of Anomalies (LODA), Local Outlier Factor (LOF) and One-Class Support Vector
Machine (OCSVM), combined with their corresponding hyperparameters as a set of detectors G with 83 candidate models in our experiments.
Those detectors are available in the Python “ PyOD ” [28] package.
A complete list of the detector set G is shown in Table S1.
Note that the size of detector set G can be infinite, and our AutoMS procedure can still be used.
[Table]
B Experiment Setting with Details
To evaluate the performance of the proposed AutoMS method, we compare the proposed AutoMS with two different types of baselines: (i) model selection methods (METAOD [11]) and (ii) error-ratecontrolled methods (SRS-based methods [17]), as in Table 1.
(i) For model selection methods, we use
METAOD [11] as a representative, since it deals with automatic unsupervised outlier model selection which uses the performances of historical outlier detection benchmark datasets as prior experience to automatically do model selection via meta-learning.
(ii) For error-rate-controlled methods, we choose
SRS-based methods [17] as typical methods to give statistical guarantees for the error rate control, but there is no model selection considered in SRS-based methods.
We consider two versions of the proposed AutoMS: AutoMS-JK and AutoMS-SRS , as shown in Table 1, where AutoMS-JK uses Jackknife technique and AutoMS-SRS replaces the Jackknife technique with single-random-splitting (SRS) technique.
We explore the performances of the four methods with respect to FDR and TDR.
We give the detailed introduction for the four methods: (1) SRS [17], METAOD [11], (3) AutoMSJK and (4) AutoMS-SRS .
(1) SRS [17] employs the single-random-splitting (SRS) based method in conjunction with a fixed detector
and constructs conformal p-values to control the FDR.
We combine SRS with different detectors from the same G we use for AutoMS.
In particular, SRS-LOF means that we use LOF as the base detector in combination with SRS, while SRS-LOF-best is the best SRS-LOF detector with the largest TDR among all its candidate hyperparameters.
SRS-based methods [17] are representative error-rate-controlled methods which combine SRS with different base detectors from G as used for AutoMS.
It is worth mentioning that SRS is guaranteed with FDR control for any given detector.
But there is no model selection considered in SRS-based methods.
(2) METAOD [
11] is a model selection algorithm which uses meta-learning to learn the performance of a large body of detection models on historical outlier detection benchmark datasets, and carries over this prior experience to automatically select an effective model to be employed on a new dataset.
METAOD is a representative model selection method using meta-learning without considering error rate control, while the AutoMS can do both model selection and error rate control.
(3) AutoMS-JK is our proposed model selection procedure using Jackknife technique to select the “best” detector among a bunch of candidate detectors while controlling the FDR.
(4) AutoMS-SRS is a variant of AutoMS that replaces the Jackknife technique with single-randomsplitting (SRS) to incorporate our model selection procedure.
As the single-random-splitting (SRS) can be combined with our model selection procedure, hereafter called AutoMS-SRS.
AutoMS-SRS can be regarded as a special case of AutoMS and also has the theoretical guarantees that the selected model yields asymptotically valid FDR control.
But SRS does not fully explore the clean data and can cause randomness by data-splitting.
So we use the
Jackknife method instead of SRS to improve the accuracy and stability of the estimated p-values and enhance detection power.
Specially, when the sample size is too large, we use AutoMS-SRS to reduce the computation complexity.
C Detailed Experiment Results on 4 Real-world Datasets in the Main Paper
In this part, we provide detailed experiment results on the four real-world datasets, and we provide additional experiment results on 11 more real-world datasets in the next part.
We show the empirical FDR and TDR as performance measures of different methods on four realworld datasets with different target FDR levels α in Table S2, as a detailed extension of Tableand Table 4 in Sect.
5 of the main paper.
As introduced in Sect.
5, the empirical FDR and TDR are calculated as the expectation of FDP and TDP, respectively.
Note the performance of METAOD is same under different α since METAOD does not employ any FDR control procedure.
[Table]
Credit Card
AutoMS-JK 0.
AutoMS-SRS 0.
SRS-HBOS-best 0.
SRS-iForest-best 0.
SRS-KNN-best 0.
SRS-LODA-best 0.
SRS-LOF-best 0.
SRS-OCSVM-best 0.
METAOD 0.
Satellite
AutoMS-JK 0.
AutoMS-SRS 0.
SRS-HBOS-best 0.
SRS-iForest-best 0.
SRS-KNN-best 0.
SRS-LODA-best 0.
SRS-LOF-best 0.
SRS-OCSVM-best 0.
METAOD 0.
Shuttle
AutoMS-JK 0.
AutoMS-SRS 0.
SRS-HBOS-best 0.
SRS-iForest-best 0.
SRS-KNN-best 0.
SRS-LODA-best 0.
SRS-LOF-best 0.
SRS-OCSVM-best 0.
METAOD 0.
[Table] [Table]
[Table] [Table]
From Table S2, we see that the empirical FDR values of AutoMS (either AutoMS-JK or AutoMSSRS) and SRS-based methods are all around the target FDR levels for all datasets.
In finite samples, the empirical FDR of AutoMS is slightly larger than the given target FDR level α , but their difference does not increase as α becomes larger.
We also list the Standard Deviation for FDP and TDP of
AutoMS-JK, AutoMS-SRS, SRS-based methods and METAOD on four real-world datasets under different target FDR levels α in Table S3.
Moreover, with similar target FDR level α , AutoMS can uniformly offer much higher TDRs than
SRS-based methods across all settings, as the former automatically selects the most suitable model
[Table]
Credit Card
AutoMS-JK 0.
AutoMS-SRS 0.
SRS-HBOS-best 0.
SRS-iForest-best 0.
SRS-KNN-best 0.
SRS-LODA-best 0.
SRS-LOF-best 0.
SRS-OCSVM-best 0.
METAOD 0.
Satellite
AutoMS-JK 0.
AutoMS-SRS 0.
SRS-HBOS-best 0.
SRS-iForest-best 0.
SRS-KNN-best 0.
SRS-LODA-best 0.
SRS-LOF-best 0.
SRS-OCSVM-best 0.
METAOD 0.
Shuttle
AutoMS-JK 0.
AutoMS-SRS 0.
SRS-HBOS-best 0.
SRS-iForest-best 0.
[Table]
[Table]
SRS-KNN-best
[Table]
[Table]
SRS-LODA
[Table]
best 0.
[Table]
[Table]
[Table]
[Table]
SRS-LOF-best 0.
[Table]
[Table]
[Table]
[Table] [Table] [Table] SRS
[Table]
best
[Table]
[Table]
[Table]
[Table]
[Table] [Table]
[Table]
[Table]
METAOD 0.
while maintaining the FDR control.
On the Satellite dataset, the SRS-KNN-best method achieves the same high empirical TDP as AutoMS.
It is because the optimal algorithm and hyperparameter selected by AutoMS are almost the same as those used by SRS-KNN-best.
On the other hand, the METAOD algorithm has a higher empirical TDR (i.e. higher recall) than
AutoMS in some settings (such as “Covertype” dataset and “Credit Card” dataset), while the empirical
FDR (i.e. 1 - Precision) of METAOD can be as large as 50% , indicating that around 50% of outliers detected by METAOD are not true outliers.
That is, the recall of METAOD can be larger than AutoMS in some applications, while its precision can only be maintained around or smaller than 50% , which results from the fact that there is no error control in METAOD.
In conclution, AutoMS is a more stable novelty detection method containing model selection and FDR control.
We can see that the larger the target FDR level α , the larger the empirical TDR, which means the more relaxation of the target FDR level, the higher improvement of empirical TDR, indicating the trade-off between precision and recall of detection methods.
Adjusting the target FDR level leaves us the trade-off that with a larger target FDR level α the detection method can detect more outliers but produce more false discoveries.
So we can adjust our target FDR level α to obtain the satisfied FDR and TDR.
However, the performance of METAOD is same under different target FDR levels α since METAOD does not employ any FDR control procedure.
And AutoMS can uniformly offer much higher TDRs than
SRS-based-best methods across all FDR levels.
Therefore, our AutoMS is a practical model selection method, which takes into account FDR and TDR.
D Additional Experiment Results on More Real-world Datasets
As for the experiment scale of our AutoMS method, we are here trying to make it clear that why there is no need for our AutoMS method to go through hundreds of datasets.
As MetaOD uses meta-learning, METAOD requires a large number of datasets as the historical benchmarks to measure the similarity between the test dataset and benchmark datasets.
And the different test-bechmarkdataset-similarity will effect the results of METAOD, while our AutoMS approach has no special requirements for datasets.
We have shown experiments on 4 datasets in the main paper, and the proposed AutoMS method works on all other datasets, of course not only on these 4 datasets shown in the main paper, which can be obtained from Theorem 1 (FDR control) in our paper.
We have conducted additional experiments on 11 more real-world datasets.
The conclusion still remains the same that AutoMS can always control FDR while METAOD cannot.
To our best knowledge, is the first effort of model selection for novelty detection with theoretical guarantees in the view of FDR control.
Our proposed AutoMS method can select the best model and simultaneously control the error rate of the best model.
[Table]
We provide additional experiment results on 11 more real-world datasets which are pre-processed data in Emmott et al.
And the details of 11 real-world datasets are summarized in Table S4.
The outlier ratio of the original dataset is calculated by p =
N 1 N 0 + N 1 .
We construct the training dataset
D by random sampling m = min { 20000 , N 0 / 3 } from the total N 0 inliers.
As for the test dataset
U , we need n = min { 2000 , ( N 0 + N 1 ) / 6 } samples in total to generate the test dataset.
We fix the outlier ratio as p , then randomly sample n × (1 − p ) inliers from the remaining N 0 − m inliers
[Table]
and randomly sample n × p outliers from N 1 outliers.
When constructing the test dataset U , the outlier ratio for each dataset is the same as the original dataset as shown in Table S
[Table]
Note that there are only inliers in the training dataset D , and there are p percent of outliers in the test dataset U .
For example, for the “abalone” dataset, there are 49 .
7% outliers in its test dataset U
[Table]
and the goal of novelty detection is to detect the 49 .
7% outliers, using only clean data D .
The goal of model selection for unsupervised outlier detection is to select the best outlier detection model.
Our goal here is to select the best model and simultaneously to control the error rate of the best model.
We propose AutoMS method to achieve the goal of model selection for unsupervised outlier detection with error rate control.
[Table]
Table S5 shows the results of empirical FDR and empirical TDR of different methods (including
AutoMS-JK, AutoMS-SRS, SRS-best and METAOD) on 11 more real-world datasets when the target
FDR level α = 0 .
As introduced above, these 11 real-world datasets have different outlier ratios which are shown in Table S4.
The outlier ratio ranges from 10% to 50% .
Unsupervised novelty detection becomes more difficult when the outlier ratio goes higher, because only inliers are used in the cleaning training dataset D and more outliers need to be detected in the test dataset U .
The outlier ratio of the 11 real-world datasets ranges from 10% to 50% .
The experiment results in Table S5 show that at different outlier ratio, our AutoMS approach outperforms SRS-best and MATAOD for all the 11 real-world datasets.
Table S5 shows that the empirical TDR of AutoMS is higher than the empirical TDR of SRS-best since SRS-based methods only consider FDR control without model selection, while AutoMS selects the best detector with largest TDR while the empirical FDR is still controlled asymptotically around the FDR level.
The effectiveness of our AutoMS can be found and supported by the theories of our paper, which theoretically guarantees that AutoMS can control the FDR.
It is worth noting that METAOD requires a large number of datasets as the historical benchmark to measure the similarity between the test dataset and benchmark datasets as it uses meta-learning.
All the empirical FDRs of all datasets using METAOD are as large as 0 .
2 or even larger, indicating a very high false discovery rate, which means METAOD can not control the FDR.
However, our proposed AutoMS can improve the TDR while controlling the FDR.
For the two versions of AutoMS, sometimes AutoMS-SRS has higher empirical TDR than AutoMSJK, but the empirical FDR of AutoMS-SRS has a larger deviation above target FDR level α than
AutoMS-JK.
Table S6 gives the standard deviation of empirical FDR and empirical TDR of different methods on 11 more real-world datasets when the target FDR level α = 0 . 1 .
[Table]
skin
[Table] [Table]
spambase
[Table]
synthetic
[Table]
wave
[Table]
[Figure]
SRS−best
METAOD
[Figure]
p a g e
[Figure]
s k
i n s p a
[Figure]
a
[Figure]
e
[Figure]
y n t h e
[Figure]
i c
[Figure]
a
[Figure]
Fig.
S1 shows the performance of AutoMS-JK, AutoMS-SRS, SRS-best and METAOD on different real-world datasets when the target FDR level α = 0 .
The empirical FDR of AutoMS-JK and
AutoMS-SRS is all controlled around the given target FDR level α = 0 .
1 , The empirical FDR of
AutoMS-JK has a smaller deviation above the target FDR α than AutoMS-SRS on “pageb” and “wave” datasets.
The empirical TDR of AutoMS-JK and AutoMS-SRS is higher than the SRS-best methods at every outlier detections.
Although the empirical TDR of METAOD is much higher than
AutoMS, the empirical FDR is the worst as on error rate control has been taken into count.
Therefore, our AutoMS procedure can be more practical in real-world applications, which takes into account
FDR and TDR and selects the best detector.
E Proofs E.1 Proof of Theorem
We present the proof of the main Theorem 1 in the Appendix.
Before starting, we make clear the notations first.
Let D
and U be the training dataset and test dataset, respectively.
Let O c
⊂
U be the subset of U containing all the inliers with size n 0 , O ⊂ U be the subset of all the outliers, and O (cid:83) O c
= U .
When there is no risk of confusion, we also denote U the indices of the test dataset, O the indices of outliers, and O c the indices of inliers.
Let S i, M be the short of S M ( Z i ) for Z i ∈
U (i.e., i ∈ U ).
Note S
i, M and S k, M are i.i.d.
for each pair ( i, k ) ∈ U conditional on D and detector M .
We define
G M (
[Equation]
G
[Equation]
= inf { t ≥ 0 : G M ( t ) ≤ y } with 0 ≤ y ≤ 1 .
Define
[Equation]
where n 0 = |O c | and recall
[Equation]
where ¯ t m, M = G − m, M ( A n /n ) , G m, M ( t ) = m − 1 (cid:80) mj
=
I ( S [ − j ] j, M > t ) and A n = o ( n ) is a prespecified sequence.
In the proof part, we use c and C denote strictly positive constants that can be changed from place to place, and they do not depend on n and m .
Let ˜ G M ( t
[Equation]
E { G M
[Equation]
t ) } .
Proof of Theorem 1 .
Briefly, we first prove that the FDP of each detector can be bounded uniformly according to the uniform convergence properties Lemma C.1-C.3.
Hence, the selected detector with the largest number of discoveries will have the same non-asymptotic bounds for FDP.
Then we give the asymptotic FDR control for the optimal detector
[Equation]
By Lemma C.3-(ii) we have
[Equation]
By Lemmas C [Equation] 1 and C.
[Equation]
we have
[Equation]
hold simultaneously with probability at least 1 − 2 δ .
Under the event that Eqs.
(C.3) and (C.4) hold, we have
[Equation]
where g m,n = o (1) under Assumption 3.
Hence, with probability at least 1 − 2 δ , for sufficiently large m and n ,
[Equation]
Since G m, M ( t ) is a deceasing function of t , we have ¯ t m, M ≤ ¯ t M with high probability, and hence ¯ t m, M ∗ ≤ ¯ t M ∗ .
Similarly, ¯ t m, M ∗ ≤ ˜ t
M ∗ with at least 1 − 2 δ .
Write
[Equation]
In the case that L M ∗ = ∞ , the result holds by definition.
Thus, we only consider the case that
L M ∗ ≤ ¯ t m, M ∗ .
Using Eqs.(C.2)-(C.4) to get bounds for the terms E 1 , E 2 and E 3 , respectively,
Eq.(7) is proved.
Then, for any ϵ > 0 ,
[Equation]
which proves the second part of this theorem by using Assumption 3.
□
E.2 Some Lemmas
We need the following lemmas to prove Theorem 1.
The first lemma establishes the uniform convergence of G n, M ( t ) .
Lemma C.1 (Uniform consistency of the estimator with U ) .
Suppose Assumption 1-Assumptionhold.
Denote ¯ t M = G −M ( A n / (2 n )) .
For δ > 0 , we have with probability at least 1 − δ ,
[Equation]
Proof.
For any given M ∈ G , note that G M ( t ) is a deceasing and continuous function of t .
Let a n =
A n / 2 , z 0 ≤ z 1 · · · ≤ z h n ≤ 1 , and t k = G −M ( z k ) , where z 0 = a n /n , z k = a n /n + b n exp( k τ ) /n , h n = { log(( n − a n ) /b n ) } 1 /τ with b n /a n → 0 and 0 < τ < 1 .
Note that | G M ( t k ) /G
M ( t k +1 ) − 1 | ≤ Cb n /a n uniformly holds in k .
It is therefore enough to obtain the convergence of
[Equation]
We calculate the second moment
[Equation]
From result (C.5) and Chebyshev’s inequality, for ϵ > 0 we have
[Equation]
Moreover, observe that
[Equation]
By Assumption
[Equation]
Hence, conditional on D
[Equation]
we have with probability at least 1 − δ ,
[Equation]
where we take b n = a 2 / 3 n .
Finally, note that this holds uniformly in D , and thus we complete the proof.
Lemma C.2 (Uniform consistency of the Jackknife estimator) .
Suppose Assumption 1-Assumptionhold.
Denote ˜ t M = ˜
G −M ( A n / 2 n ) .
For δ >
[Equation]
we have with probability at least 1 − δ ,
[Equation]
Proof.
Define G [ − j ] M ( t ) =
P ( S [ − j ] M ( Z ) > t | D [ − j ] ) , where D [ − j ] =
D \ ˜ X j .
Using the same partition technique in Lemma C.1, we need to obtain a bound for
[Equation]
Again, the main step is to bound
[Equation]
By the construction of Jackknife estimators on D , { 1 ≤ j ≤ m : S [ − j ] j, M } are not independent with each other.
Observe for each pair ( j, k )
E
[Equation]
I
[Equation]
S [ − j ] j
[Equation]
M > t )
I
[Equation]
S
[Equation]
−
[Equation]
k
[Equation]
M >
[Equation]
≤
[Equation]
M > t, S [ − k ] k, M > t | D [ − j, − k ] (cid:17) − P (cid:16) S [ − j, − k ] j, M > t, S [ − j, − k ] k, M > t | D [ − j, − k ] (cid:17)(cid:12)(cid:12)(cid:12)(cid:111) + E (cid:110) P (cid:16) S [ − j, − k ] j, M > t, S [ − j, − k ] k, M > t | D [ − j, − k ] (cid:17)(cid:111) :
= Q 1 + Q 2 .
By the fact that S [ − j, − k ] j, M and S [ − j, − k ] k, M are independent conditional on D [ − j, − k ] , we have Q 2 =
E { G [ − j, − k ] M ( t ) } 2 .
In Lemma C.5, we set Y 1 = S [ − j ] j, M , Y 2 = S [ − k ] k, M , Y 3 = S [ − j, − k ] j, M , and Y 4 =
S [ − j, − k ] k, M .
Under Assumption 2, we have
Q 1 ≤ 4 c f κ m + 6 ζ m , by applying Lemma C.5.
Accordingly,
[Equation]
where
[Equation]
, where we use Lemma C.3 to get
[Equation]
Similarly, we have
[Equation]
With the above calculation and Chebyshev’s inequality, for ϵ >
[Equation]
Moreover, observe that
[Equation]
which implies that
[Equation]
By similar arguments in Lemmas C.1 and taking b n = a 2 / 3 n , we can complete the proof.
Lemma C.3.
Suppose Assumption 1 and Assumption 2 hold.
Then we have (i)
The following holds
[Equation]
uniformly in 1 ≤ j ≤ m .
(ii) It holds that E
[Equation]
M
[Equation]
Proof.
(i) Observe
[Equation]
The lemma follows immediately by using Assumption 2 and Lemma C.4 with c f > 0 .
(ii) By Lemma C.4, we have
[Equation]
and the result holds by taking expectation and using Assumption 2 again.
Lemma C.4 (Estimation difference) .
Assume that two random variables Y 1 and Y 2 have density functions bounded by a constant c > 0 , and satisfy P ( | Y 1 − Y 2 | ≤ ε ) ≥ 1 − ζ , where ε > 0 and ζ > 0 .
Then for all t > 0 , | P ( Y 1 > t ) − P ( Y 2 > t ) | ≤ 2 cε + 3 ζ .
Proof.
By the fact that
[Equation]
we have
[Equation]
where the last inequality comes from the Lipschitz condition.
Lemma C.5.
Assume that four random variables Y 1 , Y 2 , Y 3 , and Y 4 have density functions bounded by a constant c > 0 , and satisfy P ( | Y 1 − Y 3 | ≤ ε ) ≥ 1 − ζ and P ( | Y 2 − Y 4 | ≤ ϵ ) ≥ 1 − ς , where ε > 0 , ϵ > 0 , ζ >
and ς > 0 .
Then for all t > 0 ,
[Equation] [Equation]
Proof.
Define the events E 1 = {| Y 1 − Y 3 | ≤ ε } , E 2 = {| Y 2 − Y 4 | ≤ ϵ } , and E = E 1 (cid:83) E 2 .
Note that P
[Equation]
Y 3 > t, E 1 ) ≤ P ( Y 1 − ε > t, E 1 ) and P ( Y 2 + ϵ > t, E 2 ) ≤ P ( Y 4 > t, E 2 ) ≤ P ( Y 2 − ϵ > t, E 2 ) .
Then, we have
P ( Y 1 > t, Y 2 > t ) − P
( Y 3 > t, Y 4 > t ) ≤ | P ( Y 1 > t, Y 2 > t ) − P
( Y 3 > t, Y 4 > t, E )
| + P ( E c 1 )
+ P ( E c 2 ) ≤ | P
( Y 1 > t, Y 2 > t ) − P
( Y 1 − ε > t, Y 2 − ϵ > t, E ) | + | P ( Y 1 > t, Y 2 > t ) − P ( Y 1 + ε > t, Y 2 + ϵ > t, E ) | + P
( E c 1 )
+ P ( E c 2 ) ≤ | P
( Y 1 > t, Y 2 > t ) − P
( Y 1 − ε > t, Y 2 − ϵ > t ) | + | P ( Y 1 > t, Y 2 > t ) − P
( Y 1 + ε > t, Y 2 + ϵ > t | + 3 P
( E c 1 )
+ 3 P ( E c 2 ) ≤ 2 c ( ε + ϵ ) + 3( ζ + ς ) , where we use the Lipschitz condition again.