N ON -N EGATIVE B REGMAN D IVERGENCE M INIMIZATION FOR D EEP D IRECT D ENSITY R ATIO E STIMATION
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper aims to estimate the ratio of probability densities using ﬂexible models, such as state-of-the-art deep neural networks. The density ratio estimation (DRE) has garnered attention as the density ratio is useful in various machine learning tasks, such as anomaly detection and domain adaptation.
For estimating the density ratio, methods collectively known as direct DRE have been explored.
These methods are based on the minimization of the Bregman (BR) divergence between a density ratio model and the true density ratio.
However, when using ﬂexible models , such as deep neural networks, existing direct DRE suffers from serious train-loss hacking , which is a kind of over-ﬁtting caused by the form of an empirical risk function.
In this paper, we introduce a non-negative correction for empirical risk using only the prior knowledge of the upper bound of the density ratio.
This correction makes a DRE method robust against train-loss hacking. It enables the use of ﬂexible models, such as state-of-the-art deep neural networks.
In the theoretical analysis we show the generalization error bound of the BR divergence minimization.
In our experiments, the proposed methods show favorable performance in inlier-based outlier detection and covariate shift adaptation
I NTRODUCTION
The density ratio estimation (DRE) problem has attracted a great deal of attention as an essential task in data science for its various industrial applications, such as domain adaptation (Shimodaira, 2000; Plank et al., 2014; Reddi et al., 2015), learning with noisy labels (Liu & Tao, 2014; Fang et al., 2020), anomaly detection (Smola et al., 2009; Hido et al., 2011; Abe & Sugiyama, 2019), two-sample testing (Keziou & Leoni-Aubin, 2005; Kanamori et al., 2010; Sugiyama et al., 2011a), causal inference (Kato et al., 2020), change point detection in time series (Kawahara & Sugiyama, 2009), and binary classiﬁcation only from positive and unlabeled data (PU learning; Kato et al., 2019).
For example, anomaly detection is not easy to perform based on standard machine learning methods such as binary classiﬁcation since anomalous data is often scarce, but it can be solved by estimating the density ratio when training data without anomaly as well as unlabeled test data are available (Hido et al., 2008). Among various approaches for DRE, we focus on the Bregman (BR) divergence minimization framework (Bregman, 1967; Sugiyama et al., 2011b) that is a generalization of various DRE methods,
e.g., the moment matching (Huang et al., 2007; Gretton et al., 2009), the probabilistic classiﬁcation (Qin, 1998; Cheng & Chu, 2004), the density matching (Nguyen et al., 2010; Yamada et al., 2010), and the density-ratio ﬁtting (Kanamori et al., 2009).
Recently, Kato et al.
(2019) also proposed using the risk of PU learning for DRE, which also can be generalized from the BR divergence minimization viewpoint, as we show below.
However, existing DRE methods mainly adopt a linear-in-parameter model for nonparametric DRE (Kanamori et al., 2012) and rarely discussed the use of more ﬂexible models, such as deep neural networks, while recent developments in machine learning suggest that deep neural networks can signiﬁcantly improve the performances for various tasks, such as computer vision (Krizhevsky et al., 2012) and natural language processing (Bengio et al., 2001). This motivates us to use deep neural networks for DRE. However, existing DRE studies have not fully discussed using such state-of-theart deep neural networks. For instance, although Nam & Sugiyama (2015) and Abe & Sugiyama are
When using deep neural networks in combination with empirical minimization of BR divergence, we often observe a serious over-ﬁtting problem as demonstrated through experiments in Figureof Section 5.1.
We hypothesize that this is because there is no lower bound in the empirically BR divergence approximating by ﬁnite samples, i.e., we can achieve an inﬁnitely negative value in minimization.
This hypothesis is based on Kiryo et al.
(2017), which reports a similar problem in PU learning.
While Kiryo et al.
(2017) call this phenomena over-ﬁtting , we refer to it as train-loss hacking because the nuance is a bit different from the standard meaning of overﬁtting.
Here, we brieﬂy introduce the train-loss hacking discussed in the PU learning literature Kiryo et al.
In a standard binary classiﬁcation problem, we train a classiﬁer ψ by minimizing the following empirical risk using { ( y i , x i ) } ni =1 :
[Equation]
where y i ∈ {± 1 } is a binary label, x i is a feature, and (cid:96) is a loss function.
On the other hand, in PU learning formulated by du Plessis et al. (2015), because we only have positive data { ( y (cid:48) i = +1 , x (cid:48) i ) } n (cid:48)
i =1 and unlabeled data { ( x (cid:48)(cid:48) j ) } n (cid:48)(cid:48)
j =1 , we minimize the following alternative empirical risk:
[Equation]
[Table]
−
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
(cid:96)
[Equation]
) where π is a hyper-parameter representing p ( y = +1) . Note that the empirical risk (2) is unbiased to the population binary classiﬁcation risk (1) (du Plessis et al., 2015). While the the empirical risk (1) of the standard binary classiﬁcation is lower bounded under an appropiate choise of (cid:96) , the empirical risk (2) of PU learning proposed by du Plessis et al. (2015) is not lower bounded owing to the existence of the second term.
Therefore, if a model is sufﬁciently ﬂexible, we can signiﬁcantly minimize the empirical risk only by minimizing the second term − πn (cid:48) (cid:80) n (cid:48) i =1 (cid:96) ( − ψ ( x (cid:48) i )) without increasing the other terms. Kiryo et al. (2017) proposed non-negative risk correction for avoiding this problem when using neural networks.
We discuss this problem again in Section 2 and Figure 1. In existing DRE literature, this train-loss hacking has rarely been discussed, although we often face this problem when using neural networks, as mentioned in Section 5.1. One reason for this is that the existing method assumes a linear-in-parameter model for a density ratio model (Kanamori et al., 2012), which is not so ﬂexible as neural networks and do not cause the phenomenon.
To mitigate the train-loss hacking, we propose a general procedure to modify the empirical BR divergence using the prior knowledge of the upper bound of the density ratio.
Our idea of the correction is inspired by Kiryo et al.
However, their idea of non-negative correction is only immediately applicable to the binary classiﬁcation; thus we require a non-trivial rewriting of the BR divergence to generalize the approach to our problem.
We call the proposed empirical risk the non-negative BR (nnBR) divergence, and it is a generalization of the method of Kiryo et al.
Our main contributions are: (1) the proposal of a general procedure to modify a BR divergence to enable DRE with ﬂexible models, (2) theoretical justiﬁcation of the proposed estimator, and (3) the experimental validation of the proposed method using benchmark data.
P ROBLEM S ETTING
Let X nu ⊆ R d and X de ⊆
R
d be the spaces of the d -dimensional covariates (cid:8) x
nu i (cid:9) n nu i =1 and (cid:8) x de i (cid:9) n de i =1 , respectively, which are independent and identically distributed (i.i.d.) as (cid:8) x nu i (cid:9) n nu i =
i .
i .
d . ∼ p nu ( x ) and (cid:8) x de i (cid:9) n de i =1 i .
i .
d . ∼ p de ( x ) , where p nu ( x ) and p de ( x ) are probability densities over X nu and
X de , respectively.
Here, “nu” and “de” indicate the numerator and the denominator.
Our goal is to estimate the density ratio r ∗ ( x ) =
p nu ( x ) p de ( x ) .
To identify the density ratio, we assume the following: [Table]
Assumption 1.
The density p nu ( x ) is strictly positive over the space X nu , the density p de ( x ) is strictly positive over the space X de , and X nu ⊆
X de .
In addition, the density ratio r ∗ is bounded from above on X de : R = sup x ∈X de r ∗ ( x ) < ∞ .
Note that the assumption X nu ⊆
X de is typical in the context of DRE.
For instance, in anomaly detection with unlabeled test data, X de corresponds to a sample space including clean and anomaly data and X de corresponds to a sample space only with clean data.
Here, we introduce the notation of this paper. Let E nu and E de denote the expectations over p nu ( x ) and p de ( x ) , respectively.
Let ˆ E nu and ˆ E de denote the sample average over (cid:8) x
nu i (cid:9) n nu i =1 and (cid:8) x de i (cid:9) n de i =1 , respectively.
Let H ⊂ { r : R d → ( b r , B r ) } be the hypothesis class of the density ratio, where 0 ≤ b r < R < B r .
2. D ENSITY R ATIO M ATCHING UNDER THE B REGMAN D IVERGENCE
A naive way to implement DRE would be to estimate the numerator and the denominator densities separately and take the ratio.
However, according to Vapnik’s principle, we should avoid solving a more difﬁcult intermediate problem than the target problem (Vapnik, 1998).
Therefore, various methods for directly estimating the density ratio model have been proposed (Gretton et al., 2009; Sugiyama et al., 2008; Kanamori et al., 2009; Nguyen et al., 2010). Sugiyama et al.
(2011b) showed that these methods can be generalized as the density ratio matching under the BR divergence .
The BR divergence is an extension of the Euclidean distance to a class of divergences that share similar properties (Bregman, 1967).
Formally, let f : ( b r , B r ) → R be a twice continuously differentiable convex function with a bounded derivative.
Then, the point-wise BR divergence associated with f from t ∗ to t is deﬁned as BR (cid:48)
f ( t ∗ (cid:107) t ) :
= f ( t ∗ ) − f ( t ) − ∂f ( t )( t ∗ − t ) , where ∂f is the derivative of f .
Now, the discrepancy from the true density ratio function r ∗ to a density ratio model r is measured by integrating the point-wise BR divergence as follows (Sugiyama et al., 2011b):
[Equation]
We estimate the density ratio by ﬁnding a function r that minimizes the BR divergence deﬁned in (3).
Here, we subtract the constant BR = E de (cid:2) f
( r ∗ ( x )) (cid:3) from (3) to obtain BR
[Equation]
Here, Sugiyama et al.
(2012) used r ∗ ( x ) p de = p nu for removing r ∗ ( x ) , which is a common technique in the DRE literature.
Since BR is constant with respect to r , we have arg min r BR (cid:48)
f ( r ∗ (cid:107) r ) =
arg min r BR f ( r ∗ (cid:107) r ) .
Then, let us deﬁne the sample analogue of (4) as
[Equation]
For a hypothesis class H , we estimate the density ratio by solving min r ∈H (cid:100)
BR f ( r ∗ (cid:107) r ) .
2. E XAMPLES OF DRE
Sugiyama et al.
(2011b) showed that various DRE methods can be uniﬁed from the viewpoint of BR divergence minimization.
Furthermore, Menon & Ong (2016) showed an equivalence between conditional probability estimation and DRE by BR divergence minimization.
In addition, we can derive a novel method for DRE from the proposed method of du Plessis et al.
(2015) and Kato et al.
We summarize the DRE methods in Table 1.
Here, the empirical risks of least-square importance ﬁtting (LSIF), the Kullback–Leibler importance estimation procedure (KLIEP), logistic regression (LR), and PU learning with log Loss (PULogLoss) is given as (cid:100) BR LSIF ( r ) :=
− ˆ E nu
[ r ( X )]
+ 12 ˆ E de [( r ( X ))
2 ] , (cid:100) BR UKL ( r ) :
=
ˆ E de [ r ( X )] − ˆ E nu (cid:2) log (cid:0) r ( x ) (cid:1)(cid:3) , (cid:100) BR BKL ( r ) :
= − ˆ E de (cid:104) log (cid:16) 1 1+ r ( X ) (cid:17)(cid:105) − ˆ E nu (cid:104) log (cid:16) r
( X )
+ r ( x ) (cid:17)(cid:105) , and (cid:100) BR PU ( r ) := − ˆ E de (cid:2) log (cid:0) 1 − r ( x i )
(cid:1)(cid:3)
+ C ˆ E nu (cid:2) − log (cid:0) r ( x i ) (cid:1) + log (cid:0) 1 − r ( x j ) (cid:1)(cid:3) ,
where 0 < C < 1 R .
Here, (cid:100) BR LSIF ( r ) and (cid:100) BR PU ( r ) correspond to LSIF and PULogLoss, respectively.
We can derive the
KLIEP and LR from (cid:100) BR UKL ( r ) and (cid:100) BR BKL ( r ) , which are called unnormalized Kullback–Leibler (UKL) divergence and binary Kullback–Leibler (BKL) divergence, respectively (Sugiyama et al., 2011b).
In (cid:100) BR PU ( r ) , we restrict the model as r ∈ (0 , 1) and we obtain an estimator of Cr ∗ as a result of the minimization of the risk.
Details of the existing methods are shown in Appendix A.
D EEP D IRECT DRE BASED ON N ON - NEGATIVE R ISK E STIMATOR
In this section, we develop methods for DRE with ﬂexible models such as neural networks.
3. D IFFICULTIES OF DRE USING N EURAL N ETWORKS
Training neural networks for DRE by minimizing the empirical BR divergence tends to suffer from an overﬁtting phenomenon.
For example, we show in Section 5 that the LSIF with neural networks (Nam & Sugiyama, 2015) suffers from a serious overﬁtting issue.
points, a sufﬁciently ﬂexible model r can easily make the training loss, e.g., (cid:100) BR LSIF ( r ) , diverge to −∞ .
One possible cause of overﬁtting of ﬂexible models has been identiﬁed to be the train-loss hacking as shown by
Kiryo et al.
(2017) in the context of PU learning (Figure 1).
In DRE, the train-loss hacking is also sensible: the term − ˆ E nu (cid:2) ∂f (cid:0) r ( x j ) (cid:1)(cid:3) can easily diverge to −∞ if we try to minimize empirical BR divergence (5) over a highly ﬂexible hypothesis class H .
This is because the term is not lower bounded, unlike the loss functions for other machine learning tasks such as classiﬁcation.
Here, note that the other term ˆ E de (cid:104) ∂f (cid:0) r ( x i ) (cid:1) r ( x i ) − f (cid:0) r ( x i ) (cid:1)(cid:105) does not introduce a sufﬁcient trade-off to prevent the divergence to −∞ when the hypothesis class is highly ﬂexible (Figure 1).
3. N ON - NEGATIVE BR D IVERGENCE
Although DRE using ﬂexible models suffers from serious train-loss hacking, we still have a strong motivation to use those models for analyzing data such as computer vision and text data.
For example, Nam & Sugiyama (2015) and Abe & Sugiyama (2019) approximated the density ratio with neural networks, and Uehara et al.
(2016) applied direct DRE for generative adversarial nets (GANs;
Goodfellow et al., 2014), both by minimizing the empirical BR divergence (5).
To alleviate the trainloss hacking problem, we propose the non-negative BR divergence estimator .
The proposed method is inspired by Kiryo et al.
(2017), which suggested a non-negative correction to the empirical risk of PU learning based on the knowledge that a part of the population risk is non-negative.
On the other hand, in DRE, it is not obvious how to correct the empirical risk because we do not know which part of the population risk (4) is non-negative.
However, this can be alleviated by determining the upper bound R of the density ratio r ∗ .
With the prior knowledge of R , we can determine part of the risk of DRE (4) is non-negative and conduct a non-negative correction to the empirical risk (5) based on the non-negativity of the population risk.
Let us deﬁne ˜ f
to be a function such that ∂f ( t )
=
C (cid:0) ∂f ( t ) t − f ( t ) (cid:1) + ˜ f
( t ) ,
where 0 < C < 1 R and put the following assumption.
Assumption 2.
Assume that ˜ f ( t ) is bounded from above, and that there exists a constant A such that ∂f ( t ) t − f ( t ) + A ≥ 0 for t ∈ ( b r , B r ) .
Assumption 2 is satisﬁed by most of the loss functions which appear in the previously proposed
DRE methods (see Appendix B for examples).
Under Assumption 2, because ˜ f ( t ) is bounded above, the train-loss hacking − ˆ E nu (cid:104) ∂f (cid:0) r ( x j ) (cid:1)(cid:105) →
−∞ in the minimization of the empirical risk (5) is caused by − ˆ E nu (cid:104) C { ∂f ( r ( x j ))
r ( x j ) − f
( r ( x j )) } (cid:105) →
−∞ .
Thus, we succeeded in identifying the part of the empirical risk causing the train-loss hacking.
Next, to avoid the train-loss hacking, we impose a non-negative correction to the empirical risk (5).
First, we rewrite the population risk (4) as BR f ( r ∗ (cid:107) r ) =
(cid:82) (cid:110) p de ( x ) −
Cp nu ( x ) (cid:111)(cid:110) ∂f ( r ( x ))
r ( x ) − f ( r ( x ))
+ A (cid:111)
d x − (cid:82) p nu ( x ) (cid:110) ˜
f
( r ( x )) (cid:111)
d x .
Note that we introduce a constant A , which is irrelevant to the original optimization problem.
Here, let us deﬁne (cid:96) 1 ( t ) :=
∂f ( t ) t − f ( t ) + A , and (cid:96) 2 ( t ) := − ˜ f ( t ) .
In the above equation, since Assumption 2 implies (cid:96) 1 ( t ) ≥ 0 , and 0 < C < 1 R and p nu ( x ) p de ( x ) ≤ R imply p de ( x ) −
Cp nu ( x ) > 0 for all x ∈ X , we have (cid:82) (cid:110) p de ( x ) −
Cp nu ( x ) (cid:111)(cid:110) (cid:96) 1 ( r ( x )) (cid:111)
d x > 0 .
Motivated by this inequality, we propose an empirical risk with the non-negative correction deﬁned as (cid:92)
[Equation] X )) [Equation] + , where ( · ) + := max { 0 , ·} .
Our deep direct DRE (D3RE) is based on minimizing (cid:92) nnBR f ( r ) .
Remark 1 (Choice of C ) .
In practice, selecting the hyper-parameter C does not require accurate knowledge of R because any 0 < C < 1 /R is sufﬁcient to justify the non-negative correction.
However, selecting C that is too small may damage the empirical performance.
See Section 5.2.
nnBR Divergence with Existing Methods: The above strategy can be instantiated in various methods previously proposed for DRE.
Here, we introduce the non-negative BR divergence estimators that correspond to LSIF, UKL, BKL, and PULogLoss as follows:
[Equation]
nu (cid:20)
[Table]
(cid:18)
[Equation]
X
[Equation]
[Table]
[Equation]
X
[Equation]
[Table]
[Equation] [Table] [Equation] 1 + r ( X ) (cid:19)(cid:21)
+ (cid:18) − ˆ E de (cid:20) log (cid:18) 1 + r {+( X ) (cid:19)(cid:21) + C ˆ E nu (cid:20) log (cid:18) 
+}
+ r ( X ) (cid:19)(cid:21)(cid:19) + , (cid:92) [Table]
[Equation]
[Table] More detailed derivation of ˜ f is in Appendix B.
In practice, using ﬂexible models for KLIEP poses another problem that the constrained optimization is no longer convex and is difﬁcult to solve.
Therefore, we experiment with UKL, unconstrained minimization of the objective function of KLIEP.
In Appendix C, we provide the pseudo code of D3RE.
For improving the performance heuristically, we use gradient ascending.
T HEORETICAL J USTIFICATION OF D3RE
In this section, we conﬁrm the validity of the proposed method by providing a generalization error bound.
Given n ∈ N and a distribution p , we deﬁne the Rademacher complexity R pn of a function class H as R pn ( H ) :
= E p E σ (cid:2) sup r ∈H (cid:12)(cid:12) 1 n (cid:80) ni
=
σ i r ( X i ) (cid:12)(cid:12)(cid:3) , where { σ i } ni =1 are independent uniform sign variables and { X i } ni
=
i . i .
d . ∼ p .
We omit r ∗ from the notation of BR f when there is no ambiguity.
4. G ENERALIZATION E RROR B OUND ON BR D IVERGENCE
Theorem 3 in Appendix H provides a generalization error bound under the following assumption.
Assumption 3.
Let I r :=
( b r , B r ) .
Assume that there exists an empirical risk minimizer ˆ r ∈ arg min r ∈H (cid:92)
nnBR f ( r ) and a population risk minimizer ¯ r ∈ arg min r ∈H BR f
( r ) .
Assume B (cid:96) :
= sup t ∈
I r { max {| (cid:96) 1 ( t ) | , | (cid:96) 2 ( t ) |}} < ∞ .
Also assume (cid:96) 1 (resp.
(cid:96) 2 ) is L (cid:96) 1 -Lipschitz (resp.
L (cid:96) 2Lipschitz) on I r .
Assume also that inf r ∈H ( E de −[-
 α 2 ( B (cid:96) 2 /n de )+(-] C E nu ) (cid:96) 1 ( r ( X )) > 0 holds.
In order for the boundedness and Lipschitz continuity in Assumption 3 to hold for the loss functions involving a logarithm (UKL, BKL, PU), a technical assumption b r > 0 is sufﬁcient.
We obtain a theoretical guarantee for D3RE from Theorem 3 in Appendix H by additionally imposing Assumption 4 to bound the Rademacher complexities using a previously known result (Golowich et al., 2019, Theorem 1).
Assumption 4 (Neural networks with bounded complexity) .
Assume that p nu and p de have bounded supports: sup x ∈X de (cid:107) x (cid:107) < ∞ .
Also assume that H consists of real-valued neural networks of depth L over the domain X , where each parameter matrix W j has the Frobenius norm at most
B W j ≥ 0 and with 1 -Lipschitz activation functions ϕ j that are positive-homogeneous (i.e., ϕ j is applied element-wise and ϕ j ( αt ) = αϕ j ( t ) for all α ≥ 0 ).
Under Assumption 4, Lemma 4 in Appendix I reveals R p nu n nu ( H )
=
O (1 / √ n nu ) and R p de n de ( H ) =
O (1 / √ n de ) .
By combining these with Theorem 3 in Appendix H, we obtain the following theorem:
Theorem 1 (Generalization error bound for D3RE) .
Under Assumptions 3 and 4, for any δ ∈ (0 , 1) , we have with probability at least 1 − δ , [Equation] δ
[Equation]
where κ 1 , κ 2 are constants that depend on C, f, B p , L , and B W j .
By transforming the BR divergence, Theorem 1 tells us generalization error bounds for various problems.
For instance, by deﬁning f ( t ) as log (1 − t ) + Ct (log ( t ) − log (1 − t )) for 0 < t < 1 , the BR divergence BR f ( r ) becomes the same risk functional of PU learning (see Appendix A for the derivation). Then, the generalization bound is almost equal to the one shown by Kiryo et al.
(2017), i.e., the generalization bounds matches the classiﬁcation error bound of classiﬁcation only from positive and unlabeled data.
Thus, this result implies that the BR divergence minimization with
D3RE also minimize the generalization error bound of binary classiﬁcation.
Theorem 1 can be readily turned into L 2 distance bounds by applying Lemma 1, which can be more convenient for the theoretical analyses for subsequent tasks relying on the density ratio estimator (Cortes et al., 2008).
The proof can be found in Appendix J.
Lemma 1 (Generalization error bound in L 2 distance) .
Let H 2 :=
{ r : X → I r | (cid:82) | r ( x ) | 2 d x < ∞} and assume the true density ratio r ∗ is in H 2 .
If inf t ∈ I r f (cid:48)(cid:48) ( t ) > 0 , then there exists µ > 0 such that
[Equation]
[Table] [Equation]
Note that the strong-convexity assumption of Lemma 1 could be exploited to investigate the convergence rate in more detail, via ﬁner-grained proof techniques such as local Rademacher complexity.
Nevertheless, the above theoretical guarantee already sufﬁces for the basic justiﬁcation of D3RE.
E XPERIMENT WITH I MAGE D ATA
In this section, we experimentally show how the existing estimators fail to estimate the density ratio when using neural networks and that our proposed estimators succeed.
To investigate the performances, we consider PU learning.
For a binary classiﬁcation problem with labels y ∈ {− 1 , +1 } , we consider training a classiﬁer only from p ( x | y
= +1)
and p ( x ) to ﬁnd a positive data point in test data sampled from p ( x ) .
The goal is to maximize the area under the receiver operating characteristic (AUROC) curve, which is a criterion used for anomaly detection, by estimating the density ratio r ∗ ( x )
=
p ( x | y
= +1) /p
( x ) .
We construct the positive and negative dataset from CIFAR-10 1 (Krizhevsky, 2009) dataset with 10 classes.
The positive dataset com- prises ‘airplane’, ‘automobile’, ‘ship’, and ‘truck’; the negative dataset comprises ‘bird’, ‘cat’, ‘deer’, ‘dog’, ‘frog’, and ‘horse’.
We use 1 , 000 positive data sampled from p ( x | y = +1) and 1 , 000 unlabeled data sampled from p ( x ) to train the models.
Then, we calculate the AUROCs using 10 , 000 test data sampled from p ( x ) .
In this case, it is desirable to set C < 12 because
[Equation] =
[Equation]
For demonstrative purposes, we use the CNN architecture from the PyTorch tutorial (Paszke et al., 2019).
Details of the network structure are shown in Appendix D.2.
The model is trained using Adam without weight decay and the parameters ( β 1 , β 2 , (cid:15) ) in Kingma & Ba (2015) are ﬁxed at the default of PyTorch, namely (0 . 9 , 0 . 999 , 10 − 8 ) .
5. C OMPARISON WITH N AIVE E STIMATORS
First, we compare two of the proposed estimators, (cid:92) nnBR PU (nnBR-PU) and (cid:92) nnBR LSIF (nnBRLSIF), with the existing estimators, (cid:100) BR PU (PU-NN) and (cid:100) BR LSIF (uLSIF-NN).
We use the logistic loss for PULogLoss.
Additionally, we conduct an experiment with uLSIF-NN using a naively capped model ˜ r ( x ) =
min { r ( x ) , 1 /C } (Bounded uLSIF).
We ﬁx the hyperparameter C at 1 / 3 .
We report the results for two learning rates, 1 × 10 − 4 and 1 × 10 − 5 .
We conducted 10 trials, and calculated the average AUROCs.
We also compute ˆ E de [ˆ r ( X )] , which should be close toif we successfully estimate the density ratio, because (cid:82) ( p ( x | y
= +1) /p
( x )) p ( x ) dx = 1 .
The results are in Figure 2.
In all cases, the proposed estimators outperform the other methods.
In contract, the unstable behaviors of PU-NN and LISF-NN are caused by the train-loss hacking (also see (Kiryo et al., 2017) Appendix F.1).
The experiment also demonstrates that naive capping (Bounded uLSIF) fails to prevent train-loss hacking and leads to suboptimal behavior.
A naive capping is insufﬁcient because an unreasonable model such that r ( x de i ) = 0 and r ( x nu i )
= 1 /C still can be a minimizer by making one part of the empirical BR divergence largely negative, e.g., as [Equation] 0 − [Equation]
. 5. E MPIRICAL S ENSITIVITY A NALYSIS ON THE U PPER B OUNDS OF THE D ENSITY R ATIO
Next, we investigate the sensitivity of D3RE to the hyperparameter C .
We use nnBR-LSIF and nnBR-PU as in Section 5.1, but vary the hyperparameter C in { 1 / 5 , 1 / 2 , 2 / 3 } .
The other settings remain unchanged from the previous section.
These results are shown in Figure 4 in Appendix F.1.
For R = 2 . 0 , the estimator show a better performance when 1 /C is close to 2 . 0 .
5. C OMPARISON WITH V ARIOUS E STIMATORS USING NN BR D IVERGENCE
Finally, we examine the performances of nnBR-LSIF, nnBR-PU, nnBR-UKL, and nnBR-BKL.
The learning rate was 1 × 10 − 3 , and the other settings were identical to those in the previous experiments.
[Table] These results are shown in Figure 4 in Appendix F.1.
Although nnBR-UKL and nnBR-BKL show better performance in earlier epochs, nnBR-LSIF and nnBR-PU appear more stable.  I NLIER - BASED O UTLIER D ETECTION
As an applications of D3RE [Table] we introduce inlier-based outlier detection with experiments using benchmark datasets.
Moreover, we introduce other applications such as covariate shift adaptation in Appendix G.
In addition to CIFAR-10 , we use MNIST 2 (LeCun et al., 1998) and fashion-MNIST ( FMNIST ) 3 (Xiao et al., 2017).
Hido et al.
(2008; 2011) applied the direct DRE for inlier-based outlier detection: ﬁnding outliers in a test set based on a training set consisting only of inliers by using the ratio of training and test data densities as an outlier score.
Nam & Sugiyama (2015) and Abe & Sugiyama (2019) proposed using neural networks with DRE for this problem.
In relation to the experimental setting of Section 5, the problem setting can be seen as a transductive variant of PU learning (Kato et al., 2019).
We follow the setting proposed by Golan & El-Yaniv (2018) using MNIST , CIFAR-10 , and
FMNIST .
There are ten classes in each dataset; we use one class as the inlier class and all other classes as the outliers.
For example, in the case of CIFAR-10 , there are 5 , 000 train data per class.
On the other hand, there are 1 , 000 test data for each class, which consists of 1 , 000 inlier samples and 9 , 000 outlier samples.
The AUROC is used as a metric to evaluate whether an outlier class can be detected in the test data.
We compare the proposed methods with benchmark methods of deep semi-supervised anomaly detection (DeepSAD) (Ruff et al., 2020) and geometric transformations (GT) (Golan & El-Yaniv, 2018).
The details of each method are shown in Appendix E.
To compare the methods fairly, we use the same architectures of neural networks from Golan & El-Yaniv (2018) and Ruff et al.
(2020), LeNet and Wide Resnet, for D3RE.
The details of the structures are shown in Appendix D. A part of the experimental results with CIFAR-10 and FMNIST are shown in Tabledue to the limitation of the space.
The full results are shown in Table 3 in Appendix F.2.
The proposed methods result in better AUROCs than the existing methods.
The largest performance gain is seen in the CIFAR-10:
the mean AUROC is improved by 0 .
157 on average between the uLSIF-NN and nnBR-LSIF.
Although GT and DeepSAD are designed for a different problem, to the best of our knowledge, these are the state-of-the-art algorithms, which are not based on DRE.
[-= R 
-]
C ONCLUSION
We proposed a non-negative correction to the empirical BR divergence for DRE.
Using the prior knowledge of the upper bound of the density ratio, we can prevent train-loss hacking when using ﬂexible models.
In our theoretical analysis, we showed the generalization bound of the algorithm.
A D ETAILS OF E XISTING M ETHODS FOR DRE
In this section, we overview examples of DRE methods in the framework of the density ratio matching under BR divergence.
Least Squares Importance Fitting (LSIF): LSIF minimizes the squared error between a density ratio model r and the true density ratio r ∗ deﬁned as follows (Kanamori et al., 2009):
[Equation]
In the unconstrained LSIF (uLSIF) (Kanamori et al., 2009), we ignore the ﬁrst term in the above equation and estimate the density ratio by the following minimization problem:
[Equation]
(6) where R is a regularization term.
This empirical risk minimization is equal to minimizing the empirical BR divergence deﬁned in (5) with f ( t ) =
( t − 1) 2 / 2 .
Unnormalized Kullback–Leibler (UKL) Divergence and KL Importance Estimation Procedure (KLIEP):
The KL importance estimation procedure (KLIEP) is derived from the unnormalized Kullback–Leibler (UKL) divergence objective (Sugiyama et al., 2008; Nguyen et al., 2010; Tsuboi et al., 2009; Yamada & Sugiyama, 2009; Yamada et al., 2010), which uses f ( t ) =
t log( t ) − t .
Ignoring the terms which are irrelevant for the optimization, we obtain the unnormalized Kullback–Leibler (UKL) divergence objective (Nguyen et al., 2010; Sugiyama et al., 2012) as
[Equation]
Directly minimizing UKL is proposed by Nguyen et al.
The KLIEP also solves the same problem with further imposing a constraint that the ratio model r ( x ) is non-negative for all x and is normalized as
[Equation]
Then, following is the optimization criterion of KLIEP (Sugiyama et al., 2008): [Equation] r ( x ) [Equation]
[Equation]
all x .
Logistic Regression:
By using f ( t ) = log( t ) − (1 + t ) log(1 + t ) , we obtain the following BR divergence called the binary Kullback–Leibler (BKL) divergence:
[Equation]
This BR divergence is derived from a formulation based on the logistic regression (Hastie et al., 2001; Sugiyama et al., 2011b).
PU Learning with the Log Loss: Consider a binary classiﬁcation problem and let x and y ∈ {± 1 } be the feature and the label of a sample, respectively.
In PU learning, the goal is to train a classiﬁer only using positive data sampled from p ( x | y
= +1) , and unlabeled data sampled from p ( x ) in binary classiﬁcation (Elkan & Noto, 2008).
More precisely, this problem setting of PU learning is called the case-control scenario (Elkan & Noto, 2008; Niu et al., 2016).
Let G be the set of measurable functions from X to [ (cid:15), 1 − (cid:15) ] , where (cid:15) ∈ (0 , 1 / 2) is a small positive value.
For a loss function (cid:96) : R × {± 1 } → R + , du Plessis et al.
(2015) showed that the classiﬁcation risk of g ∈ G in the PU problem setting can be expressed as
[Equation]
According to Kato et al.
(2019), we can derive the following risk for DRE from the risk for PU learning (7) as follows:
[Equation]
and Kato et al.
(2019) showed that g ∗ =
arg min g ∈G BR PU ( g ) satisﬁes the following:
Proposition 1.
It holds almost everywhere that
[Equation]
where C = 1 R , D 1 = { x | Cp nu ( x ) ≥ (cid:15)p de ( x ) } , and D 2 = { x | Cp nu ( x ) ≤ (1 − (cid:15) )
p de ( x ) } .
Using this result [Equation] we deﬁne the empirical version of BR PU ( g ) as follows: [Equation]
Note that for f ( t ) in the BR divergence, we use
[Equation]
Then, we have
[Equation]
Therefore, we have
[Equation]
2 ( x i ) (cid:18)r ( x i ) + 1 − r ( x i ) (cid:19) (cid:105) − E de (cid:104) log (1 − r ( x i ))
+ Cr ( x i ) (log ( r ( x i ))
− log (1 − r ( x i )))
(cid:105) − E nu (cid:104) −
C 1 − r ( x i ) + C (log( r ( x i ))
− log(1 − r ( x i )))
+ Cr ( x i ) (cid:18)r ( x i )
+ 1 − r ( x i ) (cid:19) (cid:105)
=
E de (cid:104) − Cr ( x i ) 1 − r ( x i ) + Cr ( x i )(log( r ( x i ))
− log(1 − r ( x i )))
+ Cr ( x i ) 1 − r ( x i ) (cid:105) − E de (cid:104) log (1 − r ( x i ))
+ Cr ( x i ) (log ( r ( x i ))
− log (1 − r ( x i )))
(cid:105) − E nu (cid:104) −
C 1 − r ( x i ) + C (log( r ( x i ))
− log(1 − r ( x i )))
+
C 1 − r ( x i ) (cid:105) =
E de (cid:104) log (1 − r ( x i ))
(cid:105) − C E nu (cid:104) log
( r ( x i ))
− log(1 − r ( x i ))
(cid:105) .
Remark 2 (DRE and PU learning) .
Menon & Ong (2016) showed that minimizing a proper CPE loss is equivalent to minimizing a BR divergence to the true density ratio, and demonstrated the viability of using existing losses from one problem for the other for CPE and DRE.
Kato et al.
(2019) pointed out the relation between the PU learning and density ratio estimation and leveraged it to solve a sample selection bias problem in PU learning.
In this paper, we introduced the BR divergence with f ( t ) =
log (1 − Ct )
+
Ct (log ( Ct ) − log (1 − Ct )) , inspired by the objective function of PU learning with the log loss.
In the terminology of Menon & Ong (2016), this f results in a DRE objective without a link function .
In other words, it yields a direct DRE method.
B E XAMPLES OF ˜ f
Here, we show the examples of ˜ f such that ∂f ( t ) =
C (cid:0) ∂f ( t ) t − f ( t ) (cid:1) + ˜ f
( t ) , where ˜ f ( t ) is bounded from above, and ∂f ( t ) t − f ( t ) + A is non-negative.
First, we consider f ( t ) =
( t − 1) 2 / 2 , which results in the LSIF objective.
Because ∂f ( t ) =
t − 1 , we have
[Equation]
The function is a concave quadratic function, therefore it is upper bounded.
Note that uLSIF satisﬁes the Assumption 2 with A = 12 since ∂f ( t ) t + f ( t ) + 12 = 12 t 2 , and Assumption 3 automatically holds.
Also note that the term A is irrelevant to the optimization.
Second, we consider f ( t ) =
t log( t ) − t , which results in the UKL or KLIEP objective.
Because ∂f ( t ) = log( t ) , we have
[Equation]
We can easily conﬁrm that the function is upper bounded by taking the derivative and ﬁnding that t = 1 /C gives the maximum.
Note that UKL satisﬁes the Assumption 2 with A = 0 since ∂f ( t ) t − f ( t ) =
t log( t ) − t log( t ) + t = t , and Assumption 3 automatically holds.
Third, we consider f ( t ) =
t log( t ) − (1+ t ) log(1+ t ) , which is used for DRE based on LR or BKL.
Because ∂f ( t ) = log( t ) − log(1 + t ) , we have
[Equation]
[-− C (cid:0) log(1 + t ) (cid:1) + log( t ) − log(1 + t ) = log (cid:18) C 
-]
We can easily conﬁrm that the function is upper bounded as the terms involving t always add up to be negative.
Note that BKL satisﬁes the Assumption 2 with A = 0 since ∂f ( t ) t − f ( t ) =
(log( t ) − log(1 + t ))
t − t log( t ) + (1 + t ) log(1 + t ) =
log(1 + t ) , and Assumption 3 automatically holds.
Fourth, we consider DRE based on PULog.
By setting f ( t ) = log (1 − t )
+
Ct (log ( t ) − log (1 − t )) , we can obtain the same risk functional introduced in Kiryo et al.
Note that PULog satisﬁes the Assumption 2 with A = 0 since ∂f ( t ) t − f ( t ) =
− log(1 − t ) with t < 1 , and Assumption 3 automatically holds.
C I MPLEMENTATION
The algorithm for D3RE is described in Algorithm 1.
For training with a large amount of data, we adopt the stochastic optimization by splitting the dataset into mini-batches.
In stochastic optimization, we separate the samples into N mini-batches as ( (cid:8) x
nu i (cid:9) n nu ,j i
=1 , (cid:8) x de i (cid:9) n de ,j i =1 ) ( j = 1 , . .
N , where n nu ,j and n de ,j are the sample sizes in each mini-batch.
Then, we consider taking sample average in each mini-batch.
Let ˆ E j nu and ˆ E j de be sample averages over (cid:8) x
nu i (cid:9) n nu ,j i
=1 and (cid:8) x de i (cid:9) n de ,j i =1 .
In addition, we use regularization such as L1 and L2 penalties, denoted by R ( r ) .
For improving the performance, we heuristically employ gradient ascent from Kiryo et al.
(2017) when ˆ E de (cid:2) (cid:96) 1 ( r ( X )) (cid:3) − C ˆ E nu (cid:2) (cid:96) 1 ( r ( X )) (cid:3) becomes less than 0 , i.e., update the model in the direction that increases the term.
Let us note that our proposed method is agnostic to the optimization procedure, and other methods such as plain gradient descent can be combined with our method.
We consider that further theoretical investigation of the optimization procedure is out of scope.
D N ETWORK S TRUCTURE USED IN S ECTIONS 5 AND
We explain the structures of neural networks used in the experiments.
D. N ETWORK S TRUCTURE USED IN S ECTIONS
In Section 5, we used CIFAR-10 datasets.
The model was a convolutional net (Springenberg et al., 2015): (32 × 32 × 3)- C (3 × 6 , 3)- C (3 × 16 , 3)-128-84-1 , where the input is a 32 × 32 RGB image,
C (3 × 6 , 3) indicates that 3 channels of 3 × 6 convolutions followed by ReLU is used.
This structure has been adopted from the tutorial of Paszke et al.
D. N ETWORK S TRUCTURE USED IN S ECTIONS
Inlier-based Outlier Detection: We used the same LeNet-type CNNs proposed in Ruff et al.
In the CNNs, each convolutional module consists of a convolutional layer followed by leaky Algorithm 1 D3RE
Input: Training data (cid:8) x nu i (cid:9) n nu i =1 and (cid:8) x de i (cid:9) n de i =1 , the algorithm for stochastic optimization such as
Adam (Kingma & Ba, 2015), the learning rate γ , the regularization coefﬁcient λ . Output: A density ratio estimator ˆ r . while No stopping criterion has been met: do Create N mini-batches (cid:8)(cid:0)(cid:8) x
nu i (cid:9) n nu ,j i
=1 , (cid:8) x de i (cid:9) n de ,j i =1 (cid:1)(cid:9) N j =1 .
for i = 1 to N do if ˆ E de (cid:2) (cid:96) 1 ( r ( X )) (cid:3) − C ˆ E nu (cid:2) (cid:96) 1 ( r ( X )) (cid:3) ≥ 0 : then
Gradient decent: set gradient ∇ r (cid:8)
ˆ E j nu (cid:2) (cid:96) 2 ( r ( X )) (cid:3) + ˆ E j de (cid:2) (cid:96) 1 ( r ( X )) (cid:3) − C
ˆ E j nu (cid:2) (cid:96) 1 ( r ( X )) (cid:3) + λ R ( r ) (cid:9) .
else
Gradient ascent: set gradient ∇ r (cid:8) − ˆ E j de (cid:2) (cid:96) 1 ( r ( X )) (cid:3)
+ C ˆ E j nu (cid:2) (cid:96) 1 ( r ( X )) (cid:3) + λ R ( r ) (cid:9) .
end if
Update r with the gradient and the learning rate γ .
end for end while show the AURPC for the test data.
The bottom identical the ones displayed in
Section 5.1.
ReLU activations with leakiness α = 0 .
1 and (2 × 2) -max-pooling.
For MNIST, we employ a CNN with two modules: (32 × 32 × 3)- C (3 × 32 , 5)- C (32 × 64 , 5)- C (64 × 128 , 5)-1 .
For CIFAR-10 we employ the following architecture: (32 × 32 × 1)- C (1 × 8 , 5)- C (8 × 4 , 5)-1 with a batch normalization (Ioffe & Szegedy, 2015) after each convolutional layer.
The WRN architecture was proposed in Zagoruyko & Komodakis (2016) and it is also used in Golan & El-Yaniv (2018).
This structure improved the performance of image recognition by decreasing the depth and increasing the width of the residual networks (He et al., 2015).
We omit the detailed description of the structure here.
Covariate Shift Adaptation: We used the 5 -layer perceptron with ReLU activations.
The structure is 10000-1000-1000-1000-1000-1 .
E E XSTING M ETHODS FOR A NOMALY D ETECTION
This section introduces the existing methods for anomaly detection.
DeepSAD is a method for semisupervised anomaly detection, which tries to take advantage of labeled anomalies (Ruff et al., 2020).
GT proposed by Golan & El-Yaniv (2018) trains neural networks based on a self-labeled dataset by performing 72 geometric transformations.
The anomaly score based on GT is calculated based on the Dirichlet distribution obtained by maximum likelihood estimation using the softmax output from the trained network.
In the problem setting of the DeepSAD, we have access to a small pool of labeled samples, e.g. a subset veriﬁed by some domain expert as being normal or anomalous.
In the experimental results shown in Ruff et al.
(2020) indicate that, when we can use such samples, the DeepSAD outperforms the other methods.
However, in our experimental results, such samples are not assumed to be available, hence the method does not perform well.
The problem setting of Ruff et al.
and ours are both termed semi-supervised learning in anomaly detection, but the two settings are different.
F D ETAILS OF E XPERIMENTS
The details of experiments are shown in this section.
The description of the data is as follows:
MNIST :
The MNIST database is one of the most popular benchmark datasets for image classiﬁcation, which consists of 28 × 28 pixel handwritten digits from 0 to 9 with 60 , 000 train samples and 10 , 000 test samples (LeCun et al., 1998).
CIFAR-10 :
The CIFAR-10 dataset consists of 60 , 000 color images of size 32 × 32 fromclasses, each having 6000 .
There are 50 , 000 training images and 10 , 000 test images (Krizhevsky et al., 2012).
fashion-MNIST :
The fashion-MNIST dataset consists of 70 , 000 grayscale images of size 28 × 28 from 10 classes.
There are 60 , 000 training images and 10 , 000 test images (Xiao et al., 2017).
Amazon Review Dataset : Blitzer et al.
(2007) published the text data of Amazon review.
The data originally consists of a rating ( 0 - 5 stars) for four different genres of products in the electronic commerce site Amazon.com: books, DVDs, electronics, and kitchen appliances.
Blitzer et al.
(2007) also released the pre-processed and balanced data of the original data.
The pre-processed data consists of text data with four labels 1 , 2 , 4 , and 5 .
We map the text data into 10 , 000 dimensional data by the TF-IDF mapping with that vocabulary size.
In the experiment, for the pre-processed data, we solve the regression problem where the text data are the inputs and the ratings 1 , 2 , 4 , and 5 are the outputs.
When evaluating the performance, following Menon & Ong (2016), we calculate PD (=1-AUROUC) by regarding 4 and 5 ratings as positive labels and 1 and 2 ratings as negative labels.
F. E XPERIMENTS WITH I MAGE D ATA
We show the additional results of Section 5.
Figure 3, we show the training loss of LSIF-based methods to demonstrate the train-loss hacking phenomenon caused by the objective function without a lower bound.
In Figure 3, even though the training loss of uLSIF-NN and that of bounded uLSIF decrease more rapidly than that of nnBR-LSIF, the test AUROC score (the higher the better) either drops or fails to increase.
These graphs are the manifestations of the severe train-loss hacking in
DRE without our proposed device.
Figure 4 shows the experimental results of Sections 5.2 and 5.3.
F. E XPERIMENTS OF I NLIER - BASED O UTLIER D ETECTION
In Table 3, we show the full results of inlier-based outlier detection.
In almost all the cases, D3RE for inlier-based outlier detection outperforms the other methods.
As explained in Section E, we consider that DeepSAD does not work well because the method assumes the availability of the labeled anomaly data, which is not available in our problem setting.
Remark 3 (Benchmark Methods) .
Although GT is outperformed by our proposed method, the problem setting for the comparison is not in favor of GT as it does not assume the access to the test [Table] [Table] data.
Recently proposed methods for semi-supervised anomaly detection by Ruff et al.
(2020) did not perform well without using other side information used in Ruff et al.
On the other hand, there is no other competitive methods in this problem setting, to the best of our knowledge.
F. E XPERIMENTS OF C OVARIATE S HIFT A DAPTATION
In Table
[Table]
we show the detailed results of experiments of covariate shift adaptation.
Even when the training data and the test data follow the same distribution, the covariate shift adaptation based on
D3RE improves the mean PD.
We consider that this is because the importance weighting emphasizes the loss in the empirical higher-density regions of the test examples.
G O THER A PPLICATIONS
In this section, we explain other potential applications of the proposed method. [Table]
We consider training a model using input distribution different from the test input distribution, which is called covariate shift , (Bickel et al., 2009). To solve this problem, the density ratio has been used via importance weighting (IW) (Shimodaira, 2000; Yamada et al., 2010; Reddi et al., 2015). We use a document dataset of Amazon 4 (Blitzer et al., 2007) for multi-domain sentiment analysis (Blitzer et al., 2007).
This data consists of text reviews from four different product domains: book, electronics (elec), dvd, and kitchen.
Following Chen et al.
(2012) and Menon & Ong (2016), we transform the text data using TF-IDF to map them into the instance space X
= R 10000 (Salton &
McGill, 1986).
Each review is endowed with four labels indicating the positivity of the review, and our goal is to conduct regression for these labels.
To achieve this goal, we perform kernel ridge regression with the polynomial kernel.
We compare regression without IW (w/o IW) with regression using the density ratio estimated by PU-NN, uLSIF-NN, nnBR-LSIF, nnBR-PU, uLSIF with
Gaussian kernels (Kernel uLSIF), and KLIEP with Gaussian kernels (Kernel KLIEP).
We conduct experiments on 2 , 000 samples from one domain, and test 2 , 000 samples.
Following Menon & Ong (2016), we reduce the dimension into 100 dimensions by principal component analysis when using Kernel uLSIF, Kernel KLEIP, and regressions.
Following Menon & Ong (2016) and Cortes &
Mohri (2011), the mean and standard deviation of the pairwise disagreement (PD), 1 − AUROC , is reported.
A part of results is in Table 5.
The full results are in Appendix F.3.
The methods with D3RE show preferable performance, but the improvement is not signiﬁcant compared with the image data.
We consider this is owing to the difﬁculty of the covariate shift problem in this dataset.
f -divergence Estimation: f -divergences (Ali & Silvey, 1966; Csiszár, 1967) are the discrepancy measures of probability densities based on the density ratio, hence the proposed method can be used for their estimation.
They include the KL divergence (Kullback & Leibler, 1951), the Hellinger distance (Hellinger, 1909), and the Pearson divergence (Pearson, 1900), as examples.
Two-sample Homogeneity Test:
The purpose of a homogeneity test is to determine if two or more datasets come from the same distribution (Loevinger, 1948).
For two-sample testing, using a semiparametric f -divergence estimator with nonparametric density ratio models has been studied (Keziou., 2003; Keziou & Leoni-Aubin, 2005).
Kanamori et al.
(2010) and Sugiyama et al.
(2011a) employed direct DRE for the nonparametric DRE.
Generative Adversarial Networks: Generative adversarial networks (GANs) are successful deep generative models, which learns to generate new data with the same distribution as the training data
Goodfellow et al.
Various GAN methods have been proposed, amongst which Nowozin et al.
(2016) proposed f-GAN , which minimizes the variational estimate of f -divergence.
Uehara et al.
(2016) extended the idea of Nowozin et al.
(2016) to use BR divergence minimization for
DRE.
The estimator proposed in this paper also has a potential to improve the method of Uehara et al.
Average Treatment Effect Estimation and Off-policy Evaluation: One of the goals in causal inference is to estimate the expected treatment effect, which is a counterfactual value.
Therefore, following the causality formulated by Rubin (1974), we consider estimating the average treatment effect (ATE).
Recently, from machine learning community, off-policy evaluation (OPE) is also proposed, which is a generalization of ATE (Dudík et al., 2011; Imai & Ratkovic, 2014; Wang et al., 2017; Narita et al., 2019; Bibaut et al., 2019; Kallus & Uehara, 2019; Oberst & Sontag, 2019). OPE has garnered attention in applications such as advertisement design selection, personalized medicine, search engines, and recommendation systems (Beygelzimer & Langford, 2009; Li et al., 2010; Athey & Wager, 2017). The problem in ATE estimation and OPE is sample selection bias.
For removing the bias, the density ratio has a critical role. An idea of using the density ratio dates back to (Rosenbaum, 1987), which proposed an inverse probability weighting (IPW) method (Horvitz & Thompson, 1952) for ATE estimation.
In the IPE method, we approximate the parameter of interest with the sample average with inverse assignment probability of treatment (action), which is also called propensity score. Here, it is known that using the true assignment probability yields higher variance than the case where we use an estimated assignment probability even if we know the true value (Hirano et al., 2003; Henmi & Eguchi, 2004; Henmi et al., 2007). This property can be explained from the viewpoint of semiparametric efﬁciency (Bickel et al., 1998). While the asymptotic variance of the
IPW estimator with an estimated propensity score can achieve the efﬁciency bound, that of the IPW estimator with the true propensity score does not.
By extending the IPW estimator, more robust ATE estimators are proposed by Rosenbaum (1983), which is known as a doubly robust (DR) estimator.
The doubly robust estimator is not only robust to model misspeciﬁcation but also useful in showing asymptotic normality.
In particular, when using the density ratio and the other nuisance parameters estimated from the machine learning method, the conventional IPW and DR estimators do not have asymptotic normality (Chernozhukov et al., 2018).
This is because the nuisance estimators do not satisfy Donsker’s condition, which is required for showing the asymptotic normality of semiparametric models.
However, by using the sample splitting method proposed by Klaassen (1987), Zheng & van der Laan (2011), and Chernozhukov et al.
(2018), we can show the asymptotic normality when using the DR estimator.
Note that for the
IPE estimator, we cannot show the asymptotic normality even if using sample-splitting.
When using the IPW and DR estimator, we often consider a two-stage approach: in the ﬁrst stage, we estimate the nuisance parameters, including the density ratio; in the second stage, we construct a semiparametric ATE estimator including the ﬁrst-stage nuisance estimators.
This is also called two-step generalized method of moments (GMM).
On the other hand, from the causal inference community, there are also weighting-based covariate balancing methods (Qin & Zhang, 2007; Tan, 2010; Hainmueller, 2012; Imai & Ratkovic, 2014).
In particular, Imai & Ratkovic (2014) proposed a covariate balancing propensity score (CBPS), which simultaneously estimates the density ratio and ATE.
The idea of CBPS is to construct moment conditions, including the density ratios, and estimate the ATE and density ratio via GMM simultaneously.
Although the asymptotic property of the CBPS is the same as other conventional estimators, existing empirical studies report that
the
CBPS outperforms them (Wyss et al., 2014).
Readers may feel that the CBPS has a close relationship with the direct DRE.
However, Imai &
Ratkovic (2014) is less relevant to the context of the direct DRE.
From the DRE perspective, the method of Imai & Ratkovic (2014) boils down to the method of Gretton et al.
(2009), which proposed direct DRE through moment matching.
The research motivation of Imai & Ratkovic (2014) is to estimate the ATE with estimating a nuisance density ratio estimator simultaneously.
Therefore, the density ratio itself is nuisance parameter; that is, they are not interested in the estimation performance of the density ratio.
Under their motivation, they are interested in a density ratio estimator satisfying the moment condition for estimating the ATE, not in a density ratio estimator predicting the true density ratio well.
In addition, while the direct DRE method adopts linear-in-parameter models and neural networks (our work), it is not appropriate to use those methods with the CBPS (Chernozhukov et al., 2018).
This is because the density ratio estimator does not satisfy Donsker’s condition.
Even naive Ridge and Lasso regression estimators do not satisfy the Donsker’s condition.
Therefore, when using machine learning methods for estimating the density ratio, we cannot show asymptotic normality of an ATE estimator obtained by the CBPS; therefore, we need to use the sample-splitting method by (Chernozhukov et al., 2018).
This means that when using the CBPS, we can only use a naive parametric linear model without regularization or classic nonparametric kernel regression.
Recently, for GMM with such non-Donsker nuisance estimators, Chernozhukov et al.
(2016) also proposed a new GMM method based on the conventional two-step approach. For these reasons, the CBPS is less relevant to the direct DRE context. Off-policy Evaluation with External Validity: By the problem setting of combining causal inference and domain adaptation, Kato et al. (2020) recently proposed using covariate shift adaptation to solve the external validity problem in OPE, i.e., the case that the distribution of covariates is the same between the historical and evaluation data (Cole & Stuart, 2010; Pearl & Bareinboim, 2014).
Change Point Detection: The methods for change-point detection try to detect abrupt changes in time-series data (Basseville & Nikiforov, 1993; Brodsky & Darkhovsky, 1993; Gustafsson, 2000;
Nguyen et al., 2011). There are two types of problem settings in change-point detection, namely the real-time detection (Adams, 2007; Garnett et al., 2009; Paquet, 2007) and the retrospective detection (Basseville & Nikiforov, 1993; Yamanishi & Takeuchi, 2002).
In retrospective detection, which requires longer reaction periods, Liu et al.
(2012) proposed using techniques of direct DRE.
Whereas the existing methods rely on linear-in-parameter models, our proposed method enables us to employ more complex models for change point detection.
Similarity-based Sentiment Analysis: Kato (2019) used the density ratio estimated from PU learning for sentiment analysis of text data based on similarity.
H G ENERALIZATION ERROR BOUND
The generalization error bound can be proved by building upon the proof techniques in Kiryo et al.
(2017); Lu et al.
Notations for the Theoretical Analysis: We denote the set of real values by R and that of positive integers by N .
Let X ⊂ R d .
Let p nu ( x ) and p de ( x ) be probability density functions over X , and assume that the density ratio r ∗ ( x ) :
= p nu ( x ) p de ( x ) is existent and bounded: R := (cid:107) r ∗ (cid:107) ∞ < ∞ .
Assume
< C < 1 R .
Since R ≥ 1 (because 1 = (cid:82) p de ( x ) r ∗ ( x ) dx ≤ 1 · (cid:107) r ∗ (cid:107) ∞ ), we have C ∈ (0 , 1] and hence p mod :
= p de − Cp nu > 0 .
Problem Setup: Let the hypothesis class of density ratio be H ⊂ { r : R D
→
( b r , B r )
=:
I r } , where 0 ≤ b r < R < B r .
Let f : I r
→ R be a twice continuously-differentiable convex function with a bounded derivative.
Deﬁne ˜ f by ∂f ( t ) =
C ( ∂f ( t ) t − f ( t ))
+ ˜ f ( t ) , where ∂f is the derivative of f continuously extended to 0 and B r .
Recall the deﬁnitions (cid:96) 1 ( t ) :=
∂f ( t ) t − f ( t ) + A , (cid:96) 2 ( t ) :
= − ˜ f
( t ) , and
[Equation]
where we denoted ˆ E mod = ˆ E de − C
ˆ E nu and ρ is a consistent correction function with Lipschitz constant L ρ (Deﬁnition 1).
Remark 4.
The true density ratio r ∗ minimizes BR f .
Deﬁnition 1 (Consistent correction function Lu et al.
A function f : R → R is called a consistent correction function if it is Lipschitz continuous, non-negative and f ( x ) =
x for all x ≥ 0 .
Deﬁnition 2 (Rademacher complexity) .
Given n ∈ N and a distribution p , deﬁne the Rademacher complexity R pn ( H ) of a function class H as
[Equation]
where { σ i } ni =1 are Rademacher variables (i.e., independent variables following the uniform distribution over {− 1 , +1 } ) and { X i } ni =1 i.i.d.
∼ p .
The theorem in the paper is a special case of Theorem 2 with ρ ( · ) :
= max { 0 , ·} (in which case L ρ = 1 ) and Theorem 3.
Theorem 2 (Generalization error bound) .
Assume that B (cid:96) :
= sup t ∈
I r { max {| (cid:96) 1 ( t ) | , | (cid:96) 2 ( t ) |}} < ∞ .
Assume (cid:96) 1 is L (cid:96) 1 -Lipschitz and (cid:96) 2 is L (cid:96) 2 -Lipschitz.
Assume that there exists an empirical risk minimizer ˆ r ∈ arg min r ∈H (cid:92)
nnBR f ( r ) and a population risk minimizer ¯ r ∈ arg min r ∈H BR f
( r ) .
Also assume inf r ∈H E ˆ E mod (cid:96) 1 ( r ( X )) > 0 and that ( ρ − Id) is ( L ρ − Id ) -Lipschitz.
Then for any δ ∈ (0 , 1) , with probability at least 1 − δ , we have
[Equation]
where Φ ( C,f,ρ ) ( n nu , n de ) is deﬁned as in Lemma 3.
Proof.
Since ˆ r minimizes (cid:92) nnBR f , we have
[Equation]
We apply McDiarmid’s inequality McDiarmid (1989); Mohri et al.
(2018) to the maximal deviation term.
The absolute value of the difference caused by altering one data point in the maximal deviation term is bounded from above by 2 B (cid:96) L ρ n de if the altered point is a sample from p de and 2 B (cid:96) 1+ L ρ C n nu if it is from p nu .
Therefore, McDiarmid’s inequality implies, with probability at least 1 − δ , that we have [Equation] [Table] [Equation]
log
[Equation]
δ . Applying Lemma 2 to the expected maximal deviation term and Lemma 3 to the bias term, we obtain the assertion.
The following lemma generalizes the symmetrization lemmas proved in Kiryo et al.
(2017) and Lu et al.
Lemma 2 (Symmetrization under Lipschitz-continuous modiﬁcation) .
Let 0 ≤ a < b , J ∈ N , and { K j } Jj =1 ⊂ N .
Given i.i.d.
samples D ( j,k ) :
= { X i } n ( j,k ) i
= each from a distribution p ( j,k ) over X , consider a stochastic process ˆ S indexed by F ⊂ ( a, b ) X of the form
[Equation]
where each ρ j is a L ρ j -Lipschitz function on R , (cid:96) ( j,k ) is a L (cid:96) ( j,k ) -Lipschitz function on ( a, b ) , and ˆ E ( i,j ) denotes the expectation with respect to the empirical measure of D ( j,k ) .
Denote S ( f ) :=
E ˆ S ( f ) where E is the expectation with respect to the product measure of {D ( j,k ) } ( j,k ) .
Here, the index j denotes the grouping of terms due to ρ j , and k denotes each sample average term.
Then we have
[Equation]
Proof.
First, we consider a continuous extension of (cid:96) ( j,k ) deﬁned on ( a, b ) to [0 , b ) .
Since the functions in F take values only in ( a, b ) , this extension can be performed without affecting the values of ˆ S ( f ) or S ( f ) .
We extend the function by deﬁning the values for x ∈ [0 , a ] as (cid:96) ( j,k ) ( x ) :=
lim x (cid:48) ↓ a (cid:96) ( j,k ) ( x (cid:48) ) , where the right-hand side is guaranteed to exist since (cid:96) ( j,k ) is Lipschitz continuous hence uniformly continuous.
Then, (cid:96) ( j,k ) remains a L ρ j -Lipschitz continuous function on [0 , b ) .
Now we perform symmetrization (Vapnik, 1998), deal with ρ j ’s, and then bound the symmetrized process by Rademacher complexity.
Denoting independent copies of { X ( j,k ) } by { X (gh) j,k } ( j,k ) and the corresponding expectations as well as the sample averages with (gh) ,
[Equation]
f ( X )))
− E (gh) ρ j (
K j (cid:88) k
= ˆ E (
[Table]
)( j,k ) (cid:96) ( j
[Equation]
k ) ( f ( X (
[Table]
) ))) | ≤
J (cid:88) j
=
[Equation]
EE (
[Equation]
[Equation]
f
[Equation]
K j (cid:88) k
= ˆ E ( i,j ) (cid:96) ( j,k ) ( f ( X )))
− ρ j (
K j (cid:88) k
= ˆ E (
[Table]
)( j,
[Table]
) (cid:96) ( j,
[Table]
) ( f (
[Table]
[Table]
) ))) | ≤
[Equation]
(cid:88)
j =
L ρ j
[Equation]
EE
[Equation]
f
[Equation]
|
[Equation]
[Table]
[Equation]
[Table]
) (cid:96)
[Table]
j
[Equation]
[Table]
f
[Table]
)) −
[Table]
E
[Table]
[Table]
(cid:96) (
[Table]
[Table]
[Table]
) )) | =
J
[Equation]
j =
L
[Equation]
j
[Equation]
(cid:88) k
=
[Equation]
EE
[Equation]
f
[Equation]
( i,j ) ( (cid:96) ( j,k ) ( f ( X )) − (cid:96) ( j,k ) (0))
− ˆ E (gh)( j,k ) ( (cid:96) ( j,k ) ( f ( X (gh) ))
− (cid:96) ( j,k ) (0))
| ≤
J (cid:88) j
=
L ρ j
K j (cid:88) k
= (cid:0)
[Equation]
n ( j
[Equation]
( j
[Equation]
( { (cid:96) ( j,k ) ◦ f
[Equation]
(cid:96) ( j,k ) (0) : f ∈ F} ) (cid:1) ≤
J (cid:88) j
= L ρ [Equation] where we applied Talagrand’s contraction lemma for two-sided Rademacher complexity (Ledoux &
Talagrand, 1991; Bartlett & Mendelson, 2001) with respect to ( t (cid:55)→ (cid:96) ( j,k ) ( t ) − (cid:96) ( j,k ) (0)) in the last inequality.
Lemma 3 (Bias due to risk correction) .
Assume inf r ∈H E ˆ E mod (cid:96) 1 ( r ( X )) > 0 and that ( ρ − Id) is ( L ρ − Id ) -Lipschitz on R .
There exists α > 0 such that
[Equation]
Remark 5.
Note that we already have p mod ≥
and (cid:96) 1 ≥ 0 and hence inf r ∈H E ˆ E mod (cid:96) 1 ( r ( X )) ≥ 0 .
Therefore, the assumption of Lemma 3 is essentially referring to the strict positivity of the inﬁmum.
Here, E ˆ E mod and P ( · ) denote the expectation and the probability with respect to the joint distribution of the samples included in ˆ E mod .
Proof.
Fix an arbitrary r ∈ H .
We have
[Equation]
where 1 {·} denotes the indicator function, and we used | ˆ E mod (cid:96)
[Equation]
( r ( X
[Equation]
| ≤ (1 + C ) B (cid:96) .
Further, we have
[Equation]
where Id denotes the identity function.
On the other hand, since inf r ∈H E ˆ E mod (cid:96) 1 ( r ( X )) > 0 is assumed, there exists α > 0 such that for any r ∈ H , E ˆ E mod (cid:96) 1 ( r ( X ))
> α .
Therefore, denoting the support of a function by supp( · ) ,
[Equation]
holds.
Now we apply McDiarmid’s inequality to the right-most quantity.
The absolute difference caused by altering one data point in ˆ E mod (cid:96) 1 ( r ( X )) is bounded by B (cid:96) n de if the change is in a sample from p de and CB (cid:96) n nu otherwise.
Therefore, McDiarmid’s inequality implies
[Equation]
Theorem 3 (Generalization error bound) .
Under Assumption 3, for any δ ∈ (0 , 1) , with probability at least 1 − δ , we have BR f
(ˆ r ) − BR f
(¯ r )
[Equation]
1 R p de n de ( H ) + 8( CL (cid:96) 1 +
L (cid:96) 2 ) R p nu n nu ( H ) + 2Φ fC ( n nu , n de ) + B (cid:96) (cid:114) 8 (cid:16) 1 n de + (1+ C ) 2 n nu (cid:17) log 1 δ , where Φ fC ( n nu , n de ) :
= (1 +
C ) B (cid:96) exp (cid:16) − 2 α 2 ( B (cid:96) 2 /n de )+( C 2 B (cid:96) 2 /n nu ) (cid:17) and α > 0 is a constant determined in the proof of Lemmain Appendix H.
I R ADEMACHER C OMPLEXITY B OUND
The following lemma provides an upper-bound on the Rademacher complexity for multi-layer perceptron models in terms of the Frobenius norms of the parameter matrices.
Alternatively, other approaches to bound the Rademacher complexity can be employed.
The assertion of the lemma follows immediately from the proof of Theorem 1 of Golowich et al.
(2019) after a slight modiﬁcation to incorporate the absolute value function in the deﬁnition of Rademacher complexity.
Lemma 4 (Rademacher complexity bound (Golowich et al., 2019, Theorem 1)) .
Assume the distribution p has a bounded support: B p := sup x ∈ supp( p ) (cid:107) x (cid:107) < ∞ .
Let H be the class of real-valued networks of depth L over the domain X , where each parameter matrix W j has Frobenius norm at most B W j ≥ 0 , and with 1 -Lipschitz activation functions ϕ j which are positive-homogeneous (i.e., ϕ j is applied element-wise and ϕ j ( αt ) = αϕ j ( t ) for all α ≥ 0 ).
Then
[Equation]
Proof.
The assertion immediately follows once we modify the beginning of the proof of Theoremby introducing the absolute value function inside the supremum of the Rademacher complexity as
[Equation]
for λ > 0 .
The rest of the proof is identical to that of Theorem 1 of Golowich et al.
J S TRONG CONVEXITY AND L 2 ERROR BOUND
A generalization error bound in terms of BR f can be converted to that of an L 2 distance when the true density ratio and the density ratio model are square-integrable and f is strongly convex.
Lemma 5 ( L 2 distance bound) .
Let H :
=
{ r : X → ( b r , B r ) =: I r | (cid:82) | r ( x ) | 2 d x < ∞} and assume r ∗ ∈ H .
If inf t ∈ I r f (cid:48)(cid:48) ( t ) > 0 , then there exists µ > 0 such that for all r ∈ H ,
[Equation]
holds.
Proof.
Since µ := inf t ∈ I r f (cid:48)(cid:48) ( t ) > 0 , the function f is µ -strongly convex.
By the deﬁnition of strong convexity,
[Equation]
Lemma 6 ( (cid:96) 2 distance bound) .
Fix r ∈ H .
Given n samples { x i } ni =1 from p de , with probability at least 1 − δ , we have
[Equation]
Proof.
The assertion follows from McDiarmid’s inequality after noting that altering one sample results in an absolute change bounded by 1 n (2 R ) 2 .