Probabilities of Causation: Adequate Size of Experimental and Observational Samples
Abstract
The probabilities of causation are commonly used to solve decision-making problems.
Tian and Pearl derived sharp bounds for the probability of necessity and sufficiency (PNS), the probability of sufficiency (PS), and the probability of necessity (PN) using experimental and observational data.
The assumption is that one is in possession of a large enough sample to permit an accurate estimation of the experimental and observational distributions.
In this study, we present a method for determining the sample size needed for such estimation, when a given confidence interval (CI) is specified.
We further show by simulation that the proposed sample size delivered stable estimations of the bounds of PNS.
Introduction
The probabilities of causation are widely used in many areas of industry, marketing, and health science, to solve decision-making problems.
For example, Li and Pearl [6, 8] proposed the “benefit function”, a linear combination of the probabilities of causation, which is the payoff/cost associated with selecting an individual with given characteristics to identify a set of individuals who are most likely to exhibit a desired mode of behavior.
Mueller and Pearl [11] demonstrated, for example, that the probabilities of causation should be considered in personalized decision-making.
Li et al.
[7] showed that the probabilities of causation can improve the accuracy of machine learning algorithms.
Pearl [14] first used the structural causal model (SCM) to defined three binary probabilities of causation (i.e., PNS, PN, and PS) [3, 4, 15].
Tian and Pearl [16] then used experimental and observational data to bound those three probabilities of causation.
Li and Pearl [8, 10] established formal proof of those bounds.
Mueller, Li, and Pearl [12] proposed narrowing the bounds of PNS using covariate information and the causal structure.
Dawid et al.
[2] also proposed using covariate information to narrow the bounds of PN.
Li and Pearl [5] recently established the theoretical bounds of nonbinary probabilities of causation.
All the abovementioned works are asymptotic (i.e., assuming a adequately large sample size to estimate the experimental and observational distributions).
The proposed results in those works are relationships between the experimental and observational distributions and the probabilities of causation.
However, the adequate sample size for obtaining those probabilities of causation remains unclear, thereby creating a barrier between the theoretical results and the real-world applications.
Consider the following motivating example: a mobile carrier that wants to identify customers who are likely to discontinue their services within the next quarter based on customer characteristics (company management has access to user data, such as income, age, usage, and monthly payments).
The carrier will then offer these customers a special renewal deal to dissuade them from discontinuing their services and to increase their service renewal rate.
These offers provide considerable discounts to the customers, and the management prefers that these offers be made only to those customers who would continue to use the service if and only if they receive the offer.
The manager decides to use Li and Pearl’s unit selection model [8] but is unsure how many experimental and observational samples are required.
Are 1000 experimental and 1000 observational samples adequate to bound the benefit function such that the error of the bounds are within 0 .
We present an assessment of the “adequate" of the sample size in the sense of CI in this study.
We would then be able to answer the question, “How many samples are adequate to estimate the probability of causation?"
as “This amount of samples is adequate to obtain the bounds of the probability of causation in 95% CIs with margin of errors of 0 .
" The probabilities of causation in most cases are not identifiable; therefore, the CIs are for the bounds of the probabilities of causation in such cases.
Preliminaries
We review the definitions for the three aspects of binary causation in this section, as defined in [14].
We use the language of counterfactuals in SCM, as defined in [3, 4].
We use Y x =
y to denote the counterfactual sentence “Variable Y would have the value y , had X been x ".
For the rest of the paper, we use y x to denote the event Y x =
y , y x ′
to denote the event Y x ′ =
y , y ′ x to denote the event Y x = y ′ , and y ′ x ′ to denote the event Y x ′ =
y ′ .
We assume that experimental distribution will be summarized in the form of the causal effects such as P ( y x ) and observational distribution will be summarized in the form of the joint probability function such as P ( x, y ) .
If neither variable is specified, variable X represents treatment and variable Y represents effect.
The following are three prominent probabilities of causation:
Definition 1 (Probability of necessity (PN)) .
Let X and Y be two binary variables in a causal model
M , let x and y stand for the propositions X = true and Y = true , respectively, and x ′ and y ′ for their complements.
The probability of necessity is defined as the expression [14]
[Equation]
Definition 2 (Probability of sufficiency (PS)) .
[Equation] [Equation]
Definition 3 (Probability of necessity and sufficiency (PNS)) .
[Equation]
PNS denotes for the probability that y would respond to x both ways, and therefore measures both the sufficiency and necessity of x to produce y .
Tian and Pearl [16] used Balke’s program [1] to provide tight bounds for PNS, PN, and PS.
PNS has the following tight bounds:
[Equation]
Note that we omitted the bounds of PN and PS because this study focuses primarily on the adequate sample size for estimating the bounds of PNS, it is simple to extend to other probabilities of causation.
[Figure]
Main Results
The bounds of PNS are the linear combination of the experimental distributions P ( y x ) , P ( y x ′ ) and the observational distributions P ( x, y ) , P ( x, y ′ ) , P ( x ′ , y ) , P ( x ′ , y ′ ) from Equation 1.
Therefore, if we can obtain the CI of each of these distributions, then we can obtain the CIs of the bounds of the
PNS.
Let R be a random variable such that R = 1 if the event y x occurs and R =
if the event y ′ x occurs, then it is clear that R ∼ Bernoulli ( P ( y x )) .
Therefore, if we use the frequentist to estimate the experimental and observational distributions, we have the following theorem and corollary (the detailed proof are in the appendix):
Theorem 4.
Given m experimental samples and n observational samples, if the frequentist is used to estimate the experimental and observational distributions, then the margin error of the bounds of PNS in 1 − α confidence interval is at most z 1 − α/ 2 ( (cid:113) 1 m + (cid:113) 1 n ) , where z 1 − α/ 2 can be found on z-table of standard normal distribution.
Corollary 5.
If the frequentist is used to estimate the experimental and observational distribution, to obtain the at most 0 .
05 margin error of the bounds of PNS in 95% confidence interval, we need m experimental samples and n observational samples, where ( (cid:113)
1 m + (cid:113) 1 n ) ≤ 5 / 196 .
More specifically, if m = n , 6147 experimental and 6147 observational samples are adequate to obtain the at most 0 .
05 margin error of the bounds of PNS in 95% confidence interval.
This amount of samples ensures that the margin errors of the bounds of PNS in 95% CI are no more than 0 .
However, in practice, we usually do not need this amount of samples because there is only one term (i.e., P ( y x ) − P
( y x ′ )
+ P ( x, y ′ )
+ P ( x ′ , y ) ) in the PNS bounds of Equation 1, which consists of four distributions.
The terms such as P ( y x ) only require 385 experimental and 385 observational samples to obtain the at most 0 .
05 margin error of the 95% CI, and the terms such as P ( y x ) − P ( y x ′ ) only require 1537 experimental and 1537 observational samples to obtain the at most 0 .
05 margin error of the 95% CI.
We will illustrate the real errors of the estimations in simulated studies in the following section.
Simulation Results
Here, we present simulated studies to show that the proposed number of experimental and observational samples are adequate to obtain the desired margin errors of the bounds of PNS using two
SCMs.
4.1 Causal Model
To estimate the margin errors of the bounds, we must first understand the data generation process to have true experimental and observational distributions.
The two models we are using are shown in Figure 1 (two models have the same causal graph, but with different coefficients in SCMs; the generation method of the models is in the appendix), where X is a binary treatment, Y is a binary effect, and Z is a set of 20 independent confounders (say Z 1 , ..., Z 20 ).
The structural equations are as follow (for simplicity reason, we let x = 1 , x ′ = 0 , and y = 1 , y ′ = 0 ):
Z i =
U
Z i for i ∈ { 1 , ..., 20 } ,
X =
f X ( M X , U X ) =
(cid:26)if M X + U X > 0 .
5 , otherwize , (cid:27)
Y =
f Y ( X, M Y , U Y ) =
(cid:26)if 0 < CX + M Y
+ U Y <
1 or 1 < CX + M Y
+ U Y < 2 , otherwize , (cid:27) where, U Z i , U X , U Y are binary exogenous variables with Bernoulli distributions,
C is a constant, and M X , M Y are linear combinatations of Z 1 , ..., Z 20 .
The value of C, M X , M Y and the distributions of U X , U Y , U Z 1 , ..., U Z 20 for the two models are provided in the appendix.
4.2 Informer Data
Based on the model from the previous section, individuals are determined by 22 binary exogenous variables.
From the viewpoint of the informer, we must know the actual experimental (i.e.,
P ( y x ) , P ( y x ′ ) ) and observational distributions (i.e., P ( x, y ) , P ( x, y ′ ) , P ( x ′ , y ) ) to compute the true
PNS bounds for the comparison purpose.
Since the structural equations and the exogenous variable distributions are explicit, the experimental and observational distributions are as follows (we illustrated two terms, see the appendix for full details):
[Equation]
× × f X
( M X , u X ) f Y ( f X ( M X , u X ) , M Y , u Y ) ,
We can then obtain the informer view of the bounds of PNS using Equation 1 and the above observational and experimental distributions.
4.3 Experimental Sample
Here is how we prepared the finite experimental samples.
Again, an individual is determined by U X , U Y , and U Z 1 , ..., U Z 20 .
Therefore, we first generated ( U X , U Y , U Z 1 , ..., U Z 20 ) at random using their distributions; then we generated X at random using Bernoulli (0 . 5) ; the value of Y is then f Y ( X, M Y , U Y ) .
We then collect a experimental sample ( X, Y ) .
P ( y x ) is then estimated as ˆ P ( y x ) =
[Table]
P ( y x
[Table]
) is then estimated as ˆ P ( y x
[Table]
) =
[Table]
4.4 Observational Sample
Similarly to experimental sample, we first generated ( U X , U Y , U Z 1 , ..., U Z 20 ) at random using their distributions; the value of X is then f X ( M X , U X ) ; the value of Y is then f Y ( X, M Y , U Y ) .
Then, we collect a observational sample ( X, Y ) .
P ( x, y ) , P ( x, y ′ ) , P ( x ′ , y ) estimates are then ˆ P ( x, y )
=
the number
of observational
samples (1,1) the number
of
total
observational
samples , ˆ P ( x, y ′ )
=
the
number of observational samples
the
number of
total
observational
samples , and ˆ P ( x ′ , y )
=
the number of observational samples
1) the
number of total observational
samples .
4.5 Simulation Results
For the two SCMs, we randomly generated 385 , 1537 , and 6147 experimental and 385 , 1537 , and 6147 observational samples respectively for 1000 times as described in the previous sections.
The true bounds of the PNS obtained from true distributions were then compared to the estimated bounds obtained from finite experimental and observational samples.
Figures 2 and 3 show the comparison results.
We see that when the sample size is 385 or even 1537 , the bounds are still in a relatively large unstable situation, with some of the estimated lower bounds being even larger than the true PNS
[Figure] [Figure]
value.
When the sample size reached 6147 , the estimations appear very stable and always include the true PNS value.
We also plot the sample size v.s. the average error of estimations (i.e., | estimated bounds − true bounds | / 1000 ) as shown in Figure 4.
The average errors improved significantly before 1500 sample sizes, therefore, 1537 should be a good sample size to use in practice even though the adequate sample size from the theorem is 6147 .
Besides, starting at roughly 300 sample size, the average errors are less than the expected margin error of 0 .
This is because the proposed theorem is of adequate size and considers the worst-case scenario.
Discussion
We demonstrated how to evaluate the precision of the probability of causation estimation using PNS.
It is simple to extend to nonbinary probabilities of causation as defined in [5].
In such cases, we can still define a random variable, R , that obeys the Bernoulli distribution and whose parameter is the experimental distribution P ( y x ) .
This work can be extended to unit selection problems [6, 8], because the unit selection problems are linear combination of the probabilities of causation.
This work can be extended to estimate experimental distributions [9, 13] using observational data.
Conclusion
We showed how to assess the accuracy of a PNS bounds estimation given by a finite number of experimental and observational samples, as well as how many experimental and observational samples are adequate to achieve a certain accuracy level of a PNS bounds estimation.
Both theoretical and experimental results are provided.
The proposed method can also be extended to general probabilities of causation.
[Figure]
Acknowledgements
This research was supported in parts by grants from the National Science Foundation [#IISand #IIS-2231798], Office of Naval Research [#N00014-21-1-2351] and Toyota Research Institute of North America [#PO-000897]
A Appendix A.1 Proof of Theorem
Proof.
Let R be a random variable, such that R = 1 if the event y x occurs and R =
if the event y ′ x occrus, then R ∼ Bernoulli ( P ( y x )) .
Let R =
( R 1 , ..., R m ) be m random experimental samples of R , and
S m = (cid:80) mi
=1 R i .
Therefore, S m ∼ binomial ( m, P ( y x )) .
By Central limit theorem, we have, S m − mp √
[Equation]
0 , 1) as m
[Equation]
∞ .
We also have ˆ P ( y x ) =
S m /m
P
[Equation]
P ( y x ) ,
Thus, we have, (cid:113) P ( y x )(1 − P ( y x ))
ˆ P ( y x )(1 − ˆ P ( y x ))
P −→ 1 and ˆ P ( y x ) − P ( y x ) √
P ( y x )(1 − P ( y x )) /m
L −→ N (0 , 1) .
Therefore, we have, ˆ P ( y x ) − P ( y x ) √ ˆ P ( y x )(1 − ˆ P ( y x )) /m
[Equation]
N (0 , 1) .
We know that P ( | ˆ P ( y x ) − P ( y x ) √ ˆ
[Equation]
[Equation] [Equation]
P
[Equation]
y
[Equation]
, we have the confidence interval of ˆ P ( y x ) is [
[Equation]
The margin of error of ˆ P ( y x ) is z 1 − α/ 2 · (cid:113) ˆ P ( y x )(1 − ˆ P ( y x ))
/m , denoted as err
[Equation]
We then have,
[Equation]
Similarly, we have,
[Equation]
We also have the fact that err ( A + B ) ≤ err ( A ) + err ( B ) and err ( A − B ) ≤ err ( A ) + err ( B ) , plug into Equation 1, we obtain that err ( bounds of PNS ) ≤ z 1 − α/ 2 ( (cid:113) 1 m + (cid:113) 1 n ) .
A.2 Proof of Corollary
Proof.
Let α = 0 .
We need z 0 .
975 ( (cid:113) 1 m + (cid:113) 1 n ) ≤ 0 .
Simply plug z 0 .
975 = 1 .
in, we have, ( (cid:113) 1 m + (cid:113) 1 n ) ≤ 5 / 196 .
And if m = n , we have, (cid:113) 1 m
[Equation]
Therefore, m ≥ 6146 .
A.3 Two Causal Models
First, M X and M Y are both linear combinatation of Z 1 , ..., Z 20 .
So for each model, we need to generate 20 coefficients for M X , 20 coefficients for M Y , a constant C , and 22 Bernoulli distribution parameters for U X , U Y , U Z 1 , ..., U Z 20 .
We generated the coeefficients for M X , M Y and C uniformly from [ − 1 , 1] , and generated the Bernoulli distribution parameters unoformly from [0 , 1] .
The detailed two models are as follow:
Z i =
U
Z i for i ∈ { 1 , ..., 20 } ,
X =
f X ( M X , U X ) =
(cid:26)if M X + U X > 0 .
5 , otherwize , (cid:27)
Y =
f Y ( X, M Y , U Y ) =
(cid:26)if 0 < CX + M Y
+ U Y <
1 or 1 < CX + M Y
+ U Y < 2 , otherwize , (cid:27) where, U Z i , U X , U Y are binary exogenous variables with Bernoulli distributions.
s.t.,
U Z 1 ∼ Bernoulli (0 . 352913861526) , U Z 2 ∼ Bernoulli (0 . 460995855543) ,
U Z 3 ∼ Bernoulli (0 . 331702473392) , U Z 4 ∼ Bernoulli (0 . 885505026779) ,
U Z 5 ∼ Bernoulli (0 . 017026872706) , U Z 6 ∼ Bernoulli (0 . 380772701708) ,
U Z 7 ∼ Bernoulli (0 . 028092602705) , U Z 8 ∼ Bernoulli (0 . 220819399962) ,
U Z 9 ∼ Bernoulli (0 . 617742227477) , U Z 10 ∼ Bernoulli (0 . 981975046713) ,
U Z 11 ∼ Bernoulli (0 . 142042291381) , U Z 12 ∼ Bernoulli (0 . 833602592350) ,
U Z 13 ∼ Bernoulli (0 . 882938907115) , U Z 14 ∼ Bernoulli (0 . 542143191999) ,
U Z 15 ∼ Bernoulli (0 . 085023436884) , U Z 16 ∼ Bernoulli (0 . 645357252864) ,
U Z 17 ∼ Bernoulli (0 . 863787135134) , U Z 18 ∼ Bernoulli (0 . 460539711624) ,
U Z 19 ∼ Bernoulli (0 . 314014079207) , U Z 20 ∼ Bernoulli (0 . 685879388218) ,
U X ∼ Bernoulli (0 . 601680857267) , U Y ∼ Bernoulli (0 . 497668975278) ,
C = − 0 .
M X =
[ Z 1 Z 2 ...
Z 20 ] ×     0 .− 0 .− 0 .0 .0 .− 0 .0 .− 0 .0 .− 0 .0 .0 .− 0 .0 .− 0 .− 0 .0 .− 0 .0 .− 0 .
   , M Y =
[ Z 1 Z 2 ...
Z 20 ] ×     − 0 .0 .
7599671361470 .0 .−
.0 .−
0 .−
.− 0 .− 0 .− 0 .− 0 .0 .
  
Z i =
U
Z i for i ∈ { 1 , ..., 20 } ,
X =
f X ( M X , U X ) =
(cid:26)if M X + U X > 0 .
5 , otherwize , (cid:27)
Y =
f Y ( X, M Y , U Y ) =
(cid:26)if 0 < CX + M Y
+ U Y <
1 or 1 < CX + M Y
+ U Y < 2 , otherwize , (cid:27) where, U Z i , U X , U Y are binary exogenous variables with Bernoulli distributions.
s.t.,
U Z 1 ∼ Bernoulli (0 . 524110233482) , U Z 2 ∼ Bernoulli (0 . 689566064108) ,
U Z 3 ∼ Bernoulli (0 . 180145428970) , U Z 4 ∼ Bernoulli (0 . 317153536644) ,
U Z 5 ∼ Bernoulli (0 . 046268153873) , U Z 6 ∼ Bernoulli (0 . 340145244411) ,
U Z 7 ∼ Bernoulli (0 . 100912238566) , U Z 8 ∼ Bernoulli (0 . 772038172066) ,
U Z 9 ∼ Bernoulli (0 . 913108434869) , U Z 10 ∼ Bernoulli (0 . 364272299067) ,
U Z 11 ∼ Bernoulli (0 . 063667554704) , U Z 12 ∼ Bernoulli (0 . 454839320009) ,
U Z 13 ∼ Bernoulli (0 . 586687215140) , U Z 14 ∼ Bernoulli (0 . 018824647595) ,
U Z 15 ∼ Bernoulli (0 . 871017316787) , U Z 16 ∼ Bernoulli (0 . 164966968157) ,
U Z 17 ∼ Bernoulli (0 . 578925020078) , U Z 18 ∼ Bernoulli (0 . 983082980658) ,
U Z 19 ∼ Bernoulli (0 . 018033993991) , U Z 20 ∼ Bernoulli (0 . 074629121266) ,
U X ∼ Bernoulli (0 . 29908139311) , U Y ∼ Bernoulli (0 . 9226108109253) ,
C = 0 .
M X =
[ Z 1 Z 2 ...
Z 20 ] ×      0 .0 .− 0 .− 0 .− 0 .− 0 .− 0 .− 0 .− 0 .− 0 .− 0 .− 0 .0 .− 0 .0 .0 .0 .0 .− 0 .0 .
    , M Y =
[ Z 1 Z 2 ...
Z 20 ] ×      − 0 .0 .0 .
3126802461410 .0 .− 0 .0 .−
0 .−
.− 0 .0 .− 
 .− 0 .− 0 .0 .−
.− 0 .0 .− 0 .
   
A.4 Informer Data
Detailed informer data can be obtained via the following equations:
[Equation]
× × (1 − f X ( M X , u X ))(1 − f Y ( f X ( M X , u X ) , M Y , u Y )) .