Original: Unit Selection: Learning Benefit Function from Finite Population DataAnonymous Author(s)
Revised: Unit Selection: Learning Benefit Function from Finite Population Data

Original: AffiliationAddress

Original: Abstract
Revised: Abstract

Original: The unit selection problem is to identify a group of individuals who are most likelyto exhibit a desired mode of behavior, for example, selecting individuals whowould respond one way if incentivized and a different way if not.
Revised: The unit selection problem is to identify a group of individuals who are most likely to exhibit a desired mode of behavior, for example, selecting individuals who would respond one way if incentivized and a different way if not.

Original: The unit selectionproblem consists of evaluation and search subproblems.
Revised: The unit selection problem consists of evaluation and search subproblems.

Original: Li and Pearl defined the"benefit function" to evaluate the average payoff of selecting a certain individualwith given characteristics.
Revised: Li and Pearl defined the "benefit function" to evaluate the average payoff of selecting a certain individual with given characteristics.

Original: The search subproblem is then to design an algorithmto identify the characteristics that maximize the above benefit function.
Revised: The search subproblem is then to design an algorithm to identify the characteristics that maximize the above benefit function.

Original: Thehardness of the search subproblem arises due to the large number of characteristicsavailable for each individual and the sparsity of the data available in each cell ofcharacteristics.
Revised: The hardness of the search subproblem arises due to the large number of characteristics available for each individual and the sparsity of the data available in each cell of characteristics.

Original: In this paper, we present a machine learning framework that usesthe bounds of the benefit function that are estimable from the finite population datato learn the bounds of the benefit function for each cell of characteristics.
Revised: In this paper, we present a machine learning framework that uses the bounds of the benefit function that are estimable from the finite population data to learn the bounds of the benefit function for each cell of characteristics.

Original: Therefore,we could easily obtain the characteristics that maximize the benefit function.
Revised: Therefore, we could easily obtain the characteristics that maximize the benefit function.

Original:  Introduction
Revised:  Introduction

Original: The unit selection problem frequently appears in the industries such as health science, politicalscience, and marketing.
Revised: The unit selection problem frequently appears in the industries such as health science, political science, and marketing.

Original: In customer relationship management [1, 6, 7, 19], for example, companiesare interested in discovering customers who are possible to leave but would not if there is an incentive.
Revised: In customer relationship management [1, 6, 7, 19], for example, companies are interested in discovering customers who are possible to leave but would not if there is an incentive.

Original: The incentive should be sent very carefully because the behavior of the customers is influenced by anyincentives
Revised: The incentive should be sent very carefully because the behavior of the customers is influenced by any incentives

Original: that the companies sent before
Revised: that the companies sent before

Original: For another example in online advertisement, companiesare interested in identifying users who would view an advertisement if and only if the advertisementis prompted [2, 12, 16, 18, 20].
Revised: For another example, in online advertisement, companies are interested in identifying users who would view an advertisement if and only if the advertisement is prompted [2, 12, 16, 18, 20].

Original: The challenge in identifying these individuals stems from the factthat the desired response pattern is not observed directly but rather is defined counterfactually interms of what the individual would do under hypothetical unrealized conditions.
Revised: The challenge in identifying these individuals stems from the fact that the desired response pattern is not observed directly but rather is defined counterfactually in terms of what the individual would do under hypothetical unrealized conditions.

Original: For example, whena customer has bought a car with a considerable discount, we have no idea whether they would buythe same car if there is no such discount.
Revised: For example, when a customer has bought a car with a considerable discount, we have no idea whether they would buy the same car if there is no such discount.

Original: As defined by Li and Pearl [13] the unit selection problem entails two sub-problems, evaluationand search.
Revised: As defined by Li and Pearl [13] the unit selection problem entails two sub-problems, evaluation and search.

Original: The evaluation problem is to develop an estimable objective function that, if maximizedover the set of observed characteristics C (available for each individual), would ensure an optimalcounterfactual behavior for the selected group.
Revised: The evaluation problem is to develop an estimable objective function that, if maximized over the set of observed characteristics C (available for each individual), would ensure an optimal counterfactual behavior for the selected group.

Original: The search task is to develop a search algorithm todetermine individuals based both on their observed characteristics and the objective function devisedabove.
Revised: The search task is to develop a search algorithm to determine individuals based both on their observed characteristics and the objective function devised above.

Original: The hardness of the search task arises due to the large number of characteristics available foreach individual and the sparsity of the data available in each cell of characteristics.
Revised: The hardness of the search task arises due to the large number of characteristics available for each individual and the sparsity of the data available in each cell of characteristics.

Original: For the evaluation sub-problem, the benefit function for the unit selection problem was defined by Liand Pearl [13], and it properly captures the nature of the desired behavior.
Revised: For the evaluation sub-problem, the benefit function for the unit selection problem was defined by Li and Pearl [13], and it properly captures the nature of the desired behavior.

Original: By making the assumptionthat the treatment has no effect on the population-specific characteristics, Li and Pearl derived tightbounds of the benefit function using experimental and observational data.
Revised: By making the assumption that the treatment has no effect on the population-specific characteristics, Li and Pearl derived tight bounds of the benefit function using experimental and observational data.

Original: Li and Pearl [15] then narrowed the bounds of the benefit function using covariates information and their causal structureinspired by Mueller, Li, and Pearl [17] and Dawid et al.
Revised: Li and Pearl [15] then narrowed the bounds of the benefit function using covariates information and their causal structure inspired by Mueller, Li, and Pearl [17] and Dawid et al.

Original: [3] that the bounds of probabilities ofcausation could be narrowed using covariates information.
Revised: [3] that the bounds of probabilities of causation could be narrowed using covariates information.

Original: In addition, the unit selection problemwith nonbinary treatment and effect was studied by Li and Pearl in [10, 11].
Revised: In addition, the unit selection problem with nonbinary treatment and effect was studied by Li and Pearl in [10, 11].

Original: In this study, we focus on the search sub-problem with binary treatment and effect.
Revised: In this study, we focus on the search sub-problem with binary treatment and effect.

Original: Nonbinarysituations can be extended with [10, 11, 14, 21].
Revised: Nonbinary situations can be extended with [10, 11, 14, 21].

Original: Consider the popular motivating scenario by Li and Pearl [13]: a mobile carrier is interested inidentifying customers who are about to churn in the future based on their 15 observed characteristicssuch as gender, income, age, monthly payments, and so on.
Revised: Consider the popular motivating scenario by Li and Pearl [13]: a mobile carrier is interested in identifying customers who are about to churn in the future based on their 15 observed characteristics such as gender, income, age, monthly payments, and so on.

Original: A considerable discount will then beoffered to identified customers to encourage them to continue the service so as to increase their servicerenewal rate.
Revised: A considerable discount will then be offered to identified customers to encourage them to continue the service so as to increase their service renewal rate.

Original: The management has to be careful that only those customers who would continue theirservice if and only if they receive the offer can receive the discount, and all other types of customerscannot.
Revised: The management has to be careful that only those customers who would continue their service if and only if they receive the offer can receive the discount, and all other types of customers cannot.

Original: The management then collected the observational and experimental data of their customers(we call this population data because the data are randomized over all customers.
Revised: The management then collected the observational and experimental data of their customers (we call this population data because the data are randomized over all customers.

Original: It is hard to collectdata for each set of characteristics because 1) the total number of the set of characteristics is large,2 15 ; 2) some set of characteristics is rare to appear).
Revised: It is hard to collect data for each set of characteristics because 1) the total number of the set of characteristics is large, 2 15 ; 2) some set of characteristics is rare to appear).

Original: The simplest solution is to apply Li-Pearl’s benefit function with benefit vector (1 , − 1 , − 1 , − 1) or(1 , − 1 , − 1 , − 2) to every set of characteristics and selects all sets such that the benefit function ispositive.
Revised: The simplest solution is to apply Li-Pearl’s benefit function with benefit vector (1 , − 1 , − 1 , − 1) or (1 , − 1 , − 1 , − 2) to every set of characteristics and selects all sets such that the benefit function is positive.

Original: However, we have 2 15 sets of characteristics, and according to Li and Pearl [9], each set ofcharacteristics requires roughly 1500 experimental and 1500 observational samples to have relativelyprecise estimations.
Revised: However, we have 2 15 sets of characteristics, and according to Li and Pearl [9], each set of characteristics requires roughly 1500 experimental and 1500 observational samples to have relatively precise estimations.

Original: Therefore, at least 49152000 experimental and 49152000 observational samplesare needed to evaluate the benefit function of every set of characteristics.
Revised: Therefore, at least 49152000 experimental and 49152000 observational samples are needed to evaluate the benefit function of every set of characteristics.

Original: In addition, the experimentaland observational data are collected from the entire population (i.e., all customers); therefore, some ofthe characteristics are rare to appear.
Revised: In addition, the experimental and observational data are collected from the entire population (i.e., all customers); therefore, some of the characteristics are rare to appear.

Original: Therefore, it is impractical to evaluate every set of characteristicsgiven the finite population data.
Revised: Therefore, it is impractical to evaluate every set of characteristics given the finite population data.

Original: Li et al.
Revised: Li et al.

Original: [8] proposed that a machine learning model can learn thebounds of probabilities of causation.
Revised: [8] proposed that a machine learning model can learn the bounds of probabilities of causation.

Original: Similar work can be applied to the benefit function.
Revised: Similar work can be applied to the benefit function.

Original: In thispaper, we believe that the behavior of the customers is determined by their characteristics; therefore,we present a machine learning framework that uses the bounds of the estimable benefit function asthe label (of those sets of characteristics such that the benefit function has sufficient data to estimate),to compute the bounds of benefit function for every set of characteristics.
Revised: In this paper, we believe that the behavior of the customers is determined by their characteristics; therefore, we present a machine learning framework that uses the bounds of the estimable benefit function as the label (of those sets of characteristics such that the benefit function has sufficient data to estimate), to compute the bounds of benefit function for every set of characteristics.

Original:  Preliminaries
Revised:  Preliminaries

Original: In this section, we review Li and Pearl’s benefit function of the unit selection problem [13].
Revised: In this section, we review Li and Pearl’s benefit function of the unit selection problem [13].

Original: The language of counterfactuals in structural model semantics, as given in [4, 5], is used in this paper.
Revised: The language of counterfactuals in structural model semantics, as given in [4, 5], is used in this paper.

Original: The basic counterfactual sentence “Variable Y would have the value y , had X been x " is denoted by
Revised: The basic counterfactual sentence “Variable Y would have the value y , had X been x " is denoted by

Original: Y x = y .
Revised: Y x = y .

Original: We distinguish between experimental and observational data; the experimental data is inthe form of the causal effects denoted by P ( y x ) , and the observational data is in the form of jointprobability denoted by P ( x, y ) .
Revised: We distinguish between experimental and observational data; the experimental data is in the form of the causal effects denoted by P ( y x ) , and the observational data is in the form of joint probability denoted by P ( x, y ) .

Original: Without further specification, we use X to denote treatment and
Revised: Without further specification, we use X to denote treatment and

Original: Y to denote effect.
Revised: Y to denote effect.

Original: For simplicity purposes, Y x = y, Y x ′ =
Revised: For simplicity purposes, Y x = y, Y x ′ =

Original: y, Y x = y ′ , Y x ′ =
Revised: y, Y x = y ′ , Y x ′ =

Original: y ′ are denoted byy x , y x ′ , y ′
Revised: y ′ are denoted by y x , y x ′ , y ′

Original: x , y ′ x ′ , respectively.
Revised: x , y ′ x ′ , respectively.

Original: Same as Li and Pearl [13], individual behavior was classified into four response types: labeledcomplier, always-taker, never-taker, and defier.
Revised: Same as Li and Pearl [13], individual behavior was classified into four response types: labeled complier, always-taker, never-taker, and defier.

Original: Suppose the benefit of selecting one individual ineach category are β, γ, θ, δ respectively (i.e., the benefit vector is ( β, γ, θ, δ ) ).
Revised: Suppose the benefit of selecting one individual in each category are β, γ, θ, δ respectively (i.e., the benefit vector is ( β, γ, θ, δ ) ).

Original: Li and Pearl definedthe objective function of the unit selection problem as the average benefit gained per individual.
Revised: Li and Pearl defined the objective function of the unit selection problem as the average benefit gained per individual.

Original: Suppose x and x ′ are binary treatments, y and y ′ are binary outcomes, and c are population-specificcharacteristics, the objective function (i.e., benefit function) is following (If the goal is to evaluate theaverage benefit gained per individual for a specific population c , argmax c can be dropped.):
Revised: Suppose x and x ′ are binary treatments, y and y ′ are binary outcomes, and c are population-specific characteristics, the objective function (i.e., benefit function) is following (If the goal is to evaluate the average benefit gained per individual for a specific population c , argmax c can be dropped.):

Original: [Equation]
Revised: [Equation]

Original: Using a combination of experimental and observational data, Li and Pearl established the mostgeneral tight bounds on this benefit function as follow (which we refer to as Li-Pearl’s Theorem inthe rest of the paper).
Revised: Using a combination of experimental and observational data, Li and Pearl established the most general tight bounds on this benefit function as follow (which we refer to as Li-Pearl’s Theorem in the rest of the paper).

Original: The only constraint is that the population-specific characteristics are not adescendant of the treatment. 
Revised: The only constraint is that the population-specific characteristics are not a descendant of the treatment. Z X Y

Original: Theorem 1.
Revised: Theorem 1.

Original: Given a causal diagram G and distribution compatible with G , let C be a set of variablesthat does not contain any descendant of X in G , then the benefit function f ( c ) = βP ( y x , y ′ x′ | c )
Revised: Given a causal diagram G and distribution compatible with G , let C be a set of variables that does not contain any descendant of X in G , then the benefit function f ( c ) = βP ( y x , y ′ x

Original:  + Z
Revised: ′ | c ) + γP ( y x , y x ′ | c ) + θP ( y ′ x , y ′ x

Original: [Equation] γP [Equation]
Revised: ′ | c ) + δP ( y x ′ , y ′ x

Original:  is bounded as follows:
Revised: | c ) is bounded as follows:

Original: [Equation]
Revised: [Equation]

Original: where σ, W, L, U are given by,
Revised: where σ, W, L, U are given by,

Original: [Equation]
Revised: [Equation]

Original:  

Original: [Equation]

Original:   .
Revised:   .

Original:  Causal Model
Revised:  Causal Model

Original: Similar to the probabilities of causation in [8], the learned bounds of the benefit function need tocompare to the true bounds.
Revised: Similar to the probabilities of causation in [8], the learned bounds of the benefit function need to compare to the true bounds.

Original: The data generating model should be stated explicitly.
Revised: The data generating model should be stated explicitly.

Original: We used thesecond causal model in [9] (see the appendix for the detail).
Revised: We used the second causal model in [9] (see the appendix for the detail).

Original: The model is depicted in Figure 1.
Revised: The model is depicted in Figure 1.

Original: X isa binary treatment, Y is a binary effect, and Z is a set of 20 independent binary characteristics (say
Revised: X is a binary treatment, Y is a binary effect, and Z is a set of 20 independent binary characteristics (say

Original: Z 1 , ..., Z 20 ).
Revised: Z 1 , ..., Z 20 ).

Original: The structural equations are as follow (for simplicity reason, we let x = 1 , x ′ = 0 , andy = 1 , y ′ = 0 ):
Revised: The structural equations are as follow (for simplicity reason, we let x = 1 , x ′ = 0 , and y = 1 , y ′ = 0 ):

Original: Z [Equation]
Revised: Z i =

Original: = U Z i for i ∈ { 1 , ..., 20 } ,
Revised: U Z i for i ∈ { 1 , ..., 20 } ,

Original: X = f X ( M X , U X ) = (cid:26)if M X + U X > 0 . 5 , otherwize , (cid:27)
Revised: [Equation]

Original: Y=
Revised: Y

Original: f Y ( X, M Y , U Y ) = (cid:26)if 0 < CX + M Y + U Y < 1 or 1 < CX + M Y
Revised: [Equation]

Original: + U Y < otherwize , (cid:27) where, U Z i , U X , U Y are binary exogenous variables with Bernoulli distributions,
Revised: where, U Z i , U X , U Y are binary exogenous variables with Bernoulli distributions,

Original: C is a constant, and M X , M Y are linear combinatations of Z 1 , ..., Z 20 .
Revised: C is a constant, and M X , M Y are linear combinatations of Z 1 , ..., Z 20 .

Original: The value of C, M X , M Y and the distributions of U X , U Y , U Z 1 , ..., U Z 20 for the model are providedin the appendix.
Revised: The value of C, M X , M Y and the distributions of U X , U Y , U Z 1 , ..., U Z 20 for the model are provided in the appendix.

Original:  Benefit Function
Revised:  Benefit Function

Original: The benefit vector that we will use in the simulated study is (1 , − 1 , − 1 , − 2) , which is a common oneto encourage complier as well as avoid all other three response types.
Revised: The benefit vector that we will use in the simulated study is (1 , − 1 , − 1 , − 2) , which is a common one to encourage complier as well as avoid all other three response types.

Original: Therefore, the benefit functionis
Revised: Therefore, the benefit function is f ( c [Equation] P [Equation]

Revised: y

Original: [Equation]
Revised: [Equation]

Revised: ( y

Revised: [Equation]

Revised: y ′

Revised: [Equation]

Revised: [Equation]

Revised: P ( y ′ x , y x ′ | c ) .

Original:  Data Generating Process
Revised:  Data Generating Process

Original: Similar to Li and Pearl [8], 15 of the 20 binary characteristics were made observable (i.e., Z 1 , ..., Z 15 ),and 5 of them were made unobservable (i.e., Z 16 , ..., Z 20 ).
Revised: Similar to Li and Pearl [8], 15 of the 20 binary characteristics were made observable (i.e., Z 1 , ..., Z 15 ), and 5 of them were made unobservable (i.e., Z 16 , ..., Z 20 ).

Original: Therefore, we have 2 15 observed sets ofcharacteristics.
Revised: Therefore, we have 2 15 observed sets of characteristics.

Original: 5.1 Informer Data
Revised: 5.1 Informer Data

Original: The informer data contains the true bounds of the benefit function of each set of characteristics.
Revised: The informer data contains the true bounds of the benefit function of each set of characteristics.

Original: Fromthe SCM we have, the value of X is determined by U X and M x (denoted by f X ( M X , U X ) ) andthe value of Y is determined by X , M Y , and U Y (denoted by f Y ( X, M Y , U Y ) ).
Revised: From the SCM we have, the value of X is determined by U X and M x (denoted by f X ( M X , U X ) ) and the value of Y is determined by X , M Y , and U Y (denoted by f Y ( X, M Y , U Y ) ).

Original: We first computethe true value of the benefit function and the true experimental and observational distributions forany set of 20 characteristics z
Revised: We first compute the true value of the benefit function and the true experimental and observational distributions for any set of 20 characteristics z

Original: = ( z 1 , ..., z 20 ) .
Revised: = ( z 1 , ..., z 20 ) .

Original: Since z is fixed; therefore, M X and M Y are fixed(denoted by M X ( z ) and M Y ( z ) ).
Revised: Since z is fixed; therefore, M X and M Y are fixed (denoted by M X ( z ) and M Y ( z ) ).

Original: We then have the benefit function f
Revised: We then have the benefit function f

Original: ( z ) , experimental distribution
Revised: ( z ) , experimental distribution

Original: P ( Y | do ( X ) , z ) , and observational distribution P
Revised: P ( Y | do ( X ) , z ) , and observational distribution P

Original: ( Y | X, z ) as follows:[Equation]
Revised: ( Y | X, z ) as follows:

Original: [Equation] =
Revised: f ( z ) =

Original: P [Equation]
Revised: P ( y x , y ′ x ′ | z ) − P ( y x , y x ′ | z ) − P ( y ′ x , y ′ x ′ | z ) − 2 P ( y ′ x , y x ′ | z ) =

Original:  U [Equation]
Revised: P ( U Y = 0)

Original: [Equation]
Revised: ( T 0 − T 2 − T 4 − 2 T 6 ) + P

Original:  U [Equation] = 1) [Equation]
Revised: ( U Y = 1)( T 1 − T 3 − T 5 − 2 T 7 ) ,

Original: T 7 ) , where , T= (cid:26)if Y (0 , M Y ( z ) , 0) = 0{+
+} and Y (1 , M Y
Revised: where , T= (cid:26)if Y (0 , M Y ( z ) , 0) = {+
+} and Y (1 , M Y

Original: ( z ) , 0)
Revised: ( z ) , 0)

Original: = 1 , otherwize , (cid:27) ,
Revised: = 1 , otherwize , (cid:27) ,

Original: T= (cid:26)if Y (0 , M Y ( z ) , 1) = 0 and Y (1 , M Y ( z ) , 1) = 1 , otherwize (cid:27) ,
Revised: T= (cid:26)if Y (0 , M Y ( z ) , 1) = 0 and Y (1 , M Y ( z ) , 1) = 1 , otherwize (cid:27) ,

Original: T= (cid:26)if Y (0 , M Y ( z ) , 0)
Revised: T= (cid:26)if Y (0 , M Y ( z ) , 0)

Original: =
Revised: =

Original: and Y (1 , M Y
Revised: and Y (1 , M Y

Original: ( z ) , 0)
Revised: ( z ) , 0)

Original: = 1 , otherwize , (cid:27) ,
Revised: = 1 , otherwize , (cid:27) ,

Original: T= (cid:26)if Y (0 , M Y ( z ) , 1) = 1 and Y (1 , M Y ( z ) , 1) = 1 , otherwize (cid:27) ,
Revised: T= (cid:26)if Y (0 , M Y ( z ) , 1) = 1 and Y (1 , M Y ( z ) , 1) = 1 , otherwize (cid:27) ,

Original: T= (cid:26)if Y (0 , M Y ( z ) , 0)
Revised: T= (cid:26)if Y (0 , M Y ( z ) , 0)

Original: = 

Revised: =

Original: and Y (1 , M Y
Revised: and Y (1 , M Y

Original: ( z ) , 0)
Revised: ( z ) , 0)

Original: = 0 , otherwize , (cid:27) ,
Revised: = 0 , otherwize , (cid:27) ,

Original: T= (cid:26)if Y (0 , M Y ( z ) , 1) = 0 and Y (1 , M Y ( z ) , 1) = 0 , otherwize (cid:27) ,
Revised: T= (cid:26)if Y (0 , M Y ( z ) , 1) = 0 and Y (1 , M Y ( z ) , 1) = 0 , otherwize (cid:27) ,

Original: T= (cid:26)if Y (0 , M Y ( z ) , 0)
Revised: T= (cid:26)if Y (0 , M Y ( z ) , 0)

Original: =
Revised: =

Original: and Y (1 , M Y
Revised: and Y (1 , M Y

Original: ( z ) , 0)
Revised: ( z ) , 0)

Original: = 0 , otherwize , (cid:27) ,
Revised: = 0 , otherwize , (cid:27) ,

Original: T= (cid:26)if Y (0 , M Y ( z ) , 1) = 1 and Y (1 , M Y ( z ) , 1) = 0 , otherwize (cid:27) .
Revised: T= (cid:26)if Y (0 , M Y ( z ) , 1) = 1 and Y (1 , M Y ( z ) , 1) = 0 , otherwize (cid:27) .

Original: P ( Y = 1 | do ( X ) , z ) =
Revised: P ( Y = 1 | do ( X ) , z ) =

Original: P ( U Y = 0) ∗ Y ( X, M Y ( z ) , 0)
Revised: P ( U Y = 0) ∗ Y ( X, M Y ( z ) , 0)

Original: + P ( U Y = 1) ∗ Y ( X, M Y ( z ) , 1) .
Revised: + P ( U Y = 1) ∗ Y ( X, M Y ( z ) , 1) .

Original: P ( Y = 1 | X, z ) =
Revised: P ( Y = 1 | X, z ) =

Original: P ( U X = 0) ∗ P ( U Y = 0) ∗ Y ( X ( M X ( z ) , 0) , M Y ( z ) , 0)
Revised: P ( U X = 0) ∗ P ( U Y = 0) ∗ Y ( X ( M X ( z ) , 0) , M Y ( z ) , 0)

Original: +
Revised: +

Original: P ( U X = 0) ∗ P ( U Y = 1) ∗ Y ( X ( M X ( z ) , 0) , M Y ( z ) , 1))
Revised: P ( U X = 0) ∗ P ( U Y = 1) ∗ Y ( X ( M X ( z ) , 0) , M Y ( z ) , 1))

Original: +
Revised: +

Original: P ( U X = 1) ∗ P ( U Y = 0) ∗ Y ( X ( M X ( z ) , 1) , M Y ( z ) , 0))
Revised: P ( U X = 1) ∗ P ( U Y = 0) ∗ Y ( X ( M X ( z ) , 1) , M Y ( z ) , 0))

Original: +
Revised: +

Original: P ( U X = 1) ∗ P ( U Y = 1) ∗ Y ( X ( M X ( z ) , 1) , M Y ( z ) , 1)) .
Revised: P ( U X = 1) ∗ P ( U Y = 1) ∗ Y ( X ( M X ( z ) , 1) , M Y ( z ) , 1)) .

Original: Second, we need to compute the informer data for any 15 observed set of characteristics,say c = ( z 1 , ..., z 15 ) .
Revised: Second, we need to compute the informer data for any 15 observed set of characteristics, say c = ( z 1 , ..., z 15 ) .

Original: c =
Revised: c =

Original: ( z 1 , ..., z 15 ) consists 32 sets of 20 characteristics (say s 0 =({+
 = (+} z 1 , ..., z 15 , 0 , 0 , 0 , 0 , 0) , s 1{+
+} = ( z 1 , ..., z 15 , 0 , 0 , 0 , 0 , 1) , s 

Revised: ( z 1 , ..., z 15 ) consists 32 sets of 20 characteristics (say s {+
 = (+} z 1 , ..., z 15 , 0 , 0 , 0 , 0 , 0) , s {+
+} = ( z 1 , ..., z 15 , 0 , 0 , 0 , 0 , 1) , s

Original: = ( z 1 , ..., z 15 , 0 , 0 , 0 , 1 , 0) , ..., s 31 =({+
 = (+} z 1 , ..., z 15 , 1 , 1 , 1 , 1 , 1) ), then we have the benefit function f ( c ) , experimental distribution
Revised: = ( z 1 , ..., z 15 , 0 , 0 , 0 , 1 , 0) , ..., s {+
 = (+} z 1 , ..., z 15 , 1 , 1 , 1 , 1 , 1) ), then we have the benefit function f ( c ) , experimental distribution

Original: P ( Y | do ( X ) , c ) , and observational distribution P
Revised: P ( Y | do ( X ) , c ) , and observational distribution P

Original: ( Y | X, c ) of any observed sets of characteristics
Revised: ( Y | X, c ) of any observed sets of characteristics

Original: c are as follow:f ( c ) =
Revised: c are as follow:

Original: [Equation]
Revised: [Equation]

Original: ( Y
Revised: P [Equation] [Equation] ) =

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: s 0 )

Original: [Equation]
Revised: [Equation]

Original: 0 X =0 , Y = 1 X =
Revised: ( Z [Equation] 0) P ( Z

Original: [Equation]
Revised: [Equation]

Original: c ) =
Revised: 0) P ( [Equation]

Original:  P (s 0 ) /P
Revised: 0) P (

Original: ( c ) f
Revised: [Equation]

Original:  (s 0 )
Revised: 0) P (

Revised: [Equation]

Original: + P ( s 1 ) /P
Revised: = 1) P ( Y [Equation] 1 | [Equation]

Original: ( c )f
Revised: ( X )

Original: ( s 1 ) + P ( s 2 ) /P
Revised: [Equation] s 1 ) + ... +

Original: ( c ) f
Revised: [Equation] [Equation] [Equation]

Original: ( s 2 ) + ...
Revised: [Equation]

Original: + P ( s 31 ) /P
Revised: [Equation] [Equation] [Equation] [Equation]

Original:  ( c ) f
Revised: 1) P ( [Equation] = 1) P ( Y = [Equation]

Original: (  s 31 )  =
Revised: ( X ) , s 31 ) . P [Equation] c ) =

Revised: [Equation]

Original: P ( Z 16 = 0) P ( Z 17 = 0) P ( Z 18 = 0) P ( Z 19 = 0) P ( Z 20 = 0)
Revised: [Equation] [Equation] [Equation] [Equation]

Revised: [Equation]

Original: f ( s 0 )
Revised: s 0 )

Original: +
Revised: [Equation] P [Equation] [Equation]

Original: P ( Z 16 = 0) P ( Z17 = 0) P ( Z 18 = 0) P ( Z 19 = 0) P ( Z 20 = 1) f
Revised: [Equation] [Equation] [Equation] ( Z

Original: ( s 1 )
Revised: [Equation] [Equation] [Equation]

Original: + ... +
Revised: [Equation]

Original: P ( Z 16 = 1) P ( Z 17 = 1) P ( Z 18 = 1) P ( Z 19 = 1) P ( Z 20 = 1) f ( s 31 ) . P ( Y = 1 | do ( X[-) , c ) = P ( Z 16 = 0) P ( Z 17 = 0) P ( Z 18 = 0) P ( Z 19 = 0) P ( Z 20 = 0) P ( Y = 
-]
Revised: X[-) , c ) = P ( Z 16 = 0) P ( Z 17 = 0) P ( Z 18 = 0) P ( Z 19 = 0) P ( Z 20 = 0) P ( Y = 
-]

Original: [-| do ( X ) , s 0 ) + P ( Z 16 = 0) P ( Z 17 = 0) P ( Z 18 = 0) P ( Z 19 = 0) P ( Z 20 = 1) P ( Y = 
 | do ( X ) , s 1 ) + ... +-]
Revised: [-| do ( X ) , s 0 ) + P ( Z 16 = 0) P ( Z 17 = 0) P ( Z 18 = 0) P ( Z 19 = 0) P ( Z 20 = 1) P ( Y = 
 | do ( X ) , s 1 ) + ... +-][Equation]

Original: P ( Z 16 = 1) P ( Z 17 = 1) P ( Z 18 = 1) P ( Z 19 = 1) P ( Z 20 = 1) P ( Y = 1 | do ( X ) , s31 ) . P ( Y = 1| X, c )[-= P ( Z 16 = 0) P ( Z 17 = 0) P ( Z 18 = 0) P ( Z 19 = 0) P ( Z 20 = 0) P ( Y = 
-]
Revised: s 1 )[-= P ( Z 16 = 0) P ( Z 17 = 0) P ( Z 18 = 0) P ( Z 19 = 0) P ( Z 20 = 0) P ( Y = 
-]

Original: [-| X, s 0 ) + P ( Z 16 = 0) P ( Z 17 = 0) P ( Z 18 = 0) P ( Z 19 = 0) P ( Z 20 = 1) P ( Y = 
 | X, s 1 )-] + ...
Revised: [-| X, s 0 ) + P ( Z 16 = 0) P ( Z 17 = 0) P ( Z 18 = 0) P ( Z 19 = 0) P ( Z 20 = 1) P ( Y = 
 | X, s 1 )-] + ...

Original: +
Revised: +

Original: P ( Z 16 = 1) P ( Z 17 = 1) P ( Z 18 = 1) P ( Z 19 = 1) P
Revised: P ( Z 16 = 1) P ( Z 17 = 1) P ( Z 18 = 1) P ( Z 19 = 1) P

Original: ( Z 20 = 1) P ( Y = 1 |
Revised: ( Z 20 = 1) P ( Y = 1 |

Original: X, s 31 ) .
Revised: X, s 31 ) .

Original: Therefore, the true bounds of the benefit function f ( c ) for each set of observed characteristics couldbe obtained using Theorem 1 and above observational and experimental distributions.
Revised: Therefore, the true bounds of the benefit function f ( c ) for each set of observed characteristics could be obtained using Theorem 1 and above observational and experimental distributions.

Original: 5.2 Experimental Sample
Revised: 5.2 Experimental Sample

Original: The 5000000 finite experimental samples for training purposes can be obtained repeatedly as follows:
Revised: The 5000000 finite experimental samples for training purposes can be obtained repeatedly as follows:

Original: • Randomly generate ( U X , U Y , U Z 1 , ..., U Z 20 ) using the given Bernoulli distributions; • Randomly generate X using Bernoulli (0 . 5) ; • Compute Y =
Revised: • Randomly generate ( U X , U Y , U Z 1 , ..., U Z 20 ) using the given Bernoulli distributions; • Randomly generate X using Bernoulli (0 . 5) ; • Compute Y =

Original: f Y ( X, M Y , U Y ) ;
Revised: f Y ( X, M Y , U Y ) ;

Original: • Collect a experimental sample ( U Z 1 , U Z 2 , ..., U Z 15 , X, Y )
Revised: • Collect a experimental sample ( U Z 1 , U Z 2 , ..., U Z 15 , X, Y )

Original: (Note that Z i
Revised: (Note that Z i

Original: =
Revised: =

Original: U
Revised: U

Original: Z i in our model);
Revised: Z i in our model);

Original: 5.3 Observational Sample
Revised: 5.3 Observational Sample

Original: The 5000000 finite observational samples for training purposes can be obtained repeatedly as follows:
Revised: The 5000000 finite observational samples for training purposes can be obtained repeatedly as follows:

Original: • Randomly generate ( U X , U Y , U Z 1 , ..., U Z 20 ) using the given Bernoulli distributions; • Compute X = f X ( M X , U X ) ; • Compute Y =
Revised: • Randomly generate ( U X , U Y , U Z 1 , ..., U Z 20 ) using the given Bernoulli distributions; • Compute X = f X ( M X , U X ) ; • Compute Y =

Original: f Y ( X, M Y , U Y ) ;
Revised: f Y ( X, M Y , U Y ) ;

Original: • Collect a experimental sample ( U Z 1 , U Z 2 , ..., U Z 15 , X, Y )
Revised: • Collect a experimental sample ( U Z 1 , U Z 2 , ..., U Z 15 , X, Y )

Original: (Note that Z i
Revised: (Note that Z i

Original: =
Revised: =

Original: U Z i in ourmodel);
Revised: U Z i in our model);

Original:  Machine Learning Model 
Revised:  Machine Learning Model 6.1 Features and Label

Original: 6.1 Features and Label

Original: The training features are 15 observed characteristics.
Revised: The training features are 15 observed characteristics.

Original: The training labels are lower and upper boundsof the benefit function for the 15 observed characteristics (two separate models for lower and upperbounds).
Revised: The training labels are lower and upper bounds of the benefit function for the 15 observed characteristics (two separate models for lower and upper bounds).

Original: In order to create the labels, we need Frequentist estimates of the experimental andobservational distributions for that 15 observed characteristics.
Revised: In order to create the labels, we need Frequentist estimates of the experimental and observational distributions for that 15 observed characteristics.

Original: According to Li, Mao, and Pearl[9], 1300 of experimental and observational samples are needed for the labels of the 15 observedcharacteristics.
Revised: According to Li, Mao, and Pearl [9], 1300 of experimental and observational samples are needed for the labels of the 15 observed characteristics.

Original: Therefore, in our 5000000 experimental and 5000000 observational samples, if thesame set of 15 observed characteristics appeared more than 1300 times, we obtain a lower bound andan upper bound for that set of 15 observed characteristics by applying the Theorem 1.
Revised: Therefore, in our 5000000 experimental and 5000000 observational samples, if the same set of 15 observed characteristics appeared more than 1300 times, we obtain a lower bound and an upper bound for that set of 15 observed characteristics by applying the Theorem 1.

Original: We have 302sets of characteristics for lower bound and 321 sets of characteristics for upper bound for trainingpurposes.
Revised: We havesets of characteristics for lower bound and 321 sets of characteristics for upper bound for training purposes.

Original: The 61 sets of lower bound and 65 sets of upper bound are then split for the testing set.
Revised: The 61 sets of lower bound and 65 sets of upper bound are then split for the testing set.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 6.2 Learning
Revised: 6.2 Learning

Original: In this paper, we use neural networks as our estimator to estimate the lower and upper bounds.
Revised: In this paper, we use neural networks as our estimator to estimate the lower and upper bounds.

Original: Because the simulated data is tabular in nature, we just use a multilayer perceptron (MLP) in thefollowing experiments.
Revised: Because the simulated data is tabular in nature, we just use a multilayer perceptron (MLP) in the following experiments.

Original: Our experimental environment is an AWS p3.2xlarge instance.
Revised: Our experimental environment is an AWS p3.2xlarge instance.

Original: The keyparameters of our model are: embeddings dimension as 128, training epochs as 600, and learningrate as 0.01.
Revised: The key parameters of our model are: embeddings dimension as 128, training epochs as 600, and learning rate as 0.01.

Original:  Experimental Results
Revised:  Experimental Results

Original: To show the performance of estimation, we randomly sampled 200 sets of characteristics from all32768 instances.
Revised: To show the performance of estimation, we randomly sampled 200 sets of characteristics from all 32768 instances.

Original: We observe that the average error of the learned lower bound and upper boundare 0 . 5652 and 0 .
Revised: We observe that the average error of the learned lower bound and upper bound are 0 . 5652 and 0 .

Original: 5447 , respectively.
Revised: 5447 , respectively.

Original: ∼ 0 .
Revised: ∼ 0 .

Original: 5 is not of bad quality because the potential values of thebenefit function are from − 4 to 1 .
Revised: 5 is not of bad quality because the potential values of the benefit function are from − 4 to 1 .

Original: We show how our predicted bounds fit the true ones in Figure 2,which confirms that the predicted bounds generally capture the theoretical ones.
Revised: We show how our predicted bounds fit the true ones in Figure 2, which confirms that the predicted bounds generally capture the theoretical ones.

Original:  Discussion
Revised:  Discussion

Original: We demonstrated that the benefit function for all sets of characteristics could be learned from finitepopulation data.
Revised: We demonstrated that the benefit function for all sets of characteristics could be learned from finite population data.

Original: We further discuss some properties of our proposed framework.
Revised: We further discuss some properties of our proposed framework.

Original: First, the bounds of the benefit function were successfully learned by the machine learning model.
Revised: First, the bounds of the benefit function were successfully learned by the machine learning model.

Original: Similar to Li and Pearl in [8], this is another example that the quantities of a counterfactual querycould be learned properly if the proper counterfactual labels were fed to the machine learning model.
Revised: Similar to Li and Pearl in [8], this is another example that the quantities of a counterfactual query could be learned properly if the proper counterfactual labels were fed to the machine learning model.

Original: This is the key to applying the machine learning model to causality concepts.
Revised: This is the key to applying the machine learning model to causality concepts.

Original: Second, we also applied the simplest machine learning model.
Revised: Second, we also applied the simplest machine learning model.

Original: The benefit vector and the datagenerating process are available for researchers to apply fancy machine learning models.
Revised: The benefit vector and the datagenerating process are available for researchers to apply fancy machine learning models.

Original: Theaccuracy of the predictions still has the potential to improve.
Revised: The accuracy of the predictions still has the potential to improve.

Original:  Conclusion
Revised:  Conclusion

Original: We illustrated how to learn the bounds of the benefit function in the unit selection problem for eachset of characteristics using finite population data.
Revised: We illustrated how to learn the bounds of the benefit function in the unit selection problem for each set of characteristics using finite population data.

Original: All sets of characteristics of the bounds of thebenefit function were being learned by a machine learning model, thus being able to select the setsof characteristics that have high (or positive) benefit values.
Revised: All sets of characteristics of the bounds of the benefit function were being learned by a machine learning model, thus being able to select the sets of characteristics that have high (or positive) benefit values.

Original: Experiments showed that the benefitfunction is learnable with proper labels.Checklist
Revised: Experiments showed that the benefit function is learnable with proper labels.

Original: For all authors...

Original: (a

Original: ) Do

Original: the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: [Yes]

Original: (b) Did you describe the

Original: limitations

Original: of your work?

Original: [Yes]

Original: (c) Did you discuss any

Original: potential negative societal impacts

Original: of your work?

Original: [N/A]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?

Original: [Yes]

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] in the appendix

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?

Original: [Yes]

Original: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters,

Original: how they

Original: were chosen)?

Original: [Yes]

Original: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experi-

Original: ments multiple times)?

Original: [Yes]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type

Original: of GPUs, internal cluster, or cloud provider)?

Original: [Yes]

Original: If you are

Original: using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a)

Original: If your work uses existing assets, did you cite the creators?

Original: [N/A](b)

Original: Did you mention the license of the assets?

Original: [N/A](c)

Original: Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’reusing/curating?

Original: [N/A](e)

Original: Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?

Original: [N/A]5.

Original: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?

Original: [N/A](b)

Original: Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable? [N/A](c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation? [N/A]
Revised: Acknowledgements

Original: A AppendixA.1 The Causal Model

Original: We used the second causal model in [9]. M X , M Y and C were uniformly generated from[ − 1 , 1] ,and the Bernoulli parameters of U X , U Y , U Z 1 , ..., U Z 20 were generated uniformly from [0 , 1] . Thedetailed model is as follows: [Equation]
Revised: This research was supported in parts by grants from the National Science Foundation [#IISand #IIS-2231798], Office of Naval Research [#N00014-21-1-2351], and Toyota Research Institute of North America [#PO-000897].

Original: where, U Z i , U X , U Y are binary exogenous variables with Bernoulli distributions.

Original: s.t.,

Original: U Z 1 ∼ Bernoulli (0 . 524110233482) , U Z 2 ∼ Bernoulli (0 . 689566064108) ,

Original: U Z 3 ∼ Bernoulli (0 . 180145428970) , U Z 4 ∼ Bernoulli (0 . 317153536644) ,

Original: U Z 5 ∼ Bernoulli (0 . 046268153873) , U Z 6 ∼ Bernoulli (0 . 340145244411) ,

Original: U Z 7 ∼ Bernoulli (0 . 100912238566) , U Z 8 ∼ Bernoulli (0 . 772038172066) ,

Original: U Z 9 ∼ Bernoulli (0 . 913108434869) , U Z 10 ∼ Bernoulli (0 . 364272299067) ,

Original: U Z 11 ∼ Bernoulli (0 . 063667554704) , U Z 12 ∼ Bernoulli (0 . 454839320009) ,

Original: U Z 13 ∼ Bernoulli (0 . 586687215140) , U Z 14 ∼ Bernoulli (0 . 018824647595) ,

Original: U Z 15 ∼ Bernoulli (0 . 871017316787) , U Z 16 ∼ Bernoulli (0 . 164966968157) ,

Original: U Z 17 ∼ Bernoulli (0 . 578925020078) , U Z 18 ∼ Bernoulli (0 . 983082980658) ,

Original: U Z 19 ∼ Bernoulli (0 . 018033993991) , U Z 20 ∼ Bernoulli (0 . 074629121266) ,

Original: U X ∼ Bernoulli (0 . 29908139311) , U Y ∼ Bernoulli (0 . 9226108109253) ,

Original: C = 0 .

Original: M X =

Original: [ Z 1 Z 2 ...

Original: [-Z 20 ] ×     0 .0 .− 
-]
Revised: [-Z 20 ] ×     0 .0 .− 
-]

Original: [-.− 0 .− 0 .− 0 .− 0 .− 0 .− 0 .− 0 .− 0 .− 0 .0 .− 0 .0 .0 .0 .0 .− 
-]
Revised: [-.− 0 .− 0 .− 0 .− 0 .− 0 .− 0 .− 0 .− 0 .− 0 .0 .− 0 .0 .0 .0 .0 .− 
-]

Original: .0 .

Original:    , M Y =

Original: [ Z 1 Z 2 ...

Original: Z 20 ] ×     − 0 .0 .0 .

Original: 3126802461410 .0 .− 0 .0 .−

Original: [-0 .− 
-]
Revised: [-0 .− 
-]

Original: [-.− 0 .0 .− 
-]
Revised: [-.− 0 .0 .− 
-]

Original: [-.− 0 .− 0 .0 .− 
-]
Revised: [-.− 0 .− 0 .0 .− 
-]

Original: .− 0 .0 .− 0 .

Original:   