Hydranet: A Neural Network for the Estimation of Multi-valued Treatment Effects
Abstract
The clinical effectiveness aspect within the Health Technology Assessment process often faces causal questions where the treatment variable can take multiple values.
Nevertheless, most developments in causal inference algorithms that employ machine learning happen in binary treatment settings.
In addition, there is a big gap between the algorithmic state of the art and the applied state of the art in this field.
In this paper, we select a state-of-the-art, neural network-based algorithm for binary treatment effect estimation and generalize it to a multi-valued treatment setting, testing it with semi-synthetic data that could mimic an HTA process.
We obtain an estimator with desirable asymptotic properties and good results in experiments.
To the best of our knowledge, this work is opening ground for the benchmarking of neural network-based algorithms for multi-valued treatment effect estimation.
Introduction
Health Technology Assessment (HTA) is a multidisciplinary process that uses systematic and explicit methods to evaluate the properties and effects of a health technology [9].
This task is of crucial importance for the technical and financial sustainability of healthcare systems.
Within the process of HTA, clinical effectiveness is defined as the comparison between several health strategies or technologies regarding an outcome of interest [5].
Usually (but not always), one of those strategies or technologies is an existing standard, and the other (or others) are new.
As it can be sensed, this comparison is a causal question.
In front of such type of questions, HTA agencies usually employ the classical hierarchic pyramid of evidence, which puts the evidence generated by Randomized
Controlled Trials (RCT) at the cuspid.
Nevertheless, those agencies are facing nowadays more and more questions in which results from RCT may not be sufficient or arrive in time [11].
This, together with the exponential growth of Real-World Data (RWD) [7], constitutes a great opportunity for causal inference methods.
In the literature state of the art, machine learning-based algorithms for causal inference are mostly developed [14][6] and benchmarked [8][13] in binary treatment settings.
The HTA process, on the contrary, happens more often than not in scenarios with multiple strategies or technologies (treatments)
This is a limitation for the potential application of those algorithms for the task of HTA.
In the present work, we select a state of the art, neural network-based, binary treatment effect estimation algorithm named Dragonnet[12], and we develop and test a generalized version for n -valued treatment settings.
We present the theoretical and mathematical formulation, we develop a framework for experiments, and we provide results obtained in different scenarios employing semi-synthetic, HTArelated data.
To the best of our knowledge, this is the first attempt to establish a benchmark for this type of algorithm in multivalued-treatment settings.
The code and the data can be found in the following anonymized repository: https anonymous
Problem statement
Let the health technology or health strategy (from now on, for simplicity, the treatment) of interest be a discrete random variable T ∈ [0 ..k ] that can take k + 1 different values.
Let the outcome be a continuous random variable Y ∈ R , and let the covariates (i.e. the variables affecting both the treatment and the outcome) be a random vector X ∈ R j .
Thus, our set of data points is ( Y i , T i , X i ) , i ∈
[1 ..N ] , generated independently and identically.
This set of data points constitutes our body of observational data.
We define the causal effect of the treatment t over the outcome Y as, µ t =
E [ Y | do ( T = t )] , using Pearl’s do-calculus notation [10], which denotes intervention.
It can be shown that, if our data meets certain conditions, we can estimate causal (interventional) quantities based on observational data.
Those conditions are known as the identifiability conditions: positivity, consistency and "no hidden confounder" conditions.
For a more detailed explanation, see [10].
Under such conditions, µ t = E [ Y | X = x, T = t ] , which is a quantity that is inferrable from our body of observational data.
Along the rest of sections 2 and 3 we will assume that the identifiability conditions are fulfilled.
We define the conditional outcome as the expectation of the outcome given the treatment and the covariates, Q ( t, x ) =
E [ Y | t, x ] .
Based on Q , we can construct a simple estimator ˆ µ t of µ t as ˆ µ t = 1 N (cid:80) i Q ( t, x i ) .
In the following, we will be interested in approximating Q .
Let ˆ Q be an approximation of Q .
We define µ ˆ Qt = 1 N (cid:80) i ˆ Q ( t, x i ) as the estimator of µ t obtained replacing Q by its estimation ˆ Q .
Furthermore, we define the Generalized Propensity Score (GPS[1]), expressed as
G ( x ) = [ g 0 ( x ) , g 1 ( x ) , . . .
, g k ( x )] ∈ R k +1 , with g t ( x ) =
P ( T = t | x ) .
In a binary treatment setting, under the identifiability conditions, the Average Treatment Effect (ATE) is one of the most common causal quantities of interest, and it is defined as ψ = µ 1 − µ 0 .
Given an approximation ˆ Q of Q , we could easily estimate ψ as ψ ˆ Q = µ ˆ Q 1 − µ ˆ Q 0 .
In a multi-valued treatment setting, a wider class of causal quantities of interest can be defined, and all the conditional outcomes must be computed in order to obtain valid estimates of those quantities[1].
In this work, we define such quantities of interest as the pair-wise average differences between the several treatments and a treatment considered the control (note that, in practice, the control treatment does not necessarily mean absence of treatment).
Thus, we define a vector of ATEs ψ ∈ R k , ψ = [ ψ 1 , ψ 2 , . . .
, ψ k ] , with ψ t = µ t − µ 0 .
We can approximate these quantities in a similar fashion as shown before, the t -th element of the vector being ψ ˆ Qt = µ ˆ Qt − µ ˆ Q 0 .
Note that if the causal quantity of interest was ψ i,j = µ i − µ j , we could easily compute it based on the previous definition, as ψ i,j = ψ i − ψ j , due to the linearity of the expectation operator.
The subject of interest in this the paper is the estimation of the vector of ATEs ψ .
In the next section we generalize the estimation method provided in [12], which has the objective of estimating the ATE in the binary case, to the estimation of ψ in the multivalued treatment case presented above.
From Dragonnet to Hydranet
Dragonnet is a high-capacity, end-to-end neural network architecture for estimating binary treatment effects[12].
We present here the variation of the architecture, mathematical formulations and proofs for adapting Dragonnet to a multivalued treatment setting.
We call this adaptation Hydranet.
3.1 Architecture
The architecture of Hydranet can be seen in Figure 1.
It consists of two parts: the representation part, formed by the input layer and two hidden layers, and the heads, formed by k + 2 ends. Out of those, k + 1 correspond to the conditional outcomes, and are formed by two more hidden layers plus the output layer. The remaining head corresponds to the GPS, G ( x ) = [ g 0 ( x ) , g 1 ( x ) , . . .
, g k ( x )] ∈ R k
+ , with g t ( x ) =
P ( T = t | x ) , consisting on just the output layer.
Recall that we approximate the t -th element of the vector of ATEs as ψ
[Equation]
The baseline objective function has the shape
[Equation]
and the model parameters are
[Equation]
3.2 Targeted regularization
Now, following the reasoning in [12], we present targeted regularization. Targeted regularization is a modification of the objective function that introduces an extra parameter, epsilon.
In our setting, ϵ is a vector in R k , ϵ
=
( ϵ 1 , ϵ 2 , ...
ϵ k ) , and the new objective function is
[Equation]
with I ( T = t ) the indicator function, and thus the sought model parameters are defined by
[Equation]
But why this modification in the first place? The answer lies in semiparametric estimation theory (SET) and targeted maximum likelihood estimation (TMLE).
Very generally, SET provides us with conditions that ensure desirable properties of our estimator psi when they are fulfilled, and TMLE is an efficient method to impose the fulfillment of those conditions.
The conditions are the set of non-parametric estimating equations, defined as
[Equation]
and they employ the elements of the vector of efficient influence curves, defined as φ ∈ R k , φ = [ φ 1 , φ 2 , ...
φ k ] , with
[Equation]
Finally, recall that what we want is that the minimization of the modified objective function ensures the fulfillment of the non-parametric estimation equations.
This can be expressed mathematically as
[Equation]
and the proof can be found in the supplementary material.
This warrants the aforementioned desirable properties of the estimator psi , i.e. double robustness, fast convergence, and efficiency.
The data and the metrics
We have designed and implemented algorithms mimicking different Data Generating Processes (DGP) to produce semi-synthetic datasets to test Hydranet.
In this context, semi-synthetic means that the covariates are constituted by variables from a study with real participants, while the treatments and the outcomes have been artificially generated.
The covariates are a subset of the variables of the IHDP dataset [4] [3].
This data was collected for a real RCT carried out in 1985, and it is routinely used for benchmarking causal inference algorithms.
Despite it may contain some variables that could be considered sensitive, we have added noise and discarded some of them, so we consider that potential privacy-related issues have a low risk.
Based on those covariates, we have generated the multi-valued treatments and the continuous outcomes, in a similar fashion to [3], but adapting the DGP to our needs.
Below we explain the basics of these DGP, and a more detailed explanation can be found in the supplementary material.
For computing the treatment, we have created a function m : X → [0 ..k ] that maps a given subset of the covariates to a specific treatment in a deterministic way.
Then, in order to fulfill the positivity condition, we have drawn the treatment from a categorical distribution such that
[Equation]
We have implemented and tested different shapes of m functions.
For computing the outcome, we have defined as many outcome functions as treatment values.
These functions map some combination of the covariates and the treatment to the output space, Y =
L ( T, X ) =
[ l 0 ( T, X ) , l 1 ( T, X ) , . . .
, l k ( T, X )] .
We have added random noise to each outcome.
Thus, we can compute all the potential outcomes Y 0 , Y 1 , . . .
, Y k for each data point, i.e., we have the observable and the unobservable (counterfactual) data.
This way, we can compute the ground truth, true causal effects, for testing the performance of the algorithms.
Of course, the algorithms have been trained only with the observable data, as it would be the case in an observational study.
In addition, we have implemented and tested different shapes of the l t functions: shapes that are partially linear (linear in the treatment and nonlienar in the covariates), and shapes that are fully nonlinear. The advantage of the partially linear ones is that it allows us to easily control the difference between the biased, observable distribution, and the unbiased, unobservable distribution. We have called this difference the bias. We provide more details of this in the supplementary material.
We have generated datasets for the particular cases of 3, 5 and 8-valued treatments.
In addition, we have tested the influence over the performance of other parameters such as the dataset size, the bias size and the number of covariates.
The details of the DGP can be found in the supplementary material.
We have also generated datasets that quasiviolate or fully violate the positivity condition.
This is explained in a later subsection.
4.1 Metrics
For performance benchmarking purposes, we have employed the sum of errors of the vector of ATEs.
This is computed as the sum of the absolute values of the differences of all the estimated ATEs with respect to their true values, E = (cid:80) kt =1 | ψ t − ˆ ψ t | .
This choice allows to have a single real number as a final result, making comparisons simpler.
We have also computed the error relative to the total effect size or total ATE, tATE = (cid:80) kt
=1 | ψ t | , as a percentage: EP =
E tATE , as it gives a better intuition of the goodness of the ATE estimation.
All values have been computed as averages across several dataset realizations (50 or 100) to increase the robustness of the results, with 95% confidence intervals computed with Bootstrapping.
Experiments and results
In the case of binary treatment settings, there are de facto benchmarking datasets and metrics, i.e., datasets and metrics that are widely used in the literature and thus serve for algorithmic comparison purposes.
The IHDP dataset and the metrics presented in[3] are an example of this.
This is not the case in multi-valued treatment settings; we did not find any suitable comparator, and thus we have not been able to directly compare the performance of Hydranet with the performance of other algorithms.
Despite this, we affirm that Hydranet performs well in all the tested scenarios, reaching low or very low error values in all the 3, 5 and 8-valued treatment settings, for different dataset sizes, bias sizes, and number of covariates respectively.
These results can be found in the supplementary material.
In each setting we have computed the error and the relative error in the train and test sets, for two algorithms: baseline and targeted regularization (or t-reg).
For comparison purposes, we have also computed the results of a naive estimator, i.e., the one obtained with the biased, observable outcomes, not controlled for covariates.
This value has been calculated only for the purpose of showing the bias of the observational data without any controlling, and should not be taken as an alternative estimator.
We have included the results both in graphics and tables, avoiding the confidence intervals in the tables for simplicity.
Note that we have not tested all the possible combinations of treatment values (3, 5, 8) and variables of analysis (dataset size, bias size, number of covariates), but just one combination.
Nevertheless, based on the results, we expect the behavior to be similar across scenarios.
The optimizer selection, optimization strategy and other hyperparameters have suffered minor modifications with respect to those in [12]
5.1 Quasiviolation and violation of the positivity condition
Identifiability conditions are one of the main limitations in causal inference with observational data.
For this reason, there is a big interest in the causal inference community to build algorithms that are as robust as possible in the presence of quasiviolations of these conditions.
We have checked the performance of Hydranet in one of these situations (quasiviolation of the positivity condition), obtaining good results.
For this purpose, we have adjusted the probability of getting a particular treatment given the covariates, as
[Equation]
setting P to 0 .
95 and 1 and thus getting scenarios of quasiviolation and total violation respectively.
Table 1 shows the results of the algorithms, where it can be observed that in the total violation scenario all algorithms perform equally bad, while in the quasiviolation scenario Hydranet (both
Baseline and T-reg) is able to correctly learn the conditional outcomes and propensity scores, despite having very few examples of some of the treatment-covariate combinations. [Table] [Table]
[Figure]
5.2 Why targeted regularization?
In general, the results of Baseline and T-reg are similar.
This changes in one particular scenario: when the baseline’s estimate of the ATE is poor.
In that case, targeted regularization improves the estimation.
Figure 2 and table 2 show the absolute and relative errors of baseline and targeted regularization algorithms for varying bias size, for realizations of the dataset where the baseline’s estimation of the ATE was poor.
We have deﬁned poor estimates as those with an error above the mean error computed across all realizations.
We have set both baseline and t-reg algorithms to the same initialization, to ensure that the observed improvement is due to targeted regularization.
We do not have a theoretical explanation for this phenomenon, but the targeted regularization algorithm yields an estimator that is doubly robust and asymptotically efﬁcient, and this could point towards such explanation.
Discussion
In this work, we have generalized a state of the art, neural network-based algorithm for ATE estimation from a binary treatment setting to a n -valued treatment setting.
We have derived a doubly-robust and data-efﬁcient estimator, i.e., Hydranet with targeted regularization, obtaining very good results with semi-synthetic datasets generated ad-hoc , even in a scenario of quasi-violation of the positivity condition.
We show that, despite the generalizability to n -valued treatment settings being possible, it has its own challenges, and the behavior of the algorithms in each particular case requires its own interpretation.
As far as we know this paper is opening ground on the proposal and evaluation of methods for benchmarking and estimation of ATEs in multivalued treatment scenarios.
The main limitations of this work are twofold: ﬁrst, we have not been able to directly compare the performance of our algorithm to others, due to the lack of benchmarking data in the literature.
Also, we have not compared our algorithm against k binary-treatment algorithms.
Despite we expect our algorithm to perform better based on theoretical results, it would have been interesting to compare the experimental results directly, and thus this is both a limitation and a line of future work.
Second, we have only employed one dataset, the IHDP dataset.
It is also a line of future work to test the performance over other available adapted semi-synthetic datasets. With real-world impact in mind, we set another line of future work not in the direction of further algorithmic development but in the direction of usage of the current algorithm for real problems. There exists a big gap between the algorithmic state of the art and the applied state of the art in causal inference: some of the best pieces of applied work (causal inference studies for HTA and/or healthcare) make use of very simple methods such as regressions for modelling the data, and implicitly or explicitly claim that more advanced methods such as neural networks still have to prove their benefits. Despite we believe that this gap is, to some extent, natural, we also believe that the work of bringing the algorithmic state of the art to application is crucial for the potentially positive real-world impact of these methods.
We believe that the main potential negative societal aspect of our work could be the generation of low-quality or wrong evidence in the clinical field, due to inherent limitations of inferring causality with observational data.
This is an ongoing debate in the scientific community.
Nevertheless, we believe that good evidence can be obtained with causal inference methods by ensuring the highest possible methodological quality of these studies.
We will remain attentive to the accumulating evidence about this matter.
Acknowledgments
This work was supported by Doctorat Industrial funded by Generalitat de Catalunya [DI-2020-18] and by project CI-SUSTAIN funded by the Spanish Ministry of Science and Innovation [PID2019104156GB-I00].
Borja Velasco-Regúlez is a PhD Student of the doctoral program in Computer Science at the Universitat Autonoma de Barcelona.
A Supplementary material A.1 Hydranet
We want to prove that
[Equation]
Proof.
On one hand [Equation] using equations (3)
[Equation]
(4) and (5) we get ∂ ¯ F ∂ϵ
[Equation]
On the other hand, by substituting the definition of the efficient influence curves (8) in the set of nonparametric estimation equations (9), multiplying by β and particularizing at ˆ Q, ˆ g, ˆ ψ (the functions modelled by the neural network at the optimal point of the parameter space), we obtain an expression equal to the one in the last line of the proof.
Thus, the non-parametric estimation equations (9) are satisfied, and the proof is complete.
A.2 Details about the DGP
The DGP consists mainly on two collections of functions, the function for the treatment and the functions for the conditional outcome. The function for the treatment is [Equation]
with m : X → [0 ..k ] a function mapping a given subset of the potential covariates to an integer (the treatment).
We have implemented different shapes of m , from simple indicator functions based on one or two covariates, to a more complex function accepting a varying number of covariates in order to test the impact of this variation (as explained in A.2.3).
The conditional outcome functions are a collection of nonlinear functions of the shape Y =
L ( T, X ) =
[ l 0 ( T, X ) , l 1 ( T, X ) , . . .
, l k ( T, X )] that map some of the potential covariates, the actual covariates (both included in X ) and the treatment to the output space.
After this mapping, we have added random noise to the output.
These functions are linear and nonlinear combinations and compositions of functions such as the log function, the absolute value function, etc.
We have implemented partially linear shapes such as l t ( T, X ) =
f ( X ) + a ∗ 1 { T = t } and fully nonlinear shapes such as l t ( T, X ) =
g ( B · X + cT 2 ) .
We have generated datasets for the particular cases of 3, 5 and 8-valued treatments.
In addition, we have tested the influence over the performance of other parameters such as the dataset size, the bias size and the number of covariates.
Each dimension is explained below. A.2.1 Dataset size We have followed a data augmentation strategy and varied the size of the dataset, repeating each data point a given number of times and then adding noise to the whole dataset.
To prevent an undesired, potentially subtle form of leakage between the train and the test sets, we have avoided shuffling the dataset until the splitting in sets was done.
Leakage could appear if two augmented data points originated from the same initial data point end up in different splits of the dataset.
We speak about a subtle form of leakage because the duplicated individuals are not exactly the same due to the addition of noise.
A.2.2 Bias
We have varied the bias size by means of the treatment effect size. The bias size is the magnitude of the difference between the observable, biased distribution, and the unobservable, unbiased distribution.
An easy way to vary this magnitude is by implementing l t functions that are partially linear, i.e., linear in the treatment: l t ( T, X ) = f
( X ) + a ∗ 1 { T = t } , where f is some nonlinear function, a is the effect of the treatment and 1 { T = t } is an indicator function that takes value {+
+} if T = t and 0 otherwise.
This way, we can control the difference between the observable and the unobservable distributions of Y : the bigger a , the bigger the difference (the bias).
A.2.3 Number of covariates
We have tested the algorithms employing different numbers of covariates.
Recall that a covariate is a variable that affects both the treatment and the outcome.
In the IHDP dataset there are twenty seven variables that are potential covariates.
In the classical binary treatment example from [3], only one of those twenty seven variables is actually a covariate.
We have designed a data generating process that accepts a varying number of them, testing the cases of 4, 8, 12, 16 and 20 covariates.
A.3 Results in the test set A.3.1 3-valued treatment setting
In the 3-valued treatment setting we have studied the performance of the algorithm under different dataset sizes.
Table 3 shows the aforementioned performance metrics for each dataset size.
Figure 3 shows the trends of the errors and relative errors.
We can see the errors from both the Baseline and the T-reg algorithms decrease as the dataset size increases.
This is to be expected: up to some limit, more data means a better training and a better fit of the neural net in the test set.
There are no appreciable differences between baseline and T-reg algorithms.
A.3.2 5-valued treatment setting
For the 5-valued treatment setting, we have studied the algorithm in the presence of different bias sizes.
The results can be seen in Figure 4 and Table 4.
As expected, the errors of the naive estimator increase as the bias increases.
Both the Baseline and the T-reg algorithms perform well, with a maximum relative error around 10% in the case of the biggest bias.
There is no significant difference between T-reg and Baseline algorithms for small bias sizes, although T-reg performs slightly better for bigger bias sizes.
Note that the bias size variable is just a multiplying term in the DGP, and the number should not be taken at face value.
A.3.3 8-valued treatment setting
In the 8-valued treatment setting, we have tested the impact of different numbers of covariates.
The results can be seen in Figure 5 and Table 5.
The error of the naive estimator diminishes, then remains stable, then diminishes again when the number of covariates increases.
We hypothesize that this is due to some sort of "randomization effect": due to the shape of the functions employed for constructing the outcomes in the DGPs, the more covariates we include, the more the observable and unobservable outcome distributions resemble to a normal distribution, and thus the smaller the bias.
This would not necessarily be the case with a real-world (not synthetic) DGP.
Both Baseline and T-reg algorithms perform well, with low relative errors, below 4% in most cases.
The error of the T-reg algorithm is bigger than the Baseline algorithm for smaller number of confounders, but it becomes similar or smaller with increasing number of confounders.
[Table]
[Table] [Table]
[Figure]
[Table] [Table]
[Table]
[Figure]
[Table]
[Figure]