Learning Individualized Treatment Rules with Many Treatments: A Supervised Clustering Approach Using Adaptive Fusion
Abstract
Learning an optimal Individualized Treatment Rule (ITR) is a very important problem in precision medicine.
This paper is concerned with the challenge when the number of treatment arms is large, and some groups of treatments in the large treatment space may work similarly for the patients.
Motivated by the recent development of supervised clustering, we propose a novel adaptive fusion based method to cluster the treatments with similar treatment effects together and estimate the optimal ITR simultaneously through a single convex optimization.
The problem is formulated as balancing loss ` penalty terms with a tuning parameter, which allows the entire solution path of the treatment clustering process to be clearly visualized hierarchically.
For computation, we propose an efﬁcient algorithm based on accelerated proximal gradient and further conduct a novel group-lasso based algorithm for variable selection to boost the performance.
Moreover, we demonstrate the theoretical guarantee of recovering the underlying true clustering structure of the treatments for our method.
Finally, we demonstrate the superior performance of our method via both simulations and a real data application on cancer treatment, which may assist the decision making process for doctors.
Introduction
Data-driven individualized decision making problems have received a lot of attentions in precision medicine [1, 2, 3, 4].
Due to possible signiﬁcant heterogeneity of treatment effects among individuals, it is necessary for decision makers to precisely tailor the treatment decision rules to different subgroups of individuals rather than simply implement the traditional “one size ﬁts all” medical procedures.
The main goal of the paper is to recommend the optimal Individualized Treatment Rule (ITR), mapping from the covariate space based on patients’ characteristics to the treatment assignment, that optimizes the expected value of a speciﬁed reward, known as the value function [5].
There are many existing machine learning based approaches for estimating the optimal ITR.
Some methods estimate the treatment effect related functions under some prespeciﬁed outcome mean models.
The estimated ITR is induced by maximizing the estimated treatment effects conditional on patients’ covariates.
These methods include Q-learning [5], A-learning [6, 7], weighted ordinary least square based method [8] and D-learning [9].
Some other methods circumvent modeling the treatment effect functions by directly estimating the ITR that maximizes the Inverse Probability
Weighting (IPW) version of the value function [10, 11, 12, 13, 14].
To overcome the potential misspeciﬁcation problems for these methods, [15, 16, 17, 18] proposed the doubly robust IPW methods.
Problem description.
This paper is concerned with the following two questions: (1) Given a large number of treatment options but limited observations of some speciﬁc treatments, how can we effectively estimate the optimal ITR?
(2) Some treatments in the large treatment space may work similarly for patients.
How can we identify this homogeneous treatment structure and cluster the treatments with similar treatment effects together to reduce the dimension of the treatment space?
For question (1), the estimated ITR using either the model-based methods or the IPW-based methods can become inaccurate with large variability and numerical instability due to the insufﬁcient number of observations for certain treatments.
For question (2), it is common that the drugs are developed based on intervening the same disease symptoms and mechanisms but may be produced by different pharmaceutical companies.
[16] and [19] directly combined the treatments with similar treatment effects into certain classes of treatments based on some prior knowledge .
In general, it is desirable to propose data-driven methods to identify the homogeneous treatment structure automatically.
However, to our best of knowledge, few existing methods deal with clustering treatments because most literature only consider binary or a moderate number of treatments.
Related work and our method.
Our goal is to cluster the heterogeneous treatment effects and estimate the ITR under the Q-learning framework [20, 5].
Some methods have been developed to address the problem of identifying subgroups with homogeneous relationships.
In supervised learning, with the special focus on exploring the homogeneous relations between the covariates and response, [21], [22] and [23] estimated the group structure for the regression coefﬁcients of covariates with various fusion penalties.
To identify subgroups from a heterogeneous population, [24] and [25] assumed the population comes from a mixture of subgroups with their own distributions and utilized the mixture model analysis to classify the observations.
However, the mixture model may cause misspeciﬁcation problems in practice since it needs to specify the underlying distribution for the data and the number of mixture components in the population.
Moreover, these supervised learning methods only utilized the information of response and covariates without considering the treatment assignment.
Hence, they are not applicable to the ITR problem.
In unsupervised learning, clustering analysis is a popular tool.
It is usually used to detect the similarities of observations using a predeﬁned distance measure.
In particular, a popular visualization method is to draw a dendrogram of the hierarchical clustering using a “bottom up” approach [26].
Regarding to our problem of clustering the treatment effects, we would like to adapt the idea of latent supervised clustering [27, 28].
We are interested in clustering the relationship characterized by the regression of “response „ covariates ˆ treatments” where “ ˆ ” refers to the interactions between covariates and treatments.
To achieve this goal, we model the treatment effects as the regression problem with treatment-speciﬁc coefﬁcient vectors.
We implement an adaptive pairwise fusion penalty of the ℓ 1 ´ distance between the treatment-speciﬁc coefﬁcient vectors in order to merge these coefﬁcient vectors with similar values together.
This is equivalent to clustering the treatments with similar treatment effects.
We formulate the clustering process as a convex minimization problem involving loss ` fusion penalty , with a tuning parameter balancing these two terms.
With this convex formulation, we achieve to maximize the goodness of ﬁt for estimating the heterogeneous treatment effects, while at the same time clustering the treatments without the need of prior knowledge.
Contribution.
The main contributions of our paper can be summarized as follows.
(1) We propose the Supervised Clustering approach using the Adaptive Fusion (SCAF) to identify possible homogeneous group structure within the large treatment space and estimate the ITR in a more effective way.
Compared with the two-step method [29], we simultaneously cluster the treatments with similar treatment effects and estimate the optimal ITR.
Our method combines the unsupervised and supervised learning together within a single convex optimization problem.
(3) We do not need to specify the number of treatment clusters and the entire solution path of the clustering process can be visualized with a new dendrogram by using a “bottom up” approach.
Although [27] and [28] also adopted the supervised clustering technique, they focused on identifying the heterogeneous subpopulations and aimed to cluster the subject-speciﬁc regression coefﬁcients.
In contrast, SCAF only needs to estimate the treatment-speciﬁc coefﬁcients and the dimension of the estimated parameters is dramatically reduced.
(5) For computation, we propose an effective algorithm to solve the convex minimization problem based on the accelerated proximal gradient algorithm [30].
Specifically, in order to boost the performance of our fusion algorithm, we propose a novel group-lasso based algorithm to classify the covariates into homogeneous variables and heterogeneous variables respectively.
Thus, the fusion penalty only needs to be imposed to the heterogeneous variables that have interaction with treatments.
For the theoretical study, we prove the consistency of the regression coefﬁcients and demonstrate that SCAF is able to recover the true underlying clustering structure with probability 1.
(7) We conduct both simulation studies and a real data analysis on cancer treatment to illustrate the superior performance of SCAF.
Methodology
We ﬁrst introduce the ITR problem under the regression-based framework.
Consider the training data p
z i , a i , y i q for i “ 1 , . . . , n as i.i.d.
realizations from the joint distribution of p Z, A, Y q , where z i P Z Ď R p denotes the patient’s prognostic variables, a i P A “ t 1 , 2 , . .
, M u is the treatment assignment, and y i P R is the observed reward for each patient i .
Let ` Y p a q ˘ a P A P R M be the potential outcome.
In addition, deﬁne the propensity scores of treatment p p a |
z q : “ P p A “ a | Z “ z q for a P A .
An ITR D P D is a map from the covariate space Z to the treatment space A .
Here, D Ď A Z is a prespeciﬁed ITR class.
The value function of an ITR is deﬁned as V p D q “
E r Y ` D p Z q ˘ s .
Assuming that a larger reward is better, our goal is to ﬁnd the optimal ITR D ˚ P D that maximizes the value function, i.e., D ˚ P arg max D P D V p D q .
We assume the following identiﬁability assumptions are satisﬁed [31]: (1) consistency:
Y “ ř a P A
I r A “ a s Y p a q ; (2) no unmeasured confounders: for each a P A , Y p a q KK A | Z ; (3) positivity: p p a | z q ě ϵ ą 0 for any z P Z .
Note that the no unmeasured confounders assumption can be relaxed.
If there are possible confounding issues as in the observational studies, our proposed method can be further generalized to deal with such issues by using propensity scores.
Based on the three identiﬁability assumptions, the value function can be written as V p
[Equation] Z, A “ a s ‰ .
Therefore, the optimal ITR D ˚ can be derived from D ˚ p
z q P arg max a P A E r Y | Z “ z , A “ a s for each z P Z .
This motivates us to estimate the conditional treatment effect E r Y | Z, A s .
Speciﬁcally, we consider the following regression model:
[Equation]
where the redundant function M 0 p Z q is the main effect of treatments, and T p Z ; ζ a q is the interaction effect between treatment a and the covariates.
We assume that the interaction effect for each treatment a P A can be characterized by the treatment-speciﬁc parameter ζ a P R p that has the same dimension as the covariates Z .
Here, we consider that the covariates Z contain the intercept term so that the treatment-speciﬁc effect is included in T p Z ; ζ a q .
Note that a sum-tozero constraint for the interaction terms is assumed for identiﬁability of the regression function.
Given an estimated parameter p ζ “ p p ζ ⊺ 1 , . . .
, p ζ ⊺ M q ⊺ , the estimated ITR can be equivalently written as p D p z q P arg max a P A T p z ; p ζ a q for each z P Z .
With Model (1) in place, we propose our SCAF method to cluster the treatments with similar treatment effects into a combined treatment group.
To achieve this, we aim to estimate and fuse the ζ a ’s into several groups.
The group structure of the ζ a ’s can equivalently represent the clustering results for the treatments.
Note that the main effect function M 0 p Z q can be estimated from the weighted parametric or nonparametric regression models [32].
More details about estimating the main effect can be found in Appendix A.1 of supplementary materials.
Denote the estimation of M 0 as x M 0 , and let Y be the residual Y ´ x M 0 p Z q .
In this article, we focus on linear interaction effects, i.e., T p z , ζ a q “ z ⊺ ζ a .
Let ζ “ p ζ ⊺ 1 , . .
, ζ ⊺ M q ⊺ .
Denote the true value of ζ to [-beζ 
-]be ζ 0 “ p ζ 0 ⊺ 1 , . . .
, ζ 0 ⊺ M q ⊺ P R Mp , where ζ 0 a P R p is the true parameter for treatment a “ 1 , . . .
, M .
In order to estimate and cluster ζ a ’s, we consider the following optimization problem by imposing a pairwise fusion penalty to each pair of the treatment-speciﬁc parameters:
[Equation]
where L p¨ , ¨q is the prespeciﬁed loss function to characterize the goodness of ﬁt, }¨} 1 is the ℓ 1 norm of a vector, p λ n is the penalty function to encourage fusing p ζ a ’s into groups, and λ n is the tuning parameter.
Here, E n denotes the empirical mean of the training data.
Speciﬁcally, the optimization problem (2) can be interpreted by maximizing the goodness of ﬁt, while at the same time minimizing the heterogeneity among treatments.
When the true treatment effects ζ 0 a ’s have homogeneous clustering structure, the estimated parameter p
ζ a ’s are expected to recover this structure.
In practice, one may assume that only certain elements of Z P R p (denoted as X P R d ) contribute to the interaction term T p Z, ζ a q while others (denoted as V P R p ´ d ) only show up in the main effects M 0 p Z q [28].
Without loss of generality, assume the ﬁrst d components of Z are the heterogeneous variables X and the remaining p ´ d variables are the homogeneous variables V , i.e., Z “ p X ⊺ , V ⊺ q ⊺ .
Let the coefﬁcients for treatment
a
be ζ a “ ` p ζ a, 1 , . . .
, ζ a,d q , p ζ a,d ` 1 . . .
, ζ a,p q ˘ ⊺ “ p β ⊺ a , γ ⊺ a q ⊺ , where ζ a,k is the regression coefﬁcient of covariate Z k ( k “ 1 , . . . , p ), and β a and γ a are the coefﬁcients for X and V respectively.
By removing the main effect M 0 p Z q and consider the residual Y , the true regression coeffecient of V for each treatment should be 0 , i.e., γ 0 a “for all a P A .
In this case, the penalty term should only be imposed on the coefﬁcients of X , i.e., β a .
If prior knowledge of whether the covariate belongs to X or V is unknown, we propose the group-lasso based algorithm in Section 3 to classify Z into homogeneous variables V and heterogeneous variables X .
Denote the training data as p x
i , v i , a i , y i q for i “ 1 , . . . , n , where x i P R d and v i P R p
´ d .
We rewrite Model (1) by decomposing T p Z ; ζ a q into two parts:
[Equation]
After distinguishing X and V from the whole variables Z , the optimization problem in (2) becomes p β p λ n q “
[Equation]
where Q n p β ; λ n q is the optimization function, and p β p λ n q is the estimated coefﬁcients based on the training data.
Note that if we do not detect the homogeneous variables X from Z , then the pairwise penalty in (2) would be applied to the p -dimensional vector ζ a ’s. For comparison, in (4), only the subvector, i.e., the d -dimensional vectors β a ’s are included in the penalty term.
Our empirical results show that our fusing algorithm becomes more effective to solve the optimization problem since the low dimensional vector is easier to be merged together.
The minimization problem (4) can be viewed as the supervised clustering process.
In the regressionbased Q-learning framework, supervised clustering can be interpreted by clustering the treatment effects (relationships described by the regression model: reward Y „ covariates X ˆ treatments A ) with the penalized regression problem in (4), which is formulated by balancing loss ` penalty terms with the tuning parameter λ n .
The pairwise fusion penalty encourages β a ’s to merge together so that the treatments with similar treatment effects can be clustered into the same group.
Algorithms for SCAF
Group-lasso based algorithm to classify Z into X and V .
After obtaining the residual Y by subtracting x M 0 p
Z q from Y , we propose the following group-lasso based algorithm to identify X and V .
Recall that ζ “ p ζ ⊺ 1 , . . .
, ζ ⊺ M q ⊺ and ζ a “ p ζ a, 1 , . . .
, ζ a,p q ⊺ for treatment a P A .
Note that ζ a,k is the regression coefﬁcient of covariate Z k p k “ 1 , . .
. , p q for treatment a P A .
We rewrite ζ by sorting it based on the covariate order, i.e., ξ “ p ξ ⊺ 1 , . . .
, ξ ⊺ p q ⊺ , where ξ k “ p ζ 1 ,k , . . .
, ζ M,k q ⊺ for k “ 1 , . . .
, p , is the coefﬁcient vector of Z k for all treatments.
By the deﬁnition of the homogeneous variables V P R p ´ d , the true regression coefﬁcient ζ 0 a,k “ 0 for all k “ d ` 1 , . .
, p and all a P A .
Hence, the whole true coefﬁcient vector can be written by the following sparse group structure, i.e, ξ [Equation] [Equation]
⊺ q ⊺ , where the coefﬁcients of each covariate are considered as a group.
This motivates us to implement group-lasso penalty to shrink the whole group vector ξ k that corresponds to covariate Z k to 0 to achieve group sparsity.
If p ξ k “ 0 , then Z k should be classiﬁed as
V , otherwise it should belong to X .
In particular, we consider the following optimization problem:
[Equation]
where }¨} b is a given base norm for each ξ k and λ glasso is the tuning parameter.
Typical choices of base norms include the }¨} 2 or }¨} 8 norm, so that the group-lasso penalty term becomes the block ℓ 1 { ℓ 2 or ℓ 1 { ℓ 8 norm, i.e., the ℓ 1 norm on all groups with the ℓ 2 or ℓ 8 norm for each group.
Adaptive proximal gradient algorithm.
Next we introduce our main algorithm to solve problem (4).
We consider the ℓ 2 -loss function for L p¨ , ¨q and choose the adaptive pairwise fusion penalty weighted by ω for p λ n p¨q .
Let β “ p β ⊺ 1 , . . .
, β ⊺ M q ⊺ .
Speciﬁcally, the optimization problem (4) can be written in the following compact form
[Equation]
where
[Equation]
Here, ω l,t is the weight of penalizing the ℓ 1 -distance between β l and β t .
We iteratively solve the problem coming from (6) and (7).
In practice, we ﬁnd that ω l,t “ min ␣ B ω , 1 {} r β l ´ r β t } 1 ( can be a good option where r β “ p r β ⊺ 1 , . . .
, r β ⊺ M q ⊺ is an estimate for β from the last iteration.
The initial estimate for β can be obtained from (5) in the group-lasso step.
Moreover, B ω is a prespeciﬁed constant to upper bound ω l,t to handle small distance of the pair p r β l , r β t q .
The adaptive weight helps to adjust for the potential bias created by the ℓ 1 -penalty.
Intuitively, large weight is expected to be assigned to treatments within the same group, otherwise, small weights are implemented.
Note that both L n p β q and P n p β q are convex functions for β , and L n p β q has Lipschitz gradient.
Thus, we can utilize the accelerated proximal gradient algorithm [30] to iteratively solve the problem coming from (6) and (7).
Denote the gradient vector of L n p β q as ∇ L n p β q and the Lipschitz constant of ∇ L n p β q as l n .
The computational cost of our algorithm mainly includes two following parts: (1) calculate l n and evaluate the gradient ∇ L n p β q for the updated β ; (2) compute the proximal operator of P n , which is deﬁned as prox s n P n p β q : “ arg min s β ␣ P n p s β q ` 12 s n } s β ´ β } 22 ( , for any updated β and the step size s n ą 0 .
For a ﬁxed λ n , the main steps of the proposed algorithm for SCAF are summarized in Algorithm 1.
The tuning parameter λ n can be tuned by cross validation based on the IPWE of the value function performed on the validation data.
For z P Z , the ﬁnal estimated ITR p D p z q is obtained from randomly sampling one treatment from the optimal estimated treatment group with probability proportional to the propensity score.
The tuning criteria is E n “ I p p D p Z q“ A q p p
A | Z q Y ‰L E n “ I p p D
p Z q“ A q p p
A | Z q ‰ , which is larger the better [33, 10].
Here, E n denotes the empirical mean for the validation data.
Theoretical properties
In this section, we establish the theoretical guarantee for consistency of the estimated regression coefﬁcients.
For Model (3), consider the M treatments can be partitioned into K disjoint treatment groups t G k u Kk “ 1 , where the effects of treatments within the same treatment group are identical.
Here, the group number K is typically unknown in practice.
Let β 0 “ p β 0 ⊺ 1 , β 0 ⊺ 2 , . . .
, β 0 ⊺ M q ⊺ be the true treatment-speciﬁc regression coefﬁcients in (3).
The values of β 0 a ’s from the same treatment group
G k for k “ 1 , . . .
, K are the same.
In particular, we aim to develop the convergence of p β p λ n q Ñ βas n Ñ 8 , which equivalently demonstrates consistency of recovering the true group structure.
Let X P R n ˆ p be the design matrix of X .
Denote X a P R n a ˆ p as the submatrix of X where the observations in X a are assigned to treatment a .
Here, n a is the number of patients receiving treatment a and hence ř Ma “ 1 n a “ n .
Further denote U “ diag p X 1 , X 2 , . . .
, X M q P R n ˆ Mp .
Under the true group structure, let α 0 “ p α 0 ⊺ 1 , α 0 ⊺ 2 , . . .
, α 0 ⊺ K q ⊺
P R
Kp be the distinct values of β 0 , where α 0 k P R p is the true treatment group-speciﬁc coefﬁcients for k “ 1 , . . .
, K .
Denote N k to be the number of patients whose assigned treatments belong to treatment group G k for k “ 1 , . .
, K .
Then we have ř
Kk “ 1 N k “ n .
In addition, we use | G k | to denote the number of treatments in the k -th group where ř
Kk “ 1 | G k | “ M .
For simplicity of notations, we assume that the treatment index has been well sorted based on the true group structure.
Then, under the true group memberships G 1 , . .
, G K , let
H k P
R
N
k
ˆ p be the submatrix of X where the observations in H k receive treatments in group G k for k “ 1 , . . .
, K .
Further denote H “ diag p H 1 , H 2 , . .
, H K q P R n ˆ Kp .
In particular, under the true group structure, deﬁne the oracle estimator for α 0 to be p α or : “ arg min α 12 } ¯ y ´ H α } 22 .
Equivalently, p α or “ p H ⊺ H q ´ 1 H ⊺ ¯ y .
The corresponding oracle estimator for β , denoted as p β or , is obtained by expanding p α or based on the true group structure.
Hereafter, we allow the number of treatments M , the true treatment group number K , and the covariates’ dimension p to grow as the sample size n increases.
We use M n , K n , and p n to denote them respectively.
In addition, for any given matrix G “ p G ij q s,ti “ 1 ,j “ 1 , denote } G } 8 : “ max 1 ď i ď s ř tj “ 1 | G ij | .
For a vector g “ p g 1 , . . .
, g s q ⊺ P R s , denote } g } 8 : “ max 1 ď i ď s | g i | .
We use l n " k n to denote l ´ 1 n k n “ o p 1 q .
We ﬁrst establish the concentration bound for consistency of the oracle estimator when the observation numbers in each treatment group employ a suitable distribution pattern.
Speciﬁcally, let
N min : “ min k “ 1 ,...,K N k be the minimum number of observations receiving the treatments from the same treatment group.
We make the following regularity assumptions in the high-dimensional statistical literature [27, 35]:
Assumption 1.
Assume } X j } 2 “ ?
n for j “ 1 , . .
. , p n , and } U ⊺ U } 8 ď B 1 M n p n where B 1 ą 0 .
Assumption 2.
The minimum eigen value λ min p H ⊺ H q ě C 1 N min where C 1 ą 0 .
Assumption 3.
Let ϵ “ p ϵ 1 , . . .
, ϵ n q ⊺ P R n be the error vector in model (3).
Suppose ϵ has subGaussian tail bound, i.e., there exists a constant c 1 P p 0 , `8q such that, for any vector a P R n and any t ą 0 , we have Pr p| a ⊺ ϵ | ą } a } 2 t q ă 2 exp p´ c 1 t 2 q .
Theorem 1 (Consistency of oracle estimator) .
Suppose Assumptions 1-3 hold.
If K n p n “ o p n q , and
N min " a p K n p n q n log n, then with probability at least 1 ´ 2 K n p n { n , we have ››› p β
or ´ β 0 ››› 8 “ ››p α or ´ α 0 ›› 8 ď ϕ n , where ϕ n “ c ´ 1 {
[Equation]
Since a p K n p n q n log n !
N min ď n { K n , we have K n a p K n p n q “ o p a n { log n q .
Hence,
K n must satisfy K n “ o p n 1 { 3 {p log n q 1 { 3 q .
It indicates that we cannot have too many treatment groups.
If we set N min “ δn { K n where δ P p 0 , 1 q , then the bound ϕ n “
[Equation]
[Equation]
Our next step is to show that the oracle estimator p β or is a local minimizer of the object function
Q n p β ; λ n q in (6) with probability 1 by giving a lower bound of the minimum signal difference between treatment groups.
Then the convergence of p β p λ n q Ñ β 0 can be established.
To achieve that, we need to assume the penalty function p λ n p¨q in (4) has sharp derivative around 0 to adjust the bias of the fusion penalty.
Some popular penalties considered in [36], [37] and [38] satisfy similar properties as the following assumption.
Assumption 4.
Suppose p λ n p¨q is symmetric around 0 for each λ n , and there exists a constant a ąsuch that p λ n p t q is a constant for all t ě a 2 λ n .
Assume p λ n p 0 q “ 0 , and it is differentiable aroundwith its derivative satisfying p 1 λ n p t q ě O p?
n log n q{ inf 1 ď k ď K n | G k | for all {+
+}
ă t ď 2 ϕ n .
{+Let b n “ inf i P G k ,j P G k 1 ,k ‰ k 1 } β 
 i ´ β 0 j } 1 “ inf k ‰ k 1 } α 0 k+}
´ α 0 k 1 } 1 be the minimal difference of the regression coefﬁcients between two groups.
Intuitively, we need the distances between two different groups to be large enough so that we can recover the group membership.
In particular, we make the following assumption.
A similar assumption was also used in [27] and [39].
Assumption 5.
Suppose b n { p n ą aλ n and λ n " ϕ n .
Theorem 2.
Suppose Assumptions 1-5 hold, then there exists a local minimizer p β p λ n q of the objective function Q n p β ; λ n q in (6) such that Pr ` p β p λ n q “ p β or ˘ Ñ 1 .
Theorem 1 together with Theorem 2 establish the convergence property of p β p λ n q Ñ β 0 .
For our implementation in Section 3, we iteratively solve the minimization problem (4) with the adaptive fusion penalty shown in (7).
We use p β p i q “ p p β p i q 1 , . .
, p β p i q M q ⊺ to denote the estimation of β in the i -th iteration.
For l, t P A , the weight in the i -th iteration ω p i q l,t is speciﬁed as min t B ω , 1 {} p β p
i
q l ´ p β p i ` 1 q t } 1 u .
Hence, the derivative of p λ n p¨q evaluated at } p β p i q l ´ p β p i q t } 1 , which equals to λ
n ω p i q l,t , will become extremely large when } p β p
i
q l ´ p β p i q t } 1 is close to 0 , and it will be relatively small if the pair p p β p i q l , p β p i q t q are not fused together.
Consequently, if p β is close to β 0 , then the derivative of p λ n p¨q will approximately satisfy the properties in Assumption 4 within a neighbourhood around β 0 .
Therefore, based on Theorem 2, the oracle estimator p β or can still be the local minimizer of
Q n p β ; λ n q within a neighbourhood around β 0 when our adaptive fusion penalty is utilized.
Experiments
We evaluate the ﬁnite-sample performance of our method using simulation and a real data application to the Patient-Derived Xenograft (PDX) study.
We compare our method with the state-of-art methods for the ITR problem: (a) RF (Random Forest, [40]); (b) PLS ( ℓ 1 -Penalized Least Square method, [5]); (c) AD (Multi-armed Angle-based Direct learning method, [9]).
However, none of these existing methods consider the treatment group structure in a large treatment space.
Data generation for simulation.
We generate 10 -dimensional independent feature variables
Z 1 , . .
, Z 10 , following U r´ 1 , 1 s .
The outcome Y is normally distributed with E r Y | Z,
A s “ 1 ` 2 Z 1 ` Z 2 ` 0 .
5 Z 3 ` T 0 p Z, A q and standard deviation 1, where T 0 p Z, A q reﬂects the interaction between the treatment and the feature variables.
Speciﬁcally, we conduct the following three scenarios where the treatment effects have the homogeneous grouping structure G :
Scenario 1.
M “ 10 , K “ 2 , G “ t G 1 , G 2 u “ ␣ t 1 , 2 , 3 , 4 , 5 u , t 6 , 7 , 8 , 9 , 10 u ( , X “ p 1 , Z 1 , Z 2 q , and T 0 p
Z, A q “ 1 . 8 ` 0 .
2 ´
Z 1 ´ Z 2 ˘` I r A P G 1 s ˆ p´
q ` I r A P G 2 s ˆ 1 ˘ ; Scenario 2.
M “ 15 , K “ 3 , G “ t G 1 , G 2 , G 3 u “ ␣ t 1 , 2 , 3 , 4 , 5 u , t 6 , 7 , 8 , 9 , 10 u , t 11 , 12 , 13 , 14 , 15 u ( , X “ p 1 , Z 1 , Z 2 q , and T 0 p
Z, A q “ 5 ` p´ 0 .
2 ` Z 1 ` 2 Z 2 q I r A P G 1 s`p 0 . 3 ` 2 Z 1 ` Z 2 q I r A P G 2 s`p´ 0 .
2 ` 3 Z 1 q I r A P G 3 s ˘ ; Scenario 3.
M “ 20 , K “ 5 , G “ t G 1 , G 2 , G 3 , G 4 , G 5 u “ ␣ t 1 , 2 , 3 , 4 u , t 5 , 6 , 7 , 8 u , t 9 , 10 , 11 , 12 u , t 13 , 14 , 15 , 16 u , t 17 , 18 , 19 , 20 u ( , X “ p 1 , Z 1 , Z 2 , Z 3 , Z 4 q , and T 0 p
Z, A q “ 3 . 5 ` p 0 .
5 ` 0 .
5 Z 1 ` 1 .
5 Z 2 ` 2 Z 3 ` {+1 . 
+}
Z 4 q
I r A P G 1 s ` p 1 ´ 2 Z 1 ´ Z 2 ´ 3 Z 3 ´ 2 Z 4 q I r A P G 2 s ` p´ 1 ` Z 1 ´ 2 Z 2 ` Z 3 ´ Z 4 q I r A P G {+3 s ` p´ 
+}
´
Z 1 ` Z 2 ´ Z 3 ` Z 4 q
I r A P G 4 s `
[Table]
As the number of treatments M increases from 10 , 15 to 20 , the above three scenarios correspond to the treatments having 2 , 3 and 5 treatment groups respectively.
For each scenario, the true decision boundaries are linear under the group domain.
Take Scenario 1 for example.
For any z P R 10 , the optimal ITR D ˚ p
z q would recommend any treatment in G 1 when 0 .
2 ´
z 1 ´ z
2 ă 0 , and any treatment in G {+
 when+} 0 .
2 ´
z 1 ´ z 2 ě 0 .
The treatments in all the scenarios are assigned in the unbalanced structure, which means the propensity score of speciﬁc treatments can be small.
The tuning parameter λ n is selected by the 5 -fold cross-validation.
For each scenario, the training sample sizes vary from 200, 400 to 600 and we replicate the simulations for 200 times.
On an independently generated testing data of size 10 , 000 , we evaluate the above methods using (a) empirical value function; (b) group-based misclassiﬁcation rate between the estimated group decision rules and the optimal group decision rules.
The empirical value function is calculated using
E
[Equation] [33, 10], where E n denotes the empirical mean for the testing data.
Since the treatment effects are identical within the same treatment group, the misclassiﬁcation rate under the group domain is equivalent to that under the individual treatment domain.
Results for simulation.
The boxplots of the empirical value and the misclassiﬁcation rate evaluated on the independent testing data are shown in Figure 1 and Figure C.1 in the supplementary materials.
In most cases, compared with RF, AD and PLS, our proposed SCAF method has superior performance with higher empirical values, smaller misclassiﬁcation rates, and extremely lower variabilities for both evaluation criteria.
When the treatment space becomes large, RF, AD and PLS suffer from inaccurate estimation of the individual treatment effect due to the small amount of observations for some speciﬁc treatments.
Moreover, these methods do not consider the possible group structure in the treatment space and hence employ relatively large variance.
As a comparison, SCAF is able to recover the treatment group structure and reduce the dimension of treatment space by clustering the treatments with similar treatment effects into the same treatment group.
We also show the ratio of exactly recovering the true treatment group structure of SCAF among the 200 replications in Figure C.2 in the supplementary materials.
Furthermore, we use Scenario 1 to demonstrate the advantage of our group-lasso step for ﬁnding the heterogeneous variables X from Z .
By implementing the group-lasso before the fusion step, as shown in Figure C.3 in the supplementary materials,
SCAF with the group-lasso step has better performance than that without the group-lasso step since we only need to fuse the lower dimensional subvectors β a ’s rather than ζ a
’s
[Figure]
[Figure] [Figure] Figure 2 demonstrates the solution path of the estimated treatment group structure as the tuning parameter λ increases based on one of the replications for the three scenarios.
When λ “ 0 , SCAF is equivalent to RF and PLS since the fusion penalty is not imposed and does not show any clustering pattern.
As λ increases, the fusion penalty encourages the treatments to merge together and cluster them with the expected group structure.
When λ becomes large enough, all of the treatments will be merged together and only one treatment group will be formed.
With the best tuned λ , 2 , 3 andgroups are correctly selected for the three scenarios respectively.
Application to Patient-Derived Xenograft (PDX) study.
Due to the complexity of human cancer, there’s signiﬁcant heterogeneity of treatment effects among the high throughput patients’ genomic biomarkers such as RNA and DNA sequencing [41].
Utilizing the transferred tumor pieces from patients to mice, the PDX study [42, 43, 44] aims to personalize the optimal cancer treatment for ﬁve types of cancer among a large number of FDA-approved preclinical cancer therapies, given the assigned treatments, the observed genomic features and responses.
In particular, among ﬁve types of cancer, we focus on the ColoRectal Cancer (CRC), which includes 43 PDX lines with complete genomic information.
For CRC, 847 mice expanded from different PDX lines were treated with unique treatments among M “ 20 single or combination treatments.
The response Y is measured by the scaled maximum observed tumor shrinkage from the baseline time, where the larger value is preferable.
After the preprocessing and supervised screening steps shown in [29], we selectsigniﬁcant genomic biomarkers among all features. [Figure] [Table]
[Figure]
Our goal is to identify the possible homogeneous group structure among thesetreatments and cluster the treatments with similar treatment effects in order to boost the performance of the estimated ITR.
The boxplot of the observed response for the 20 treatments in PDX data is shown in Figure 3.
We can see that, with the highest mean and median of the response, the combination treatment BYL+BIN is superior than other treatments.
For all methods, we randomly split the data into six folds with 200 replications where ﬁve folds are used to train the model in each replication.
We use the remaining fold to compute the empirical value based on the estimated ITR.
The tuning parameters are selected based on the three-fold cross validation.
Results for the PDX study.
We ﬁrst include all the single and combination treatments in the model.
As shown in the left panel of Figure 4, with the highest empirical value, SCAF can provide more effective ITR than other comparison methods.
On the overall dataset, we plot the solution path of the hierarchical clustering for all treatments in the left panel of Figure 5.
With the best tuned λ , it can be seen that the 20 treatments can be clustered into three groups.
Most of the combination treatments belong to the same group (the green one).
One possible explanation is that they include BYL as a common component.
It is interesting to point out that the combination treatment BYL+BIN forms a treatment group itself.
Moreover, SCAF recommends BYL+BIN as the optimal treatment to 92 .
1% patients among the PDX data, which is consistent with the outstanding performance of
BYL+BIN shown in Figure 3.
This also explains why BYL+BIN itself forms a group because its recommendation proportion nearly dominates the other treatments.
To better examine the group structure of the treatment effect, we exclude the combination treatments and only consider the 13 single treatments due to the superior performance of BYL+BIN.
Based on the right panel of Figure 4, the overall values of all four methods decrease compared with the left panel because we drop the combination treatments and reduce the treatment space, while our proposed SCAF still has the best performance.
Similarly, we draw the solution path of the hierarchical clustering for the single treatments in the right panel of Figure 5.
The best tuned λ further suggests that the single treatments can be clustered into three groups where the second treatment group (the green one) is recommended as the optimal to 94 .
1% patients in the PDX data.
The outstanding performance of the second group matches with Figure 3 because the treatments HDM and LKA in the ﬁrst group (the pink one) and CGM, ENC and LJC in the third group (the blue one) are the treatments with smallest mean but highest variance among the 13 single treatments.
In particular,
−0. R e s pon s e . S c a l ed With All Treatments −1.
−0.
R e s pon s e .
S c a l ed
Without Combination Treatments
Methods
RF
AD
PLS
SCAF
Figure C.4 in the supplementary materials shows the path of empirical value function of the testing data for both cases.
As λ becomes larger, the empirical value reaches the maximum point due to the helpful clustering structure and ﬁnally dramatically decreases because of the excessive merging pattern imposed by the large λ .
The results in Figure C.4 are consistent with Figure 5. [Figure]
Discussion
In this paper, we consider the challenging ITR problem with a large number of treatments which may have homogeneous group structure.
We propose the SCAF method to identify this structure and cluster the treatments in order to learn the optimal ITR more effectively.
In particular, we adapt the idea of supervised clustering and formulate the problem as convex optimization that consists of loss ` penalty .
We are able to cluster the treatments and estimate the optimal ITR simultaneously within a single optimization problem.
The whole clustering process can be intuitively visualized with a dendrogram plot similar to the hierarchical clustering.
An efﬁcient algorithm is proposed to solve the problem and the numerical results demonstrate the superior performance of SCAF.
The proposed method can be extended in several aspects.
First, we consider the continuous outcome in this paper.
One interesting future direction is to extend SCAF to deal with various types of outcomes, such as discrete outcome and survival outcome.
Second, our method can be extended to learn the group structures of treatments for multi-stage dynamic treatment regimes [6, 11].
We leave these interesting directions for future research.
Acknowledgments and Disclosure of Funding
The authors would like to thank the helpful and constructive comments from the reviewers which led to a much improved presentation of this paper.
Yufeng Liu was supported in part by NSF grant
DMS 2100729 and NIH grant R01GM126550.
Donglin Zeng was partially supported by NIH
R01GM124104, R01MH123487 and R01NS073671.
Any opinions, ﬁndings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reﬂect the views of National Science Foundation or National Institute of Health.