Interactive rank testing by betting
Abstract
In order to test if a treatment is perceptibly different from a placebo in a randomized experiment with covariates, classical nonparametric tests based on ranks of observations/residuals have been employed (eg: by Rosenbaum), with ﬁnite-sample valid inference enabled via permutations.
This paper proposes a different principle on which to base inference: if — with access to all covariates and outcomes, but without access to any treatment assignments — one can form a ranking of the subjects that is sufﬁciently nonrandom (eg: mostly treated followed by mostly control), then we can conﬁdently conclude that there must be a treatment effect.
Based on a more nuanced, quantiﬁable, version of this principle, we design an interactive test called i-bet: the analyst forms a single permutation of the subjects one element at a time, and at each step the analyst bets toy money on whether that subject was actually treated or not, and learns the truth immediately after.
The wealth process forms a real-valued measure of evidence against the global causal null, and we may reject the null at level α if the wealth ever crosses 1 /α .
Apart from providing a fresh “game-theoretic” principle on which to base the causal conclusion, the i-bet has other statistical and computational beneﬁts, for example (A) allowing a human to adaptively design the test statistic based on increasing amounts of data being revealed (along with any working causal models and prior knowledge), and (B) not requiring permutation resampling, instead noting that under the null, the wealth forms a nonnegative martingale, and the type-1 error control of the aforementioned decision rule follows from a tight inequality by Ville.
Further, if the null is not rejected, new subjects can later be added and the test can be simply continued, without any corrections (unlike with permutation p-values).
Numerical experiments demonstrate good power under various heterogeneous treatment effects.
We ﬁrst describe i-bet test for two-sample comparisons with unpaired data, and then adapt it to paired data, multi-sample comparison, and sequential settings; these may be viewed as interactive martingale variants of the Wilcoxon, Kruskal-Wallis, and Friedman tests.
1. Introduction
The problem of testing whether a treatment has any effect in a randomized experiment without parametric assumptions is frequently encountered in biology, medical research, and social sciences (see, for example, Olive et al.
; Aguilera et al.
(2017); Rastinehad et al.
A classical nonparametric method is the Wilcoxon test, and there have been several proposed extensions that adjust for covariates, in order to better detect the treatment effect.
For example, suppose we want to evaluate a medication by comparing the blood pressure (outcome) of subjects who take the medication (treatment) with that of subjects who do not (control).
The blood pressure could be affected by the subject’s gender, age, etc.—accounting for these would help increase power, especially when the medication only affects a subpopulation.
In this paper, we use a novel “game-theoretic” principle of guessing and betting on the treatment assignments using all data except the truth assignments, and conclude there is an effect if most guesses are correct.
Our proposed test is “interactive” — it allows an analyst to look at (progressively revealed) data and adaptively explore arbitrary working models for making the bets — which improves power especially under heterogeneous treatment effects.
1.1. Problem setup
Consider a sample with n subjects.
Let the outcome of subject i be Y
i , the covariates be X
i , and the treatment assignments be indicators A i for i ∈ [ n ] ≡ { 1 , 2 , . . . , n } .
The null hypothesis of interest is that there is no difference between treated and control outcomes conditional on the covariates 1 :
[Equation]
Rejecting the above null means that there exist some subjects who respond differently when treated or not.
We do not further identify which subject respond differently.
Testing the above global null may appear in an exploratory analysis to see whether the treatment has any effect on any person, or as a building block within a closed testing procedure.
For our interactive algorithm that we propose later to succeed in rejecting the global null, it must indeed implicitly learn which part of the covariate space exhibits this difference between treatment and placebo, and if the global null is rejected, one may use this information to design follow-up studies or analyses focused on other goals.
This paper deals with classic randomized experiments, and in particular we assume that
(i) (random assignment) the treatment assignments are independent and randomized: {+P ( A i = 
+}
| X i ) =
µ i ≡ µ i ( X i ) ∈ (0 , 1) , where µ i is known for all i ∈ [ n ]; (ii) (no interference) the outcome of any subject Y i 1 depends only on their assignment
A
i 1 and does not depend on the assignment
A
i 2 for any i 1 (cid:54) =
i 2 ∈ [ n ] .
Note that the above considers the case where the probabilities of receiving treatment { µ i ( X i ) } ni = are known (but the total number of treated subjects is not ﬁxed).
The methods are easily extended to the case where the number of treatments is ﬁxed: (cid:80) ni =1 A i = m , and they are assigned to a random subset of subjects (see Remark 5).
To enable us to effectively adjust for covariates, we use the following “working model”:
[Equation]
where ∆( X i ) is the treatment effect, f ( X i ) as the control outcome, and U i is zero mean ‘noise’ (unexplained variance).
When working with such a model, we effectively want to detect if ∆( X i ) is nonzero.
Importantly, model (2) only exists on the analyst’s computer, and it need not be correctly speciﬁed or accurately reﬂect reality in order for the tests in this paper to be valid (but the more ill-speciﬁed or inaccurate the model is, the more test power may be hurt).
1.2. Rosenbaum’s covariance-adjusted Wilcoxon test
Recall that the Wilcoxon rank-sum test (also referred to as the Mann–Whitney U-test) calculates
[Equation]
where rank ( Z i ) is the rank of Z i amongst { Z i } ni =1 .
When the treatment effect is large, the subjects receiving treatment ( A i = 1) tend to have larger outcomes, and hence W ori would be large.
Rank-based statistics have been explored in many directions: see Lehmann and D’Abrera (1975) for a review.
Recent work focuses on how to incorporate covariate information to improve power.
Zhang et al.
(2012) develop an optimal statistic to detect constant treatment effect; in multi-sample comparison, Ding and Keele (2018) numerically compare rank statistics of outcomes or residuals from linear models; Rosenblum and Van Der Laan (2009) and Vermeulen et al.
(2015) focus on related testing problems for conditional average effect and marginal effect; Rosenbaum (2010) and
Howard and Pimentel (2020) use generalizations of rank tests for sensitivity analysis in observational studies.
Here, we focus on improving power under heterogeneous treatment effects.
Rosenbaum (2002) proposed the covariance-adjusted Wilcoxon test that considers the residuals of regressing the outcome Y i on covariates
X i (without assignment A i ).
Speciﬁcally, denote the residual for subject i as R i :
[Equation]
(3) where (cid:98) Y ( X i ) the prediction of Y
i using X i via any modeling and R i can be viewed as an approximation of the treatment effect after accounting for heterogeneous control outcome.
The covarianceadjusted Wilcoxon test replaces the outcomes with the residuals:
[Equation]
(4) abbreviated as CovAdj Wilcoxon test in the rest of the paper.
The rejection rule is based on permutation.
Note that under the null, the assignment A i is independent of other data information { Y i , X i } .
The permutation test estimates the null distribution of W by permuting the treatment assignments { A i } ni =1 , described as follows:
Input: Outcomes, treatment assignments, and covariates { Y i , A i , X i } ni =1 , target Type-I error rate α ; Procedure: 1.
Calculate W using the observed data { Y i , A i , X i } ni =1 ; 2.
Let W 1 = W
and for b = 2 , . .
, B , generate a random permutation of the treatment assignments ( A b 1 , . . .
, A bn )
; and calculate W b using the permuted data { Y i , A bi , X i } ni =1 ; 3.
Let W ( j ) be the j -th largest order statistic.
Reject the null if W > W ( (cid:98) α · B (cid:99) ) Algorithm 1:
The signed-rank test offers a general formula to construct permutation tests with various forms of test statistics W for two-sample comparison, which we discuss in Appendix G.
. Interactively constructing a ranking, and betting on it
In contrast to one-step tests such as CovAdj Wilcoxon test, we propose a multi-step test that progressively guesses and bet on the treatment assignments.
The intuitive idea is that if based on all covariates and outcomes, a human analyst can guess most treatment assignments correctly, then the treatment must have an effect and we can reject the null.
Several advantages of taking the above betting perspective include: (a) the ﬂexibility for the analyst to use combine (partial) data, prior knowledge, and arbitrary modeling for guessing and betting on the treatment assignments; (b) the bets are used to construct a sequence of test statistics and form a multi-step protocol, during which the analyst can monitor the current algorithm’s performance and is allowed to make adjustments to their working model at any step; (c) the constructed test statistics form a nonnegative martingale, and the type-I error control follows from a martingale property (detailed later), avoiding the high computation cost in data permutation for the rejection rule.
Despite allowing human interaction in (a) and (b), the proposed test always maintains valid type I error control without assuming any working model speciﬁed by the analyst to be correct.
Our proposed test by betting can be viewed as separating the information used for betting and interactive algorithm design and that for testing, via “masking and unmasking” (Figure 1).
Masking means we hide the information of treatment assignments { A i } ni =1 from the analyst.
Unmasking refers to the process of revealing the masked assignments one at a time to the analyst.
Consider a simple case where the treatment is assigned to each subject independently with 1 / 2 probability.
The test considers the cumulative products
[Equation]
(5) where { π j } nj =1 denotes an ordering interactively decided by the analyst, and w j ∈ [ − 2 , 2] is a user-deﬁned bet on the treatment assignment, both of which can be based on all the revealed information { Y i , X i } ni =1 and the true treatment assignments of all previous subjects in the ordering
A π 1 , . . .
A π j − 1 .
Regardless of the speciﬁc choices of the ordering and binary estimations and weights, w j · (2 (cid:98)
A π j − 1) is independent of A π j under the null (but not under the alternative); and thus, the process ( M t ) is a nonnegative martingale under the null.
We reject the null as soon as M t ≥ 1 /α for some t ∈ [ n ] (see the precise description of our procedure in Algorithm 2 of
Section 2). Type-1 error control is guaranteed by Ville’s (often attributed to Doob) maximal inequality (Ville, 1939), which states that with probability 1 − α , a nonnegative martingale with initial value one (which M t is, under the null) will never exceed 1 /α :
[Equation]
For a self-contained proof of this fact, see (Howard et al., 2020).
Ville’s inequality holds with equality for continuous path martingales (only possible in continuous time) and in discrete time, the only looseness is due to “overshoot” and is typically negligible (Howard et al., 2020), meaning that the inequality almost holds with equality, which is important so that the test is not too conservative.
Remark 1 There is nothing special about the use of 1 / 2 above, and one can simply use µ if there is some other probability of random assignment.
Further, if subject i was randomized with probability µ i ≡ µ ( X i ) possibly depending on its covariates, or in some kind of stratiﬁed manner, then we can simply use that µ i in (5) instead of 1 / 2 , retaining the required martingale property; we return to this formally later.
(In this case, the range of the bet w j must be adjusted to [ − 1 / (1 − µ i ) , 1 /µ i ] to ensure nonnegativity.)
It appears that in the last case, the exchangeability amongst subjects has been destroyed and so the permutation test in Algorithm 1 does not directly apply — this can be addressed using a recent sophisticated concept called weighted exchangeability that has utility in other settings (Tibshirani et al., 2019), but this leads us far astray; our point is simply that our procedure retains its simplicity under more complex randomization schemes.
Remark 2 Though we do not necessarily recommend viewing i-bet in this way, it is possible to view our interactive test as a computational shortcut for the permutation framework.
In short, whenever the test statistic is a nonnegative martingale (by design), permutations can be avoided.
To elaborate, one could imagine calibrating the test statistic W = max 1 ,...,n M t using algorithm 1.
What Ville’s inequality implies is that the α -upper quantile of the permutation null distribution for W (obtained by permuting the data and construct { W b } Bb =1 , with B = n !
for example), will be at most 1 /α .
The above test retains validity amidst signiﬁcant ﬂexibility.
For example, the analyst could employ any probabilistic working model or predictive machine-learning algorithm to guess the treatment assignments (cid:98)
A i ∈ { 0 , 1 } , perhaps along with an associated level of conﬁdence such as a posterior probability or a score ν i ∈
[0 , 1] , for each subject i that have not yet been included in the ordering.
Then, at step t of the algorithm, the next subject in the ordering π t could be the one where the analyst is most conﬁdent, and w t could equal 2(2 (cid:98)
A t − 1) ∈ {− 2 , 2 } or 2(2 ν t − 1) ∈ [ − 2 , 2] .
Intuitively, our algorithm tests whether there exist any non-nulls by examining whether we can succeed at guessing the treatment assignments better than random chance, and this is reﬂected by our ability to form “smart, nonrandom” bets that cause M t to grow faster than a martingale, and cross 1 /α as soon as possible.
When all subjects are nulls and we have that the treatment assignments are independent of the outcomes and covariates, we cannot distinguish subjects who are treated or not based on the outcomes and covariates, and no algorithm can result in M t losing this martingale property (and thus having controlled growth).
In contrast, if the null is false, we hope that our algorithm will be able to correctly guess the assignments, especially for subjects we are most conﬁdent about (ordered upfront), so that the cumulative products ( M t ) nt =1 grow large.
Interaction enters in the process of unmasking.
Intuitively, to construct large M t and reject the null, the analyst should guess whether a subject receives treatment while the assignments { A i } ni = are hidden.
She can guess the treatment assignments using the revealed data information { Y i , X i } ni = and { A π j } t − 1 j =1 (for the t -th iteration), and any prior knowledge, and she is free to use any algorithms or models.
Even if the model chosen initially is inaccurate because of masking, the interactive test progressively reveals the assignments (of the ﬁrst t − 1 subjects at step t ) to the analyst, so that she can improve her understanding of the data and update the model or heuristic for estimating the treatment assignments at any step.
We call our proposed procedure the i-bet test.
Our contribution is to provide a “game-theoretic” principle for the causal hypothesis testing problem, and demonstrate a new class of interactive multistep algorithms that, by masking some of the data and progressively revealing it to the scientist, can combine the strengths of (automated) statistical modeling and (human-guided) scientiﬁc knowledge, in order to reject the global null while not suffering from any p-hacking or data-dredging concerns despite a great deal of ﬂexibility provided to the scientist. Directly related work.
The game-theoretic principles of testing by betting stems from the books by Shafer and Vovk (2019, 2005) as elucidated in a recent paper by Shafer (2020). Recently, these { Y i , X i } ni = Prior information
Select the t -th subject and bet on its treatment assignment
Obtain M t
Reject Initialize t =
If M t > 1 /α
If M t ≤ 1 /α
Reveal A π t and update working model ( t ← t + 1 ) ideas have been successfully applied to election auditing (Waudby-Smith et al., 2021), and for constructing concentration bounds (Waudby-Smith and Ramdas, 2020).
Our work connects the betting perspective with the causal null hypothesis, introducing the various advantages as outlined in the abstract.
The advantage of optional continuation of experiments (equivalently, optional stopping) has been highlighted in particular by Gr¨unwald et al.
and Howard et al.
The idea of interactive testing was recently proposed by Lei and Fithian (2018) and Lei et al.
(2020), in the context of multiple testing problems to control FDR (the false discovery rate), followed by several works for other error metrics in multiple testing.
Our interactive test for twosample comparison relates most with the work of controlling the global type-I error (Duan et al., 2020), where the individual null hypothesis is zero effect for each subject, and the global null corresponds to the null of no treatment effect as null hypothesis (1).
Previous development of the interactive tests typically focuses on generic multiple testing problems, which operate directly on multiple p -values; in other words, the units of inference were p -values (for different hypotheses) rather than data points (to test a single hypothesis).
Here, interactive testing is directly applied to the observed data, expanding the potential of interactive tests.
For the related problem of two-sample testing, Lh´eritier and Cazals (2018) developed the same idea of constructing the test statistics as a nonnegative martingale.
One can view our paper as an extension of their work to causal inference settings, developing the core idea further along methodological, theoretical and practical fronts (for example, extending to sampling without replacement in order to handle different types of randomization beyond independent Bernoulli assignment).
We emphasize the ﬂexibility allowed to a human analyst to utilize arbitrary contextual knowledge prior to and during the test, with or without the aid of probabilistic modeling.
We also develop some extensions in Appendix E. Other related work on testing weaker null hypotheses is in Appendix F.
Outline.
The rest of the paper is organized as follows.
In Section 2, we describe the i-bet test in detail, followed by numerical experiments to demonstrate its advantage over standard methods in Section 3.
Section 4 concludes the paper with a discussion on the potential of interactive rank tests.
Extensions to various settings, such as paired data, are deferred to Appendix E.
2. An interactive rank test with covariates (i-bet)
To account for covariates through a ﬂexible algorithm that involves human interaction, we propose the i-bet test.
In short, the analyst decides the ordering of subjects { π j } nj =1 and the bets { w j } nj =  progressively: at step t , she selects the t -th subject from the to-be-ordered subjects [ n ] \{ π j } t − 1 j = and decides the bet w t , based on an increasing amount of data information starting from the assignments { A i } ni =1 masked and then gradually revealed.
Note that the bet can be unreliable, and in turn hurt the power, at the ﬁrst few steps if all the assignments are masked.
Thus, we reveal the complete data for a random subset of subjects at t = 0 , denoted as set I
(10% of all subjects for example).
At each iteration, we select subject π t and expand the set I t
= I t − 1 ∪ { π t } whose complete data is then revealed.
Mathematically, the data information available to the analyst at the end of step t is denoted by the ﬁltration:
[Equation]
The choice of π t and w t are predictable (measurable) with respect to F t − 1 , while the analyst is allowed to explore and choose arbitrary models or heuristics to form the ordering and get the bets.
After each iteration of selecting π t ∈ [ n ] \I t − 1 and choosing w t ∈ [ − [Equation] 1 µ
[Equation]
] , the test calculates
[Equation]
and the iteration stops once M t reaches the boundary 1 /α , or all the subjects in [ n ] \I 0 are ordered.
We summarize the i-bet test in Algorithm 2.
Input: Outcomes, assignments, covariates { Y i , A i , X i } ni =1 , target type-I error α , holdout ratio γ ;
Procedure: 1.
Random select a set I 0 ∈
[ n ] with size γ · n ; for t = 1 , · · · , | [ n ] \I 0 | do 2. Using F t − 1 , pick any π t ∈ [ n ] \I t − 1 and obtain an arbitrary bet w t ∈ [ − − µ πt , 1 µ πt ] ; 3.
Reveal A π t and update I t and F t ;
Algorithm 2: Framework for the interactive rank test (i-bet test)
2.1. Important remarks
Remark
We deﬁned the problem as testing the global null (1) of no treatment effect at a predeﬁned level α .
Instead, we could ask the test to output a p -value for the global null, or even better, to output an anytime-valid p -value, which is a sequence of p -values { p t } nt =1 such that for an arbitrary stopping time τ , p τ is also a valid p -value (its distribution is stochastically larger than uniform if the null is true).
Luckily, this is easy: p t = inf s ≤ t 1 /M s ﬁts the bill, once again due to Ville’s inequality.
Further, being a nonnegative martingale, the optional stopping theorem implies that the wealth process at any stopping time has expectation at most one under the null; this makes the wealth process an e-process (Ramdas et al., 2021; Gr ¨ unwald et al., 2019; Shafer, 2020).
The relationship between these objects is detailed in Ramdas et al.
Remark
The anytime-validity discussed above implies that the experiment can be extended to a larger size if the smaller size did not provide sufﬁcient evidence (meaning the wealth did not exceed 1 /α ), as discussed in Gr¨unwald et al.
and Howard et al.
It is indeed a remarkable property that if the null cannot be rejected using the current dataset, we can just continue experimentation: randomly assign l more people to treatment and control, reveal their covariates and outcomes, and continue the betting on the new l subjects starting with the wealth M n , extending the original ranking as if we had all l + n subjects from the start.
This optional continuation does not require adjustment for multiple testing before or after collecting new samples, because the wealth continues to be a nonnegative martingale under the null, the p -value is anytime-valid, and the probability the wealth ever exceeds 1 /α is at most α .
In game-theoretic terms, no amount of betting can make us signiﬁcantly rich in a fair game (characterized by the martingale property under the null) — even if we ﬁrst chose to play n rounds, and then later added l more rounds.
Remark
The i-bet test can be extended to a completely randomized experiment, where the number of treatments is ﬁxed and known as m at the beginning, and m subjects are randomly chosen to be treated.
In such a case, the revealed information additionally has the sum of treatment assignments
[Equation]
We would then construct the nonnegative martingale as M t = (cid:81) tj =1 (cid:2) 1 + w j · ( A π j − µ j ) (cid:3) , where w t ∈ [ − 1 1 − µ t , 1 µ t ] and µ t = (cid:16) m − (cid:80) i ∈I t − 1 A i (cid:17) /
( n − |I t − 1 | ) is the expected treatment assignment given the revealed information in F t − 1 .
Remark 6 Despite high ﬂexibility in choosing the weight w t and the order π t , good choices can increase power.
The choice of the ordering π t affects the test power, when taken together with the choice of weight w t .
So let us ﬁrst note that a desirable weight w t should ideally have the same sign as A π t ; this would allow M t to increase to sooner reach the rejection threshold 1 /α .
Therefore, we recommend practitioners to order upfront the subjects for which they (or the algorithm acting on their behalf) are most conﬁdent about their treatment assignment.
As an example, we provide an automatic approach to choose weight w t and order π t in Section 2.2.
Theorem 7 As long as an analyst explores and updates working models at any step t using only the information in F t , the i-bet test controls type-I error for null hypothesis (1) under assumptions (i),(ii) of randomized experiments.
In fact, the error control holds conditionally on { X i , Y i } ni =1 .
Although more information is revealed to the analyst after each step, the error control is valid, because under the null, the increment A π t for testing is independent of the revealed information:
[Equation]
The complete proof is in Appendix A.
The i-bet allows the analyst to incorporate covariates and various types of domain knowledge for ordering and choosing weights.
However, manually picking π t at every step could be tedious.
The analyst can instead design an automated algorithm for choosing π t and w t , such as the example we provide in the next section, and still keeps the ﬂexibility to modify it at any step.
A concrete, automated, instantiation of i-bet
We can infer the treatment assignments by exploring various models to ﬁt the (partial) data.
An example is to model the outcome as a mixture of the distributions for treatment and control groups:
[Equation]
, when A
[Equation]
, when A
[Equation]
with µ ji = θ j
( X i ) for j =
[Equation]
where θ j could be linear functions of the covariates and their second-order interaction terms.
The masked treatment assignments can be viewed as missing values, and by the EM algorithm (details in Appendix B), we get an estimated posterior probability of receiving the treatment for each subject.
The estimated probability of receiving treatment, denoted as (cid:98) q i , provides an estimation of the assignment and an approach to select π t .
Recall that we hope to order upfront the subject whose estimated assignment we are most conﬁdent, which can be measured by | (cid:98) q i − 0 .
5 | , so we could select π t = arg max i ∈ [ n ] \{ π j } t − 1 j =1 {| (cid:98) q i − 0 .
5 |} .
For the chosen subject, we bet on the treatment assignment by w j = 0 .
8(2 (cid:98)
A π j − 1) , where the estimated assignment (cid:98)
A π j := 1 { (cid:98) q π j > 0 . 5 } is a function of the estimated probability of receiving treatment.
We summarize this automated procedure in
Algorithm 3.
Input
Procedure
for t = 1 , · · · , | [ n ] \I 0 | do
Estimate
q i for subjects in n
− 1 ; 3. − Reveal A π t [-j =1 (cid:2) 
 + 1 .-]and update I t and F t ; if
+ 0 . q π > 0 .
−
A π j − > /α
then
By design, w j · ( A π j − 1 / 2) is +0 .
if the estimated assignment is consistent with the truth; and − 0 .
4 otherwise 2 .
Ideally, when the null is false, we could guess most assignments correctly and order them upfront, leading to a larger M t that could exceed the boundary 1 /α .
As the test proceeds and more actual assignments get revealed for interaction, we reﬁt the above model and update the estimation of posterior probabilities for every (cid:98) n 5 (cid:99) steps (say).
Keep in mind that the validity of the error control does not require model (10) to be correct.
The analyst can choose other models such as logistic regression for θ j if the revealed data or prior knowledge suggests so.
3. Numerical experiments
Though the primary contribution of our paper is the construction and derivation of a conceptually interesting test (or framework, since there is signiﬁcant ﬂexibility left to the analyst) for the global causal null, we attempt below to convince the reader that the ﬂexibility afforded by our interactive setup sufﬁces to deliver high power.
We view the simulations with i-bet as a thought experiment: the reader must imagine that we perhaps chose a poor model for all methods at the start.
Using a poor (uninformative or barely better than chance) model, our bet w j would possibly have a random sign for early subjects, and our wealth may ﬂuctuate up and down, rather than increase reasonably steadily.
The multi-step i-bet test can have higher power than other single-shot tests because, in the midst of this testing, the analyst can observe the poor start, explore and evaluate various models using the complete data for completed bets and the masked-assignment data for every other point, and try to ﬁnd a better model (details in the paragraph of “Illustration of adaptive modeling”).
Simulation setup.
To evaluate the performance of the automated algorithm, we simulatesubjects ( n = 500) .
Suppose each subject is recorded with two binary attributes (e.g., female/male and senior/junior) and one continuous attribute (e.g., body weight), all of which are denoted as a vector X i =
( X i (1) , X i (2) , X i (3)) ∈ { 0 , 1 } 2 × R .
Among n subjects, the binary attributes are marginally balanced, and the subpopulation with X i (1)
= 1 and X i (2) = 1 is of size n 0 (see
Table 1), where we set n 0 = 30 .
The continuous attribute is independent of the binary ones and follows the distribution of a standard Gaussian.
[Table]
X i (2) =nn/
2 − nn/
X i (2) =
n/ 2 − nnn/ Totals n/n/n
The outcomes are simulated as a function of the covariates X i and the treatment assignment
A i following the generating model (2), where we vary the functions for the treatment effect ∆ and the control outcome f to evaluate the performance of the i-bet test.
Recall that earlier, we used model (2) as a working model, which is not required to be correctly speciﬁed.
Here, we generate data from such a model in simulation to provide various types of underlying truth for a clear evaluation of the considered methods 3 .
Alternative tests for comparison.
In addition to the CovAdj Wilcoxon test, we compare the ibet test with a semi-parametric test derived from the literature of estimating conditional average treatment effect (CATE), which we refer to as the linear-CATE-test.
Here, the nonparametric testing problem is transformed into testing a parameter, potentially considering a less stringent null.
Speciﬁcally, null hypothesis (1) implies that
[Equation]
Assume that the outcome difference is a linear function of covariates
X i , the method for CATE provides an asymptotic conﬁdence interval for ψ ∗ , and the null is rejected if the conﬁdence interval does not include zero (see Appendix C for an explicit form of the test).
Note that the test has valid error control even if the outcome difference is not linearly correlated with X i , in which case, however, the power would be low.
The presented methods (the CovAdj Wilcoxon test, the linear-CATE-test, and the automated algorithm of the i-bet test) all involve some working model of the outcomes, but the extent of ﬂexibility varies.
The linear-CATE-test requires us to specify the parametric model before looking at the data; the CovAdj Wilcoxon test allows model exploration given partial data { Y i , X i } ni = before testing; and the i-bet test further permits the analyst to interactively change the model as the test proceeds and more assignments A i become available for modeling.
Test performances when the default model is a good ﬁt.
Consider outcomes from the generating model (2) with the treatment effect ∆ and the control outcome f speciﬁed as:
[Equation]
where S ∆ encodes the signal strength of the effect.
Intuitively, all subjects have some Gaussiandistributed effect correlated with X (3) and the subjects with X (1)
= 1 and X (2)
= 1 additionally have a constant positive effect.
In such a setting, all the methods with their working models speciﬁed as linear functions should ﬁt the data well.
For two-sided heterogeneous treatment effects, the CovAdj Wilcoxon test has low power because the positive effects cancel out with the negative effects in the sum statistics (4), while the linear-CATE-test and the i-bet test can accumulate the effect of both signs.
The linear-CATE-test has higher power as it targets the speciﬁc alternative of nonzero parameters in the linear model (11), although the i-bet test also achieves reasonable power in Figure 2.
Note that the three methods we compare (CovAdj Wilcoxon test, linear-CATE, i-bet test) all have valid type-I error control for the same global null H 0 , while these methods implicitly target alternatives in different directions.
Recall that we do not make any assumptions on the distribution of non-nulls — that is, if some people do respond to treatment, no assumption is made on how they respond — or how informative the covariates are.
It is well known that in such nonparametric settings, there is no universally most powerful test; for example, Janssen (2000) discusses this phenomenon when testing goodness of ﬁt.
We demonstrate next that i-bet could have higher power when the initial working model may be incorrect, among other situations.
Illustrations of adaptive modeling.
One advantage of the interactive test is that it allows exploration and adaptation of the working model using the revealed data.
Here, we present an example where model (10) might not ﬁt the data well; the reader must imagine that we do not suspect this at the start, so we begin by utilizing it anyway.
However, suppose the poor guidance provided by the incorrect model results in the algorithm making mistakes guessing the assignments at the ﬁrst few steps itself (even though we are ordering them from most to least conﬁdent!).
By a mistake, we mean that our bet w j had the wrong sign, and we lose some wealth.
The multi-step i-bet test can have higher power than other single-shot tests because, in the midst of this testing, the analyst can observe the poor start, explore and evaluate various models, and ﬁnd a reasonably good ﬁt using all the available revealed data.
(Practitioners using the one-shot methods may change their model and thus their test statistic after observing its performance on the full data, for example whether it rejects or not, but this technically invalidates their type-I error guarantee.)
[Figure]
Suppose the control outcome is nonlinearly correlated with the attributes by specifying function f in the generating model (2) as
[Equation]
[Equation] where the distribution of potential control outcomes is skewed (treatment effect ∆ is the same as before in (12)).
When we ﬁt the default working model (10) with linear functions (along with a few revealed treatment assignments), the QQ-plot and Cook’s distance indicate a poor ﬁt because of possible outliers in the outcomes (Figures 3( a ) and 3( b )).
An easy ﬁx is to use robust linear regression (Huber, 2004), which leads to signiﬁcant power improvement compared with the default algorithm (see Figure 3( c )).
(In practice, we recommend using robust regression from the very beginning anyway since it keeps good power when the working model is correct while it improves power when the control outcome has a skewed distribution.
The robust regression is also observed to improve power under heavy-tailed noise (see Appendix D.1).
However, for the purpose of this illustrative example, presume that we switch to it after observing a few early mistakes.)
Another example of model exploration considers treatment effect as a quadratic function of the covariates, and a robust regression with the quadratic term can perform better (see Appendix D.2).
Real-data application to the effect of screening ASB on reduction of low birth weight.
We use the data in Gehani et al.
(2021), which investigates whether antenatal screening of asymptomatic bacteriuria (ASB) can reduce low birth weight.
The study collected 240 participants and randomly selected 120 of them into the treated group (and the rest in the control group). The treated group additionally screened for ASB with a novel rapid test; and the control group did not receive such a test. The outcome Y i is the birth weight. The covariates X
i include trimester at the time of enrolment, gestation age, and the number of previous pregnancies.
We use a robust linear model in the i-bet test, as recommended in Figure 3.
The ﬁnal wealth M n after betting for all participants is 10 15 (see the path of log-wealth in Appendix D.3), and the p -value is 10 − 16 (it is the inverse of the maximum wealth, not ﬁnal wealth).
As a comparison, the original paper evaluates the effect by a binary indicator for low birth weight, and they also detect statistically signiﬁcant difference between treated and control group.
To summarize, the i-bet test has valid error control without any parametric assumption on the outcomes and allows exploration of working models so that the algorithm can adapt to different underlying data distributions.
In practice, the working model can also be changed in the middle of the testing procedure, for example, if it ﬁts the data worse as more treatment assignments get revealed.
The ﬂexibility of interactive data-dependent model design with the freedom of adjustment on the ﬂy makes the i-bet test with parametric working models practical and promising.
4. Summary
For randomized trials, we have proposed the i-bet test, which takes the perspective of betting and incorporates the recent idea of allowing human interaction via the procedure of “masking” and “unmasking”.
The interactive tests encourage the analyst to explore various working models before and during the testing procedure, so that the test can integrate the observed data information with prior knowledge of various types and even a human’s subjective belief in a highly ﬂexible manner.
Due to space, we have only discussed in depth the setting of two-sample comparisons with unpaired data, and our test can be extended to various problem settings: two/multi-sample comparison with/without block structure, and a dynamic setting with subjects or mini-batches of subjects arrive sequentially (see Appendix E). The current i-bet test can have unstable results when implemented on real datasets with binary outcomes.
As a future direction, we hope to polish the models for deciding the ordering π t and bet w t in related settings, such as with binary outcomes, or with the proportion of treated subjects being small.
We remark that no test, interactive or otherwise, can be run twice from scratch (with a tweak made the second time to boost power) after the entire data has been examined; this amounts to p -hacking.
Our interactive tests—that can be continued with additional experimentation—are one step towards enabling experts (scientists and statisticians) to work together with statistical models and machine learning algorithms in order to discover scientiﬁc insights with rigorous guarantees. Acknowledgments We thank the anonymous reviewers for their helpful suggestions. AR acknowledges support from
NSF DMS 1916320, and NSF CAREER 1945266. LW acknowledges support from NSF DMS 1713003. This work used the Extreme Science and Engineering Discovery Environment (XSEDE) (Towns et al., 2014), which is supported by National Science Foundation grant number ACI-1548562. Speciﬁcally, it used the Bridges system Nystrom et al.
(2015), which is supported by NSF award number ACI-1445606, at the Pittsburgh Supercomputing Center (PSC)
Proof We argue that the product { M t } | [ n ] \I 0 | t = is a nonnegative martingale with respect to the ﬁltration {F t − 1 } | [ n ] \I 0 | t = .
First, the product M t is measurable with respect to F t − 1 , because M t = (cid:81) tj =1 (cid:2) 1 + w j · ( A π j − µ π j ) (cid:3) , where the t -th selected subject π t and its bet w t are all F t − 1 -measurable.
Second, we show that E ( M t | F t − 1 ) =
M t − 1 .
Note that M t − 1 is ﬁxed given F t − 1 , so E ( M t |
F t − 1 )
=
M t − 1 · E [1 + w t · ( A π t − µ π t ) | F t − 1 ] , and E ( M t | F t − 1 ) =
M t − 1 holds when E ( w t · ( A π t − µ π t ) | F t − 1 ) = 0 , which is implied when
[Equation]
The above can be veriﬁed because
[Equation]
µ π t , where ( a ) holds because w t is F t − 1 -measurable, and ( b ) is because π t is F t − 1 -measurable, and ( c ) stems from the fact that A j is independent of all the outcomes and other treatment assignments and covariates under the global null.
Also, the fact that A π j ∈ { 0 , 1 } and w j ∈ [ − 2 , 2] ensures M t to be nonnegative at any time t .
Thus, we conclude that { M t } | [ n ] \I 0 | t = is a nonnegative martingale.
The error control follows by Ville’s inequality (6).
Appendix B. Estimation of the posterior probability of receiving treatment
Under working model (10), we view the treatment assignments of to-be-ordered subjects as hidden variables and apply the EM algorithm.
At step t , the hidden variables are A i for subjects i / ∈
{ π j } t − 1 i =1 .
And the rest of the complete data { Y i , A i , X i } ni =1 is the observed data, denoted by σ -ﬁeld
F t − 1 as deﬁned in (7).
In the working model (10), the log-likelihood of { Y i , A i , X i } ni =1 is
[Equation]
where φ ( · ) is the density of standard Gaussian and g ( · ) denotes the density of the covariates.
In the
E-step, we update the hidden variable
A i for i / ∈ { π j } t − 1 i =1 as
[Equation]
In the M-step, we update the (parametric) functions θ 0 and θ 1 as
[Equation]
=
[-( Y i − θ 
 ( X i ))-][Equation]
θ [-
-]new
[Equation]
which are least square regressions with weights.
The posterior probability of receiving treatment is estimated as E ( A i | F t − 1 ) for i / ∈ { π j } t − 1 i =1 .
Appendix C. The linear-CATE-test
We ﬁrst describe the general framework of CATE without specifying the working model (see
Vansteelandt and Joffe (2014) for a review).
Suppose ψ ∗ is a vector of parameters, and a predeﬁned function h satisﬁes h ( ψ ∗ , x )
=
if ψ ∗ = 0 , for which a standard choice is a linear function of the covariates, h ( ψ ∗ , x ) = x T ψ ∗ .
One ﬁrst posits that the difference in conditional expectations satisﬁes
[Equation]
Thus, a valid test for null hypothesis (1) can be developed by testing ψ ∗ = 0 .
Note that the test is model-free (regardless of the correctness of h ) since ψ ∗ = 0 is implied by null hypothesis (1) for any function h speciﬁed as above.
The inference on ψ ∗ uses an observation that for any function g of the covariates and the assignment, we have
[Equation]
where E ( Y i | X i , A i ) =
A i · h ( ψ ∗ , X i )
+ E
( Y i |
X i , A i = 0) because of (16).
To estimate ψ ∗ , we need to specify functions h and g , and estimate E ( g ( X i , A i )
| X i ) and E ( Y i |
X i , A i = 0) .
Notice that in a randomized experiment, E ( g ( X i , A i ) |
X i ) is known given g , which guarantees that equation (17) holds regardless of whether E ( Y i |
X i , A i = 0) is correctly speciﬁed (double robustness).
In the following, we choose functions h , g and estimate E
( Y i |
X i , A i = 0) without being concerned about the validity of equation (17).
After getting an estimator of ψ ∗ , we present the test for ψ ∗ = 0 in the end.
For fair comparison with the i-bet test that uses linear model by default, we set h to be a linear function of the covariates and their second-order interaction terms.
Let X (cid:48)
i be the vector of covariates
X i and the interaction terms, then h =
( X (cid:48) i )
T ψ ∗ .
In such as case, a good choice of function g is X (cid:48)
i · A i (Vansteelandt and Joffe, 2014).
Because other methods in our comparison use linear models by default, we estimate E
( Y i |
X i , A i = 0) by a linear model of X (cid:48)
i , denoted as ( X (cid:48)
i ) T (cid:98) β
(note that (cid:98) β can be learned by regressing Y i on X i without involving A i since under the null,
E ( Y i |
X i , A i = 0) =
E ( Y i |
X i , A i = 1) =
E ( Y i | X i ) ).
With the above choices, equation (17) can be written as
[Equation]
which is denoted as E ( b i )
=
E ( B i ) ψ ∗ for simplicity.
Let P n b be the sample average of { b i } ni =1 and
P n B be the sample average of { B i } ni =1 .
A consistent estimator of ψ ∗ is
[Equation]
The test statistic is proposed based on ψ ∗ and its variance estimator.
Notice that the asymptotic variance of (cid:98) ψ is B − 1 Var ( b )
( B − 1 ) T conditional on { X i , A i } ni =1 , for which a consistent estimator is
[Equation]
where (cid:99)
Var ( b ) denotes the sample covariance of { b i } ni =1 .
Thus, the test statistic is proposed as
[Equation]
The limiting distribution of S under the null is χ 2 p where p is the dimension of X (cid:48) i , because E [ b i ] =
E ( B i ) ψ ∗ = 0 under the null.
The linear-CATE-test rejects the null if
[Equation]
where b i is deﬁned in (18); and P n and (cid:99) Var denotes sample average and sample covariance matrix; and χ 2 p (1 − α ) is the 1 − α quantile of a chi-squared distribution with p degrees of freedom.
Appendix D. Experiments for the i-bet test D.1. Heavy-tailed noise
In the automated algorithm of i-bet test, we recommend using the robust regression because it is less sensitive to skewed control outcomes, as shown by Figure 3( c ).
Here, we show that the robust regression also makes the i-bet test more robust to heavy-tailed noise (see Figure 4).
function of control outcome f are linear as deﬁned in (12) and (13).
Instead of Gaussian noise in Section 3, the noise U i is now Cauchy distributed.
The i-bet test with robust linear regression has higher power than that using regular linear regression under heavy-tailed noise.
For a fair comparison, the CovAdj Wilcoxon test is also implemented with robust linear regression.
D.2. Quadratic treatment effect
Another example of adaptive modeling considers the treatment effect as a quadratic function of the covariates, by specifying the function ∆ in the generating model (2) as
[Equation] [Equation]
The control outcome is linearly correlated with the attributes as deﬁned in (13).
We observe that with the robust linear regression, the residuals have a nonlinear trend (see Figure 5( a )), indicating that the linear functions of covariates might not be accurate.
If we add a quadratic term of X 2 i (3) in the robust regression, the trend in residuals is less obvious, and the model ﬁts better (see Figure 5( b )).
As a result, the power is higher than the test using robust linear regression (see Figure 5( c )).
using the robust linear regression
[Figure] [Figure] [Figure] D.3. Wealth path in the real-data application
In the real-data application in Section 3, we report the ﬁnal wealth after betting for all participants as 10 15 .
Here we show the path of log-wealth log( M t ) for each iteration t in Figure 6.
The increase in wealth comes from correct guesses of the masked treatment assignments based on the outcomes and covariates (and the treatment assignments of the revealed subjects).
We cumulate large wealth because most of the treatment assignments can be guessed correctly, indicating the treatment has an effect.
The p -value is the inverse of the maximum wealth, which equals 10 − 16 .
Appendix E. Extensions of i-bet to various other experimental settings E.1. Two-sample comparison with paired data
Suppose there are n pairs of subjects.
Let the outcomes of subjects in the i -th pair be Y ij , the treatment assignments be indicators A ij , the covariates be vector X ij for j = 1 , 2 and i ∈
[ n ] .
The null hypothesis of interest is that there is no difference between treatment and control outcomes conditional on covariates:
[Equation]
Consider a simple case of randomized experiments where
(i) the treatment assignments are independent across pairs, and randomized within each pair: [Equation]
i
[Equation] [Equation] ; (ii) the outcome of one subject Y i 1 ,j 1 is independent of the treatment assignment of another subject
A i 2 ,j 2 for any ( i 1 , j 1 ) (cid:54) =
( i 2 , j 2 ) ∈ [ n ] × [2] .
Under the null, observe that
[Equation]
[Equation]
which implies the independence between A i 1 − A i 2 and all outcomes and covariates.
We can compress the paired data to an “unpaired” form, by treating the difference of paired assignments (after rescaling) (cid:101)
A i :=
( A i 1 − A i 2 + 1) / 2 as the pseudo treatment assignment, and the difference in the paired outcomes (cid:101) Y
i :=
Y i 1 − Y i 2 as the pseudo outcome, and the union of the covariates as the pseudo covariates (cid:101) X i :=
{ X i 1 , X i 2 } .
In such a way, algorithm 2 can be applied with pseudo data { (cid:101) Y i , (cid:101)
A i , (cid:101)
X i } ni
=1 , and guarantee valid error control for paired data.
Meanwhile, under the alternative with positive (negative) effect, the outcome difference (cid:101) Y
i is positively (negatively) correlated with the (rescaled) assignment difference (cid:101)
A i , so our proposed tests can have nontrivial power.
For example, in the i-bet test, the outcome difference (cid:101) Y i can be used along with the union of covariates (cid:101)
X i to gather pairs with positive (cid:101)
A i , as described in Algorithm 2 once we replace the input data with { (cid:101) Y i , (cid:101) A i , (cid:101)
X i } ni =1 .
Interestingly, we can derive another set of corresponding tests for the paired data from a different perspective.
Rosenbaum (2002) and Howard and Pimentel (2020) consider the treatment-minuscontrol difference of the outcome, denoted as D i :=
( A i 1 − A i 2 )( Y i 1 − Y i 2 ) .
Observe that under the null,
[Equation]
for all i ∈ [ n ] , [Equation] 25) because ( A i 1 − A i 2 ) has equal probability to be positive or negative as in (24).
Note that here, we assume the outcomes are continuous to avoid nonzero probability of sign ( D i )
= 0 .
Under the alternative, the treatment-minus-control difference D i can bias to positive (or negative) value.
Therefore, all the discussed methods can be applied to the data {| D i | , sign ( D i ) , (cid:101)
X i } n i =1 where sign ( D i ) is viewed as the pseudo treatment assignment (if rescaled), and | D i | as the pseudo outcome.
E.2. Multi-sample comparison without block structure
In multi-sample comparison, the case where subjects are not matched is often referred to as data without block structure, for which a classical test is the Kruskal-Wallis test (Kruskal and Wallis, 1952).
We call the interactive test in this setting the i-Kruskal-Wallis test.
Follow the notation of two-sample comparison with unpaired data in the previous section, where the treatment assignment
A i now takes values in [ k ] ≡ { 1 , . . .
, k } for k -sample comparison.
The null hypothesis asserts that there is no difference between outcomes of any two treatments conditional on covariates:
[Equation]
Consider a simple case with a randomized experiment, where we assume that
(i) the treatment assignments are independent and randomized
[Equation]
(ii) the outcome of one subject Y i 1 is independent of the assignment of another A i 2 for any i 1 (cid:54) =
i 2 ∈ [ n ] .
Before introducing the interactive test, we ﬁrst brieﬂy describe the classical Kruskal-Wallis test.
The Kruskal-Wallis test
The Kruskal-Wallis test considers the ranks of all observations.
For subjects with treatment a , let the sample size be N a = (cid:80) ni
=1 1 ( A i = a ) and the average rank be
RK ( a ) =
1 N a (cid:80) ni
=1 rank ( Y i ) 1 ( A i = a ) .
Denote the overall averaged rank as RK = 1 n (cid:80) ni
=1 rank ( Y i ) .
The test statistic is
[Equation]
(27) which measures the relative variation across blocks and is expected to be large under the alternative.
Thus, the Kruskal-Wallis test rejects the null if H is larger than a threshold.
The threshold is obtained from the null distribution of H , which can be derived if the sample size is small; otherwise, it is approximated by a chi-squared distribution.
An interactive Kruskal-Wallis test
The interactive test for multi-sample comparison is similar to the case of two-sample comparison.
Both cases have the critical property that under the null,
A i is independent of { Y i , X i } with a known distribution.
A difference from comparing two samples is that under the alternative, the association between the outcome Y i and the treatment
A i can have various patterns depending on the underlying truth.
Here, we consider an example of the i-KruskalWallis test that targets a speciﬁc type of alternative.
Given three treatments ( k = 3 ), suppose we wish to target the alternative of increasing outcomes:
[Equation]
where Y 1 (cid:22) Y 2 means that Y 1 is stochastically smaller than Y 2 .
The i-Kruskal-Wallis test can then use { Y i , X i } to bet on whether A i is larger than its expected value µ i =
E
( A i |
X i )
= 2 .
The complete procedure follows Algorithm 2 with µ i = 2 for all i ∈ [ n ] and w t ∈ [ − 1 , 1] .
Under the null, M t is a nonnegative martingale regardless of the bets and the error control is guaranteed.
E.3. Multi-sample comparison with block structure
Suppose we want to compare k treatments with n blocks of data; a “block” is a group of k subjects each of whom receives a different treatment (each treatment is assigned to exactly one subject).
A classical test is the Friedman test (Friedman, 1937), and we call the interactive test as the i-Friedman test.
For block i ∈ [ n ] and subject j ∈ [ k ] , denote the outcome as Y ij , the treatment assignment as A ij , and the covariates as X ij .
The null hypothesis states that there is no difference between the outcome of any two treatments conditional on covariates:
[Equation]
ij = a 2 , X ij ) for all j ∈ [ k ] and i ∈ [ n ] and a 1 , a 2 ∈ [ k ];
[Equation]
Consider a simple case of the randomized experiments where
(i) the treatment assignment A ij takes value 1 , . .
, k such that (a) { A i 1 , . . . ,
A ik } is equally likely to be any permutation of { 1 , . .
. , k } , and (b) the treatment assignments are independent across blocks; (ii) the outcome of one subject Y i 1 ,j 1 is independent of the assignment of another subject
A i 2 ,jfor any ( i 1 , j 1 ) (cid:54) =
( i 2 , j 2 ) ∈ [ n ] × [ k ] .
The Friedman test
The Friedman test considers the ranks within each block { Y i 1 , . . .
, Y ik } , denoted as rank ( Y ij ) .
Let the rank of the subjects with treatment a averaged over n blocks be
RK [Equation] a ) = [Equation]
n [Equation] kj =1 rank ( Y ij )
( A ij = a ) , and its expected value under the null is 1+ k 2 .
Under the alternative, the outcomes for one of the treatment could be larger (or smaller) than those for other treatments and the averaged rank would be higher (or lower).
The Friedman test computes:
[Equation]
and reject the null if F is larger than a threshold obtained by the null distribution of F , which is approximated by a chi-square when n or k is large.
An interactive Friedman test
The interactive test for multi-sample comparison with block structure integrates the data within each block, similar to the case of paired sample for two-sample comparison.
Consider the vector of treatment assignments within each block i ordered by the outcomes, denoted as A
i = (cid:0) A i, (1) , . . .
, A i, ( k ) (cid:1) , where Y i, (1) ≥ . . .
≥ Y i, ( k ) .
Because the assignments are independent of the outcomes under the null, we claim that
[Equation]
for all a ∈ permute ([ k ]) and i ∈ [ n ] , (30) where permute ([ k ]) denotes the set of all possible permutations of [ k ] .
Under the alternative, the conditional distribution of A i can bias to a certain ordering depending on the underlying truth.
As an example to compare three treatments ( k = 3 ), suppose we wish to detect the following alternative:
[Equation]
in which case A i are more likely to be (1 , 2 , 3) .
To develop an interactive test, we encode the vector of assignments by a scalar (pseudo assignment (cid:101)
A i ) such that it takes larger value when A i is more “similar” to the ideal permutation (1 , 2 , 3) .
Speciﬁcally, the similarity (distance) between A i and (1 , 2 , 3) can be measured by the number of exchange operations needed to convert A i to (1 , 2 , 3) .
We deﬁne (cid:101)
A i as:
[Equation]
  1 , if A i = (1 , 2 ,
[Equation]
1 , if A i =
(2 , 1 , 3) , 1 , if A i = (
[Equation]
[Equation]
[Equation]
34) (35) (36) (37) where the ordered assignments (33) and (34) need one exchange operation to be converted to (1 , 2 , 3) ; (35) and (36) need two; and (37) is the opposite of the ideal permutation, which needs three exchange operations.
This design of (cid:101)
A i takes binary values, but it can also take different values for each ordering of A i .
We present the above deﬁnition because it has a simple form and leads to relatively high power for a broad range of alternatives in simple simulations.
With the above transformation from a vector of assignments to a scalar (cid:101) A i for each block i , we can view the blocks as individuals in the interactive test.
That is, we use the pseudo assignment (cid:101)
A i for testing while ordering the blocks using the revealed data { Y ij , X ij } i = n,j = k i =1 ,j =1 and the actual assignments { A ij } kj =1 once block i is ordered.
In other words, let the pseudo assignment (cid:101)
A i be deﬁned in (32)-(37), the pseudo outcome be the union within each block
, (cid:101) Y i
= { Y ij } kj =1 , and same for the pseudo covariates (cid:101)
X i =
{ X ij } kj =1 .
The i-Friedman test follows Algorithm 2 with the input data replaced by { (cid:101) Y i , (cid:101)
A i , (cid:101)
X i } ni
=1 and µ i by
for all i ∈ [ n ] .
E.4. Sample comparison in dynamic settings
We have proposed interactive tests for two/multi-sample comparison with unpaired/paired data, all of which are in the batch setting where the sample size is ﬁxed before testing.
Nonetheless, in many applications, one hopes to monitor the null of zero treatment effect as more subjects are collected, so that the experiment can stop once there is enough evidence to reject the null.
In this section, we consider a sequential setting where an unknown and potentially inﬁnite number of subjects (or pairs) arrive sequentially in a stream and introduce the sequential interactive tests.
As a demonstration, we propose the seq-bet test for a two-sample comparison with unpaired data.
Because the subjects arrive one by one, it is hard to order them on the ﬂy, and we instead propose to ﬁlter the subjects to be cumulated in the product M t .
At time t + 1 when a new subject arrives, the analyst can interactively decide whether to include A t +1 in current M t .
Denote the decision by an indicator I t +1 , and the product is
[Equation]
The available information to decide I t +1 and weight w t +1 includes the complete data information of the ﬁrst t subjects and the revealed data of the ( t + 1) -th subject, denoted by the ﬁltration:
[Equation]
where the complete data { Y i , A i , X i } ti =1 can be used for modeling and guide the decision of I
t +1 .
Under the null, we have
[Equation]
so the product M t +1 is a martingale.
Also, the martingale is nonnegative with bets in the range w i ∈ [ − 1 1 − µ i , 1 µ i ] .
Thus, with the same argument as in Appendix A for the batch setting, Algorithm E. has valid error control as it stops and rejects the null when M t reaches the boundary 1 /α . Input ; Procedure , do 1.
Using G α then else end In practice, to get a reasonably good model for our ﬁltering process, we can ﬁrst collectsubjects (say) and reveal their complete data { Y i ,
A i , X i } for modeling and then apply the seq-bet test from the 51 -th subject.
Note that Algorithm E.4 also applies to the sequential setting with paired data or multi-sample comparison when we replace the input data by pseudo sample { (cid:101)
Y t , (cid:101) A t , (cid:101)
X t } deﬁned in previous sections.
Appendix F. Related work on weak null hypotheses
There are many works that focus on a less strict null hypothesis H (cid:48) 0 than our global null H 0 in (1), which of course has pros and cons.
These related methods would continue valid for the global null hypothesis of our interest, but they could have lower power especially when H (cid:48) 0 is true and H 0 is not true.
Our strong global null is still sometimes of scientiﬁc interest, for example when certain quantiles of the distribution may be different under two different treatments (without the means differing), or one may be interested in the heavy-tailed case when the means may not even exist.
We elaborate on the related work as follows.
While several works study treatment with multiple levels, for simplicity we describe them in the case with two levels (treated or not) in our discussion below.
Akritas et al.
(2000) assess the treatment effect by comparing the outcome CDF of treated and control group, denoted as F Tx ( y ) and F Cx ( y ) where x is the given covariate value.
Let G ( x ) be a prespeciﬁed distribution for the covariate or its empirical distribution.
The null hypothesis concerns marginal CDF after averaging over the covariate:
[Equation]
which is implied by the global null H 0 in our discussion.
Fan and Zhang (2017) also study the above null hypothesis (41), and propose an alternative test statistic to incorporate covariates.
Wang and
Akritas (2006) consider several extensions in the type of null hypothesis and suggest the possibility of testing whether the conditional outcome CDF given the covariates is identical:
[Equation]
which is equivalent to the global null H 0 in our discussion, but no explicit test is provided for this null hypothesis.
Similar null hypotheses are discussed in the work of Edgar Brunner (such as Akritas et al.
; Bathke and Brunner (2003)), which focus on factorial design and develop tests for the effect of one factor conditional on the level of the other factors.
Thus, their methods can be used to test our global null H
when the covariate takes a ﬁnite number of values.
Hettmansperger and
McKean (2010) focus on testing the global null H 0 when the treatment effect is a linear function of the covariates, and discusses inference such as conﬁdence intervals of the involved parameters.
Along a different line of work, Thas et al.
(2012) considers outcome Y and covariates Z (which include the treatment assignment A and other covariates X in our context) and let two instances ( Y, Z ) and ( Y ∗ , Z ∗ ) be independently distributed.
The outcomes Y and Y ∗ are compared by estimating the probabilistic index P ( Y > Y ∗ | Z, Z ∗ )
+ 12 P ( Y =
Y ∗ | Z, Z ∗ ) .
Their results imply a test for the null hypothesis of the probabilistic index being 1 / 2 , which can be used in our context:
[Equation]
which is true when our global null H 0 is true; hence, their method is valid for our problem of interest.
Aside from different target null hypotheses, several features distinguish our proposed algorithms from most existing work: (a) previous methods often commit to a single ﬁxed procedure, while the i-bet test we propose can employ arbitrary working models, and the working model can be changed by a human analyst at any iteration to improve power; (b) most other methods mentioned above guarantee type-I error asymptotically, whereas our interactive methods have exact type-I error control (without any parametric or model assumptions on the outcomes); (c) we demonstrate through numerical experiments that the advantage of our proposed methods is more evident when a treatment effect exists only for a few subjects, whereas the above methods do not speciﬁcally focus on such sparse effects.
Appendix G. Options for adjusting Wilcoxon’s signed-rank test for covariates
The Wilcoxon signed-rank test is a simple and efﬁcient nonparametric test with a known null distribution.
Of course, rank-based statistics have been explored in many directions: see Lehmann and
D’Abrera (1975) for a review of classical methods.
Recent work focuses on how to incorporate covariate information to improve power.
Zhang et al.
(2012) develop an optimal statistic to detect constant treatment effect; in multi-sample comparison, Ding and Keele (2018) numerically compare rank statistics of outcomes or residuals from linear models; Rosenblum and Van Der Laan (2009) and Vermeulen et al.
(2015) focus on related testing problems for conditional average effect and marginal effect; Rosenbaum (2010) and Howard and Pimentel (2020) use generalizations of rank tests for sensitivity analysis in observational studies.
Here, we introduce variants of the signed-rank test for two-sample comparison in a randomized trial, which can improve the power of Rosenbaum’s
CovAdj Wilcoxon test under heterogeneous treatment effect.
The signed-rank test offers a general formula to construct tests for two-sample comparison.
We note that the signed-rank test is perhaps more frequently used for paired data; but it can also be applied to unpaired data because the error control is also based on a decoupling between the sign and the rank.
For each subject i ∈ [ n ] , let E i be any statistic that is larger when subject i has treatment effect.
We compute
[Equation]
[Equation]
and the null is rejected when W is large.
As an example, Rosenbaum (2002) proposed the covarianceadjusted signed-rank test by specifying E i as [Equation]
[Equation]
where recall R i is the residual of regressing Y i on X i without using A i as a predictor.
(The covariance-adjusted signed-rank test is slightly different from the covariance-adjusted Wilcoxon rank-sum test (4), but they had similar power in most of our experiments.)
The null distribution of
W depends on E i , but one can use a permutation test that is valid for any choice of E i , as described in Algorithm 1.
Ideally, statistic E i should be designed to take a larger value when subject i has a larger treatment effect.
In the following, we discuss the question of whether the original choice of
E i =
E R ( X ) i can be improved, and which choice of E i should we prefer given different types of treatment effect.
G.1. Existing statistics and their drawbacks
Aside from Rosenbaum’s design of E i as E R ( X ) i , we can ﬁnd several other alternatives to detect treatment effects in the causal inference literature.
For example, one can construct a conﬁdence interval for the ATE, which implies a test for zero ATE.
However, the null of zero ATE is not the focus of this paper, as we are interested in the null of zero effect for any subpopulation.
Lin (2013) suggests modeling Y i by a linear function of A i and X i (recently extended in a preprint by Guo and
Basse (2021) to other parametric models), and construct the estimator for ATE as an average over subjects:
[Equation]
where (cid:98) Y ( · ; · ) denotes a ﬁtted outcome using X i ,
A i and (cid:98) Y
( X i ; 1 − A i ) predicts using the false assignment.
This estimator provides a design of E i that calculates the residual of predicting Y i using covariates X i and the false assignment 1 − A i as follows:
[Equation]
) where (cid:98) Y ( X i ; 1 − A i ) can be the prediction via any black-box algorithm, such as a random forest.
There is also a rich literature on doubly-robust methods (see, for example, Robinson (1988);
Robins et al.
; Cao et al.
; Chernozhukov et al.
to estimate ATE when the probability of receiving treatment varies with X i .
In a randomized experiment, the estimator boils down to
[Equation]
which suggests a design of E i as (2 A i − 1)
( Y i − (cid:98) Y ( X [Equation] − (cid:98) [Equation]
( X [Equation] This design leads to similar power as E R ( X, 1 − A )
i in most experiments and hence is omitted from this paper.
To examine the performance of tests using the statistics E R ( X ) i and E R ( X, 1 − A ) i , we simulate outcomes from the generating model (2) where the function for treatment effect ∆ and that for control outcome f are constructed with different features (e.g., dense/sparse effect and bell-shaped/skewed control outcome):
[Equation]
The dense (sparse) effect is set to be weak (strong) since otherwise, all methods have power near one (zero).
We intentionally let the treatment effect and control outcome be nonlinear functions of the covariates because our discussion focuses on methods using nonparametric working models.
In the rest of this paper, we employ random forests (with default parameters in the R package randomForest ) as our working model since it usually generates good predictions for various data distributions (Breiman, 2001).
Although both methods have high power under a well-behaved distribution where the treatment effect is dense, the control outcome is bell-shaped, and the noise is standard Gaussian (solid lines in Figure 7( a )), they show different weak points when the effect is harder to detect—the test using E R ( X ) i tends to have lower power when the treatment effect is sparse (Figure 7( b )); and the test using E R ( X, 1 − A ) i tends to be less robust when the control outcome is skewed (Figure 7( c )).
When the noise is heavy-tailed, both tests have lower power as expected, but the one using E R ( X, 1 − A ) i appears to be more sensitive (Figure 7( a )).
Broadly, the aforementioned pros and cons may be traced to two characteristics in the design of E i :
(i) the prediction model that uses both X i and A i as in E R
( X, 1 − A ) i accounts for heterogeneous treatment effect (by the interaction terms between X i and A i ), leading to high power for sparse effects;
( a ) Power when the treatment effect is dense and the control outcome is bell-shaped, and the noise varies as Gaussian and Cauchy (heavy-tailed).
( b ) Power when the treatment effect is sparse, the control outcome is bell-shaped, and the noise is Gaussian. ( c ) Power when the treatment effect is dense, the control outcome is skewed, and the noise is Gaussian. (ii) the residuals in E R ( X ) i only uses X i as predictors so that it effectively reduces the outcome variation that is not caused by the treatment, making the test robust under skewed control outcome.
Next we propose other designs of E i that combine the advantages of the above two characteristics.
G.2.
Improve robustness under skewed control outcome by predicting residuals
R
i
Because residuals R i can downsize the noise caused by skewed control outcome, we propose to measure the treatment effect via a prediction on R i .
That is, we compute the statistic E i by two steps of prediction:
(i) obtain residuals R i by predicting Y i using X i (without A i ); (ii) ﬁt a prediction model for R i using X i and A i , denoted as (cid:98) R ( · , · ) ; (iii) get E i from the prediction error of R i using covariates X i and the false assignment 1 − A i : E R − (cid:98) R ( X, 1 − A ) i
[Equation]
Notice that E R − (cid:98) R ( X, 1 − A ) i has a similar form as E R ( X, 1 − A ) i , where { R i } ni =1 can be viewed as “denoised” outcomes: a large Y i could stem from skewness in the control outcome, but a large R i is more likely to indicate large treatment effect, and hence achieves higher robustness to skewed control outcome.
Numerical experiments coincide with our intuition: the power of using E R − (cid:98) R ( X, 1 − A ) i improves from that using E R ( X, 1 − A ) i when the control outcome is skewed (see Figure 8).
[Figure]
G.3.
Improve robustness under heavy-tailed noise using difference in the prediction error
Treating residuals R i as the pseudo outcomes is useful to account for variation in the control outcome, but R i can still contain much irrelevant variation, such as when the random noise U i in model (2) is Cauchy.
Under heavy-tailed noise, the prediction model (cid:98) R ( · , · ) in E R − (cid:98) R ( X, 1 − A ) i could be inaccurate; and a large prediction error of using the false assignment as in E R − (cid:98) R ( X, 1 − A )
i could result from heavy-tailed noise, while it is supposed to be evidence of large treatment effect.
So how to remove the large prediction error caused by poor modeling?
We propose to consider the difference between the prediction error of using the false assignment | (cid:98) R
( X i , 1 − A i ) − R
( X i ) |
and that using the true assignment | (cid:98) R ( X i , A i ) − R ( X i ) | : E | (cid:98) R ( X, 1 − A ) − R |−| (cid:98) R ( X,A ) − R | i := | (cid:98)
[Equation] [Equation] [Equation] −
[Equation]
) −
[Equation]
) |
[Equation]
[Equation]
Intuitively, when the prediction model (cid:98) R ( · , · ) is a good ﬁt, the prediction error using true assignment
| (cid:98) R
( X i , A i ) − R
( X i ) | should be close to zero, and the proposed statistic is similar to
E R − (cid:98) R ( X, 1 − A ) i .
The advantage shows when the modeling is poor, such as under heavy-tailed noise.
Here, the prediction error is large using either true or false assignment, so taking their difference as in E
| (cid:98) R
( X, 1 − A ) − R |−| (cid:98) R
( X,A ) − R | i can help rule out the variation caused by noise, letting the variation from treatment effect stand out.
In the experiment with sparse effect (47), the test using E | (cid:98) R ( X, 1 − A ) − R |−| (cid:98) R ( X,A ) − R | i has similar power as that using E [Equation] −
[Equation] i when data is welldistributed (see Figure 9( a )), while it can achieve higher power under Cauchy noise or skewed control outcome (see Figure 9( b ) and 9( c )), consistent with our intuition.
[Figure]
under
Gaussian noise and bell-shaped control outcome.
[Figure]
under
Cauchy noise and bell
[Figure]
control outcome.
[Figure]
under
Gaussian noise and skewed control outcome.
[Figure]
[Figure]
[Figure] [Figure]
effect under
Gaussian noise and bell [Figure] control
[Figure]
Gaussian noise and skewed control
[Figure] come. Remark 8 Note that E | (cid:98)
[Table]
leads to high power when we want to detect a sparse and strong effect.
However, when the effect is dense and weak as in model (46) , Rosenbaum’s
Wilcoxon test using E R ( X ) i is more robust to peculiar noise or control outcomes (see Figure 10).
It is because E | (cid:98) R ( X, 1 − A ) − R |−| (cid:98)
R ( X,A ) − R | i uses a prediction model for R i , which can be less informative for weak effect, especially when the noise is large.
In practice, one may have some anticipation on the population properties of the treatment effect (density or strength), and choose the statistic accordingly.
We summarize our recommendations under different settings in ﬂowchart (56) .
G.4. On one-sided versus two-sided effects
The statistic of difference in the prediction error leads to high power for two-sided effects.
A major distinction between E | (cid:98) R
( X, 1 − A ) − R |−| (cid:98) R
( X,A ) − R | i and the statistics discussed previously is that it takes large value for both positive and negative effects.
It is because the difference in the prediction error of using opposite assignments is large as long as the assignment is a signiﬁcant predictor for the outcome, regardless of the direction of effect.
Therefore, the test using E | (cid:98) R ( X, 1 − A ) − R |−| (cid:98) R
( X,A ) − R | i can cumulate effects of both signs while they cancel out in other statistics, leading to high power even when the average effect is close to zero.
As some examples, we construct the following treatment effect:
[Equation]
(Sparse strong positive effect and dense weak negative effect); ∆
[Equation]
(Sparse strong effect of both signs); ∆( X i ) =
S ∆
[Equation]
[Equation]
(cid:21) (Dense weak effect of
both signs).
[Equation]
In all examples, only the test using E | (cid:98) R ( X,
[Equation]
A [Equation] |
[Equation]
| [Equation] R
[Equation]
X [Equation] has nontrivial power (see the ﬁrst row in Figure 11).
Such sensitivity may or may not be desirable depending on the problem context.
For example, we would hope to reject the null when the positive effect is strong for a subpopulation as in (52).
However, one might want to treat a weak effect in both directions (54) as noise and leave the null unrejected.
Next, we propose a modiﬁcation of E | (cid:98) R
( X, 1 − A ) − R |−| (cid:98) R
( X,A ) − R | i with such behavior.
Targeting one-sided effects.
To differentiate between positive and negative effects, we modify the statistic E | (cid:98) R
( X, 1 − A ) − R |−| (cid:98) R
( X,A ) − R | i by incorporating a sign that indicates the direction of the treatment effect.
Consider the sign of two other statistics that approximate the treatment effect:
[Equation]
X
[Equation]
A i )) ≥ 0 } , and combine them to get
S i :
= 1 { S 1 i > 0 or S 2 i > 0 } .
We then deﬁne
[Equation]
which is large when the treatment effect is large and positive .
We tried using only S 1 i or S 2 i for the sign, but the combined one is more robust in experiments.
The essential idea is to construct S i using some statistics that have a consistent sign with the treatment effect, while keeping the advantage of
E | (cid:98) R
( X, 1 − A ) − R |−| (cid:98) R
( X,A ) − R | i under skewed control outcome and heavy-tailed noise.
As desired, the test using E S · ( | (cid:98) R ( X, 1 − A ) − R |−| (cid:98) R
( X,A ) − R | ) i is less sensitive to weak effect of both signs (Figure 11( c )) and keeps high power for sparse strong positive effect (Figure 11( d )).
Note that the signed statistic is more sensitive to noise because the signs are generated from less robust statistics (Figures 11( e ), 11( f )).
Nonetheless, among statistics that are insensitive to two-sided effect, E S [Equation] [Equation] X,
[Equation] −
[Equation] ,A [Equation] − R | ) i leads to high power for sparse effect, irrespective of whether the control outcome and the noise are well-distributed or have outliers.
[Figure]
[Figure]
[Figure] E [Equation] ( X
[Equation]
i , E [Equation] (cid:98)
[Equation]
− A
[Equation]
i ,
E [Equation] [Equation] , 1 − A ) − R |−| (cid:98) R
( X,A ) − R | i and E S · ( | (cid:98) R ( X, 1 − A ) − R |−| (cid:98) R ( X,A ) − R | ) i . In the ﬁrst row where the treatment effect can be positive or negative, only the test using E | (cid:98) R ( X, 1 − A )
− R |−| (cid:98) R
( X,A ) − R | i has nontrivial power.
In the second row, the treatment effect is sparse and positive, and the control outcome and noise vary. The test using E S · ( | (cid:98) R ( X, 1 − A ) − R |−| (cid:98) R ( X,A ) − R | ) i can have high power without being too sensitive to the weak effect in both directions (see subplot 11( c )).
G.5.
Summarizing the observations made in this section
In this section, we proposed several variants of Rosenbaum’s covariate-adjusted Wilcoxon as follows: (i) Instead of predicting the outcomes , using the prediction model (cid:98) R ( · , · ) for residuals R i can improve power under skewed control outcome.
This is because the residuals R i , which are themselves obtained by regressing Y i only on X i (without A i ), can remove much variation caused by the control outcome, and in turn highlight the treatment effect (see Appendix G.2).
(ii)
The evidence of treatment effect can be measured by the prediction error using the false assignment, but a large prediction error could also be a result of a poorly ﬁt model, such as when the noise is heavy-tailed.
In contrast, the difference in the prediction error of using true and false assignments can eliminate most of the prediction error that is irrelevant to the treatment, including that from poorly ﬁt models, and thus improve the power (see Appendix G.3).
(iii)
The difference in prediction error detects both positive and negative effects with no distinction, so it can arguably be too sensitive (if there is such a thing) to a weak effect in both directions.
If one wishes to target one-sided effects while maintaining the robustness achieved by “difference in prediction error”, we propose to multiply it with an estimated sign of the effect (see Appendix G.4).
In summary, we recommend choosing one out of the three test statistics discussed in this section—
E
[Equation]
X
[Equation]
i
[Equation]
, and E S
[Equation]
—depending on one’s prior belief of the population properties of treatment effect (if one exists), as shown below:
Nonzero effect   Effect of both signs → E | (cid:98) R ( X
[Equation]
Note that the i-bet test is not included here because its performance depends on the interaction and progressive updates to the initial working model made by the analyst based on revealed data.
The ﬂexibility makes the i-bet test a potentially more robust and promising method compared with the aforementioned methods that also use a parametric (or semiparametric) working model.