Original: A Conditional Randomization Test for Sparse Logistic Regression in High-Dimension
Revised: A Conditional Randomization Test for Sparse Logistic Regression in High-Dimension

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Identifying the relevant variables for a classification model with correct confidencelevels is a central but difficult task in high-dimension.
Revised: Identifying the relevant variables for a classification model with correct confidencelevels is a central but difficult task in high-dimension.

Original: Despite the core roleof sparse logistic regression in statistics and machine learning, it still lacks agood solution for accurate inference in the regime where the number of featuresp is as large as or larger than the number of samples n .
Revised: Despite the core roleof sparse logistic regression in statistics and machine learning, it still lacks agood solution for accurate inference in the regime where the number of featuresp is as large as or larger than the number of samples n .

Original: Here we tackle thisproblem by improving the Conditional Randomization Test (CRT).
Revised: Here we tackle thisproblem by improving the Conditional Randomization Test (CRT).

Original: The original
Revised: The original

Original: CRT algorithm shows promise as a way to output p-values while making fewassumptions on the distribution of the test statistics.
Revised: CRT algorithm shows promise as a way to output p-values while making fewassumptions on the distribution of the test statistics.

Original: As it comes with a prohibitivecomputational cost even in mildly high-dimensional problems, faster solutionsbased on distillation have been proposed.
Revised: As it comes with a prohibitivecomputational cost even in mildly high-dimensional problems, faster solutionsbased on distillation have been proposed.

Original: Yet, they rely on unrealistic hypothesesand result in low-power solutions.
Revised: Yet, they rely on unrealistic hypothesesand result in low-power solutions.

Original: To improve this, we propose CRT-logit , analgorithm that combines a variable-distillation step and a decorrelation step thattakes into account the geometry of  ℓ 1 -penalized logistic regression problem.
Revised: To improve this, we propose CRT-logit , analgorithm that combines a variable-distillation step and a decorrelation step thattakes into account the geometry of the ℓ 1 -penalized logistic regression problem.

Original: Weprovide a theoretical analysis of this procedure, and demonstrate its effectivenesson simulations, along with experiments on large-scale brain-imaging and genomicsdatasets.
Revised: Weprovide a theoretical analysis of this procedure, and demonstrate its effectivenesson simulations, along with experiments on large-scale brain-imaging and genomicsdatasets.

Original:  Introduction
Revised:  Introduction

Original: Logistic regression is one of the most popular tools in modern applications of statistics and machinelearning, partly due to its relative algorithmic simplicity.
Revised: Logistic regression is one of the most popular tools in modern applications of statistics and machinelearning, partly due to its relative algorithmic simplicity.

Original: The method belongs to the class ofgeneralized linear models that handle discrete outcomes, i.e. classification problems.
Revised: The method belongs to the class ofgeneralized linear models that handle discrete outcomes, i.e. classification problems.

Original: Here, we focuson the binary classification problem, where one observation of the responses y ∈ { 0 , 1 } and the datavectors x ∈ R p follows the relationship:
Revised: Here, we focuson the binary classification problem, where one observation of the responses y ∈ { 0 , 1 } and the datavectors x ∈ R p follows the relationship:

Original: [Equation]
Revised: [Equation]

Original: where g ( x ) = 1 / (1 + exp( − x )) is the sigmoid function, and β 0 the vector of true regressioncoefficients.
Revised: where g ( x ) = 1 / (1 + exp( − x )) is the sigmoid function, and β 0 the vector of true regressioncoefficients.

Original: In the classical setting, in which the number of samples n is greater than the number offeatures p , an estimate ˆ β of the true signals β 0 can be obtained using maximum likelihood estimation(MLE).
Revised: In the classical setting, in which the number of samples n is greater than the number offeatures p , an estimate ˆ β of the true signals β 0 can be obtained using maximum likelihood estimation(MLE).

Original: The asymptotic behaviour and derivation of the test statistic, confidence intervals and p-valuesof the MLE have been well studied, e.g. in [10].
Revised: The asymptotic behaviour and derivation of the test statistic, confidence intervals and p-valuesof the MLE have been well studied, e.g. in [13].

Original: The availability of p-values for the test statisticsmakes it possible to rely on multiple hypothesis testing, where one wants to test which variables havea non-zero effect on the outcome, conditionally to the remaining variables.
Revised: The availability of p-values for the test statisticsmakes it possible to rely on multiple hypothesis testing, where one wants to test which variables havea non-zero effect on the outcome, conditioned on the remaining variables.

Original: Unfortunately, this lineof analysis cannot be applied to the high-dimensional regime, where p is larger than n , as arguedin [21, 26, 28].
Revised: Unfortunately, this lineof analysis cannot be applied to the high-dimensional regime, where p is larger than n , as arguedin [25, 32, 34].

Original: These works show that in the regime where lim n,p →∞ n/p = κ , the MLE estimatorexists only when κ > 2 .
Revised: These works show that in the regime where lim n,p →∞ n/p = κ , the MLE estimatorexists only when κ > 2 .

Original: However, we note that this type of analysis is done without the additionof ℓ 1 -regularization to the likelihood function, i.e. without using a penalized estimator to enforcesparsity.
Revised: However, we note that this type of analysis is done without the additionof ℓ 1 -regularization to the likelihood function, i.e. without using a penalized estimator to enforcesparsity.

Original: Motivation Our focus in this paper is to do inference with statistical guarantees on high-dimensionalsparse logistic regression, where p is larger or much larger than n .
Revised: Motivation Our focus in this paper is to do inference with statistical guarantees on high-dimensionalsparse logistic regression, where p is larger or much larger than n .

Original: This setting is typical inmodern applications of pattern recognition, e.g. in brain-imaging or genomics [2], with p aslarge as hundreds of thousands –compressible to thousands– but n stays at most few thousand.
Revised: This setting is typical inmodern applications of pattern recognition, e.g. in brain-imaging or genomics [3], with p aslarge as hundreds of thousands –compressible to thousands– but n stays at most few thousand.

Original:  2 Theoretical Quantiles
Revised:  2 Theoretical Quantiles

Original: S a m p l e
Revised: S a m p l e

Original: Q u a n t i l e s dCRT  2 Theoretical Quantiles
Revised: Q u a n t i l e s dCRT  2 Theoretical Quantiles

Original: S a m p l e
Revised: S a m p l e

Original: Q u a n t i l e s
Revised: Q u a n t i l e s

Original: CRT-Logit
Revised: CRT-Logit

Original: The family of methods we consider is the Conditional Randomization Test (CRT) [8].
Revised: The family of methods we consider is the Conditional Randomization Test (CRT) [10].

Original: CRT relieson generating multiple noisy copies of originalvariables to output empirical p-values in highdimensional inference problems.
Revised: CRTrelies on generating multiple noisy copies oforiginal variables to output empirical p-values inhigh-dimensional inference problems.

Original: However, prohibitive computational cost makes CRT impractical, as discussed at length in [8, 22, 6, 16].
Revised: However,prohibitive computational cost makes CRT impractical, as discussed at length in [10, 26, 7, 19].

Original: Therehave been several lines of research attemptingto fix this problem, most notably the distilled Conditional Randomization Test (dCRT) [16].
Revised: There have been several lines of research attempting to fix this problem, most notablythe distilled Conditional Randomization Test(dCRT)

Original: This work introduced a distillation step as a replacement for the randomized sampling step tocompute the importance statistics (see Section 2for more details).
Revised: This work introduced a distillation step as a replacement for the randomizedsampling step to compute the importance statistics (see Section 2 for more details).

Original: It provides a way to outputp-values for multiple types of regression andclassification problems, assuming convergenceto Gaussian  of the test statistic in large-sampleregime.
Revised: It providesa way to output p-values for multiple types of regression and classification problems, assumingconvergence to Gaussian distribution of the teststatistic in large-sample regime.

Original: Yet, as shown in the left panel of Figure 1, the originally proposed dCRT test-statisticfor logistic regression does not behave as wellas intended.
Revised: Yet, as shownin the left panel of Figure 1, the originally proposed dCRT test-statistic for logistic regressiondoes not behave as well as intended.

Original: In particular, its null distribution deviates markedly from standard normal in highdimension whenever n/p ≤ 1 .
Revised: In particular, its null distribution deviates markedly fromstandard normal in high-dimension whenever n/p ≤ 1 .

Original: Contribution We propose a correction for the dCRT, inspired by the decorrelation method presentedin [19].
Revised: Contribution We propose a correction for the dCRT, inspired by the decorrelation method presentedin [21].

Original: The decorrelation step makes the null-distribution of the test statistics much closer to standardnormal, as shown on the right panel of Figure 1, and thus increases the statistical power of the method.
Revised: The decorrelation step makes the null-distribution of the test statistics much closer to standardnormal, as shown on the right panel of Figure 1, and thus increases the statistical power of the method.

Original: We provide asymptotic analysis of this method, which shows that CRT-logit produces standard normaltest-statistic in  large-sample regime.
Revised: We provide asymptotic analysis of this method, which shows that CRT-logit produces standardnormal test-statistics in the large-sample regime.

Original: In addition, we validate the high performance of CRT-logit onlarge-scale brain-imaging and genetics datasets, thus showing its usefulness in practical applications.
Revised: In addition, we validate the high performance of CRT-logit on large-scale brain-imaging and genetics datasets, thus showing its usefulness in practicalapplications.

Original: Related works The closest cousin of the Conditional Randomization Test is Knockoff Filter[3, 8], a recent breakthrough in the False Discovery Rate (FDR) control literature.
Revised: Related works The closest cousin of the Conditional Randomization Test is Knockoff Filter [4, 10],a recent breakthrough in the False Discovery Rate (FDR) control literature.

Original: It relies on thecreation of additional noisy features, called knockoffs, to calculate variable-importance statistics.
Revised: It relies on the creationof additional noisy features, called knockoffs, to calculate variable-importance statistics.

Original: Another extension of vanilla CRT is the Holdout Randomization Test (HRT)
Revised: Anotherextension of vanilla CRT is the Holdout Randomization Test (HRT)

Original: While stillrequiring multiple sampling of noisy variables, HRT solves the computational issue of original CRTby doing heavy model fitting only once on one part of the dataset, and test statistics calculationon the other part, without refitting the model.
Revised: While still requiringmultiple samplings of noisy variables, HRT solves the computational issue of original CRT bydoing heavy model fitting only once on one part of the dataset, and test statistics calculation onthe other part, without refitting the model.

Original: However, this method relies on sample-splitting,hence inherently suffers from a loss of statistical power.
Revised: However, this method relies on sample-splitting, andhence inherently suffers from a loss of statistical power.

Original: A parallel line of work has introducedthe Conditional Permutation Test (CPT) [6], a non-parametric alternative to CRT that relies ona random shuffling mechanism applied to original variables, instead of multiple sampling of newvariables.
Revised: A parallel line of work has introducedthe Conditional Permutation Test (CPT) [7], a non-parametric alternative to CRT that relies ona random shuffling mechanism applied to original variables, instead of multiple sampling of newvariables.

Original: This potentially makes CPT more robust to model mis-specification.
Revised: This potentially makes CPT more robust to model mis-specification.

Original: [26] recently proposeda method called SLOE, which adapts the analysis of [28], but in the regime different from what we areconsidering, where lim n,p →∞ n/p → κ ∈ (1 , 2) , and more importantly without sparsity-inducingpenalty.
Revised: [32] recently proposeda method called SLOE, which adapts the analysis of [34], but in a regime different from what we areconsidering, where lim n,p →∞ n/p → κ ∈ (1 , 2) , and more importantly without sparsity-inducingpenalty.

Original: On a separate note, we notice the similarity of dCRT [16] with debiased Lasso [13, 23, 27].
Revised: On a separate note, we notice the similarity of dCRT [19] with debiased Lasso [16, 28, 33].

Original: This line of work proposed a debiasing formula for the estimator, which makes the asymptoticdistribution of ( ˆ β LASSO − β 0 ) standard normal, so that one can compute the test statistic and p-valueassociated with each variable.
Revised: This line of work proposed a debiasing formula for the estimator, which makes the asymptoticdistribution of ( ˆ β LASSO − β 0 ) standard normal, so that one can compute the test statistic and p-valueassociated with each variable.

Original:  Background
Revised:  Background

Original: Notation We denote matrices, vectors, scalars and sets by bold uppercase, bold lowercase, scriptlowercase , and calligraphic letters, respectively, e.g. X , x , x , and X .
Revised: Notation We denote matrices, vectors, scalars and sets by bold uppercase, bold lowercase, scriptlowercase , and calligraphic letters, respectively, e.g. X , x , x , and X .

Original: The i -th row of a matrix
Revised: The i -th row of a matrix

Original: X will be denoted X i, ∗ , the j -th column X ∗ ,j and the ( i, j ) -th
Revised: X will be denoted X i, ∗ , the j -th column X ∗ ,j and the ( i, j ) -th

Original: element X i,j .
Revised: element X i,j .

Original: For any naturalnumber p , we denote the set [ p ] def.
Revised: For any naturalnumber p , we denote the set [ p ] def.

Original: = { 1 , . . .
Revised: = { 1 , . . .

Original: , p } .
Revised: , p } .

Original: For each x ∈ R p and j ∈ [ p ] , we denote x − j def.
Revised: For each x ∈ R p and j ∈ [ p ] , we denote x − j def.

Original: ={ x 1 , x 2 , . . .
Revised: ={ x 1 , x 2 , . . .

Original: , x j − 1 , x j +1 , . . .
Revised: , x j − 1 , x j +1 , . . .

Original: , x p } a p − 1 dimensional observation after removing the j -th variable.
Revised: , x p } a p − 1 dimensional observation after removing the j -th variable.

Original: Correspondingly, X − j is the data matrix X ∈ R n × p with column X ∗ ,j removed.
Revised: Correspondingly, X − j is the data matrix X ∈ R n × p with column X ∗ ,j removed.

Original: The cumulativedistribution function (CDF) of the standard Gaussian distribution will be denoted Φ( · ) .
Revised: The cumulativedistribution function (CDF) of the standard Gaussian distribution will be denoted Φ( · ) .

Original: The indicatorfunction of a random event A will be denoted 1 A .
Revised: The indicatorfunction of a random event A will be denoted 1 A .

Original: For any two positive sequences x n and y n , wewrite x n ≍ y n if cy n ≤ x  ≤ Cy n for all n , for some positive constants c and C .
Revised: For any two positive sequences x n and y n , wewrite x n ≍ y n if cy n ≤ x n ≤ Cy n for all n , for some positive constants c and C .

Original: For a vector x , ∥ x ∥ pdenotes its ℓ p norm.
Revised: For a vectorx , ∥ x ∥ p denotes its ℓ p norm.

Original: For a function f : R p
Revised: For a function f : R p

Original: → R , ∇ j f denotes its gradient w.r.t.
Revised: → R , ∇ j f denotes its gradient w.r.t.

Original: the j -th variable, forj ∈ [ p ] .
Revised: the j -thvariable, for j ∈ [ p ] .

Original: Problem setting We consider exclusively binary classification, where the response vector is denotedy ∈ { 0 , 1 } n and the data matrix X ∈ R n × p consists of n p
Revised: Problem setting We consider exclusively binary classification, where the response vector is denotedy ∈ { 0 , 1 } n and the data matrix X ∈ R n × p consists of n p

Original: -dimensional samples.
Revised: -dimensional samples.

Original: Throughout thepaper, we assume the data { X i, ∗ } ni =1 are i.i.d.
Revised: Throughout thepaper, we assume the data { X i, ∗ } ni =1 are i.i.d.

Original: and follow a distribution with zero mean and populationcovariance matrix Σ .
Revised: and follow a distribution with zero mean and populationcovariance matrix Σ .

Original: Moreover, we assume that X i, ∗ and y i follow the logistic relationship in Eq.
Revised: Moreover, we assume that X i, ∗ and y i follow the logistic relationship in Eq.

Original: We denote the support set S def.
Revised: We denote the support set S def.

Original: = { j ∈ [ p ] : β 0 j ̸ = 0 } and assume that it is sparse, i.e. card( S ) =
Revised: = { j ∈ [ p ] : β 0 j ̸ = 0 } and assume that it is sparse, i.e. card( S ) =

Original: s ∗ ≪ p , where card denotes the cardinality of a set.
Revised: s ∗ ≪ p , where card denotes the cardinality of a set.

Original: Furthermore, ˆ S def.
Revised: Furthermore, ˆ S def.

Original: = { j ∈ [ p ] : ˆ β j ̸ = 0 } indicatesan estimation of S , where ˆ β j is an estimate of the true signal β 0 j .
Revised: = { j ∈ [ p ] : ˆ β j ̸ = 0 } indicatesan estimation of S , where ˆ β j is an estimate of the true signal β 0 j .

Original: We try to obtain it through aℓ 1 -penalized logistic estimator:
Revised: We try to obtain it through aℓ 1 -penalized logistic estimator:

Original: [Equation]
Revised: [Equation]

Original: We denote I def.
Revised: We denote I def.

Original: =
Revised: =

Original: E β 

Revised: E β

Original: [ ∇ 2 ℓ ( β 0 )]
Revised: [ ∇ 2 ℓ ( β 0 )]

Original: the Fisher information matrix, and I j |− j the partial Fisher information, defined by I j |− j def.
Revised: the Fisher information matrix, and I j |− j the partial Fisher information, defined by I j |− j def.

Original: =
Revised: =

Original: E [ ∇ 2 jj ℓ ( β 0 ) − [ ∇ 2 j, − j ℓ ( β 0 )]
Revised: E [ ∇ 2 jj ℓ ( β 0 ) − [ ∇ 2 j, − j ℓ ( β 0 )]

Original: ⊤ [ ∇ 2 − j, − j ℓ ( β 0 )] − 1 ∇ 2 − j,j ℓ ( β 0 )]
Revised: ⊤ [ ∇ 2 − j, − j ℓ ( β 0 )] − 1 ∇ 2 − j,j ℓ ( β 0 )]

Original: =
Revised: =

Original: I jj − I j, − j I − 1 − j, − j I − j,j , where I j, − j is the row-vector made with the j th-row and the columnscorresponding to β − j , I − j, − j the sub-matrix of I made with the rows and columns corresponding toβ − j .
Revised: I jj − I j, − j I − 1 − j, − j I − j,j , where I j, − j is the row-vector made with the j th-row and the columnscorresponding to β − j , I − j, − j the sub-matrix of I made with the rows and columns corresponding toβ − j .

Original: This quantity, defined following [10, pp. 323], plays an important role in our proposed method,detailed in Section 3.
Revised: This quantity, defined following [13, pp. 323], plays an important role in our proposed method,detailed in Section 3.

Original: Statistical control with False Discovery Rate To quantify statistical errors, we consider the False
Revised: Statistical control with False Discovery Rate To quantify statistical errors, we consider the False

Original: Discovery Rate , introduced in [4].
Revised: Discovery Rate , introduced in [5].

Original: Given an estimate of the support ˆ S , the false discovery proportion(FDP) is the ratio of the number of selected features that do not belong to the true support S , dividedby the total number of selected features.
Revised: Given an estimate of the support ˆ S , the false discovery proportion(FDP) is the ratio of the number of selected features that do not belong to the true support S , dividedby the total number of selected features.

Original: The False Discovery Rate is the expectation of the FDP:
Revised: The False Discovery Rate is the expectation of the FDP:

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: FDR [Equation] ˆ
Revised: FDR( ˆ

Original: [Equation]
Revised: [Equation]

Original: [FDP
Revised: [FDP(

Original: [Equation]
Revised: [Equation]

Original: ˆ [Equation] [Equation] Conditional Randomization Test (CRT) and Distillation CRT (dCRT)
Revised: Conditional Randomization Test (CRT) and Distillation CRT (dCRT)

Original: The concept of Conditional Randomization Test was originally proposed in the model-X knockoff paper [8] as a wayto output valid empirical p-values using knockoff variables.
Revised: The concept of Conditional Randomization Test was originally proposed in the model-X knockoff paper [10] as a wayto output valid empirical p-values using knockoff variables.

Original: The principle of the knockoff filteris first to sample noisy copies ˜ X ∗ ,j of variable X ∗ ,j , given a known sampling mechanism P j |− j .
Revised: The principle of the knockoff filteris first to sample noisy copies ˜ X ∗ ,j of variable X ∗ ,j , given a known sampling mechanism P j |− j .

Original: One advantage of the knockoff filter is that no specific assumption is placed on the distribution ofthe inferred test statistic.
Revised: One advantage of the knockoff filter is that no specific assumption is placed on the distribution ofthe inferred test statistic.

Original: However, this means that, in general, there is no mechanism to derivep-values from the knockoff statistic.
Revised: However, this means that, in general, there is no mechanism to derivep-values from the knockoff statistic.

Original: This motivates the introduction of CRT, which requires runninghigh-dimensional inference for each variable j B times.
Revised: This motivates the introduction of CRT, which requires runninghigh-dimensional inference for each variable j B times.

Original: However, the computation cost of CRT isprohibitive when p grows large: assuming that we use the Lasso program with coordinate descent tocompute T CRT j , its runtime would be O
Revised: However, the computation cost of CRT isprohibitive when p grows large: assuming that we use the Lasso program with coordinate descent tocompute T CRT j , its runtime would be O

Original: ( Bp 4 ) [12, pp. 93].
Revised: ( Bp 4 ) [15, pp. 93].

Original: Moreover, CRT requires decently large
Revised: Moreover, CRT requires decently large

Original: B to make the empirical distribution of the p-values smooth enough.
Revised: B to make the empirical distribution of the p-values smooth enough.

Original: Reducing the computationalcost of CRT is the main motivation of several works [6, 16, 22].
Revised: Reducing the computationalcost of CRT is the main motivation of several works [7, 19, 26].

Original: One of them is the introductionof distillation-CRT (dCRT)
Revised: One of them is the introductionof distillation-CRT (dCRT)

Original: The main appeal of this method is that it can output p-valuesanalytically, therefore bypassing the multiple knockoffs sampling steps used to infer on each variable,and leads to a reasonable reduction of the computation cost.
Revised: The main appeal of this method is that it can output p-valuesanalytically, therefore bypassing the multiple knockoffs sampling steps used to infer on each variable,and leads to a reasonable reduction of the computation cost.

Original: Distillation operation The key addition of dCRT is the distillation operation: for each variable j ,we want to distill all the conditional information of the remaining variables
Revised: Distillation operation The key addition of dCRT is the distillation operation: for each variable j ,we want to distill all the conditional information of the remaining variables

Original: X − j to x j and to y vialeast-squares minimization with ℓ 1 -regularization to enforce sparsity.
Revised: X − j to x j and to y vialeast-squares minimization with ℓ 1 -regularization to enforce sparsity.

Original: For each variable j , we firstsolve the lasso problem by regressing X ∗ ,j on X − j ,
Revised: For each variable j , we firstsolve the lasso problem by regressing X ∗ ,j on X − j ,

Original: [Equation]
Revised: [Equation]

Original: For distillation of variable j and the binary response y with logistic relationship, [16] briefly suggestedto solve a penalized estimation problem, similar to Eq.
Revised: For distillation of variable j and the binary response y with logistic relationship, [19] briefly suggestedto solve a penalized estimation problem, similar to Eq.

Original: [Equation]
Revised: [Equation]

Original: Then, a test statistic is calculated for each j = 1 , . .
Revised: Then, a test statistic is calculated for each j = 1 , . .

Original: , p :
Revised: , p :

Original: [Equation]
Revised: [Equation]

Revised: Intuitively, Eq.

Revised: (5) is the correlation of the regression residuals, calculated from Eq.

Revised: (3) and (4), scaledby a factor of √ n .

Original: which, under the null hypothesis, and more importantly, assuming linear relationship between X i, ∗and y ,  follows standard normal distribution asymptotically, conditional to y and X − j .
Revised: Under the null hypothesis, and more importantly, assuming linear relationshipbetween X i, ∗ and y , this quantity follows standard normal distribution asymptotically, conditionalto y and X − j .

Original: It then followsthat we can output a p-value for each variable j by taking (cid:98) p
Revised: It then follows that we can output a p-value for each variable j by taking (cid:98) p

Original: j = 2 [1 − Φ ( T j )] .
Revised: j =2 [1 − Φ ( T j )] .

Original: However, the formulation of test statistics in Eq.
Revised: However, the formulation of test statistics in Eq.

Original: (5) is not truly satisfactory in the setting of sparselogistic regression.
Revised: (5) is not truly satisfactory in the setting of sparselogistic regression.

Original: More specifically, both the calculation of regression residuals y − X − j ˆ β d y ,jand test statistics T j do not take into account the non-linear relationship between X and the binaryresponse y .
Revised: More specifically, both the calculation of regression residuals y − X − j ˆ β d y ,jand test statistics T j do not take into account the non-linear relationship between X and the binaryresponse y .

Original: The first row of Figure 2 plots the qq-plot of the test statistics T j for logistic regression,which shows that even in the classical regime where n > p , its distribution is far from standardnormal.
Revised: The first row of Figure 6 plots the qq-plot of the test statistics T j for logistic regression,which shows that even in the classical regime where n > p , its distribution is far from standardnormal.

Original:  Decorrelating Test-Statistics for High-Dimensional Logistic Regression
Revised:  Decorrelating Test-Statistics for High-Dimensional Logistic Regression

Original: As we have elaborated, the formulation of dCRT is not well-suited for problems other than penalizedleast-squares regression.
Revised: As we have elaborated, the formulation of dCRT is not well-suited for problems other than penalizedleast-squares regression.

Original: We therefore propose an adaptation of dCRT in the case of logistic regression,inspired by the classical work of [10] and by [19].
Revised: We therefore propose an adaptation of dCRT in the case of logistic regression,inspired by the classical work of [13] and by [21].

Original: First, note that when testing H j 0 : β
Revised: First, note that when testing H j 0 : β

Original: j = 0 underthe case where n > p , we have the classical Rao’s test statistic, defined by
Revised: j = 0 underthe case where n > p , we have the classical Rao’s test statistic, defined by

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original:  where ∇ j ℓ (0 , ˆ β − j )def. = ∇
Revised: ) where ∇ j ℓ (0 , ˆ β − j )

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: =

Original: [Equation]
Revised: [Equation]

Original: β j =0 is the Fisher score.
Revised: is the Fisher score.

Original: Here ˆ β − j def.
Revised: Here ˆ β − j def.

Original: =
Revised: =

Original: argmin β − j ∈ R p − 1 ℓ ( β j , β − j ) is the constrained maximum-likelihood estimator of β − j with fixedβ j , and ˆ I j |− j is a consistent estimator of the partial Fisher information
Revised: argmin β − j ∈ R p − 1 ℓ ( β j , β − j ) is the constrained maximum-likelihood estimator of β − j with fixedβ j , and ˆ I j |− j is a consistent estimator of the partial Fisher information

Original: I j |− j .
Revised: I j |− j .

Original: The appearance of theterm ˆ I − 1 / 2 j |− j is due to the fact that under the null hypothesis H j 0 , we have, by [10, Chapter 9], [20],
Revised: The appearance of theterm ˆ I − 1 / 2 j |− j is due to the fact that under the null hypothesis H j 0 , we have, by [13, Chapter 9], [24],

Original: [Equation]
Revised: [Equation]

Original: which makes the asymptotic distribution of T Rao j standard normal.
Revised: which makes the asymptotic distribution of T Rao j standard normal.

Original: However, in the high-dimensioncase, where n < p , we do not reach this convergence in distribution.
Revised: However, in the high-dimensioncase, where n < p , we do not reach this convergence in distribution.

Original: To see this, consider the Taylorexpansion of the Fisher score of variable j around any given estimator (cid:101)
Revised: To see this, consider the Taylorexpansion of the Fisher score of variable j around any given estimator (cid:101)

Original: β − j of the true β 0 − j :
Revised: β − j of the true β 0 − j :

Original: [Equation]
Revised: [Equation]

Original: On the right-hand side, the first term converges weakly to a normal distribution due to the Central
Revised: On the right-hand side, the first term converges weakly to a normal distribution due to the Central

Original: Limit Theorem, the remainder term becomes negligible using ℓ 1 regularization to induce sparsity, butthe second term does not, due to estimation bias and sparsity effect of (cid:101) β − j [11].
Revised: Limit Theorem, the remainder term becomes negligible using ℓ 1 regularization to induce sparsity, butthe second term does not, due to estimation bias and sparsity effect of (cid:101) β − j [14].

Original: Adapting distillation operation for sparse logistic regression Fortunately, Eq.
Revised: Adapting distillation operation for sparse logistic regression Fortunately, Eq.

Original: (7) suggests thatfor each variable j , we can debias the Fisher score by correcting the impact of other terms.
Revised: (7) suggests thatfor each variable j , we can debias the Fisher score by correcting the impact of other terms.

Original: Inparticular, for each variable j , we replace the Fisher score by
Revised: Inparticular, for each variable j , we replace the Fisher score by

Original: [Equation]
Revised: [Equation]

Original: The inversion of the large matrix I − j, − j ∈ R ( p − 1) × ( p − 1) is computationally prohibitive, but we canestimate the term I j, − j I − 1 − j, − j straightforwardly by solving
Revised: The inversion of the large matrix I − j, − j ∈ R ( p − 1) × ( p − 1) is computationally prohibitive, but we canestimate the term I j, − j I − 1 − j, − j straightforwardly by solving

Original: [Equation]
Revised: [Equation]

Original: for each variable j , where ˆ β is given with Eq.
Revised: for each variable j , where ˆ β is given with Eq.

Original: Moreover, since we have the closed-form ofthe derivatives of the logistic loss ℓ ( ˆ β ) , a simple derivation from Eq.
Revised: Moreover, since we have the closed-form ofthe derivatives of the logistic loss ℓ ( ˆ β ) , a simple derivation from Eq.

Original: (9) suggests the following x j -distillation operation, adapted for logistic regression :
Revised: (9) suggests the following x j -distillation operation, adapted for logistic regression :

Original: [Equation]
Revised: [Equation]

Original: where the extra term (second-order derivative of the sigmoid function) g ′′ ( Xi,
Revised: where the extra term (second-order derivative of the sigmoid function) g [Equation] ( X

Original: [Equation]
Revised: [Equation]

Original: appears from differentiating twice the loss function ℓ ( ˆ β ) , and ˆ β = ˆ β λ is defined in Eq.
Revised: appears from differentiating twice the loss function ℓ ( ˆ β ) , and ˆ β = ˆ β λ is defined in Eq.

Original: On the other hand, we can obtain ˆ β d y ,j j from ˆ β by simply omitting the j -th coefficient from it, i.e.
Revised: On the other hand, we can obtain ˆ β d y ,j j from ˆ β by simply omitting the j -th coefficient from it, i.e.

Original: [Equation]
Revised: [Equation]

Original: Finally, the equation for decorrelated test score, adapted from both Eq.
Revised: Finally, the equation for decorrelated test score, adapted from both Eq.

Original: (5) and (6), reads
Revised: (5) and (6), reads

Original: [Equation]
Revised: [Equation]

Original:  where the formula for  empirical partial Fisher information is ˆI j |− j = n − 1 (cid:80) ni
Revised: (11) where the formula for the empirical partial Fisher information is ˆ

Original: =1 g ′′ ( X i, ∗ ˆ β )( X i,j − X i, − j ˆ β d X ∗ ,j ) X i,j .
Revised: [Equation] = [Equation]

Original: A summary of the full procedure, which we call CRT-logit, can be found in Algorithm 1.
Revised: A summary of the full procedure, whichwe call CRT-logit, can be found in Algorithm 1.

Original: Notice that the runtime of CRT-logit is the same as dCRT, which means in generalslower than KO and HRT.
Revised: Notice that the runtime of CRT-logit is the same asdCRT, which means in general slower than KO and HRT.

Original: To speedup inference time, we introduce a variable-screening step thateliminates potentially unimportant variables before distillation, similar to dCRT.
Revised: To speedup inference time, we introduce avariable-screening step that eliminates potentially unimportant variables before distillation, similar todCRT.

Original: We provide empiricalbenchmark of  runtime of each method in Section 4.4.
Revised: We provide empirical benchmark of the runtime of each method in Section 4.4.

Original: Setting ℓ 1 -regularization parameter λ and λ dx
Revised: Setting ℓ 1 -regularization parameter λ and λ dx

Original: In general, we advise to use cross-validationfor obtaining ˆ β λ in Eq.
Revised: In general, we advise to use cross-validation forobtaining ˆ β λ in Eq.

Original: (2) and for X ∗ ,j -distillation operator, as defined by Eq.
Revised: (2) and for X ∗ ,j -distillation operator, as defined by Eq.

Original: This is inlinewith the theoretical argument for dCRT [16, Lemma 1 and Theorem 3].
Revised: This is in linewith the theoretical argument for dCRT [19, Lemma 1 and Theorem 3].

Original: However, we also observeempirically that choosing the ℓ 1 -regularization parameters of the distillation step can strongly affecthow variables are selected by CRT-logit.
Revised: However, we also observeempirically that choosing the ℓ 1 -regularization parameters of the distillation step can strongly affecthow variables are selected by CRT-logit.

Original: We provide more details in the Supplementary Material,and leave further theoretical investigations of this phenomenon for future work.
Revised: We provide more details in the Supplementary Material,and leave further theoretical investigations of this phenomenon for future work.

Original: Eq
Revised: Algorithm β

Original: 180 Asymptoticanalysis of the Decorrelated Test Statistic We now provide theoretical analysisof CRT-logit in large-sample regime.
Revised: 183 Asymptotic We now provide theoretical analysis of CRT-logit in large-sample regime.

Original: All the proofs can be found in the Supplementary Material.
Revised: All the proofs can be found in the Supplementary Material.

Original: Without writing it explicitly, in our analysis, we consider p = p ( n ) , and the following assumption.
Revised: First,we introduce the following assumption.

Original: Assumption 3.1 (Regularity conditions) .
Revised: Assumption 3.1 (Regularity conditions) .

Original: Assume that
Revised: Assume that

Original: (A1) λ min ( I ) ≥ κ 2 for some constant κ > 0 .
Revised: (A1) λ min ( I ) ≥ K for some constant K > 0 .

Original: (A2) Sparsity of β 0 and w 0 ,j , with w 0 ,j the ground truth weights for the distillation of x j inEq.
Revised: (A2) Sparsity of β 0 and w 0 ,j , with w 0 ,j the ground truth weights for the distillation of x j inEq.

Original: (10) : |S| = s ∗ and ∥ w 0 ,j ∥ 

Revised: (10) : |S| = s ∗ and ∥ w 0 ,j ∥

Original: = s ′ with s ∗ =
Revised: = s ′ with s ∗ =

Original: o (cid:0) n 1 / 2 / log( p ) (cid:1) and s ′
Revised: o (cid:0) n 1 / 2 / log( p ) (cid:1) and s ′

Original: =
Revised: =

Original: o (cid:0) n 1 / 2 / log( p ) (cid:1) .(A3)
Revised: o (cid:0) n 1 / 2 / log( p ) (cid:1) .(A3)

Original: For all i ∈ [ n ] ,∥ X i ∥ ∞ ≤ K and | X i, − j w 0 ,j | ≤ K  almost surely, for some constant K  .
Revised: For all i ∈ [ n ] , X i, ∗ and ( − y i + g ′ ( X i, ∗ β )) are sub-exponential random variables, and| X i, − j w 0 ,j | ≤ K ′ almost surely, for some constant K ′ > 0 .

Original: We then have the following result, that states that the asymptotic distribution of the decorrelated testscores is standard normal.
Revised: We then have the following result, that states that the asymptotic distribution of the decorrelated testscores is standard normal.

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: Let j ∈ [ p ] , and let T decorr j be defined as in Eq.
Revised: Let j ∈ [ p ] , and let T decorr j be defined as in Eq.

Original: (11) , with λ ≍ λ dx ≍ (cid:112) n − 1 log( p ) .
Revised: (11) , with λ ≍ λ dx ≍ (cid:112) n − 1 log( p ) .

Original: Then, if Assumption 3.1 holds true,  under the null hypothesis H j 0 : β 0 j = 0 , we have[Equation] R ,
Revised: Then, if Assumption 3.1 holds true, and if we consider p = p ( n ) , under the null hypothesis H j 0 : β 0 j =0 , we have

Original: [Equation]
Revised: [Equation]

Original: where Φ( · ) is the CDF of the standard Gaussian distribution.
Revised: where Φ( · ) is the CDF of the standard Gaussian distribution.

Original: Moreover, for each j ∈ [ p ] , if wedefine (cid:98) p j def.
Revised: Moreover, for each j ∈ [ p ] , if wedefine (cid:98) p j def.

Original: = 2 [1 − Φ ( T j )] , i.e. (cid:98) p
Revised: = 2 [1 − Φ ( T j )] , i.e. (cid:98) p

Original: j is the output of Algorithm 1, then, under the null hypothesis
Revised: j is the output of Algorithm 1, then, under the null hypothesis

Original: H j 0 : β 0 j = 0 , we have
Revised: H j 0 : β 0 j = 0 , we have

Original: [Equation]
Revised: [Equation]

Original: all t ∈ [ [Equation] , 1] , that is, the p-values output by Algorithm 1 are valid asymptotically. 
Revised: that is, the p-values output by Algorithm 1 are valid asymptotically. Remark 3.1.

Revised: We also show in the proof of Theorem 3.1 (in Suppementary Material) that the rate ofconvergence is O (1 / √ n ) .

Revised: Compared with some of the related works ( e.g. , knockoffs) that come withfinite sample guarantees, our theoretical analysis only works in the asymprotic regime.

Revised: We leave thefinite sample analysis as one of the directions for future works.

Original: FDR control with CRT-logit We now state the second main result, which establishes that the FDRof the test is controlled when using  Benjamini-Yekutieli procedure [5] with the p-values output from Algorithm 1, assuming that the number of tests p is fixed.
Revised: FDR control with CRT-logit As a consequence of Theorem 3.1, we have the following result,which establishes that the FDR of the test is controlled when using the Benjamini-Yekutieli procedure[6] with the p-values output from Algorithm 1, assuming that the number of tests p is fixed.

Original: Theorem 3.2.
Revised: Corollary 3.1.

Original: Under Assumptions 3.1 and logistic model defined in Eq (1) , with λ ≍ λ dx ≍ 200 (cid:112) n − 1 log( p ) , assume n − 1 / 2 ( s ′ ∨ s ∗ ) log( p ) =
Revised: Under Assumptions 3.1 and logistic model defined in Eq (1) , with λ ≍ λ dx ≍ 209 (cid:112) n − 1 log( p ) , assume n − 1 / 2 ( s ′ ∨ s ∗ ) log( p ) =

Original: o (1) , and assume the number of tests p is fixed.
Revised: o (1) , and assume the number of tests p is fixed.

Original: Letα ∈ (0 , 1) and (cid:98) S BY-CRT be given by applying following the Benjamini-Yekutieli FDR-controllingprocedure to the CRT-logit p-values { (cid:98) p j } j ∈ [ p ] , output from Algo.1.
Revised: Letα ∈ (0 , 1) and (cid:98) S BY-CRT be given by applying following the Benjamini-Yekutieli FDR-controllingprocedure to the CRT-logit p-values { (cid:98) p j } j ∈ [ p ] , output from Algo.1.

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: Remark 3.1.
Revised: Remark 3.2.

Original: Assumption 3.1 is also assumed in [19, 23], which also provide a detailed discussionof this regularity assumption in generalized linear models.
Revised: Assumption 3.1 is also assumed in [21, 28], which also provide a detailed discussionof this regularity assumption in generalized linear models.

Original: This assumption, in turn, is built on theregularity assumption in the classic work [10, Chapter 9] to establish asymptotic normality of Rao’stest statistic.
Revised: This assumption, in turn, is built on theregularity assumption in the classic work [13, Chapter 9] to establish asymptotic normality of Rao’stest statistic.

Original: Theorem 3.1 is an adaptation of [19, Theorem 3.1], specialized for the case of sparselogistic regression and the p-values output from CRT-logit.
Revised: Theorem 3.1 is an adaptation of [21, Theorem 3.1], specialized for the case of sparselogistic regression and the p-values output from CRT-logit.

Original:  Empirical Results
Revised:  Empirical Results

Original: We provide benchmarks of the proposed CRT-logit algorithm along with most other methods mentioned in the introduction, in particular: model-X Knockoff (KO) [8], Debiased Lasso (dLasso)[27, 13], original CRT with 1000 samplings [8], Holdout Randomization Test with 5000 samplings[22], and Lasso-Distillation CRT (dCRT)[16].
Revised: We provide benchmarks of the proposed CRT-logit algorithm along with most other methods mentioned in the introduction, in particular: model-X Knockoff (KO) [10], Debiased Lasso (dLasso)[33, 16], original CRT with 1000 samplings [10], Holdout Randomization Test with 5000 samplings[26], and Lasso-Distillation CRT (dCRT)

Original: We did not include SLOE and CPT as the providedopen-source implementation are particularly unstable and do not fit in the sparse-regression setting(for SLOE), or implementation are not available (for CPT).
Revised: We did not include SLOE and CPT as the providedopen-source implementation are particularly unstable and do not fit in the sparse-regression setting(for SLOE), or implementation is not available (for CPT).

Original: For the lack of space, we leave the extraexperiment with a genome-wide association study in the Supplementary Material.
Revised: For the lack of space, we leave the extraexperiment with a genome-wide association study in the Supplementary Material.

Original: Remark 4.1.
Revised: Remark 4.1.

Original: As a slight caveat, in the simulated and semi-realistic experiment sections (Sections 4.1,4.2 and 4.3), we introduce an additional noise term to the logistic relationship of Eq. 
Revised: As a slight caveat, in the simulated and semi-realistic experiment sections (Sections 4.1,4.2 and 4.3), we introduce an additional noise term to the logistic relationship of Eq. [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: (12) where ξ i ∼ N (0 , 1) is a Gaussian noise and σ > 0 the noise magnitude.
Revised: where ξ i ∼ N (0 , 1) is a Gaussian noise and σ > 0 the noise magnitude.

Original: The formula in Eq.
Revised: The formula in Eq.

Original: (12) hasbeen used in previous works, e.g. [7].
Revised: (12) hasbeen used in previous works, e.g. [8].

Original: There is a clear justification to this: in most of the applicationswe consider, data are collected with measurement errors.
Revised: There is a clear justification to this: in most of the applicationswe consider, data are collected with measurement errors.

Original: In the case of brain-imaging, for example,recording the brain signal of the human subjects by scanners often includes noise caused eitherfrom the machine, or from the movement of the subjects [15].
Revised: In the case of brain-imaging, for example,recording the brain signal of the human subjects by scanners often includes noise caused eitherfrom the machine, or from the movement of the subjects [18].

Original: Moreover, in general, this settingcorresponds to a model mis-specification, which the CRT-logit is robust to under Assumption 3.1,following the same argument as in [19, Section 5].
Revised: Moreover, in general, this settingcorresponds to a model mis-specification, which the CRT-logit is robust to under Assumption 3.1,following the same argument as in [21, Section 5].

Original: Remark 4.2.
Revised: Remark 4.2.

Original: We use Benjamini-Hochberg step-up procedure [4] to control FDR with the p-valuesin all the empirical experiments in Section 4.2 and App.
Revised: We use Benjamini-Hochberg step-up procedure [5] to control FDR with the p-values inall the empirical experiments in Section 4.2 and App.

Original: G, as we observe empirically the FDR isusually controlled with this procedure, without compromising power with the conservative BY bound.
Revised: G, as we observe that the FDR is empiricallycontrolled with this procedure, without compromising power with the conservative BY bound.

Original: 4.1 Effectiveness of the decorrelation step
Revised: 4.1 Effectiveness of the decorrelation step

Original: To show how decorrelating the test statistics helps, we set up a simulation with matrix X of p = 400features and vary the number of samples n ∈ { 200 , 400 , 800 , 4000 } .
Revised: To show how decorrelating the test statistics helps, we set up a simulation with matrix X of p = 400features and vary the number of samples n ∈ { 200 , 400 , 800 , 4000 } .

Original: The binary response vectory is created following Eq.
Revised: The binary response vectory is created following Eq.

Original: (12), and the design matrix X is sampled from a multivariate normaldistribution with zero mean, while the covariance matrix Σ ∈ R p × p is a symmetric Toeplitz matrix, where the parameter ρ ∈ (0 , 1) controls the correlation between neighboring features: correlationdecreases quickly when the distance between feature indices increases.
Revised: (12), and the design matrix X is sampled from a multivariate normaldistribution with zero mean, while the covariance matrix Σ ∈ R p × p is a symmetric Toeplitz matrix,where the parameter ρ ∈ (0 , 1) controls the correlation between neighboring features: correlationdecreases quickly when the distance between feature indices increases.

Original: The true signal β 0 is pickedwith a sparsity parameter κ = s ∗ /p that controls the proportion of non-zero elements with magnitude2.0, i.e. β j = 2 . 0 for all j ∈ S .
Revised: The true signal β 0 is pickedwith a sparsity parameter κ = s ∗ /p that controls the proportion of non-zero elements with magnitude2.0, i.e. β j = 2 . 0 for all j ∈ S .

Original: For the specific purpose of this experiment, non-zero indices of Sare kept fixed.
Revised: For the specific purpose of this experiment, non-zero indices of Sare kept fixed.

Original: The noise ξ is i.i.d.
Revised: The noise ξ is i.i.d.

Original: normal N (0 , Id n ) with magnitude σ = ∥
Revised: normal N (0 , Id n ) with magnitude σ = ∥

Original: X β 0 ∥ 2 / ( √ n SNR ) ,controlled by the SNR parameter.
Revised: X β 0 ∥ 2 / ( √ n SNR ) ,controlled by the SNR parameter.

Original: In short, the three main parameters controlling this simulationare correlation ρ , sparsity degree κ and signal-to-noise ratio SNR.
Revised: In short, the three main parameters controlling this simulationare correlation ρ , sparsity degree κ and signal-to-noise ratio SNR.

Original: We generate randomly 1000datasets, and run dCRT and CRT-logit algorithm to obtain one sample of test statistics { T j } pj =1 and{ T decorr j } pj =1 .
Revised: We generate randomly 1000datasets, and run dCRT and CRT-logit algorithm to obtain one sample of test statistics { T j } pj =1 and{ T decorr j } pj =1 .

Original: Then, we pick 1000 samples of one null test statistic T j and T decorr j , defined in Eq.
Revised: Then, we pick 1000 samples of one null test statistic T j and T decorr j , defined in Eq.

Original: (5)and (11), respectively, and plot the qq-plot of their empirical quantile versus the standard normalquantile.
Revised: (5)and (11), respectively, and plot the qq-plot of their empirical quantile versus the standard normalquantile.

Original: From the results in Figure 2, we observe that the empirical null distribution of the test
Revised: From the results in Figure 6, we observe that the empirical null distribution of the test

Original: [Figure]
Revised: [Figure]

Revised: (a) n =2.

Original: 5 Theoretical Quantiles
Revised: 2.5 Theoretical Quantiles

Original: S a m p l e
Revised: S a m p l e

Original: Q u a n t i l e s dCRT 2.
Revised: Q u a n t i l e s dCRT 2.

Revised: 2.5 Theoretical Quantiles

Revised: S a m p l e

Revised: Q u a n t i l e s

Revised: CRT-Logit (b) n =2.

Original: 2.5 Theoretical Quantiles
Revised: 2.5 Theoretical Quantiles

Original: S a m p l e
Revised: S a m p l e

Revised: Q u a n t i l e s dCRT 2.

Revised: 2.5 Theoretical Quantiles

Revised: S a m p l e

Original: Q u a n t i l e s
Revised: Q u a n t i l e s

Original: CRT-Logit (c) n =2.
Revised: CRT-Logit (c) n =2.

Original: 2.5 Theoretical Quantiles
Revised: 2.5 Theoretical Quantiles

Original: S a m p l e
Revised: S a m p l e

Original: Q u a n t i l e s dCRT 2.
Revised: Q u a n t i l e s dCRT 2.

Original: 2.5 Theoretical Quantiles
Revised: 2.5 Theoretical Quantiles

Original: S a m p l e
Revised: S a m p l e

Original: Q u a n t i l e s
Revised: Q u a n t i l e s

Original: CRT-Logit ( [Figure]
Revised: CRT-Logit (d) n =

Original: 244 statistic is much closer to a standard normal when adding the decorrelation step.
Revised: 253 statistic is much closer to a standard normal when adding the decorrelation step.

Original: In particular, whenthe sample size n increases to 400, the decorrelated test statistic has empirical quantiles almost inlinewith the theoretical quantiles of the standard normal distribution, while dCRT test score strays awayfrom the 45-degree line.
Revised: In particular, whenthe sample size n increases to 400, the decorrelated test statistic has empirical quantiles almost inlinewith the theoretical quantiles of the standard normal distribution, while dCRT test score strays awayfrom the 45-degree line.

Original: Again, we emphasize that the normality of T j is crucial for the p-valuescalculation.
Revised: Again, we emphasize that the normality of T j is crucial for the p-valuescalculation.

Original: This outlines the importance of the decorrelating step on T j .
Revised: This outlines the importance of the decorrelating step on T j .

Original: 4.2 High-dimensional scenario with varying simulation parameters
Revised: 4.2 High-dimensional scenario with varying simulation parameters

Original: To see how each algorithm performs under different settings, we follow the same simulation scenarioas in Sec. 4.1, but vary each of the three simulation parameters, while keeping the others unchangedat  default value of SNR = 2 . 0 , ρ = 0 .
Revised: To see how each algorithm performs under different settings, we follow the same simulation scenarioas in Sec. 4.1, but vary each of the three simulation parameters, while keeping the others unchangedat a default value of SNR = 2 . 0 , ρ = 0 .

Original: 5 , κ = 0 .
Revised: 5 , κ = 0 .

Original: We target a control of FDR at level 0 .
Revised: We target a control of FDR at level 0 .

Original: 1 , using Benjamini-Hochberg procedure.
Revised: 1 , usingthe Benjamini-Hochberg procedure.

Original: Results in Figure 3 show that CRT-logit is the most powerful methodwhile still controlling the FDR.
Revised: The results in Figure 3 show that CRT-logit is the most powerfulmethod while still controlling the FDR.

Original: Moreover, in the presence of higher correlations between nearbyvariables ( ρ > 0 . 6 ), other methods suffer a drop in average power, but this is not as severe for CRTlogit.
Revised: Moreover, in the presence of higher correlations betweennearby variables ( ρ > 0 . 6 ), other methods suffer a drop in average power, but this is not as severefor CRT-logit.

Original: The original CRT, in general, is conservative.
Revised: The original CRT, in general, is conservative.

Original: We believe that this is due to using only B = 500samplings to generate empirical p-values for the two methods, due to prohibitive average runtimeof the algorithm with larger B (which we provide in Section 4.4).
Revised: We believe that this is due to usingonly B = 500 samplings to generate empirical p-values for the two methods, due to prohibitiveaverage runtime of the algorithm with larger B (which we provide in Section 4.4).

Original: For HRT, the conservativeness isexpected, due to the usage of only half of the sample for test-statistics calculation – even though thenumber of samplings is bigger than original CRT ( B = 5000 ).
Revised: For HRT, theconservativeness is expected, due to the usage of only half of the sample for test-statistics calculation –even though the number of samplings is bigger than original CRT ( B = 5000 ).

Original: We note that, perhaps surprisingly, thedebiased lasso ( cdlasso ) is the most conservative.
Revised: We note that, perhapssurprisingly, the debiased lasso ( cdlasso ) is the most conservative.

Original: It controls FDR well in all settings.
Revised: It controls FDR well in allsettings.

Original: This mightbe due to the fact that dlasso also relies on the choice of the ℓ 1 -regularization λ in the nodewise Lasso operation, similar to the X ∗ ,j -distillation of dCRT, as noted in Section 1.
Revised: This might be due to the fact that dlasso also relies on the choice of the ℓ 1 -regularization λ in the nodewise Lasso operation, similar to the X ∗ ,j -distillation of dCRT, as noted in Section 1.

Original: What makes thedifference is that instead of using cross-validation for setting λ for each variable j , a fixed value ofλ = 10{+
+} − 2 λ max is used in the implementation of dlasso .
Revised: What makes the difference is that instead of using cross-validation for setting λ for each variable j ,a fixed value of λ = {+
+} − 2 λ max is used in the implementation of dlasso .

Original: We strongly suspect this fixed value is notoptimal, which makes the procedure powerless.
Revised: We strongly suspect thisfixed value is not optimal, which makes the procedure powerless.

Revised: 4.3 Application: large-scale analysis on brain-imaging dataset

Original: [Figure]
Revised: [Figure]

Original: Description The Human Connectome Projectdataset (HCP) is a collection of brain imagingdata on healthy young adult subjects with ageranging from 22 to 35.
Revised: Description The Human Connectome Projectdataset (HCP) is a collection of brain imagingdata on healthy young adult subjects with ageranging from 22 to 35.

Original: More specifically, the input X is a set of 2mm statistical maps of 400 subjects across 8 cognitive tasks.
Revised: More specifically, the input X is a set of 2mm statistical maps of 400 subjects across 8 cognitive tasks.

Original: These are calledz-maps, as the data follow a standard normal distribution under the null hypothesis.
Revised: These are calledz-maps, as the data follow a standard normaldistribution under the null hypothesis.

Original: Each task inturn features 2 different contrasts, which effectively form binary responses y ∈ { 0 , 1 } n .
Revised: Each taskin turn features 2 different contrasts, which effectively form binary responses y ∈ { 0 , 1 } n .

Original: Wepropose to fit y through distributed brain signals and identify relevant brain locations.
Revised: Inshort, the goal of this fMRI data analysis is toidentify voxels with task-related levels of activity by fitting y through distributed brain signals.

Original: Thesetting is high-dimensional with n = 800 samples, corresponding to 400 subjects, while thetotal number of variables is p ≈ 200 , 000 brainvoxels.
Revised: The setting is high-dimensional with n = 800samples, corresponding to 400 subjects, whilethe total number of variables is p ≈ 200 , 000brain voxels.

Original: Following [9, 17], we use a hierarchicalclustering scheme to group the variables into C = 1000 spatially connected clusters.
Revised: Following [11, 20], we use a hierarchical clustering scheme to group the variablesinto C = 1000 spatially connected clusters.

Original: We provide details of the pre-processing step in Supplementary Material.
Revised: Weprovide details of the pre-processing step in Supplementary Material.

Original: Creating semi-realistic ground-truth and response labels Since there is no ground truthfor this dataset, we create synthetic true signalsby fitting the data X and response y with anℓ 1 -penalized logistic classifier.
Revised: Creating semi-realistic ground-truth and response labels Since there is no ground truthfor this dataset, we create synthetic true signalsby fitting the data X and response y with an ℓ 1penalized logistic classifier.

Original: In other words,the estimator ˆ β logreg will serve as true regression coefficients for each task.
Revised: In other words, theestimator ˆ β logreg will serve as true regressioncoefficients for each task.

Original: Then, to avoidbias in simulating label ˆ y , the z-maps matrix X of one task are used in conjunction withthe discriminative pattern map ˆ β logreg fromthenext task in the  following order: relational,gambling , emotion , social .
Revised: Then, to avoid bias insimulating label ˆ y , the z-maps matrix X of onetask are used in conjunction with the discriminative pattern map ˆ β logreg from the next task inthe following order: relational , gambling ,emotion , social .

Original: For instance, weuse ˆ β logreg of gambling with z-maps data matrix of relational , i.e. for all i = 1 , . . . , n , given x i, relational , 
Revised: For instance, we use ˆ β logregof gambling with z-maps data matrix of relational , i.e. for all i = 1 , . . . , n , given x i, relational , [Equation]

Original: [Equation]
Revised: [Equation]

Original: (13) where Bern ( a ) is a Bernoulli probability mass function that takes a value 1 with probability a , σ is anoise magnitude and ξ i is a standard normal noise.
Revised: where Bern ( a ) is a Bernoulli probability mass function that takes a value 1 with probability a , σ is anoise magnitude and ξ i is a standard normal noise.

Original: Finally, we apply all inference algorithms on thesemi-synthetic data ( X , ˆ y ) , and we evaluate their performance using the ground-truth ˆ β logreg .
Revised: Finally, we apply all inference algorithms on thesemi-synthetic data ( X , ˆ y ) , and we evaluate their performance using the ground-truth ˆ β logreg .

Original: Thissimulation setting is similar to [9], except that here we consider a classification and not a regressionproblem.
Revised: Thissimulation setting is similar to [11], except that here we consider a classification and not a regressionproblem.

Original: It allows us to calculate the False Discovery Rate and average power with multiple runs ofthe inference procedure (across tasks).
Revised: It allows us to calculate the False Discovery Rate and average power with multiple runs ofthe inference procedure (across tasks).

Original: Remark 4.3.
Revised: Remark 4.3.

Original: The i.i.d.
Revised: The i.i.d.

Original: assumption is formally violated in this experiment, where for each subjectwe analyze two sample image that are not independent.
Revised: assumption is formally violated in this experiment, where for each subjectwe analyze two sample images that are not independent.

Original: Yet, this remains a short-range correlationstructure, and is thus not a strong challenge to the i.i.d. assumption.
Revised: Yet, this remains a short-range correlationstructure, and is thus not a strong challenge to the i.i.d. assumption.

Original: Results Results in Figure 4 show that CRT-logit achieves a better recovery compared to KO ororiginal CRT/dCRT/HRT, which results in higher statistical power.
Revised: Results The results in Figure 4 show that CRT-logit achieves a better recovery compared to KOor original CRT/dCRT/HRT, which results in higher statistical power.

Original: This gain comes with a goodcontrol of the FDR under desired level α = 0 .
Revised: This gain comes with a goodcontrol of the FDR under desired level α = 0 .

Original: On a related note, the only analysis where dCRTmakes more discoveries than CRT-Logit is in  emotion task, but at the cost of failing to control FDRat nominal level. 
Revised: On a related note, the only analysis where dCRTmakes more discoveries than CRT-Logit is in the emotion task, but at the cost of failing to control FDR at the nominal level. [Figure]

Original: 4.4 Average runtime of benchmarked methods
Revised: 4.4 Average runtime of benchmarked methods

Revised: Methods

Revised: Simulated (Sec. 4.2)

Revised: HCP

Revised: [Table]

Revised: semi

Original: Besides statistical performance, it is equally important to assess the computational cost of inferenceprocedures.
Revised: [Table] Sec [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: dCRT[screening=

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: ) dCRT[screening

Revised: [Table]

Revised: False] [19]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: The average runtime in Table 1 from the two experiments shows that the original CRT is
Revised: CRT-logit[screening= True] (this work) [Table] [Table]

Revised: [Table]

Original: [Figure]
Revised: [Table]

Revised: 55) CRT-logit[screening=

Original: [Table]
Revised: [Table]

Original: not suitable for large-scale inference: it is over 2000 times slower than the fastest method (Knockoff
Revised: Besides statistical performance, it is equally important to assess the computational cost of inferenceprocedures. The average runtime in Table 1 from the two experiments shows that the original CRT isnot suitable for large-scale inference: it is over 2000 times slower than the fastest method (Knockoff

Original: Filter), and over 150 times slower than dCRT/CRT-logit.
Revised: Filter), and over 150 times slower than dCRT/CRT-logit.

Original: The empirical runtime also confirms theeffectiveness of the screening step before doing distillation/decorrelation of the test-statistics: thestep makes CRT-logit and dCRT 20 times faster than without.
Revised: The empirical runtime also confirms theeffectiveness of the screening step before doing distillation/decorrelation of the test-statistics: thestep makes CRT-logit and dCRT 20 times faster than without.

Original: On a related note, although in theory
Revised: On a related note, although in theory

Original: Debiased Lasso, dCRT and CRT-logit (both without screening) share the same runtime complexity,the latter two are slower due to the use of cross-validation to estimate the sparsity hyperparameter λand λ dx (detailed in Section 3).
Revised: Debiased Lasso, dCRT and CRT-logit (both without screening) share the same runtime complexity,the latter two are slower due to the use of cross-validation to estimate the sparsity hyperparameter λand λ dx (detailed in Section 3).

Original:  Discussion
Revised:  Discussion

Original: We proposed an adaptation of the Conditional Randomization Test (CRT) for sparse logistic regressionin the high-dimensional regime.
Revised: We proposed an adaptation of the Conditional Randomization Test (CRT) for sparse logistic regressionin the high-dimensional regime.

Original: A major improvement of CRT-logit, our proposed algorithm,compared to original CRT, comes from the decorrelation of test statistics to make their distributioncloser to standard normal.
Revised: A major improvement of CRT-logit, our proposed algorithm,compared to original CRT, comes from the decorrelation of test statistics to make their distributioncloser to standard normal.

Original: Indeed, results from synthetic experiments in Figure 2 show that in highdimension (when 0 . 5 ≤ n/p ≤ 1 . 0 ), the empirical null distribution of CRT-logit’s test statistic T decorris much more similar to a standard normal compared to the original CRT test statistic.
Revised: Indeed, results from synthetic experiments in Figure 6 show that in highdimension (when 0 . 5 ≤ n/p ≤ 1 . 0 ), the empirical null distribution of CRT-logit’s test statistic T decorris much more similar to a standard normal compared to the original CRT test statistic.

Original: Moreover,empirical benchmarks in Section 4 demonstrate that CRT-logit performs better than related statisticalinference methods, such as the Debiased Lasso or Model-X Knockoffs.
Revised: Moreover,empirical benchmarks in Section 4 demonstrate that CRT-logit performs better than related statisticalinference methods, such as the Debiased Lasso or Model-X Knockoffs.

Original: In particular, CRT-logit is themost powerful method in our synthetic experiment with high-dimensional datasets in Section 4.2,while still keeping FDR controlled under predefined level α = 0 .
Revised: In particular, CRT-logit is themost powerful method in our synthetic experiment with high-dimensional datasets in Section 4.2,while still keeping FDR controlled under predefined level α = 0 .

Revised: 1 (with a slight caveat of using BHinstead of BY procedure, as elaborated in Remark 4.2).

Original: We note that there exists some limitations to CRT-logit.
Revised: We note that there exist some limitations to CRT-logit.

Original: The computational cost of CRT-logit, whilelower than vanilla CRT, is still larger than alternative methods such as Knockoff Filter and Holdout Randomization Test.
Revised: The computational cost of CRT-logit, while lower than vanilla CRT, is still larger thanalternative methods such as Knockoff Filter and Holdout Randomization Test.

Original: Moreover, tuning the ℓ 1 − regularization λ dx parameter by cross-validation, as isoften done, can further increase the computational cost of CRT-logit (and dCRT).
Revised: Moreover, tuningthe ℓ 1 − regularization λ dx parameter by cross-validation, as is often done, can further increasethe computational cost of CRT-logit (and dCRT).

Original: Despite this, our empirical benchmarks on both simulated and real data show real promises of CRTlogit.
Revised: Despite this, our empirical benchmarks on bothsimulated and real data show real promises of CRT-logit.

Original: Henceforth, we believe CRT-logit is competitive for practical settings that involve structureddata, such as brain-imaging and genomics applications.
Revised: Henceforth, we believe CRT-logit iscompetitive for practical settings that involve structured data, such as brain-imaging and genomicsapplications.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes](b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [No] We believeour work provides no potential negative societal impacts.
Revised: [No] We believe our work provides no potential negative societal impacts.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results.
Revised: If you are including theoretical results...

Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Table]
Revised: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] The proofs can be
Revised: [Yes] The proofs can be

Original: found in Supplementary Material
Revised: found in Supplementary Material

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main exper-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main exper-

Original: imental results (either in the supplemental material or as a URL)?
Revised: imental results (either in the supplemental material or as a URL)?

Original: [Yes] Except for datasets of realistic applications, which are not included, but references are provided
Revised: [Yes] Except fordatasets of realistic applications, which are not included, but references are provided

Original: (b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes] Both in main text and in Supplementary Material
Revised: [Yes] Both in main text and in Supplementary Material

Original: (c) Did you report error bars (e
Revised: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experiments multiple times)?
Revised: g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] With the simulated and semi-realistic experiment
Revised: [Yes] With the simulated and semi-realistic experiment

Original: (d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [No]
Revised: [No]

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [No]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A
Revised: [N/A]

Original: [Table]