Original: A Conditional Randomization Test for Sparse Logistic Regression in High-Dimension
Revised: A Conditional Randomization Test for Sparse Logistic Regression in High-Dimension

Original: Abstract
Revised: Abstract

Original: Identifying the relevant variables for a classification model with correct confidence levels is a central but difficult task in high-dimension.
Revised: Identifying the relevant variables for a classification model with correct confidence levels is a central but difficult task in high-dimension.

Original: Despite the core role of sparse logistic regression in statistics and machine learning, it still lacks a good solution for accurate inference in the regime where the number of features p is as large as or larger than the number of samples n .
Revised: Despite the core role of sparse logistic regression in statistics and machine learning, it still lacks a good solution for accurate inference in the regime where the number of features p is as large as or larger than the number of samples n .

Original: Here we tackle this problem by improving the Conditional Randomization Test (CRT).
Revised: Here we tackle this problem by improving the Conditional Randomization Test (CRT).

Original: The original
Revised: The original

Original: CRT algorithm shows promise as a way to output p-values while making few assumptions on the distribution of the test statistics.
Revised: CRT algorithm shows promise as a way to output p-values while making few assumptions on the distribution of the test statistics.

Original: As it comes with a prohibitive computational cost even in mildly high-dimensional problems, faster solutions based on distillation have been proposed.
Revised: As it comes with a prohibitive computational cost even in mildly high-dimensional problems, faster solutions based on distillation have been proposed.

Original: Yet, they rely on unrealistic hypotheses and result in low-power solutions.
Revised: Yet, they rely on unrealistic hypotheses and result in low-power solutions.

Original: To improve this, we propose CRT-logit , an algorithm that combines a variable-distillation step and a decorrelation step that takes into account the geometry of the ℓ 1 -penalized logistic regression problem.
Revised: To improve this, we propose CRT-logit , an algorithm that combines a variable-distillation step and a decorrelation step that takes into account the geometry of the ℓ 1 -penalized logistic regression problem.

Original: We provide a theoretical analysis of this procedure, and demonstrate its effectiveness on simulations, along with experiments on large-scale brain-imaging and genomics datasets.
Revised: We provide a theoretical analysis of this procedure, and demonstrate its effectiveness on simulations, along with experiments on large-scale brain-imaging and genomics datasets.

Original:  Introduction
Revised:  Introduction

Original: Logistic regression is one of the most popular tools in modern applications of statistics and machine learning, partly due to its relative algorithmic simplicity.
Revised: Logistic regression is one of the most popular tools in modern applications of statistics and machine learning, partly due to its relative algorithmic simplicity.

Original: The method belongs to the class of generalized linear models that handle discrete outcomes, i e. classification problems.
Revised: The method belongs to the class of generalized linear models that handle discrete outcomes, i e. classification problems.

Original: Here, we focus on the binary classification problem, where one observation of the responses y ∈ { 0 , 1 } and the data vectors x ∈ R p follows the relationship:
Revised: Here, we focus on the binary classification problem, where one observation of the responses y ∈ { 0 , 1 } and the data vectors x ∈ R p follows the relationship:

Original: [Equation]
Revised: [Equation]

Original: where g ( x ) = 1 / (1 + exp( − x )) is the sigmoid function, and β 0 the vector of true regression coefficients.
Revised: where g ( x ) = 1 / (1 + exp( − x )) is the sigmoid function, and β 0 the vector of true regression coefficients.

Original: In the classical setting, in which the number of samples n is greater than the number of features p , an estimate ˆ β of the true signals β 0 can be obtained using maximum likelihood estimation (MLE).
Revised: In the classical setting, in which the number of samples n is greater than the number of features p , an estimate ˆ β of the true signals β 0 can be obtained using maximum likelihood estimation (MLE).

Original: The asymptotic behaviour and derivation of the test statistic, confidence intervals and p-values of the MLE have been well studied, e.g. in [13].
Revised: The asymptotic behaviour and derivation of the test statistic, confidence intervals and p-values of the MLE have been well studied, e.g. in [13].

Original: The availability of p-values for the test statistics makes it possible to rely on multiple hypothesis testing, where one wants to test which variables have a non-zero effect on the outcome, conditioned on the remaining variables.
Revised: The availability of p-values for the test statistics makes it possible to rely on multiple hypothesis testing, where one wants to test which variables have a non-zero effect on the outcome, conditioned on the remaining variables.

Original: Unfortunately, this line of analysis cannot be applied to the high-dimensional regime, where p is larger than n , as argued in [25, 32, 34].
Revised: Unfortunately, this line of analysis cannot be applied to the high-dimensional regime, where p is larger than n , as argued in [25, 32, 34].

Original: These works show that in the regime lim n,p →∞ n/p = κ , the MLE estimator exists only when κ > 2 .
Revised: These works show that in the regime lim n,p →∞ n/p = κ , the MLE estimator exists only when κ > 2 .

Original: However, we note that this type of analysis is done without the addition of ℓ 1 -regularization to the likelihood function, i.e. without using a penalized estimator to enforce sparsity.
Revised: However, we note that this type of analysis is done without the addition of ℓ 1 -regularization to the likelihood function, i.e. without using a penalized estimator to enforce sparsity.

Original: Motivation Our focus in this paper is to do inference with statistical guarantees on high-dimensional sparse logistic regression, where p is larger or much larger than n .
Revised: Motivation Our focus in this paper is to do inference with statistical guarantees on high-dimensional sparse logistic regression, where p is larger or much larger than n .

Original: This setting is typical in modern applications of pattern recognition, e.g. in brain-imaging or genomics [3], with p as large as hundreds of thousands –compressible to thousands– but n stays at most few thousand.
Revised: This setting is typical in modern applications of pattern recognition, e.g. in brain-imaging or genomics [3], with p as large as hundreds of thousands –compressible to thousands– but n stays at most few thousand.

Original: [Figure]
Revised: [Figure]

Original: The family of methods we consider is the Conditional Randomization Test (CRT) [10].
Revised: The family of methods we consider is the Conditional Randomization Test (CRT) [10].

Original: CRT relies on generating multiple noisy copies of original variables to output empirical p-values in high-dimensional inference problems.
Revised: CRT relies on generating multiple noisy copies of original variables to output empirical p-values in high-dimensional inference problems.

Original: However, prohibitive computational cost makes CRT impractical, as discussed at length in [10, 26, 7, 19].
Revised: However, prohibitive computational cost makes CRT impractical, as discussed at length in [10, 26, 7, 19].

Original: There have been several lines of research attempting to fix this problem, most notably the distilled Conditional Randomization Test (dCRT) [19].
Revised: There have been several lines of research attempting to fix this problem, most notably the distilled Conditional Randomization Test (dCRT) [19].

Original: This work introduced a distillation step as a replacement for the randomized sampling step to compute the importance statistics (see Section 2 for more details).
Revised: This work introduced a distillation step as a replacement for the randomized sampling step to compute the importance statistics (see Section 2 for more details).

Original: It provides a way to output p-values for multiple types of regression and classification problems, assuming convergence to Gaussian distribution of the test statistic in large-sample regime.
Revised: It provides a way to output p-values for multiple types of regression and classification problems, assuming convergence to Gaussian distribution of the test statistic in large-sample regime.

Original: Yet, as shown in the left panel of Figure 1, the originally proposed dCRT test-statistic for logistic regression does not behave as well as intended.
Revised: Yet, as shown in the left panel of Figure 1, the originally proposed dCRT test-statistic for logistic regression does not behave as well as intended.

Original: In particular, its null distribution deviates markedly from standard normal in high-dimension whenever n/p ≤ 1 .
Revised: In particular, its null distribution deviates markedly from standard normal in high-dimension whenever n/p ≤ 1 .

Original: Contribution We propose a correction for the dCRT, inspired by the decorrelation method presented in [21].
Revised: Contribution We propose a correction for the dCRT, inspired by the decorrelation method presented in [21].

Original: The decorrelation step makes the null-distribution of the test statistics much closer to standard normal, as shown on the right panel of Figure 1, and thus increases the statistical power of the method.
Revised: The decorrelation step makes the null-distribution of the test statistics much closer to standard normal, as shown on the right panel of Figure 1, and thus increases the statistical power of the method.

Original: We provide asymptotic analysis of this method, which shows that CRT-logit produces standard normal test-statistics in the large-sample regime.
Revised: We provide asymptotic analysis of this method, which shows that CRT-logit produces standard normal test-statistics in the large-sample regime.

Original: In addition, we validate the high performance of
Revised: In addition, we validate the high performance of

Original: CRT-logit on large-scale brain-imaging and genetics datasets, thus showing its usefulness in practical applications.
Revised: CRT-logit on large-scale brain-imaging and genetics datasets, thus showing its usefulness in practical applications.

Original: Related works The closest cousin of the Conditional Randomization Test is Knockoff Filter [4, 10], a recent breakthrough in the False Discovery Rate (FDR) control literature.
Revised: Related works The closest cousin of the Conditional Randomization Test is Knockoff Filter [4, 10], a recent breakthrough in the False Discovery Rate (FDR) control literature.

Original: It relies on the creation of additional noisy features, called knockoffs, to calculate variable-importance statistics.
Revised: It relies on the creation of additional noisy features, called knockoffs, to calculate variable-importance statistics.

Original: Another extension of vanilla CRT is the Holdout Randomization Test (HRT)
Revised: Another extension of vanilla CRT is the Holdout Randomization Test (HRT)

Original: While still requiring multiple samplings of noisy variables, HRT solves the computational issue of original CRT by doing heavy model fitting only once on one part of the dataset, and test statistics calculation on the other part, without refitting the model.
Revised: While still requiring multiple samplings of noisy variables, HRT solves the computational issue of original CRT by doing heavy model fitting only once on one part of the dataset, and test statistics calculation on the other part, without refitting the model.

Original: However, this method relies on sample-splitting, and hence inherently suffers from a loss of statistical power.
Revised: However, this method relies on sample-splitting, and hence inherently suffers from a loss of statistical power.

Original: A parallel line of work has introduced the Conditional Permutation Test (CPT)
Revised: A parallel line of work has introduced the Conditional Permutation Test (CPT)

Original: [7], a non-parametric alternative to CRT that relies on a random shuffling mechanism applied to original variables, instead of multiple sampling of new variables.
Revised: [7], a non-parametric alternative to CRT that relies on a random shuffling mechanism applied to original variables, instead of multiple sampling of new variables.

Original: This potentially makes CPT more robust to model mis-specification.
Revised: This potentially makes CPT more robust to model mis-specification.

Original: [32] recently proposed a method called SLOE, which adapts the analysis of [34], but in a regime different from what we are considering, where lim n,p →∞ n/p → κ ∈ (1 , 2) , and more importantly without sparsity-inducing penalty.
Revised: [32] recently proposed a method called SLOE, which adapts the analysis of [34], but in a regime different from what we are considering, where lim n,p →∞ n/p → κ ∈ (1 , 2) , and more importantly without sparsity-inducing penalty.

Original: On a separate note, we notice the similarity of dCRT [19] with debiased Lasso [16, 28, 33].
Revised: On a separate note, we notice the similarity of dCRT [19] with debiased Lasso [16, 28, 33].

Original: This line of work proposed a debiasing formula for the estimator, which makes the asymptotic distribution of ( ˆ β LASSO − β 0 ) standard normal, so that one can compute the test statistic and p-value associated with each variable.
Revised: This line of work proposed a debiasing formula for the estimator, which makes the asymptotic distribution of ( ˆ β LASSO − β 0 ) standard normal, so that one can compute the test statistic and p-value associated with each variable.

Original:  Background
Revised:  Background

Original: Notation We denote matrices, vectors, scalars and sets by bold uppercase, bold lowercase, script lowercase , and calligraphic letters, respectively, e.g. X , x , x , and X .
Revised: Notation We denote matrices, vectors, scalars and sets by bold uppercase, bold lowercase, script lowercase , and calligraphic letters, respectively, e.g. X , x , x , and X .

Original: The i -th row of a matrix
Revised: The i -th row of a matrix

Original: X will be denoted X i, ∗ , the j -th column X ∗ ,j and the ( i, j ) -th
Revised: X will be denoted X i, ∗ , the j -th column X ∗ ,j and the ( i, j ) -th

Original: element X i,j .
Revised: element X i,j .

Original: For any natural number p , we denote the set [ p ] def.
Revised: For any natural number p , we denote the set [ p ] def.

Original: = { 1 , . . .
Revised: = { 1 , . . .

Original: , p } .
Revised: , p } .

Original: For each x ∈ R p and j ∈ [ p ] , we denote x − j def.
Revised: For each x ∈ R p and j ∈ [ p ] , we denote x − j def.

Original: = { x 1 , x 2 , . . .
Revised: = { x 1 , x 2 , . . .

Original: , x j − 1 , x j +1 , . . .
Revised: , x j − 1 , x j +1 , . . .

Original: , x p } a p − 1 dimensional observation after removing the j -th variable.
Revised: , x p } a p − 1 dimensional observation after removing the j -th variable.

Original: Correspondingly, X − j is the data matrix X ∈ R n × p with column X ∗ ,j removed.
Revised: Correspondingly, X − j is the data matrix X ∈ R n × p with column X ∗ ,j removed.

Original: The cumulative distribution function (CDF) of the standard Gaussian distribution will be denoted Φ( · ) .
Revised: The cumulative distribution function (CDF) of the standard Gaussian distribution will be denoted Φ( · ) .

Original: The indicator function of a random event A will be denoted 1 A .
Revised: The indicator function of a random event A will be denoted 1 A .

Original: For any two positive sequences x n and y n , we write x n ≍ y n if cy n ≤ x n ≤ Cy n for all n , for some positive constants c and C .
Revised: For any two positive sequences x n and y n , we write x n ≍ y n if cy n ≤ x n ≤ Cy n for all n , for some positive constants c and C .

Original: For a vector x , ∥ x ∥ p denotes its ℓ p norm.
Revised: For a vector x , ∥ x ∥ p denotes its ℓ p norm.

Original: For a function f : R p
Revised: For a function f : R p

Original: → R , ∇ j f denotes its gradient w.r.t.
Revised: → R , ∇ j f denotes its gradient w.r.t.

Original: the j -th variable, for j ∈ [ p ] .
Revised: the j -th variable, for j ∈ [ p ] .

Original: Problem setting We consider exclusively binary classification, where the response vector is denoted y ∈ { 0 , 1 } n and the data matrix X ∈ R n × p consists of n p -dimensional samples.
Revised: Problem setting We consider exclusively binary classification, where the response vector is denoted y ∈ { 0 , 1 } n and the data matrix X ∈ R n × p consists of n p -dimensional samples.

Original: Throughout the paper, we assume the data { X i, ∗ } ni =1 are i.i.d.
Revised: Throughout the paper, we assume the data { X i, ∗ } ni =1 are i.i.d.

Original: and follow a distribution with zero mean and population covariance matrix Σ .
Revised: and follow a distribution with zero mean and population covariance matrix Σ .

Original: Moreover, we assume that X i, ∗ and y i follow the logistic relationship in Eq.
Revised: Moreover, we assume that X i, ∗ and y i follow the logistic relationship in Eq.

Original: We denote the support set S def.
Revised: We denote the support set S def.

Original: = { j ∈ [ p ] : β 0 j ̸ = 0 } and assume that it is sparse, i.e. card( S ) =
Revised: = { j ∈ [ p ] : β 0 j ̸ = 0 } and assume that it is sparse, i.e. card( S ) =

Original: s ∗ ≪ p , where card denotes the cardinality of a set.
Revised: s ∗ ≪ p , where card denotes the cardinality of a set.

Original: Furthermore, ˆ S def.
Revised: Furthermore, ˆ S def.

Original: = { j ∈ [ p ] : ˆ
Revised: = { j ∈ [ p ] : ˆ

Original: β j ̸ = 0 } indicates an estimation of S , where ˆ β j is an estimate of the true signal β 0 j .
Revised: β j ̸ = 0 } indicates an estimation of S , where ˆ β j is an estimate of the true signal β 0 j .

Original: We try to obtain it through a ℓ 1 -penalized logistic estimator:
Revised: We try to obtain it through a ℓ 1 -penalized logistic estimator:

Original: [Equation]
Revised: [Equation]

Original: We denote I def.
Revised: We denote I def.

Original: =
Revised: =

Original: E β 

Revised: E β

Original: [ ∇ 2 ℓ ( β 0 )]
Revised: [ ∇ 2 ℓ ( β 0 )]

Original: the Fisher information matrix, and I j |− j the partial Fisher information, defined by I j |− j def.
Revised: the Fisher information matrix, and I j |− j the partial Fisher information, defined by I j |− j def.

Original: =
Revised: =

Original: E [ ∇ 2 jj ℓ ( β 0 ) − [ ∇ 2 j, − j ℓ ( β 0 )]
Revised: E [ ∇ 2 jj ℓ ( β 0 ) − [ ∇ 2 j, − j ℓ ( β 0 )]

Original: ⊤ [ ∇ 2 − j, − j ℓ ( β 0 )] − 1 ∇ 2 − j,j ℓ ( β 0 )]
Revised: ⊤ [ ∇ 2 − j, − j ℓ ( β 0 )] − 1 ∇ 2 − j,j ℓ ( β 0 )]

Original: =
Revised: =

Original: I jj − I j, − j I − 1 − j, − j I − j,j , where I j, − j is the row-vector made with the j th-row and the columns corresponding to β − j , I − j, − j the sub-matrix of I made with the rows and columns corresponding to β − j .
Revised: I jj − I j, − j I − 1 − j, − j I − j,j , where I j, − j is the row-vector made with the j th-row and the columns corresponding to β − j , I − j, − j the sub-matrix of I made with the rows and columns corresponding to β − j .

Original: This quantity, defined following [13, pp. 323], plays an important role in our proposed method, detailed in Section 3.
Revised: This quantity, defined following [13, pp. 323], plays an important role in our proposed method, detailed in Section 3.

Original: Statistical control with False Discovery Rate To quantify statistical errors, we consider the False
Revised: Statistical control with False Discovery Rate To quantify statistical errors, we consider the False

Original: Discovery Rate , introduced in [5].
Revised: Discovery Rate , introduced in [5].

Original: Given an estimate of the support ˆ S , the false discovery proportion (FDP) is the ratio of the number of selected features that do not belong to the true support S , divided by the total number of selected features.
Revised: Given an estimate of the support ˆ S , the false discovery proportion (FDP) is the ratio of the number of selected features that do not belong to the true support S , divided by the total number of selected features.

Original: The False Discovery Rate is the expectation of the FDP:
Revised: The False Discovery Rate is the expectation of the FDP:

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Conditional Randomization Test (CRT) and Distillation CRT (dCRT)
Revised: Conditional Randomization Test (CRT) and Distillation CRT (dCRT)

Original: The concept of Conditional Randomization Test was originally proposed in the model-X knockoff paper [10] as a way to output valid empirical p-values using knockoff variables.
Revised: The concept of Conditional Randomization Test was originally proposed in the model-X knockoff paper [10] as a way to output valid empirical p-values using knockoff variables.

Original: The principle of the knockoff filter is first to sample noisy copies ˜
Revised: The principle of the knockoff filter is first to sample noisy copies ˜

Original: X ∗ ,j of variable X ∗ ,j , given a known sampling mechanism P j |− j .
Revised: X ∗ ,j of variable X ∗ ,j , given a known sampling mechanism P j |− j .

Original: One advantage of the knockoff filter is that no specific assumption is placed on the distribution of the inferred test statistic.
Revised: One advantage of the knockoff filter is that no specific assumption is placed on the distribution of the inferred test statistic.

Original: However, this means that, in general, there is no mechanism to derive p-values from the knockoff statistic.
Revised: However, this means that, in general, there is no mechanism to derive p-values from the knockoff statistic.

Original: This motivates the introduction of CRT, which requires running high-dimensional inference for each variable j B times.
Revised: This motivates the introduction of CRT, which requires running high-dimensional inference for each variable j B times.

Original: However, the computation cost of CRT is prohibitive when p grows large: assuming that we use the Lasso program with coordinate descent to compute T CRT j , its runtime would be O
Revised: However, the computation cost of CRT is prohibitive when p grows large: assuming that we use the Lasso program with coordinate descent to compute T CRT j , its runtime would be O

Original: ( Bp 4 ) [15, pp. 93].
Revised: ( Bp 4 ) [15, pp. 93].

Original: Moreover, CRT requires decently large
Revised: Moreover, CRT requires decently large

Original: B to make the empirical distribution of the p-values smooth enough.
Revised: B to make the empirical distribution of the p-values smooth enough.

Original: Reducing the computational cost of CRT is the main motivation of several works [7, 19, 26].
Revised: Reducing the computational cost of CRT is the main motivation of several works [7, 19, 26].

Original: One of them is the introduction of distillation-CRT (dCRT)
Revised: One of them is the introduction of distillation-CRT (dCRT)

Original: The main appeal of this method is that it can output p-values analytically, therefore bypassing the multiple knockoffs sampling steps used to infer on each variable, and leads to a reasonable reduction of the computation cost.
Revised: The main appeal of this method is that it can output p-values analytically, therefore bypassing the multiple knockoffs sampling steps used to infer on each variable, and leads to a reasonable reduction of the computation cost.

Original: Distillation operation The key addition of dCRT is the distillation operation: for each variable j , we want to distill all the conditional information of the remaining variables
Revised: Distillation operation The key addition of dCRT is the distillation operation: for each variable j , we want to distill all the conditional information of the remaining variables

Original: X − j to x j and to y via least-squares minimization with ℓ 1 -regularization to enforce sparsity.
Revised: X − j to x j and to y via least-squares minimization with ℓ 1 -regularization to enforce sparsity.

Original: For each variable j , we first solve the lasso problem by regressing X ∗ ,j on X − j ,
Revised: For each variable j , we first solve the lasso problem by regressing X ∗ ,j on X − j ,

Original: [Equation]
Revised: [Equation]

Original: For distillation of variable j and the binary response y with logistic relationship, [19] briefly suggested to solve a penalized estimation problem, similar to Eq.
Revised: For distillation of variable j and the binary response y with logistic relationship, [19] briefly suggested to solve a penalized estimation problem, similar to Eq.

Original: [Equation]
Revised: [Equation]

Original: Then, a test statistic is calculated for each j = 1 , . .
Revised: Then, a test statistic is calculated for each j = 1 , . .

Original: , p :
Revised: , p :

Original: [Equation]
Revised: [Equation]

Original: Intuitively, Eq.
Revised: Intuitively, Eq.

Original: (5) is the correlation of the regression residuals, calculated from Eq.
Revised: (5) is the correlation of the regression residuals, calculated from Eq.

Original: (3) and (4), scaled by a factor of √ n .
Revised: (3) and (4), scaled by a factor of √ n .

Original: Under the null hypothesis, and more importantly, assuming linear relationship between X i, ∗ and y , this quantity follows standard normal distribution asymptotically, conditional to y and X − j .
Revised: Under the null hypothesis, and more importantly, assuming linear relationship between X i, ∗ and y , this quantity follows standard normal distribution asymptotically, conditional to y and X − j .

Original: It then follows that we can output a p-value for each variable j by taking (cid:98) p
Revised: It then follows that we can output a p-value for each variable j by taking (cid:98) p

Original: j = 2 [1 − Φ ( T j )] .
Revised: j = 2 [1 − Φ ( T j )] .

Original: However, the formulation of test statistics in Eq.
Revised: However, the formulation of test statistics in Eq.

Original: (5) is not truly satisfactory in the setting of sparse logistic regression.
Revised: (5) is not truly satisfactory in the setting of sparse logistic regression.

Original: More specifically, both the calculation of regression residuals y − X − j ˆ β d y ,j and test statistics T j do not take into account the non-linear relationship between X and the binary response y .
Revised: More specifically, both the calculation of regression residuals y − X − j ˆ β d y ,j and test statistics T j do not take into account the non-linear relationship between X and the binary response y .

Original: The first row of Figure 6 plots the qq-plot of the test statistics T j for logistic regression, which shows that even in the classical regime where n > p , its distribution is far from standard normal.
Revised: The first row of Figure 6 plots the qq-plot of the test statistics T j for logistic regression, which shows that even in the classical regime where n > p , its distribution is far from standard normal.

Original:  Decorrelating Test-Statistics for High-Dimensional Logistic Regression
Revised:  Decorrelating Test-Statistics for High-Dimensional Logistic Regression

Original: As we have elaborated, the formulation of dCRT is not well-suited for problems other than penalized least-squares regression.
Revised: As we have elaborated, the formulation of dCRT is not well-suited for problems other than penalized least-squares regression.

Original: We therefore propose an adaptation of dCRT in the case of logistic regression, inspired by the classical work of [13] and by [21].
Revised: We therefore propose an adaptation of dCRT in the case of logistic regression, inspired by the classical work of [13] and by [21].

Original: First, note that when testing H j 0 : β
Revised: First, note that when testing H j 0 : β

Original: j = 0 under the case where n > p , we have the classical Rao’s test statistic, defined by
Revised: j = 0 under the case where n > p , we have the classical Rao’s test statistic, defined by

Original: [Equation]
Revised: [Equation]

Original: where ∇ j ℓ (0 , ˆ β − j ) def.
Revised: where ∇ j ℓ (0 , ˆ β − j ) def.

Original: = ∇
Revised: = ∇

Original: [Equation]
Revised: [Equation]

Original: is the Fisher score.
Revised: is the Fisher score.

Original: Here ˆ β − j def.
Revised: Here ˆ β − j def.

Original: =
Revised: =

Original: argmin β − j ∈ R p − 1 ℓ ( β j , β − j ) is the constrained maximum-likelihood estimator of β − j with fixed β j , and ˆ I j |− j is a consistent estimator of the partial Fisher information
Revised: argmin β − j ∈ R p − 1 ℓ ( β j , β − j ) is the constrained maximum-likelihood estimator of β − j with fixed β j , and ˆ I j |− j is a consistent estimator of the partial Fisher information

Original: I j |− j .
Revised: I j |− j .

Original: The appearance of the term ˆ
Revised: The appearance of the term ˆ

Original: I − 1 / 2 j |− j is due to the fact that under the null hypothesis H j 0 , we have, by [13, Chapter 9], [24],
Revised: I − 1 / 2 j |− j is due to the fact that under the null hypothesis H j 0 , we have, by [13, Chapter 9], [24],

Original: [Equation]
Revised: [Equation]

Original: which makes the asymptotic distribution of T Rao j standard normal.
Revised: which makes the asymptotic distribution of T Rao j standard normal.

Original: However, in the high-dimension case, where n < p , we do not reach this convergence in distribution.
Revised: However, in the high-dimension case, where n < p , we do not reach this convergence in distribution.

Original: To see this, consider the Taylor expansion of the Fisher score of variable j around any given estimator (cid:101)
Revised: To see this, consider the Taylor expansion of the Fisher score of variable j around any given estimator (cid:101)

Original: β − j of the true β 0 − j :
Revised: β − j of the true β 0 − j :

Original: [Equation]
Revised: [Equation]

Original: On the right-hand side, the first term converges weakly to a normal distribution due to the Central
Revised: On the right-hand side, the first term converges weakly to a normal distribution due to the Central

Original: Limit Theorem, the remainder term becomes negligible using ℓ 1 regularization to induce sparsity, but the second term does not, due to estimation bias and sparsity effect of (cid:101) β − j [14].
Revised: Limit Theorem, the remainder term becomes negligible using ℓ 1 regularization to induce sparsity, but the second term does not, due to estimation bias and sparsity effect of (cid:101) β − j [14].

Original: Adapting distillation operation for sparse logistic regression Fortunately, Eq.
Revised: Adapting distillation operation for sparse logistic regression Fortunately, Eq.

Original: (7) suggests that for each variable j , we can debias the Fisher score by correcting the impact of other terms.
Revised: (7) suggests that for each variable j , we can debias the Fisher score by correcting the impact of other terms.

Original: In particular, for each variable j , we replace the Fisher score by
Revised: In particular, for each variable j , we replace the Fisher score by

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The inversion of the large matrix I − j, − j ∈ R ( p − 1) × ( p − 1) is computationally prohibitive, but we can estimate the term I j, − j I − 1 − j, − j straightforwardly by solving
Revised: The inversion of the large matrix I − j, − j ∈ R ( p − 1) × ( p − 1) is computationally prohibitive, but we can estimate the term I j, − j I − 1 − j, − j straightforwardly by solving

Original: [Equation]
Revised: [Equation]

Original: for each variable j , where ˆ β is given with Eq.
Revised: for each variable j , where ˆ β is given with Eq.

Original: Moreover, since we have the closed-form of the derivatives of the logistic loss ℓ ( ˆ β ) , a simple derivation from Eq.
Revised: Moreover, since we have the closed-form of the derivatives of the logistic loss ℓ ( ˆ β ) , a simple derivation from Eq.

Original: (9) suggests the following x j -distillation operation, adapted for logistic regression :
Revised: (9) suggests the following x j -distillation operation, adapted for logistic regression :

Original: [Equation]
Revised: [Equation]

Original: where the extra term (second-order derivative of the sigmoid function) g ′′ ( X i
Revised: where the extra term (second-order derivative of the sigmoid function) g ′′ ( X i

Original: [Equation]
Revised: [Equation]

Original: β
Revised: β

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: appears from differentiating twice the loss function ℓ ( ˆ β ) , and ˆ β = ˆ β λ is defined in Eq.
Revised: appears from differentiating twice the loss function ℓ ( ˆ β ) , and ˆ β = ˆ β λ is defined in Eq.

Original: On the other hand, we can obtain ˆ β d y ,j j from ˆ β by simply omitting the j -th coefficient from it, i.e.
Revised: On the other hand, we can obtain ˆ β d y ,j j from ˆ β by simply omitting the j -th coefficient from it, i.e.

Original: [Equation]
Revised: [Equation]

Original: Finally, the equation for decorrelated test score, adapted from both Eq.
Revised: Finally, the equation for decorrelated test score, adapted from both Eq.

Original: (5) and (6), reads
Revised: (5) and (6), reads

Original: [Equation]
Revised: [Equation]

Original: 11) where the formula for the empirical partial
Revised: 11) where the formula for the empirical partial

Original: Fisher information is ˆ I
Revised: Fisher information is ˆ I

Original: [Equation]
Revised: [Equation]

Original: |− j = n −
Revised: |− j = n −

Original: [Equation]
Revised: [Equation]

Original: ni =
Revised: ni =

Original: [Equation]
Revised: [Equation]

Original: g
Revised: g

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: ˆ β )( X i,j − X i, − j ˆ β d X
Revised: ˆ β )( X i,j − X i, − j ˆ β d X

Original: [Equation]
Revised: [Equation]

Original: j ) X i,j .
Revised: j ) X i,j .

Original: A summary of the full procedure, which we call CRT-logit, can be found in Algorithm 1.
Revised: A summary of the full procedure, which we call CRT-logit, can be found in Algorithm 1.

Original: Notice that the runtime of CRT-logit is the same as dCRT, which means in general slower than KO and HRT.
Revised: Notice that the runtime of CRT-logit is the same as dCRT, which means in general slower than KO and HRT.

Original: To speedup inference time, we introduce a variable-screening step that eliminates potentially unimportant variables before distillation, similar to dCRT.
Revised: To speedup inference time, we introduce a variable-screening step that eliminates potentially unimportant variables before distillation, similar to dCRT.

Original: We provide empirical benchmark of the runtime of each method in Section 4.5.
Revised: We provide empirical benchmark of the runtime of each method in Section 4.5.

Original: Setting ℓ 1 -regularization parameter λ and λ dx
Revised: Setting ℓ 1 -regularization parameter λ and λ dx

Original: In general, we advise to use cross-validation for obtaining ˆ β λ in Eq.
Revised: In general, we advise to use cross-validation for obtaining ˆ β λ in Eq.

Original: (2) and for X ∗ ,j -distillation operator, as defined by Eq.
Revised: (2) and for X ∗ ,j -distillation operator, as defined by Eq.

Original: This is in line with the theoretical argument for dCRT [19, Lemma 1 and Theorem 3].
Revised: This is in line with the theoretical argument for dCRT [19, Lemma 1 and Theorem 3].

Original: However, we also observe empirically that choosing the ℓ 1 -regularization parameters of the distillation step can strongly affect how variables are selected by CRT-logit.
Revised: However, we also observe empirically that choosing the ℓ 1 -regularization parameters of the distillation step can strongly affect how variables are selected by CRT-logit.

Original: We provide more details in the Supplementary Material, and leave further theoretical investigations of this phenomenon for future work.
Revised: We provide more details in the Supplementary Material, and leave further theoretical investigations of this phenomenon for future work.

Original: Algorithm 1:
Revised: Algorithm 1:

Original: , X ∗ , − j ) //
Revised: , X ∗ , − j ) //

Original: Using Eq.
Revised: Using Eq.

Original: (10) ˆ β d y ,j ← (ˆ β 1 , ˆ β 2 , . . .
Revised: (10) ˆ β d y ,j ← (ˆ β 1 , ˆ β 2 , . . .

Original: , ˆ β j − 1 , ˆ β j +1 , . . .
Revised: , ˆ β j − 1 , ˆ β j +1 , . . .

Original: , ˆ β p )
Revised: , ˆ β p )

Original: T decorr j ← decorrelated_test_score ( j, X , y , ˆ β d X ∗ ,j , ˆ β d y ,j ) //
Revised: T decorr j ← decorrelated_test_score ( j, X , y , ˆ β d X ∗ ,j , ˆ β d y ,j ) //

Original: Using Eq (11) (cid:98) p j ← 2[1 − | Φ (cid:0) T decorr j (cid:1) | ] 10 end 11 for j / ∈ ˆ S SCREENING do (cid:98) p
Revised: Using Eq (11) (cid:98) p j ← 2[1 − | Φ (cid:0) T decorr j (cid:1) | ] 10 end 11 for j / ∈ ˆ S SCREENING do (cid:98) p

Original: j = 1 13 end
Revised: j = 1 13 end

Original: Asymptotic analysis of the Decorrelated Test Statistic We now provide theoretical analysis of
Revised: Asymptotic analysis of the Decorrelated Test Statistic We now provide theoretical analysis of

Original: CRT-logit in large-sample regime.
Revised: CRT-logit in large-sample regime.

Original: All the proofs can be found in the Supplementary Material.
Revised: All the proofs can be found in the Supplementary Material.

Original: First, we introduce the following assumption.
Revised: First, we introduce the following assumption.

Original: Assumption 3.1 (Regularity conditions) .
Revised: Assumption 3.1 (Regularity conditions) .

Original: Assume that
Revised: Assume that

Original: (A1) λ min ( I ) ≥ K for some constant K > 0 .
Revised: (A1) λ min ( I ) ≥ K for some constant K > 0 .

Original: (A2) Sparsity of β 0 and w 0 ,j , with w 0 ,j the ground truth weights for the distillation of x j in Eq.
Revised: (A2) Sparsity of β 0 and w 0 ,j , with w 0 ,j the ground truth weights for the distillation of x j in Eq.

Original: (10) : |S| = s ∗ and ∥ w 0 ,j ∥ 

Revised: (10) : |S| = s ∗ and ∥ w 0 ,j ∥

Original: = s ′ with s ∗ =
Revised: = s ′ with s ∗ =

Original: o (cid:0) n 1 / 2 / log( p ) (cid:1) and s ′
Revised: o (cid:0) n 1 / 2 / log( p ) (cid:1) and s ′

Original: =
Revised: =

Original: o (cid:0) n 1 / 2 / log( p ) (cid:1) .
Revised: o (cid:0) n 1 / 2 / log( p ) (cid:1) .

Original: (A3)
Revised: (A3)

Original: For all i ∈ [ n ] , X i, ∗ and ( − y i + g ′
Revised: For all i ∈ [ n ] , X i, ∗ and ( − y i + g ′

Original: ( X i, ∗ β )) are sub-exponential random variables, and | X i, − j w 0 ,j | ≤ K ′ almost surely, for some constant K ′ > 0 .
Revised: ( X i, ∗ β )) are sub-exponential random variables, and | X i, − j w 0 ,j | ≤ K ′ almost surely, for some constant K ′ > 0 .

Original: We then have the following result, that states that the asymptotic distribution of the decorrelated test scores is standard normal.
Revised: We then have the following result, that states that the asymptotic distribution of the decorrelated test scores is standard normal.

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: Let j ∈ [ p ] , and let T decorr j be defined as in Eq.
Revised: Let j ∈ [ p ] , and let T decorr j be defined as in Eq.

Original: (11) , with λ ≍ λ dx ≍ (cid:112) n − 1 log( p ) .
Revised: (11) , with λ ≍ λ dx ≍ (cid:112) n − 1 log( p ) .

Original: Then, if Assumption 3.1 holds true, and if we consider p = p ( n ) , under the null hypothesis H j 0 : β 0 j = 0 , we have
Revised: Then, if Assumption 3.1 holds true, and if we consider p = p ( n ) , under the null hypothesis H j 0 : β 0 j = 0 , we have

Original: [Equation]
Revised: [Equation]

Original: where Φ( · ) is the CDF of the standard Gaussian distribution.
Revised: where Φ( · ) is the CDF of the standard Gaussian distribution.

Original: Moreover, for each j ∈ [ p ] , if we define (cid:98) p j def.
Revised: Moreover, for each j ∈ [ p ] , if we define (cid:98) p j def.

Original: = 2 [1 − Φ ( T j )] , i.e. (cid:98) p
Revised: = 2 [1 − Φ ( T j )] , i.e. (cid:98) p

Original: j is the output of Algorithm 1, then, under the null hypothesis
Revised: j is the output of Algorithm 1, then, under the null hypothesis

Original: H j 0 : β 0 j = 0 , we have
Revised: H j 0 : β 0 j = 0 , we have

Original: [Equation]
Revised: [Equation]

Original: ∈
Revised: ∈

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: that is, the p-values output by Algorithm 1 are valid asymptotically.
Revised: that is, the p-values output by Algorithm 1 are valid asymptotically.

Original: Remark 3.1.
Revised: Remark 3.1.

Original: We also show in the proof of Theorem 3.1 (in Suppementary Material) that the rate of convergence is O (1 / √ n ) .
Revised: We also show in the proof of Theorem 3.1 (in Suppementary Material) that the rate of convergence is O (1 / √ n ) .

Original: Compared with some of the related works ( e.g. , knockoffs) that come with finite sample guarantees, our theoretical analysis only works in the asymprotic regime.
Revised: Compared with some of the related works ( e.g. , knockoffs) that come with finite sample guarantees, our theoretical analysis only works in the asymprotic regime.

Original: We leave the finite sample analysis as one of the directions for future works.
Revised: We leave the finite sample analysis as one of the directions for future works.

Original: FDR control with CRT-logit As a consequence of Theorem 3.1, we have the following result, which establishes that the FDR of the test is controlled when using the Benjamini-Yekutieli procedure [6] with the p-values output from Algorithm 1, assuming that the number of tests p is fixed.
Revised: FDR control with CRT-logit As a consequence of Theorem 3.1, we have the following result, which establishes that the FDR of the test is controlled when using the Benjamini-Yekutieli procedure [6] with the p-values output from Algorithm 1, assuming that the number of tests p is fixed.

Original: Corollary 3.1.
Revised: Corollary 3.1.

Original: Under Assumptions 3.1 and logistic model defined in Eq (1) , with λ ≍ λ dx ≍ (cid:112) n − 1 log( p ) , assume n − 1 / 2 ( s ′ ∨ s ∗ ) log( p ) =
Revised: Under Assumptions 3.1 and logistic model defined in Eq (1) , with λ ≍ λ dx ≍ (cid:112) n − 1 log( p ) , assume n − 1 / 2 ( s ′ ∨ s ∗ ) log( p ) =

Original: o (1) , and assume the number of tests p is fixed.
Revised: o (1) , and assume the number of tests p is fixed.

Original: Let α ∈ (0 , 1) and (cid:98) S BY-CRT be given by applying following the Benjamini-Yekutieli FDR-controlling procedure to the CRT-logit p-values { (cid:98) p j } j ∈ [ p ] , output from Algo.1.
Revised: Let α ∈ (0 , 1) and (cid:98) S BY-CRT be given by applying following the Benjamini-Yekutieli FDR-controlling procedure to the CRT-logit p-values { (cid:98) p j } j ∈ [ p ] , output from Algo.1.

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: Remark 3.2.
Revised: Remark 3.2.

Original: Assumption 3.1 is also assumed in [21, 28], which also provide a detailed discussion of this regularity assumption in generalized linear models.
Revised: Assumption 3.1 is also assumed in [21, 28], which also provide a detailed discussion of this regularity assumption in generalized linear models.

Original: This assumption, in turn, is built on the regularity assumption in the classic work [13, Chapter 9] to establish asymptotic normality of Rao’s test statistic.
Revised: This assumption, in turn, is built on the regularity assumption in the classic work [13, Chapter 9] to establish asymptotic normality of Rao’s test statistic.

Original: Theorem 3.1 is an adaptation of [21, Theorem 3.1], specialized for the case of sparse logistic regression and the p-values output from CRT-logit.
Revised: Theorem 3.1 is an adaptation of [21, Theorem 3.1], specialized for the case of sparse logistic regression and the p-values output from CRT-logit.

Original:  Empirical Results
Revised:  Empirical Results

Original: We provide benchmarks of the proposed CRT-logit algorithm along with most other methods mentioned in the introduction, in particular: model-X Knockoff (KO) [10], Debiased Lasso (dLasso) [33, 16], original CRT with 1000 samplings [10], Holdout Randomization Test with 5000 samplings [26], and Lasso-Distillation CRT (dCRT)
Revised: We provide benchmarks of the proposed CRT-logit algorithm along with most other methods mentioned in the introduction, in particular: model-X Knockoff (KO) [10], Debiased Lasso (dLasso) [33, 16], original CRT with 1000 samplings [10], Holdout Randomization Test with 5000 samplings [26], and Lasso-Distillation CRT (dCRT)

Original: We did not include SLOE and CPT as the provided open-source implementation are particularly unstable and do not fit in the sparse-regression setting (for SLOE), or implementation is not available (for CPT).
Revised: We did not include SLOE and CPT as the provided open-source implementation are particularly unstable and do not fit in the sparse-regression setting (for SLOE), or implementation is not available (for CPT).

Original: For the lack of space, we leave the extra experiment with a genome-wide association study in the Supplementary Material.
Revised: For the lack of space, we leave the extra experiment with a genome-wide association study in the Supplementary Material.

Original: Remark 4.1.
Revised: Remark 4.1.

Original: As a slight caveat, in the simulated and semi-realistic experiment sections (Sections 4.1, 4.2 and 4.3), we introduce an additional noise term to the logistic relationship of Eq.
Revised: As a slight caveat, in the simulated and semi-realistic experiment sections (Sections 4.1, 4.2 and 4.3), we introduce an additional noise term to the logistic relationship of Eq.

Original: [Equation]
Revised: [Equation]

Original: where ξ i ∼ N (0 , 1) is a Gaussian noise and σ > 0 the noise magnitude.
Revised: where ξ i ∼ N (0 , 1) is a Gaussian noise and σ > 0 the noise magnitude.

Original: The formula in Eq.
Revised: The formula in Eq.

Original: (12) has been used in previous works, e.g. [8].
Revised: (12) has been used in previous works, e.g. [8].

Original: There is a clear justification to this: in most of the applications we consider, data are collected with measurement errors.
Revised: There is a clear justification to this: in most of the applications we consider, data are collected with measurement errors.

Original: In the case of brain-imaging, for example, recording the brain signal of the human subjects by scanners often includes noise caused either from the machine, or from the movement of the subjects [18].
Revised: In the case of brain-imaging, for example, recording the brain signal of the human subjects by scanners often includes noise caused either from the machine, or from the movement of the subjects [18].

Original: Moreover, in general, this setting corresponds to a model mis-specification, which the CRT-logit is robust to under Assumption 3.1, following the same argument as in [21, Section 5].
Revised: Moreover, in general, this setting corresponds to a model mis-specification, which the CRT-logit is robust to under Assumption 3.1, following the same argument as in [21, Section 5].

Original: Remark 4.2.
Revised: Remark 4.2.

Original: We use Benjamini-Hochberg step-up procedure [5] to control FDR with the p-values in all the empirical experiments in Section 4.2 and App. 4.4, as we observe that the FDR is empirically controlled with this procedure, without compromising power with the conservative BY bound.
Revised: We use Benjamini-Hochberg step-up procedure [5] to control FDR with the p-values in all the empirical experiments in Section 4.2 and App. 4.4, as we observe that the FDR is empirically controlled with this procedure, without compromising power with the conservative BY bound.

Original: 4.1 Effectiveness of the decorrelation step
Revised: 4.1 Effectiveness of the decorrelation step

Original: To show how decorrelating the test statistics helps, we set up a simulation with matrix X of p =features and vary the number of samples n ∈ { 200 , 400 , 800 , 4000 } .
Revised: To show how decorrelating the test statistics helps, we set up a simulation with matrix X of p =features and vary the number of samples n ∈ { 200 , 400 , 800 , 4000 } .

Original: The binary response vector y is created following Eq.
Revised: The binary response vector y is created following Eq.

Original: (12), and the design matrix X is sampled from a multivariate normal distribution with zero mean, while the covariance matrix Σ ∈ R p × p is a symmetric Toeplitz matrix, where the parameter ρ ∈ (0 , 1) controls the correlation between neighboring features: correlation decreases quickly when the distance between feature indices increases.
Revised: (12), and the design matrix X is sampled from a multivariate normal distribution with zero mean, while the covariance matrix Σ ∈ R p × p is a symmetric Toeplitz matrix, where the parameter ρ ∈ (0 , 1) controls the correlation between neighboring features: correlation decreases quickly when the distance between feature indices increases.

Original: The true signal β 0 is picked with a sparsity parameter κ = s ∗ /p that controls the proportion of non-zero elements with magnitude 2.0, i.e. β j = 2 . 

Revised: The true signal β 0 is picked with a sparsity parameter κ = s ∗ /p that controls the proportion of non-zero elements with magnitude 2.0, i.e. β j = 2 .

Original: for all j ∈ S .
Revised: for all j ∈ S .

Original: For the specific purpose of this experiment, non-zero indices of S are kept fixed.
Revised: For the specific purpose of this experiment, non-zero indices of S are kept fixed.

Original: The noise ξ is i.i.d.
Revised: The noise ξ is i.i.d.

Original: normal N (0 , Id n ) with magnitude σ = ∥
Revised: normal N (0 , Id n ) with magnitude σ = ∥

Original: X β 0 ∥ 2 / ( √ n SNR ) , controlled by the SNR parameter.
Revised: X β 0 ∥ 2 / ( √ n SNR ) , controlled by the SNR parameter.

Original: In short, the three main parameters controlling this simulation are correlation ρ , sparsity degree κ and signal-to-noise ratio SNR.
Revised: In short, the three main parameters controlling this simulation are correlation ρ , sparsity degree κ and signal-to-noise ratio SNR.

Original: We generate randomlydatasets, and run dCRT and CRT-logit algorithm to obtain one sample of test statistics { T j } pj =1 and { T decorr j } pj =1 .
Revised: We generate randomlydatasets, and run dCRT and CRT-logit algorithm to obtain one sample of test statistics { T j } pj =1 and { T decorr j } pj =1 .

Original: Then, we pick 1000 samples of one null test statistic T j and T decorr j , defined in Eq.
Revised: Then, we pick 1000 samples of one null test statistic T j and T decorr j , defined in Eq.

Original: (5) and (11), respectively, and plot the qq-plot of their empirical quantile versus the standard normal quantile.
Revised: (5) and (11), respectively, and plot the qq-plot of their empirical quantile versus the standard normal quantile.

Original: From the results in Figure 6, we observe that the empirical null distribution of the test
Revised: From the results in Figure 6, we observe that the empirical null distribution of the test

Original: [Figure]
Revised: [Figure]

Original: e
Revised: e

Original: Q
Revised: Q

Original: [Figure]
Revised: [Figure]

Original: t i l e
Revised: t i l e

Original: [Figure]
Revised: [Figure]

Original: n
Revised: n

Original: [Figure]
Revised: [Figure]

Original: e
Revised: e

Original: [Figure]
Revised: [Figure]

Original: u
Revised: u

Original: [Figure]
Revised: [Figure]

Original: e s
Revised: e s

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: S a m p l e
Revised: S a m p l e

Original: Q u a n t i l e s
Revised: Q u a n t i l e s

Original: CRT
Revised: CRT

Original: [Figure]
Revised: [Figure]

Original: Logit (b) n =
Revised: Logit (b) n =

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: S a m p l e
Revised: S a m p l e

Original: Q u a n t i l e s dCRT
Revised: Q u a n t i l e s dCRT

Original: [Figure]
Revised: [Figure]

Original: p l e
Revised: p l e

Original: Q u a n t i l e s
Revised: Q u a n t i l e s

Original: CRT
Revised: CRT

Original: [Figure]
Revised: [Figure]

Original: Logit (c) n =
Revised: Logit (c) n =

Original: [Figure]
Revised: [Figure]

Original: e
Revised: e

Original: [Figure]
Revised: [Figure]

Original: t i l e s dCRT
Revised: t i l e s dCRT

Original: [Figure]
Revised: [Figure]

Original: a m p l e
Revised: a m p l e

Original: Q u a n t i l e s
Revised: Q u a n t i l e s

Original: CRT
Revised: CRT

Original: [Figure]
Revised: [Figure]

Original: statistic is much closer to a standard normal when adding the decorrelation step.
Revised: statistic is much closer to a standard normal when adding the decorrelation step.

Original: In particular, when the sample size n increases to 400, the decorrelated test statistic has empirical quantiles almost inline with the theoretical quantiles of the standard normal distribution, while dCRT test score strays away from the 45-degree line.
Revised: In particular, when the sample size n increases to 400, the decorrelated test statistic has empirical quantiles almost inline with the theoretical quantiles of the standard normal distribution, while dCRT test score strays away from the 45-degree line.

Original: Again, we emphasize that the normality of T j is crucial for the p-values calculation.
Revised: Again, we emphasize that the normality of T j is crucial for the p-values calculation.

Original: This outlines the importance of the decorrelating step on T j .
Revised: This outlines the importance of the decorrelating step on T j .

Original: 4.2 High-dimensional scenario with varying simulation parameters
Revised: 4.2 High-dimensional scenario with varying simulation parameters

Original: To see how each algorithm performs under different settings, we follow the same simulation scenario as in Sec. 4.1, but vary each of the three simulation parameters, while keeping the others unchanged at a default value of SNR = 2 . 0 , ρ = 0 .
Revised: To see how each algorithm performs under different settings, we follow the same simulation scenario as in Sec. 4.1, but vary each of the three simulation parameters, while keeping the others unchanged at a default value of SNR = 2 . 0 , ρ = 0 .

Original: 5 , κ = 0 .
Revised: 5 , κ = 0 .

Original: We target a control of FDR at level 0 .
Revised: We target a control of FDR at level 0 .

Original: 1 , using the Benjamini-Hochberg procedure.
Revised: 1 , using the Benjamini-Hochberg procedure.

Original: The results in Figure 3 show that CRT-logit is the most powerful method while still controlling the FDR.
Revised: The results in Figure 3 show that CRT-logit is the most powerful method while still controlling the FDR.

Original: Moreover, in the presence of higher correlations between nearby variables ( ρ > 0 . 6 ), other methods suffer a drop in average power, but this is not as severe for CRT-logit.
Revised: Moreover, in the presence of higher correlations between nearby variables ( ρ > 0 . 6 ), other methods suffer a drop in average power, but this is not as severe for CRT-logit.

Original: The original CRT, in general, is conservative.
Revised: The original CRT, in general, is conservative.

Original: We believe that this is due to using only B = 500 samplings to generate empirical p-values for the two methods, due to prohibitive average runtime of the algorithm with larger B (which we provide in Section 4.5).
Revised: We believe that this is due to using only B = 500 samplings to generate empirical p-values for the two methods, due to prohibitive average runtime of the algorithm with larger B (which we provide in Section 4.5).

Original: For HRT, the conservativeness is expected, due to the usage of only half of the sample for test-statistics calculation – even though the number of samplings is bigger than original CRT ( B = 5000 ).
Revised: For HRT, the conservativeness is expected, due to the usage of only half of the sample for test-statistics calculation – even though the number of samplings is bigger than original CRT ( B = 5000 ).

Original: We note that, perhaps surprisingly, the debiased lasso ( cdlasso ) is the most conservative.
Revised: We note that, perhaps surprisingly, the debiased lasso ( cdlasso ) is the most conservative.

Original: It controls FDR well in all settings.
Revised: It controls FDR well in all settings.

Original: This might be due to the fact that dlasso also relies on the choice of the ℓ 1 -regularization λ in the nodewise Lasso operation, similar to the X ∗ ,j -distillation of dCRT, as noted in Section 1.
Revised: This might be due to the fact that dlasso also relies on the choice of the ℓ 1 -regularization λ in the nodewise Lasso operation, similar to the X ∗ ,j -distillation of dCRT, as noted in Section 1.

Original: What makes the difference is that instead of using cross-validation for setting λ for each variable j , a fixed value of λ = 

Revised: What makes the difference is that instead of using cross-validation for setting λ for each variable j , a fixed value of λ =

Original: − 2 λ max is used in the implementation of dlasso .
Revised: − 2 λ max is used in the implementation of dlasso .

Original: We strongly suspect this fixed value is not optimal, which makes the procedure powerless.
Revised: We strongly suspect this fixed value is not optimal, which makes the procedure powerless.

Original: 4.3 Application: large-scale analysis on brain-imaging dataset
Revised: 4.3 Application: large-scale analysis on brain-imaging dataset

Original: [Figure]
Revised: [Figure]

Original: Description The Human Connectome Project
Revised: Description The Human Connectome Project

Original: dataset (HCP) is a collection of brain imaging data on healthy young adult subjects with age ranging from 22 to 35.
Revised: dataset (HCP) is a collection of brain imaging data on healthy young adult subjects with age ranging from 22 to 35.

Original: More specifically, the input X is a set of 2mm statistical maps of 400 subjects across 8 cognitive tasks.
Revised: More specifically, the input X is a set of 2mm statistical maps of 400 subjects across 8 cognitive tasks.

Original: These are called z-maps, as the data follow a standard normal distribution under the null hypothesis.
Revised: These are called z-maps, as the data follow a standard normal distribution under the null hypothesis.

Original: Each task in turn features 2 different contrasts, which effectively form binary responses y ∈ { 0 , 1 } n .
Revised: Each task in turn features 2 different contrasts, which effectively form binary responses y ∈ { 0 , 1 } n .

Original: In short, the goal of this fMRI data analysis is to identify voxels with task-related levels of activity by fitting y through distributed brain signals.
Revised: In short, the goal of this fMRI data analysis is to identify voxels with task-related levels of activity by fitting y through distributed brain signals.

Original: The setting is high-dimensional with n =samples, corresponding to 400 subjects, while the total number of variables is p ≈ 200 ,brain voxels.
Revised: The setting is high-dimensional with n =samples, corresponding to 400 subjects, while the total number of variables is p ≈ 200 ,brain voxels.

Original: Following [11, 20], we use a hierarchical clustering scheme to group the variables into C = 1000 spatially connected clusters.
Revised: Following [11, 20], we use a hierarchical clustering scheme to group the variables into C = 1000 spatially connected clusters.

Original: We provide details of the pre-processing step in Supplementary Material.
Revised: We provide details of the pre-processing step in Supplementary Material.

Original: Creating semi-realistic ground-truth and response labels Since there is no ground truth for this dataset, we create synthetic true signals by fitting the data X and response y with an ℓ 1penalized logistic classifier.
Revised: Creating semi-realistic ground-truth and response labels Since there is no ground truth for this dataset, we create synthetic true signals by fitting the data X and response y with an ℓ 1penalized logistic classifier.

Original: In other words, the estimator ˆ β logreg will serve as true regression coefficients for each task.
Revised: In other words, the estimator ˆ β logreg will serve as true regression coefficients for each task.

Original: Then, to avoid bias in simulating label ˆ y , the z-maps matrix X of one task are used in conjunction with the discriminative pattern map ˆ β logreg from the next task in the following order: relational , gambling , emotion , social .
Revised: Then, to avoid bias in simulating label ˆ y , the z-maps matrix X of one task are used in conjunction with the discriminative pattern map ˆ β logreg from the next task in the following order: relational , gambling , emotion , social .

Original: For instance, we use ˆ β logreg of gambling with z-maps data matrix of relational , i.e. for all i = 1 , . . . , n , given x i, relational ,
Revised: For instance, we use ˆ β logreg of gambling with z-maps data matrix of relational , i.e. for all i = 1 , . . . , n , given x i, relational ,

Original: [Equation]
Revised: [Equation]

Original: (13) where Bern ( a ) is a Bernoulli probability mass function that takes a value 1 with probability a , σ is a noise magnitude and ξ i is a standard normal noise.
Revised: (13) where Bern ( a ) is a Bernoulli probability mass function that takes a value 1 with probability a , σ is a noise magnitude and ξ i is a standard normal noise.

Original: Finally, we apply all inference algorithms on the semi-synthetic data ( X , ˆ y ) , and we evaluate their performance using the ground-truth ˆ β logreg .
Revised: Finally, we apply all inference algorithms on the semi-synthetic data ( X , ˆ y ) , and we evaluate their performance using the ground-truth ˆ β logreg .

Original: This simulation setting is similar to [11], except that here we consider a classification and not a regression problem.
Revised: This simulation setting is similar to [11], except that here we consider a classification and not a regression problem.

Original: It allows us to calculate the False Discovery Rate and average power with multiple runs of the inference procedure (across tasks).
Revised: It allows us to calculate the False Discovery Rate and average power with multiple runs of the inference procedure (across tasks).

Original: Remark 4.3.
Revised: Remark 4.3.

Original: The i.i.d.
Revised: The i.i.d.

Original: assumption is formally violated in this experiment, where for each subject we analyze two sample images that are not independent.
Revised: assumption is formally violated in this experiment, where for each subject we analyze two sample images that are not independent.

Original: Yet, this remains a short-range correlation structure, and is thus not a strong challenge to the i.i
Revised: Yet, this remains a short-range correlation structure, and is thus not a strong challenge to the i.i

Original: [Figure]
Revised: [Figure]

Original: d. assumption.
Revised: d. assumption.

Original: Results
Revised: Results

Original: The results in Figure 4 show that CRT-logit achieves a better recovery compared to KO or original CRT/dCRT/HRT, which results in higher statistical power.
Revised: The results in Figure 4 show that CRT-logit achieves a better recovery compared to KO or original CRT/dCRT/HRT, which results in higher statistical power.

Original: This gain comes with a good control of the FDR under desired level α = 0 .
Revised: This gain comes with a good control of the FDR under desired level α = 0 .

Original: On a related note, the only analysis where dCRT makes more discoveries than CRT-Logit is in the emotion task, but at the cost of failing to control
Revised: On a related note, the only analysis where dCRT makes more discoveries than CRT-Logit is in the emotion task, but at the cost of failing to control

Original: FDR at the nominal level.
Revised: FDR at the nominal level.

Original: [Figure]
Revised: [Figure]

Original: 4.4 Application: genome-wide association study with Human Brain Cancer Dataset
Revised: 4.4 Application: genome-wide association study with Human Brain Cancer Dataset

Original: Description The last in our benchmark is a Genome-wide Association Study (GWAS) on the The
Revised: Description The last in our benchmark is a Genome-wide Association Study (GWAS) on the The

Original: Cancer Genome Atlas (TCGA) dataset [30, 31].
Revised: Cancer Genome Atlas (TCGA) dataset [30, 31].

Original: We choose to analyze the Glioma cohort, which consists of n = 1026 patients across a wide age range, diagnosed with this type of brain tumor, with a total of p = 24776 genes in the data matrix recorded as copy number variations (CNVs) at the gene level in log ratio format.
Revised: We choose to analyze the Glioma cohort, which consists of n = 1026 patients across a wide age range, diagnosed with this type of brain tumor, with a total of p = 24776 genes in the data matrix recorded as copy number variations (CNVs) at the gene level in log ratio format.

Original: As with the brain-imaging inference in Section 4.3, we use clustering to reduce the dimension to C = 1000 clusters.
Revised: As with the brain-imaging inference in Section 4.3, we use clustering to reduce the dimension to C = 1000 clusters.

Original: However, we use different criterion to merge variables (genes) to clusters of variables, which is the pairwise Linkage Disequilibrium, following [1, Section 4] (with available R library).
Revised: However, we use different criterion to merge variables (genes) to clusters of variables, which is the pairwise Linkage Disequilibrium, following [1, Section 4] (with available R library).

Original: For the response, a long-term survivor (LTS) is defined as a patient who survived more than five years after diagnosis and would be labeled y = 0 , and otherwise would be a short-term survivor (STS), labeled y = 1 .
Revised: For the response, a long-term survivor (LTS) is defined as a patient who survived more than five years after diagnosis and would be labeled y = 0 , and otherwise would be a short-term survivor (STS), labeled y = 1 .

Original: The objective is to identify significant genes that contribute to classification of the LTS/STS status.
Revised: The objective is to identify significant genes that contribute to classification of the LTS/STS status.

Original: Similar to the Human Connectome Project dataset, there is no real ground-truth for the TCGA Glioma.
Revised: Similar to the Human Connectome Project dataset, there is no real ground-truth for the TCGA Glioma.

Original: However, we have the list of mutations and the frequency of those detected in the diagnosed patients.
Revised: However, we have the list of mutations and the frequency of those detected in the diagnosed patients.

Original: We therefore select the 1000 most frequent gene mutations that appeared in this list, i
Revised: We therefore select the 1000 most frequent gene mutations that appeared in this list, i

Original: [Table]
Revised: [Table]

Original: e. the ground truth list consists of 1000 genes (variables)
Revised: e. the ground truth list consists of 1000 genes (variables)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Result
Revised: Result

Original: The result from Table 1 shows that CRT-logit finds the largest number of genes.
Revised: The result from Table 1 shows that CRT-logit finds the largest number of genes.

Original: Moreover, most of selected genes in this table are detected in the list of mutated genes found on recorded patients.
Revised: Moreover, most of selected genes in this table are detected in the list of mutated genes found on recorded patients.

Original: Some genes are detected by all the benchmarked methods, most prominently SPEN , which is found on over 10 % of patients in the cohort.
Revised: Some genes are detected by all the benchmarked methods, most prominently SPEN , which is found on over 10 % of patients in the cohort.

Original: Furthermore, this gene is known to be associated not only with brain cancer, but also with other types of cancer in The Human Protein Atlas project [17].
Revised: Furthermore, this gene is known to be associated not only with brain cancer, but also with other types of cancer in The Human Protein Atlas project [17].

Original: Note that, in the absence of a ground-truth, this does not guarantee all genes found are associated with glioma
Revised: Note that, in the absence of a ground-truth, this does not guarantee all genes found are associated with glioma

Original: [Table]
Revised: [Table]

Original: but this experiment demonstrates the capability of CRT-logit in GWAS studies.
Revised: but this experiment demonstrates the capability of CRT-logit in GWAS studies.

Original: 4.5 Average runtime of benchmarked methods
Revised: 4.5 Average runtime of benchmarked methods

Original: [Table]
Revised: [Table]

Original: Besides statistical performance
Revised: Besides statistical performance

Original: [Table]
Revised: [Table]

Original: it is equally important to assess the computational cost of inference procedures.
Revised: it is equally important to assess the computational cost of inference procedures.

Original: The average runtime in Table 2 from the two experiments shows that the original CRT is not suitable for large-scale inference: it is over 2000 times slower than the fastest method (Knockoff
Revised: The average runtime in Table 2 from the two experiments shows that the original CRT is not suitable for large-scale inference: it is over 2000 times slower than the fastest method (Knockoff

Original: Filter), and over 150 times slower than dCRT/CRT-logit.
Revised: Filter), and over 150 times slower than dCRT/CRT-logit.

Original: The empirical runtime also confirms the effectiveness of the screening step before doing distillation/decorrelation of the test-statistics: the step makes CRT-logit and dCRT 20 times faster than without.
Revised: The empirical runtime also confirms the effectiveness of the screening step before doing distillation/decorrelation of the test-statistics: the step makes CRT-logit and dCRT 20 times faster than without.

Original: On a related note, although in theory
Revised: On a related note, although in theory

Original: Debiased Lasso, dCRT and CRT-logit (both without screening) share the same runtime complexity, the latter two are slower due to the use of cross-validation to estimate the sparsity hyperparameter λ and λ dx (detailed in Section 3).
Revised: Debiased Lasso, dCRT and CRT-logit (both without screening) share the same runtime complexity, the latter two are slower due to the use of cross-validation to estimate the sparsity hyperparameter λ and λ dx (detailed in Section 3).

Original:  Discussion
Revised:  Discussion

Original: We proposed an adaptation of the Conditional Randomization Test (CRT) for sparse logistic regression in the high-dimensional regime.
Revised: We proposed an adaptation of the Conditional Randomization Test (CRT) for sparse logistic regression in the high-dimensional regime.

Original: A major improvement of CRT-logit, our proposed algorithm, compared to original CRT, comes from the decorrelation of test statistics to make their distribution closer to standard normal.
Revised: A major improvement of CRT-logit, our proposed algorithm, compared to original CRT, comes from the decorrelation of test statistics to make their distribution closer to standard normal.

Original: Indeed, results from synthetic experiments in Figure 6 show that in highdimension (when 0 . 5 ≤ n/p ≤ 1 . 0 ), the empirical null distribution of CRT-logit’s test statistic T decorr is much more similar to a standard normal compared to the original CRT test statistic.
Revised: Indeed, results from synthetic experiments in Figure 6 show that in highdimension (when 0 . 5 ≤ n/p ≤ 1 . 0 ), the empirical null distribution of CRT-logit’s test statistic T decorr is much more similar to a standard normal compared to the original CRT test statistic.

Original: Moreover, empirical benchmarks in Section 4 demonstrate that CRT-logit performs better than related statistical inference methods, such as the Debiased Lasso or Model-X Knockoffs.
Revised: Moreover, empirical benchmarks in Section 4 demonstrate that CRT-logit performs better than related statistical inference methods, such as the Debiased Lasso or Model-X Knockoffs.

Original: In particular, CRT-logit is the most powerful method in our synthetic experiment with high-dimensional datasets in Section 4.2, while still keeping FDR controlled under predefined level α = 0 .
Revised: In particular, CRT-logit is the most powerful method in our synthetic experiment with high-dimensional datasets in Section 4.2, while still keeping FDR controlled under predefined level α = 0 .

Original: 1 (with a slight caveat of using BH instead of BY procedure, as elaborated in Remark 4.2).
Revised: 1 (with a slight caveat of using BH instead of BY procedure, as elaborated in Remark 4.2).

Original: We note that there exist some limitations to
Revised: We note that there exist some limitations to

Original: CRT-logit.
Revised: CRT-logit.

Original: The computational cost of CRT-logit, while lower than vanilla CRT, is still larger than alternative methods such as Knockoff Filter and Holdout Randomization Test.
Revised: The computational cost of CRT-logit, while lower than vanilla CRT, is still larger than alternative methods such as Knockoff Filter and Holdout Randomization Test.

Original: Moreover, tuning the ℓ 1 − regularization λ dx parameter by cross-validation, as is often done, can further increase the computational cost of CRT-logit (and dCRT).
Revised: Moreover, tuning the ℓ 1 − regularization λ dx parameter by cross-validation, as is often done, can further increase the computational cost of CRT-logit (and dCRT).

Original: Despite this, our empirical benchmarks on both simulated and real data show real promises of CRT-logit.
Revised: Despite this, our empirical benchmarks on both simulated and real data show real promises of CRT-logit.

Original: Henceforth, we believe CRT-logit is competitive for practical settings that involve structured data, such as brain-imaging and genomics applications.
Revised: Henceforth, we believe CRT-logit is competitive for practical settings that involve structured data, such as brain-imaging and genomics applications.

Original: Acknowledgements
Revised: Acknowledgements

Original: BN, BT and SA acknowledged the support of the French "Agence Nationale de la Recherche" under the project ANR-17-CE23-0011 (FastBig) and ANR-20-CHIA-0025-01 (KARAIB AI chair).
Revised: BN, BT and SA acknowledged the support of the French "Agence Nationale de la Recherche" under the project ANR-17-CE23-0011 (FastBig) and ANR-20-CHIA-0025-01 (KARAIB AI chair).

Original: BN was also supported by Research Chair DSAIDIS (Data Science and Artificial Intelligence for Digitalized
Revised: BN was also supported by Research Chair DSAIDIS (Data Science and Artificial Intelligence for Digitalized

Original: Indutry and Services) of Télecom Paris.
Revised: Indutry and Services) of Télecom Paris.

Original: A Proofs of theoretical results in Section

Original: We first present some technical lemmas that are useful for the proof of the main theorem.

Original: From now on, let ≾ and ≿ denote inequalities with a hidden constant factor, i.e. x ≾ y means that with high probability, there exists an absolute constant C > 0 such that x ≤ Cy , and vice versa.

Original: As mentioned in the main text, in what follows, without writing it explicitly, we consider p = p ( n ) .

Original: Lemma A.1 (Lemma E.1, [21]) .

Original: Assume Assumption 3.1, under the logistic model, we have

Original: [Equation]

Original: where s ∗ =

Original: ∥ β 0 ∥ 0 .

Original: In addition, we also have

Original: [Equation]

Original: where g ( x ) = 1 / (1 + exp( x )) is the sigmoid function.

Original: Lemma A.2 (Lemma E.2 [21], concentration of the gradient and Hessian of the logistic loss function) .

Original: Assume Assumption 3.1 holds, under the logistic model, we have, with v ∗ def.

Original: = (1 , − w 0 ,j ) ∈ R p ,

Original: [Equation]

Original: log p, and

Original: [Equation]

Original: Lemma A.3 (Lemma E.3, [21]) .

Original: Assume Assumption 3.1 holds, under logistic model, we have

Original: [Equation]

Original: where s ∗ =

Original: ∥ β 0 ∥ 0 and s ′ =

Original: ∥ w 0 ,j ∥ 0 .

Original: In addition, we also have

Original: [Equation]

Original: Lemma A.4 (Lemma E.4, [21], local smoothness conditions on the loss function) .

Original: Let ˆ β null =

Original: (0 , ˆ β − j ) ∈

Original: R p , where ˆ β is an estimator of β 0 .

Original: It holds that

Original: [Equation]

Original: for both β = ˆ β null and β = ˆ β , where ˆ v def.

Original: = (1 , ( ˆ β d X ∗ ,j ) ⊤ ) .

Original: Remark A.1.

Original: We make a slight abuse of notation in the definition of ˆ v and v ∗ , which formally corresponds to the j = 1 case.

Original: We use the fact that permuting the corresponding variable indices position of ˆ v , v ∗ , and ∇ ℓ ( β ) simultaneously does not change the value of v T ∇ ℓ ( β ) , and hence will not change the proofs.

Original: Proof of Theorem 3.1.

Original: The following proof is an adaptation from [21].

Original: Notice that our version of the proof is shorter, with specific consideration on sparse logistic regression, and with elaboration on the convergence rate of the decorreleated test score, which is missing from [21].

Original: Denote ˆ v def.

Original: = (1 , − ( ˆ β d X ∗ ,j ) ⊤ ) , then the decorrelated test score can be written in a more general from as

Original: T

Original: [Equation]

Original: Moreover, denote ˆ β null def.

Original: = (0 , ˆ β − j ) and v ∗ def.

Original: = (1 , − w 0 ,j ) , then we have, under the null hypothesis,

Original: [Equation]

Original: where we use the triangle inequality in the last step.

Original: From Lemma A.4, we have

Original: [Equation]

Original: n where the second inequality is by Holdër inequality, and the last inequality is due to Lemma A. and A.2.

Original: Similarly, we can bound A 2 , by using Lemma A.3 and Lemma A.

Original: [Equation]

Original: n

Original: This implies that,

Original: [Equation]

Original: The remaining part of the proof is to bound ˆ I j |− j − I j |− j , where, by definition

Original: [Equation]

Original: Evaluating the difference between ˆ I j |− j and I j |− j gives ˆ I j |− j - I j |− j =n n (cid:88)

Original: [Equation]

Original: ≤

Original: [Equation]

Original: where the last step follows from triangle inequality

Original: [Equation]

Original: We have, by Cauchy-Schwartz inequality

Original: [Equation]

Original: by Lemma A.

Original: [Equation]

Original: and by the fact that g ′′ ( x ) ∈ (0 , 1) for every x ∈ R ; and X i, − j , X i,j is sub-exponential by Assumption 3.1:

Original: [Equation]

Original: Similarly, to bound B 2 , we have, again by Cauchy-Schwartz inequality,

Original: [Equation]

Original: where the second inequality comes from using the self-concordance property of the sigmoid function (discussed at length in [2] and extended further in [22]), that is, | g ′′ ( t 1 ) − g ′′ ( t ) | ≤ | t 1 − t | g ′′ ( t ) for a fixed constant t , and for every t 1 ∈ R such that t 1 converges to t , with t 1 = ˆ β , and t = β 0 .

Original: By

Original: Assumption 3.1-A3 that X

Original: i,j is sub-exponential, applying Bernstein inequality leads to

Original: [Equation]

Original: To bound B 3 , by direct application of Hoeffding inequality, we have B 3 ≾ (cid:114) ( s ∗

Original: [Equation]

Original: s ′ ) log p n .

Original: This implies

Original: [Equation]

Original: Putting Equation (15) and (16) together, we have, under the null hypothesis,

Original: [Equation]

Original: with convergence rate O ( n − 1 / 2 ) .

Original: Finally, by noting that we can decompose ∇ ℓ ( β 0 ) =n

Original: [Equation]

Original: ( β 0 ) , and each ∇ ℓ i ( β 0 ) has bounded first, second, and third moment, a direct application of Berry-Esseen theorem give convergence in distribution of T ∗ j to a standard normal law, with rate O ( n − 1 / 2 ) .

Original: We also arrive at the second conclusion of Theorem 3.1 by noting that it is a straightforward byproduct of the result on normality of the distribution of decorrelated test score under null hypothesis, based on the formula for the p-values of CRT-logit algorithm.

Original: Proof of Corollary 3.1.

Original: The proof of this result is a straightforward adaptation from [6].

Original: For shorter notation, we denote (cid:98) S def.

Original: = (cid:98) S BY-CRT and (cid:98)

Original: k def.

Original: = (cid:98) k

Original: BY .

Original: If we denote ¯ α def.

Original: = α p (cid:80) pi

Original: =1 1 /i ∈

Original: (0 , 1) , then step 1 in the procedure defined in Definition B.2 is equivalent to finding (cid:98) k such that

Original: [Equation]

Original: For every i, j, k ∈ [ p ] , let us define

Original: [Equation]

Original: if

Original: [Equation]

Original: Then, since i ∈ (cid:98) S and | (cid:98) S| = k implies that (cid:98) p i ≤ (cid:98) p (cid:98) k ≤ (cid:98) k

Original: ¯ α

Original: =

Original: k

Original: ¯ α , we have | (cid:98) S ∩ S c

Original: [Equation]

Original: | (cid:98)

Original: [Equation]

Original: Taking the expectation and writing that

Original: [Equation]

Original: we get

Original: E

Original: [Equation]

Original: k

Original: [Equation]

Original: ∈S c

Original: [Equation]

Original: (cid:88) k

Original: [Equation]

Original: j k p

Original: [Equation]

Original: k

Original: [Equation]

Original: ∈S c

Original: [Equation]

Original: j =

Original: [Equation]

Original: p

Original: [Equation]

Original: k = j

Original: [Equation]

Original: p

Original: [Equation]

Original: = p

Original: [Equation]

Original: j =  j (cid:88) i ∈S c p

Original: [Equation]

Original: k = j p i,j,k (cid:124) (cid:123)(cid:122) (cid:125) A .

Original: Denote F ( j ) def.

Original: = (cid:80) i ∈S c (cid:80) jj ′

Original: =1 (cid:80) pk =1 p

Original: [-i,j ′ ,k for all j ∈ { 1 , . . . , p } , and remark that p i,j ′ ,k = 
-]
Revised: [-i,j ′ ,k for all j ∈ { 1 , . . . , p } , and remark that p i,j ′ ,k = 
-]

Original: if j ′ > k , by definition of (cid:98) S BY-CRT .

Original: We then have

Original: A = F (1) + p (cid:88)

Original: j =  j (cid:2) F ( j ) − F ( j − 1) (cid:3) =

Original: p − 1 (cid:88)

Original: j

Original: = (cid:18)j − j +(cid:19)

Original: F ( j ) + F ( p ) p .

Original: This leads to

Original: E

Original: [Equation]

Original: [Equation]

Original: (cid:19)

Original: F

Original: [Equation]

Original: j )

Original: [Equation]

Original: By the definition of p i,j,k in Eq.

Original: (18), we have

Original: [Equation]

Original: and

Original: [Equation]

Original: Therefore

Original: E

Original: [Equation]

Original: Taking the limit where n → ∞ and p fixed, we have, using the result in Theorem 3.1,

Original: [Equation]

Original: We conclude the proof by noting that ¯ α def.

Original: = α p (cid:80) pj =1 1 /j .

Original: B Controlling False Discovery Rate Procedures

Original: Definition B.1 (Benjamini-Hochberg procedure [5]) .

Original: Let α ∈ (0 , 1) be the predefined FDR control level.

Original: Let ˆ p 1 , . . .

Original: , ˆ p m be output p-values from an inference algorithm, e.g. Algorithm 1.

Original: We reorder them ascendingly, denoted by ˆ p (1) ≤ ˆ p (2) ≤ · · · ≤ ˆ p ( p ) and H (1)0 , . . .

Original: , H ( p ) 0 , then

Original: Find (cid:98) k

Original: BH such that

Original: [Equation]

Original: max

Original: [Equation]

Original: If (cid:98) k

Original: BH exists, take (cid:98) S

Original: = { j ∈

Original: [ p ] : ˆ p ( j ) ≤ ˆ p (cid:98) k BH } .

Original: Otherwise (cid:98) S = ∅ .

Original: Definition B.2 (Benjamini-Yekutieli procedure [6]) .

Original: Let α ∈ (0 , 1) be the predefined FDR control level.

Original: Let ˆ p 1 , . . .

Original: , ˆ p m be output p-values from Algorithm 1.

Original: We reorder them ascendingly, denoted by ˆ p (1) ≤ ˆ p (2) ≤ · · · ≤ ˆ p ( p ) and H (1)0 , . . .

Original: , H ( p ) 0 , then

Original: Find (cid:98) k

Original: BY such that

Original: [Equation]

Original: k BY = max

Original: [Equation]

Original: p (cid:80) pi

Original: [Equation]

Original: 1 /i (cid:27) .

Original: If (cid:98) k

Original: BY exists, take (cid:98) S

Original: = { j ∈

Original: [ p ] : ˆ p ( j ) ≤ ˆ p (cid:98) k BY } .

Original: Otherwise (cid:98) S = ∅ .

Original: C Setting the ℓ 1 − Regularization Parameter of the X ∗ ,j -distillation

Original: A core issue is the dependency of the statistical power and FDR of CRT-logit on the ℓ 1 − regularization parameter λ dx when doing Lasso distillation on x j in Eq.

Original: One might choose the reference value λ univ =

Original: (cid:112) n − 1 log p with theoretical validity, as suggested in [21, 28].

Original: However, experimental results in Fig.

Original: [-5 show that at λ dx = λ univ (or log 10 λ/λ univ = 0 . 
-]
Revised: [-5 show that at λ dx = λ univ (or log 10 λ/λ univ = 0 . 
-]

Original: with the labeling of the figure), we do not have the best possible FDR/Power with CRT-logit inference.

Original: For this experiment, we average the inference results of 100 simulations (with similar setting in Section 4.1) for different values of n and λ dx , with p fixed.

Original: There is a clear phase transition in both FDR and average power when the regularization parameter λ dx increases.

Original: In other words, we have found empirically that both FDR and power of the method are sensitive to the ℓ 1 − regularization parameter.

Original: Preferably, one wants to return a high statistical power while controlling FDR under predefined level.

Original: Hence, it is necessary to choose λ dx wisely.

Original: In practice, we advise using cross-validation for X ∗ ,j -distillation operator, as defined by Eq.

Original: This means we would have to find p different values of λ dx with cross-validation, and we reemphasize the importance of the screening step to reduce the number of computations.

Original: [Figure]

Original: D Pseudocode for CRT-logit and Related Algorithms

Original: Algorithm 2: Conditional Randomization Test [10]

Original: ; 4 ˆ f θ ← model_fitting ( X train , y train ) ; 5 for p do

Original: B do

Original: Generate

Original: ← f

Original: ,j )) ; end

Original: Compute the empirical p-value (cid:98) p

Original: j = 1 + (cid:80)

Original: Bb =1 1 ˜ T ( b ) j ≥ T

Original: j 1 + B 12 end

Original: E Time complexity of Related Methods

Original: We present the time complexity of benchmarked methods in Table 3.

Original: [Table]

Original: References

Original: [Table]

Original: F Additional Details on Experiments in SectionF.1 Preprocessing of the brain-imaging dataset

Original: The Human Connectome Project dataset (HCP) is a collection of brain imaging data on healthy young adult subjects with age ranging from 22 to 35.

Original: The participants performed different tasks while being scanned by a magnetic resonance imaging (MRI) device to record blood oxygenation level dependent (BOLD) signals of the brain.

Original: The aim of this analysis is to investigate which areas of the brain can predict cognitive activity across participants, while taking into account the information from other brain regions.

Original: The brain imaging modalities include, among others, resting-state fMRI (R-fMRI) and task-evoked fMRI (T-fMRI).

Original: In this work, we only deal with decoding the task-evoked fMRI dataset.

Original: The four classification problems we are working with are as follows.

Original: • Relational: predict whether the participant matches figures or identified feature similarities.

Original: • Gambling: predict whether the participant gains or loses gambles.

Original: • Emotion: predict whether the participant watches an angry face or a geometric shape.

Original: • Social: predict whether the participant watches a movie with social behavior or not.

Original: To perform dimension reduction, we apply a clustering scheme that preserves the spatial structure of the data.

Original: This is achieved with data-driven parcellation along with a spatially constrained clustering algorithm, following the conclusions by [29] and [27].

Original: The hierarchical clustering scheme that we use recursively merges pair of clusters of features based on a criterion that minimized the within-cluster variance.

Original: This algorithm is implemented in scikit-learn [23], a popular package for applied machine learning.

Original: G Example of decoding maps in semi-realistic brain-analysis experiment of Section 4.

Original: L

Original: R

Original: L

Original: R - -    (a) Debiased Lasso ( dlasso )

Original: L

Original: R

Original: L

Original: R (b) Knockoffs ( KO )

Original: L

Original: R

Original: L

Original: R (c) CRT ( CRT ), B =

Original: L

Original: R

Original: L

Original: R (d) Distilled-CRT ( dCRT )

Original: L

Original: R

Original: L

Original: R - -    (e) CRT-

Original: [Figure]

Original: H Ineffectiveness of CRT in extremely high-dimensional problems

Original: When the number of observations n is too small compared to the number of variables p , e.g. when n/p < 0 .

Original: 2 as shown in Figure 7, the inference problem becomes too ill-posed.

Original: Indeed, the statistical power of both the original dCRT and our proposed solution CRT-logit decrease dramatically from a large value in the easy setting ( p < n ) to zero when p > 1600 .

Original: The failure to detect any significant variable when the dimension of the problem becomes too high hints on future direction of performing statistical inference on clusters of variables.

Original: For instance, the works of [9, 12] have provided detailed discussions on this matter.

Original: 2 04 08 01 6 03 2 06 4 0p 0.

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: A v e

Original: [Figure]