M EASURING CAUSAL INFLUENCE WITH BACK - TO - BACK REGRESSION : THE LINEAR CASE
Anonymous authors
Paper under double-blind review
A BSTRACT
Identifying causes from observations can be particularly challenging when i) potential factors are difﬁcult to manipulate individually and ii) observations are complex and multi-dimensional.
To address this issue, we introduce “Back-to-Back” regression (B2B), a method designed to efﬁciently measure, from a set of co-varying factors, the causal inﬂuences that most plausibly account for multidimensional observations.
After proving the consistency of B2B and its links to other linear approaches, we show that our method outperforms least-squares regression and cross-decomposition techniques (e.g. canonical correlation analysis and partial least squares) on causal identiﬁcation.
Finally, we apply B2B to neuroimaging recordings of 102 subjects reading word sequences.
The results show that the early and late brain representations, caused by low- and high-level word features respectively, are more reliably detected with B2B than with other standard techniques.
I NTRODUCTION
Natural sciences are tasked to ﬁnd, from a set of hypothetical factors, the minimal subset that sufﬁces to reliably predict novel observations.
This endeavor is impeded by two major challenges.
First, causal and non-causal factors may be numerous and collinear.
This issue becomes increasingly pronounced as the number of potential factors increases.
In neuroscience, for example, identifying whether the frequency of a word presented on a subject’s retina modulates brain activity can be surprisingly difﬁcult.
Indeed, the frequency of words in natural language covaries with other factors such as their length (short words are more frequent than long words), their categories (determinants are more frequent than adverbs) and so forth (Kutas & Federmeier, 2011; Pegado et al., 2014).
Instead of selecting a set of words that control for all of these factors simultaneously, it is thus common to use forward modeling, i.e. to train a model to predict observations (e.g. brain activity) from a minimal combination of competing factors (e.g. word length, word frequency, e.g. (Huth et al., 2016)), and investigate, in the model, the estimated contribution of each factor (Friston et al., 1994).
The second challenge to measuring causal inﬂuence is that observations can be large and complex.
The relationship between causes and effects is thus often considered in a backward manner, by training models to maximally predict causes from multidimensional observations.
For example, brain activity is often recorded with hundreds or thousands of sensors simultaneously.
As multiple sensors may be affected by common noise sources, it is common to use backward regression, by, for example, ﬁtting a support vector machine across multiple sensors to decode the category of a stimulus (Cichy et al., 2014; Kriegeskorte et al., 2008; Norman et al., 2006).
Both forward and backward modeling have competing beneﬁts and drawbacks.
Speciﬁcally, forward modeling disentangles the independent contribution of collinear factors, but does not combine multidimensional observations.
By contrast, backward modeling combines multiple observations, but does not disentangle collinear factors (Weichwald et al., 2015; Hebart & Baker, 2018; King et al., 2018).
To combine the beneﬁts of forward and backward modeling, several authors have proposed to use cross-decomposition techniques such as Partial Least Squares (PLS) and Canonical Correlation
Analysis (CCA) (de Cheveigne et al., 2019).
CCA and PLS aim to ﬁnd, from two sets of data X and Y , the components H and G were XH and Y G are maximially correlated or maximally covarying respectively.
Because CCA and PLS are based on a generalized eigen decomposition, their resulting coefﬁcients are mixing the features of X and Y in a way that makes them notoriously difﬁcult to interpret (Lebart et al., 1995).
= × + ×
Y ∈ [Equation] n
[Equation]
y X ∈ R n × d x
E ∈ D
d x × d x
N ∈ R n × d x
F ∈ R d x × d y
Observations
Factors
Cause selection
Noise
Cause-effectmapping
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
Here, we introduce the ‘back-to-back regression’ (B2B), which not only combines the beneﬁts of forward and backward modeling (Section 2), but also provide robust, interpretable and unidimensional causal coefﬁcients for each tested factor. The present paper focuses on the restricted issue of disentangling the causal inﬂuence of linearly correlated predictors ( X ) onto noisy multivariate observations ( Y ).
The present approach thus differs from other causal discovery algorithms based on temporal-delays and/or nonlinear interactions in systems where the directionality of causation (from X to Y or vice versa) is unknown (e.g. (Peters et al., 2017; Granger, 1969; Janzing et al., 2013;
Sch ¨ olkopf et al., 2016). After detailing B2B method and proving its convergence (Section 2.2), we show with synthetic data that it outperforms state-of-the-art forward, backward and cross-decomposition techniques in identifying causal inﬂuence (Section 3.1).
Finally, we apply B2B to a large neuroimaging dataset and reveal that distinct but collinear word features lead to distinguishable brain representations (Section 3.5).
B ACK - TO -B ACK REGRESSION
We consider the measurement of multivariate signal Y ∈ R n × d y , generated from a set of putative causes X ∈ R n × d x , via some unknown linear apparatus F ∈ R d x × d y .
Not all the variables in X exert a causal inﬂuence on Y .
By considering a square binary diagonal matrix of causal inﬂuences E ∈ D d x × d x , we denote by XE the causal factors of Y .
In summary, the problem can be formalized as:
[Equation]
where i is a given sample, and n i is a sample-speciﬁc noise drawen from a centered distribution.
While the triplet of variables X and N are independent, we allow each of them to have any form of covariance.
In practice, we observe n samples ( X, Y ) from the model.
This problem space, along with the sizes of all variables involved, is illustrated in Figure 1.
Given the model in Equation (1), the goal of Back-to-Back Regression (B2B) is to estimate the matrix of causal inﬂuences E .
2. A LGORITHM
Back-to-Back Regression (B2B) consists of two steps.
First, we estimate the linear regression coefﬁcients ˆ G from Y to X , and construct the predictions ˆ X =
Y ˆ G .
This backward regression recovers the correlations between Y and each factor of X .
Second, we estimate the linear regression coefﬁcients ˆ H from X to ˆ X .
The diagonal of the regression coefﬁcients ˆ H , denoted by ˆ E = diag ( ˆ H ) , is the desired estimate of the causal inﬂuence matrix E , as detailed in the A.1.
If using l2-regularized least-squares (Hoerl, 1959; Rifkin & Lippert, 2007), B2B has a closed form solution:
[Equation]
where Λ X and Λ Y are two diagonal matrices of regularization parameters, useful to invert the covariance matrices of X and Y if these are ill-conditioned.
Performing two regressions over the same data sample can result in overﬁtting, as spurious correlations in the data absorbed by the ﬁrst regression will be leveraged by the second one.
To avoid this issue, we split our sample ( X, Y ) into two splits ( X 1 , Y 1 ) and ( X 2 , Y 2 ) .
Then, the ﬁrst regression is performed using ( X 1 , Y 1 ) , and the second regression is performed using ( X 2 , Y 2 ) .
To compensate for the reduction in sample size caused by the split, B2B is repeated over many random splits, and the ﬁnal estimate ˆ E of the causal inﬂuence matrix is the average over the estimates associated to each split (Breiman, 1996).
To accelerate this ensembling procedure, we implemented an efﬁcient leave-one-out cross-validation scheme as detailed in (Rifkin & Lippert, 2007) , as follows:
[Equation]
where Σ X is the X kernel matrix and where G is computed with an eigen decomposition of X :
[Equation]
where Q , V and λ are the eigen vectors, eigen values and regularization, respectively.
We summarize the B2B procedure in Algorithm 1.
The rest of this section provides a theoretical guarantee on the correctness of B2B.
[Equation]
[Equation]
[Equation]
2. T HEORETICAL GUARANTEES
Theorem 1 (B2B consistency - general case) .
Consider the B2B model from Equation Y =
( XE +
N ) F , N centred and full rank noise.
Let Img ( M ) refers to the image of the matrix M .
If F and X are full-rank on the Img ( E ) , then, the solution of B2B, ˆ H , will minimize min H (cid:107)
X − XH (cid:107) 2 + (cid:107) NH (cid:107) 2 and satisfy E ˆ H
= ˆ H
Proof.
See Appendix A.1.
Since E ˆ H = ˆ H , we have
[Equation]
Assuming, without loss of generality, that the active features in E are the k {+∈ Z : k ∈ [0 , d x ] ﬁrst features, and rewriting X = ( X 1 , X 2 ) and N = ( N 1 , N 2 ) ( X 
 and N 1 containing the k ﬁrst features),+} we have:
[Equation]
where Σ AB is the covariance of A and B , and:
[Equation]
In the absence of noise, we have Σ N 1 N 1 = 0 , and so diag k ( ˆ H ) = I , and
[Equation]
Therefore, we recover E from ˆ H .
In the presence of noise, the causal factors of E correspond to the positive elements of diag ( ˆ H ) .
The methods to recover them are presented in the Appendix A.4.
E XPERIMENTS
We perform two sets of experiments to evaluate B2B: one on controlled synthetic data, and a second one on a real, large-scale magneto-encephalography (MEG) dataset.
We use scikit-learn’s PLS and
RidgeCV (Pedregosa et al., 2011) as well as pyrcca’s regularized canonical component analysis (RegCCA, (Bilenko & Gallant, 2016)) objects to compare B2B against the standard baselines.
3. S YNTHETIC DATA
We evaluate the performance of B2B throughout a series of experiments on controlled synthetic data.
The purpose of these experiments is to evaluate the ability of B2B in terms of prediction of independent and identically distributed data, as well as a method to recover causal factors.
The data generating process for each experiment constructs n = 1000 training examples according to the model Y =
( h XE + N ) F , where h is a scalar that modulates the signal-to-noise ratio.
Here, F ∈ R d x × d y contains entries drawn from N (0 , σ 2 ) where σ 2 is inversely proportional to d x , X ∈ R n × d x contains rows drawn from N (0 , Σ X ) , N ∈ R n × d x contains rows drawn from N (0 , Σ N ) , E ∈ R d x × d x is a binary diagonal matrix containing n c ones, Σ X
= AA (cid:62)
where
A ∈ R d x × d x contains entries drawn from N (0 , σ 2 ) , Σ N = BB (cid:62) where B ∈ R d x × d x contains entries drawn from N (0 , σ 2 ) , and the factor h ∈ R + .
To simulate a wide range of experimental conditions, we sample 10 values in log-space for d x , d y ∈ [10 , 100] , n c ∈ [3 , 63] , h ∈ [0 . 001 , 10] .
We discard the cases where n c > d x , limit d x , d y to 100 to keep the running time under 2 hours for each condition, and average over 5 random seeds.
We compare the performance of B2B against four competing methods, all implemented in scikit-learn (Pedregosa et al., 2011) and pyrcca (Bilenko & Gallant, 2016):
. B ASELINE MODELS
Forward regression consists of an l 2 -regularized ”ridge” regression from the putative causes X to the observations Y :
[Equation]
[Figure]
Backward regression consists of an l 2 -regularized ”ridge” regression
[Equation]
CCA ﬁnds G cca ∈ R d z ,d y and H cca ∈ R d z ,d x s.t.
X and Y are maximally correlated in a latent Z space:
[Equation]
PLS ﬁnds G pls ∈ R d z ,d y and H pls ∈ R d z ,d x s.t.
X and Y are maximally covarying in a latent Z space:
[Equation] [Equation]
We employ 5 -fold cross-validation to select the optimal number of components for CCA and PLS.
Regressions were (cid:96) 2 -regularized with a λ regularization parameters ﬁtted with the efﬁcient leave-oneout procedure implemented in scikit-learn RidgeCV (Pedregosa et al., 2011).
3. E VALUATING C AUSAL D ISCOVERY FROM MODELS ’ COEFFICIENTS
B2B leads to unbiased (i.e. zeros-centered) scalar coefﬁcients for non-causal features.
In contrast, the
Forward, Backward, CCA and PLS models lead to a loading vector H i per feature i (or one vector
G i for the backward model).
To transform such vector into an estimated causal contribution ˆ E , we take the sum of square coefﬁcients: ˆ E i =
(cid:80) j H ji
To estimate whether models accurately identify causal factors, we compute the area-under-the-curve (AUC) across factors AUC ( E, ˆ E ) .
The AUC allows evaluating the capacity of models at detecting the causal importance of factors when ground truth labels are available, as is the case in this setup.
We report AUC results in Figures 2 (top) and 5 (left, in Appendix), and compare favorably to all baselines.
3. E VALUATING C AUSAL D ISCOVERY WITH HELD - OUT PREDICTION RELIABILITY
In most cases, E is not known and AUC can thus not be estimated.
To address this issue, we assess the ability of each model to reliably predict independent and identically distributed data from Y , given all of the X features versus all-but-ones feature X −
i (i.e. ’knock-out X’).
This procedure [Figure] results in two correlation metrics R full and R knockout , whose difference ∆ R i =
R full − R knockout indicates how much each X i improves the prediction of Y .
In our ﬁgures, ∆ R is the average of ∆ R i .
A higher score means that for prediction, the model relies on individual features rather than combinations of features.
We show in Appendix A.3 pseudo-code to assess feature importance for our algorithm as well as baselines.
For the Backward Model, feature importance cannot be assessed as the X colinearity is never taken into account.
We show in Figures 2 (bottom) and 5 (right, in Appendix) that our method outperforms baselines.
3. M AGNETOENCEPHALOGRAPHY DATA
Next, we apply our method to brain imaging data from the anonymized multimodal neuroimaging “Mother Of all Uniﬁcation Studies” (MOUS) dataset (Schoffelen et al., 2019).
The dataset contains magneto-encephalography (MEG) recordings of 102 healthy native-Dutch adults who participated in a reading task.
Twelve subjects were excluded from the analysis because of corrupted ﬁle headers.
Subjects were exposed to a rapid serial visual presentation of Dutch words.
The word lists consisted of 120 sentences, and scrambled lists of the same words.
Each word was presented on the computer screen for 351ms on average (min: 300ms, max: 1400ms).
Successive words were separated by a blank screen for 300ms, and successive sentences were separated by an empty screen for a few (3-4) seconds.
3.5. MEG PREPROCESSING
The raw MEG data was bandpass-ﬁltered between 0.1 and 40Hz using MNE-Python default parameters (Gramfort et al., 2013; 2014).
Speciﬁcally, we used a zero-phase ﬁnite impulse response ﬁlter (FIR) with a Hamming window and with transition bands of 0.1Hz and 10Hz for the low and high cut-off frequencies.
The raw data was then segmented 100ms before word onset and 1s after word onset ( t =
ms corresponds to word onset).
Finally, each resulting segment was baseline-corrected between -100ms and 0ms, and decimated by 5 and thus led a sampling frequency of 240Hz.
The average responses across words is displayed in Figure 3.
For each subject and each time sample relative to word onset, we build an observation matrix Y ∈ R n × d y of n ≈ 2700 words by d y =
MEG channels (273 magnetometers and 28 compensation channels).
Each of the columns of Y is normalized to have zero mean and unit variance.
3.5. F EATURE DEFINITION
We aim to identify the word features that cause a variation in brain responses.
We consider four distinct but colinear features.
First, ’Word Length’ refers to the total number of letters.
Word Length is expected to speciﬁcally cause a variation in the early evoked MEG responses (i.e. fromms after stimulus onset) elicited by the retinotopically-tuned visual cortices (e.g. (Pegado et al., 2014).).
Second, ’Word Frequency’ indexes how frequently each word appears in Dutch and was derived with the the Zipf logarithmic scale of (Van Heuven et al., 2014) provided by the WordFreq package (Speer et al., 2018).
Word Frequency is expected to speciﬁcally cause a variation in the late evoked MEG responses (i.e. from 400 ms), because it variably engages semantic processes in the temporal cortices (Kutas & Federmeier, 2011).
Third, ’Word Function’ indicates whether each word is a content word (i.e. a noun, a verb, an adjective or an adverb) or a function word (i.e. a preposition, a conjunction, a determinant, a pronoun or a numeral), and was derived from Spacy’s part of speech tagger (Honnibal & Montani, 2017).
To our knowledge, this feature has not been thouroughly investigated with MEG.
Its causal contribution to reading processes in the brain thus remains unclear.
Finally, to verify that B2B and other methods would not inadequately identify non-causal features, we added a dummy feature, constructed from a noisy combination of Word
Length and Word Frequency:
dummy = z ( length ) + z ( frequency ) + N , where z normalizes features and N is a random vector sampling Gaussian distribution (all terms thus have a zero-mean and a unit-variance).
This procedure yields an X ∈ R n × d x matrix of n ≈ 2700 words by d x =features for each subject.
Each of the columns of X is normalized to have a mean and a standard deviation of 0 and 1 respectively.
3.5. M ODELS AND STATISTICS
We compare B2B to four standard methods: Forward regression, Backward regression, CCA and PLS, as implemented in scikit-learn (Pedregosa et al., 2011) and (Bilenko & Gallant, 2016), and optimized with nested cross-validation over twenty l 2 regularization parameters logarithmically spaced between 10 − 4 and 10 4 (for regression and CCA methods) or 1 to 4 canonical components (for PLS).
We used the feature importance described in Algorithm 2 to assess the extent to which each feature
X i speciﬁcally improves the prediction of held-out Y data, using a 5-fold cross-validation (with shufﬂed trials to homogeneize the distributions between the training and testing splits).
Each model was implemented for each subject and each time sample independently.
Pairwise comparison between models were performed using a two-sided Wilcoxon test across subjects (n=90) using the average ∆ R across time.
Corresponding effect sizes are shown in Figure 4, and p-values are reported below.
3.5. R ESULTS
We compared the ability of Forward regression, Backward regression, CCA, PLS and B2B to estimate the causal contribution of four distinct but collinear features on brain evoked responses to words.
As expected, the Backward model reveals a similar decoding time course for Word Length and
Word Frequency, even though these features are known to speciﬁcally inﬂuence early and late MEG responses respectively (Kutas & Federmeier, 2011). In addition, the same decoding time course was observed for the dummy variable.
These results illustrate that backward modelling cannot be used to estimate the causal contribution of collinear features.
We thus focus on the four remaining methods (i.e. Forward Regression, PLS, CCA, and B2B) and estimate their ∆ R (i.e. the improvement of Y prediction induced by the introduction of a given feature into the model,as described in Algorithm 2).
Contrary to the Backward Model, none of the models predicted the Dummy Variable to improve the Y prediction: all ∆ R < 0 (all p > . 089 ).
Figure 4 shows, for each model, the effects obtained across time (left) and subjects (right).
Word Length and Word Frequency improved the prediction performance of all methods: ∆ R >for all models (all p < 0 . 0001 ).
As expected, the time course associated with Word Length and Word Frequency rose from ≈ 100 ms and from ≈ 400 ms respectively.
Furthermore, Word Function improved the prediction performance of all models (all p < 0 . 0002 ) except for PLS ( p = 0 . 7989 ).
Overall, these results conﬁrm that Word Length, Word Frequency and Word Function causally inﬂuence speciﬁc periods of brain responses to words.
To assess which model would be most sensitive to these causal discoveries, we compared B2B to other models across subjects (Figure 4 right).
For Word Length B2B outperforms all models (all p < 0 . 00001 ) but CCA ( p = 0 . 0678 ).
For Word Frequency, B2B outperforms all models (all p < 0 . 0006 ).
For ”Word Function”, B2B outperforms all models (all p < 0 . 0015 ).
Overall, these results show that B2B reliably outperforms standard methods, especially when the effects are difﬁcult to detect.
R ELATED WORK
[Figure]
Forward
PLS RegCCA B2B Forward and cross-decomposition models have been used to identify the causal contribution of collinear features onto multi-dimensional observations (e.g. (Naselaris et al., 2011)).
These approaches typically lead to multiple coefﬁcients for each features (i.e. one per dimension of Y or one per component respectively).
Furthermore, these coefﬁcients can be difﬁcult to summarize into a single causal estimate.
By contrast, B2B quickly (Fig. 6) leads to a single unbiased scalar values ˆ E tending towards 1 and 0 for causal and non-causal features respectively.
A variety of other statistical methods applied to neuroimaging data have been proposed to clarify what is being represented in brain responsesi.e. what feature causes speciﬁc brain activity.
One of the popular linear method is Representational Similarity Analysis (RSA) (Kriegeskorte et al., 2008), and consists in analyzing the similarity of brain responses associated with speciﬁc categorical conditions (e.g. distinct images), by (1) ﬁtting one-against-all classiﬁers on each condition and (2) testing whether these classiﬁers can discriminate all other conditions.
The resulting confusion matrix is then analyzed in an unsupervised manner to reveal which conditions lead to similar brain activity patterns.
B2B differs from RSA in that (1) it uses regressions instead of classiﬁcations, and can thus generalize to new items and new contexts and (2) it is fully supervised.
Finally, CCA has been used in neuroimaging for a variety of purposes such as denoising and subject alignment (Hotelling, 1936; de Cheveigne et al., 2019). While CCA relates to B2B, these two methods diverge in several ways. First, CCA and B2B have different objectives: CCA aims to ﬁnd the potentially numerous and poorly interpretable components where X and Y are maximally correlated, whereas B2B aims to recover the causal factors from X to Y. Second, B2B is not symmetric between
X and Y : it aims to identify speciﬁc causal features by ﬁrst optimizing over the decoders G and then over H .
By contrast, CCA is symmetric between X and Y , and aims to ﬁnd G and H such that they project X and Y on maximally correlated dimensions.
Third, CCA is based an eigen decomposition of XH and Y G - the corresponding canonical components are thus mixing the X features in way that limit interpretability and potentially dilute the impact of each feature onto multiple components.
In contrast B2B assesses each feature X j on a single Y component speciﬁcally selected to maximize signal-to-noise ratio of that feature j .
Fourth, and unlike B2B, CCA does not separately optimize two distinct regularization parameters for G and H .
Finally, CCA does not use different data splits to estimate G and H .
Together, these differences may explain why B2B reliably outperform CCA on estimating causal inﬂuences (Figs. 2 and 5).
C ONCLUSION
In this work, we proposed Back-to-Back (B2B) regression, a linear method to measure the causal inﬂuence of a potential set of variables generating multidimensional observations.
B2B performs two successive multidimensional regressions: one from the output domain, and another one from the input domain.
We provided a theoretical guarantee about the consistency of B2B, and compared it to several baselines in controlled synthetic experiments.
We also applied B2B to a recent brain imaging dataset, analyzing the timing of brain responses and their connection to word features.
We obtained results consistent with prior work in neuroscience literature, conﬁrming the efﬁcacy of B2B for real data analysis.
A.
P ROOF OF CONSISTENCY THEOREM
Proof of the theorem in 2.2:
Theorem 2 (B2B consistency - general case) .
Consider the B2B model from equation
[Equation]
with N centred and full rank noise.
If F and X are full-rank on Img ( E ) , then, the solution of B2B, ˆ H minimizes [Equation]
and satisﬁes
[Equation]
Proof.
Let ˆ G and ˆ H be the solutions of the ﬁrst and second regressions of B2B. Since ˆ G is the least square estimator of X from Y
[Equation]
Replacing Y by its model deﬁnition Y = ( XE + N ) F , we have [Equation] Since N is centered and independent of X , we have
[Equation] Samely, for ˆ H , we have [Equation] a positive quantity which reaches a minimum (zero) for ˆ
[Equation] = EF [Equation]
Let us now prove that EF ˆ G = F ˆ G .
Let F † be the pseudo inverse of F , and Z = F † EF ˆ G , we have FZ = FF † EF ˆ G
Since F is full rank on Img ( E ) , we have FF † E = E , and FZ = EF ˆ G
As E is a binary diagonal matrix, it is an orthogonal projection and therefore a contraction, thus
[Equation]
and
[Equation]
But since ˆ G = arg min G (cid:107) X − XEFG (cid:107) 2 + (cid:107) NFG (cid:107) 2 , we also have [Equation] Summarizing the above,
[-Therefore, (cid:107) X − XEFG (cid:107) 
 + (cid:107) NFG (cid:107) 
-][Equation] N being full rank, this yields EF ˆ G = F ˆ G .
Replacing into (14), and setting H =[-(cid:107) X − XEFG (cid:107) 
-]
[-+ (cid:107) NEFG (cid:107) 
-]EFG , we have
[-= (cid:107) X − XH (cid:107) 
 + (cid:107) NH (cid:107) 2 is minimised by ˆ G and ˆ H .-][Equation]
Finally, E ˆ H =
EEF ˆ G = EF ˆ G =
ˆ H , since E , a binary diagonal matrix, is involutive.
This completes the proof.
A. M ODELING MEASUREMENT NOISE
Equation 1 does not explicitly contain a measurement noise term.
Yet, in moast experimental cases, the problem is best described as:
[Equation]
with M ∈ R n × d y .
This equation is actually equivalent to Equation 1 given our hypotheses.
Indeed, we can rewrite
M = MF − 1 F over Img ( F ) , which leads to:
[Equation]
Consequently, assuming that F is full rank on Img ( XE ) , B2B yields the same solutions to equations 1 and 16.
A. F EATURE IMPORTANCE
For B2B, feature importance is assessed as follows: Algorithm 2: B2B feature importance. Input [Equation]
Output
[Equation] For the Forward Model, the feature importance is assessed as follows: Algorithm 3: Forward feature importance. Input
[Equation] Output [Equation] For the CCA and PLS models, the feature importance is assessed as follows:
Algorithm 4: PLS
For the Backward Model, feature importance cannot be assessed because there is no prediction. A. R ECOVERING E
In case of noise, B2B yields non binary ˆ E . Three thresholding rules can be used to binarize its values thus explicitly recover causal features. First, given known signal-to-noise ratio,the threshold above which a feature should considered to be causal can be derived analytically. Indeed, Equation 9 implies that the k ﬁrst diagonal elements of ˆ H are bounded:
[Equation]
where σ X 1 , σ X k , σ N 1 and σ N k denote the largest and smallest eigenvalues of Σ X 1 X 1 and Σ N 1 N 1 .
The average value µ of non-zero coefﬁcients of diag ( ˆ H ) is the trace of ˆ H divided by k , and can be computed as
[Equation]
The decision threshold between causal and non-causal elements is thus a fraction µ , whose proportion arbitrarily depends on the necessity to favor type I and type II errors.
In practice, we cannot use this procedure for our MEG study, because signal-to-noise ratio is unknown.
Second, diag ( ˆ H ) can be binarized with the Sonquist-Morgan criterion (Morgan & Sonquist, 1963), a non-parametric clustering procedure separating small and large values in a given set.
This procedure maximizes the ratio of inter-group variance while minimizing the intra-group variance, over all possible splits of the diagonal into p largest values and d x − p smallest values.
Let m 0 and m 1 be the average values of the two clusters, p and d x − p
their size, and v the total variance of the sample,
Sonquist-Morgan criterion maximizes (Kass, 1975):
[Equation] This procedure assumes that there exists at least one causal and at least one non-causal feature. Third, second-order statistics across multiple datasets can be used to identify the elements of diag ( ˆ H ) that are sgniﬁcantly different from 0. This procedure is detailed in the method section of our MEG experiment.
Overall, these three procedure thus varies in their additional assumptions: i.e. (1) a known signal-tonoise ratio, (2) the existence of both causal and non-causal factors or (3) independent repetitions of the experiment. [Figure] [Figure]
[Figure]
B A DDITIONAL F IGURES B. R OBUSTNESS TO INCREASING NUMBER OF FACTORS
To test whether each of the methods robustly scales to an increasingly large number of potential causes X , we enhanced the four ad-hoc features (word length, word frequency, word function, dummy variable) with another ten features. These additional features corresponds to the ﬁrst dimensions of word embedding as provided by Spacy (Honnibal & Montani, 2017).
The results shown in Figure 7, show that the feature importance of ad-hoc features as derived by B2B remain unchanged and are actually improved.