Original: C OMMS VAE: L EARNING THE BRAIN ’ S MACROSCALE COMMUNICATION DYNAMICS USING COUPLED SEQUENTIAL
Revised: C OMMS VAE: L EARNING THE BRAIN ’ S MACROSCALE COMMUNICATION DYNAMICS USING COUPLED SEQUENTIAL

Original: VAE S
Revised: VAE S

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Communication within or between complex systems is commonplace in the natural sciences and fields such as graph neural networks.
Revised: Communication within or between complex systems is commonplace in the natural sciences and fields such as graph neural networks.

Original: The brain is a perfect example of such a complex system, where communication between brain regions is constantly being orchestrated.
Revised: The brain is a perfect example of such a complex system, where communication between brain regions is constantly being orchestrated.

Original: To analyze communication, the brain is often split up into anatomical regions that each perform certain computations.
Revised: To analyze communication, the brain is often split up into anatomical regions that each perform certain computations.

Original: These regions must interact and communicate with each other to perform tasks and support higher-level cognition.
Revised: These regions must interact and communicate with each other to perform tasks and support higher-level cognition.

Original: On a macroscale, these regions communicate through signal propagation along the cortex and along white matter tracts over longer distances.
Revised: On a macroscale, these regions communicate through signal propagation along the cortex and along white matter tracts over longer distances.

Original: When and what types of signals are communicated over time is an unsolved problem and is often studied using either functional or structural data.
Revised: When and what types of signals are communicated over time is an unsolved problem and is often studied using either functional or structural data.

Original: In this paper, we propose a non-linear generative approach to communication from functional data.
Revised: In this paper, we propose a non-linear generative approach to communication from functional data.

Original: We address three issues with common connectivity approaches by explicitly modeling the directionality of communication, finding communication at each timestep, and encouraging sparsity.
Revised: We address three issues with common connectivity approaches by explicitly modeling the directionality of communication, finding communication at each timestep, and encouraging sparsity.

Original: To evaluate our model, we simulate temporal data that has sparse communication between nodes embedded in it and show that our model can uncover the expected communication dynamics.
Revised: To evaluate our model, we simulate temporal data that has sparse communication between nodes embedded in it and show that our model can uncover the expected communication dynamics.

Original: Subsequently, we apply our model to temporal neural data from multiple tasks and show that our approach models communication that is more specific to each task.
Revised: Subsequently, we apply our model to temporal neural data from multiple tasks and show that our approach models communication that is more specific to each task.

Original: The specificity of our method means it can have an impact on the understanding of psychiatric disorders, which are believed to be related to highly specific communication between brain regions compared to controls.
Revised: The specificity of our method means it can have an impact on the understanding of psychiatric disorders, which are believed to be related to highly specific communication between brain regions compared to controls.

Original: In sum, we propose a general model for dynamic communication learning on graphs, and show its applicability to a subfield of the natural sciences, with potential widespread scientific impact.
Revised: In sum, we propose a general model for dynamic communication learning on graphs, and show its applicability to a subfield of the natural sciences, with potential widespread scientific impact.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Characterizing macroscale communication between brain regions is a complex and difficult problem, but is necessary to understand the connection between brain activity and behavior.
Revised: Characterizing macroscale communication between brain regions is a complex and difficult problem, but is necessary to understand the connection between brain activity and behavior.

Original: The effect of neural systems on each other, or connectivity, has also been linked to psychiatric disorders, such as schizophrenia Friston (2002).
Revised: The effect of neural systems on each other, or connectivity, has also been linked to psychiatric disorders, such as schizophrenia Friston (2002).

Original: Hence, gaining a deeper understanding of the dynamics underlying the communication between brain regions is both from the perspective of understanding how our brain facilitates higher-order cognition and also to provide insight into and consequently how psychiatric disorders arise.
Revised: Hence, gaining a deeper understanding of the dynamics underlying the communication between brain regions is both from the perspective of understanding how our brain facilitates higher-order cognition and also to provide insight into and consequently how psychiatric disorders arise.

Original: Static functional network connectivity (sFNC) and dynamic functional network connectivity (dFNC), computed respectively as the Pearson correlation between regional activation timeseries over the full scan duration (sFNC) or on shorter sliding windows (dFNC), are among the most widely reported measures of connectivity between brain regions Hutchison et al.
Revised: Static functional network connectivity (sFNC) and dynamic functional network connectivity (dFNC), computed respectively as the Pearson correlation between regional activation timeseries over the full scan duration (sFNC) or on shorter sliding windows (dFNC), are among the most widely reported measures of connectivity between brain regions Hutchison et al.

Original: These approaches calculate the correlation between the timeseries of each brain region to find their coherence, either across the full timeseries (sFNC) or by windowing the timeseries and calculating the correlation within each window (dFNC).
Revised: These approaches calculate the correlation between the timeseries of each brain region to find their coherence, either across the full timeseries (sFNC) or by windowing the timeseries and calculating the correlation within each window (dFNC).

Original: Although extensions have been proposed Hutchison et al.
Revised: Although extensions have been proposed Hutchison et al.

Original: (2013), along with more complex connectivity measures, such as wavelet coherence Yaesoubi et al.
Revised: (2013), along with more complex connectivity measures, such as wavelet coherence Yaesoubi et al.

Revised: , multiplication of temporal derivates Shine et al.

Original:  and Granger causality Roebroeck et al.
Revised: (2015), and Granger causality Roebroeck et al.

Original: Seth et al. (2015), Pearson correlation remains the most prevalent measure of brain network connectivity.
Revised: Seth et al. (2015), Pearson correlation remains the most prevalent measure of brain network connectivity.

Original: And even the less commonly employed metrics have issues stemming from some combination of sensitivity to noise, linearity, symmetry, or coarse timescales.
Revised: And even the less commonly employed metrics have issues stemming from some combination of sensitivity to noise, linearity, symmetry, or coarse timescales.

Original: Most importantly, these approaches do not directly model the communication, but rather analyze it post-hoc.
Revised: Most im- portantly, these approaches do not directly model the communication, but rather analyze it post-hoc.

Original: The pursuit of instantaneous communication between brain regions Sporns et al.
Revised: The pursuit of instantaneous communication between brain regions Sporns et al.

Original: and generative approaches to model communication can potentially lead to models that closely resemble effective connectivity Avena-Koenigsberger et al.
Revised: and generative approaches to model communication can potentially lead to models that closely resemble effective connectivity Avena-Koenigsberger et al.

Original: An important advantage of generative models is that they allow us to move away from post-hoc inference from context-na¨ıve metrics toward simulating macroscale brain communication.
Revised: An important advantage of generative models is that they allow us to move away from post-hoc inference from context-na¨ıve metrics toward simulating macroscale brain communication.

Original: We propose the use of recurrent neural networks as generative models of communication on both simulated data and functional magnetic resonance imaging (fMRI) data to validate and demonstrate the specificity of our model.
Revised: We propose the use of recurrent neural networks as generative models of communication on both simulated data and functional magnetic resonance imaging (fMRI) data to validate and demonstrate the specificity of our model.

Original: Our method complements connectivity metrics, since it does more than quantify the aggregation of communication Avena-Koenigsberger et al.
Revised: Our method complements connectivity metrics, since it does more than quantify the aggregation of communication Avena-Koenigsberger et al.

Original: , but directly simulates it, and can thus be analyzed using those same connectivity metrics.
Revised: , but directly simulates it, and can thus be analyzed using those same connectivity metrics.

Original: Creating an accurate generative model of the macroscale communication dynamics in the brain is hard, due to its complexity.
Revised: Creating an accurate generative model of the macroscale communication dynamics in the brain is hard, due to its complexity.

Original: However, there are some general design principles the brain follows
Revised: However, there are some general design principles the brain follows

Original: Sterling & Laughlin (2015).
Revised: Sterling & Laughlin (2015).

Original: Generally, the brain tries to minimize its energy use, which is likely also true for communication in the brain.
Revised: Generally, the brain tries to minimize its energy use, which is likely also true for communication in the brain.

Original: Macroscale communication is an energy-intensive process and involves white matter tracts, which are essentially highways that connect spatially separate parts of the brain.
Revised: Macroscale communication is an energy-intensive process and involves white matter tracts, which are essentially highways that connect spatially separate parts of the brain.

Original: Hence, the amount of information and the number of times information is sent should generally be limited.
Revised: Hence, the amount of information and the number of times information is sent should generally be limited.

Original: The bits needed to convey are limited by the brain using sparse coding, which at a lower scale is how neurons encode and communicate information Olshausen & Field (2004).
Revised: The bits needed to convey are limited by the brain using sparse coding, which at a lower scale is how neurons encode and communicate information Olshausen & Field (2004).

Original: Although mechanistically different, due to metabolic and volume constraints, macroscale communication presumably exhibits strategies similar to sparse coding to efficiently transfer information between neural populations Bullmore & Sporns (2012)
Revised: Although mechanistically different, due to metabolic and volume constraints, macroscale communication presumably exhibits strategies similar to sparse coding to efficiently transfer information between neural populations Bullmore & Sporns (2012)

Original: ; Sterling & Laughlin (2015).
Revised: ; Sterling & Laughlin (2015).

Original: To incorporate this inductive bias into our model, we regularize the communication from one region in our model to another using a KL-divergence term to a Laplace distribution.
Revised: To incorporate this inductive bias into our model, we regularize the communication from one region in our model to another using a KL-divergence term to a Laplace distribution.

Original: The communication itself is modeled as a normal distribution, but by minimizing its divergence with a Laplace distribution we encourage sparser temporal communication.
Revised: The communication itself is modeled as a Laplace distribution and by minimizing its divergence from a Laplace distribution we encourage sparser temporal communication.

Original: Encouraging sparse temporal communication implies that information is only sent when necessary, and lower rates of communication lead to reductions in the brain’s energy requirements.
Revised: Encouraging sparse temporal communication implies that information is only sent when necessary, and lower rates of communication lead to reductions in the brain’s energy requirements.

Revised: As far as we are aware, this is the first model that explicitly models communication dynamics between vertices on a graph, specifically between brain regions.

Revised: The connectivity metrics that are used currently assume that the connectivity between brain regions is stationary, e.g. Granger causality, and lack a generative model.

Revised: Although non-linear generative models have been proposed for connectivity Stephan et al.

Revised: (2008), however, determining the correct model is intractable Lohmann et al.

Revised: Furthermore, most methods quantify the coupling between brain regions and do not consider potentially rapid changes in fMRI signals that can be traced from region to region.

Original: The goal of our model is to get a better idea of communication dynamics in  the brain.
Revised: In this paper, we specifically try to model the communication between brain regions, which likely relates to these abrupt and parsimonious changes in the signal. This means that our model is finding a type of interaction between brain regions, or more generally nodes on a graph, that has never been studied in this way before, to the best of our knowledge. The assumptions we make thus also differ from commonly used connectivity metrics. Firstly, we assume directionality and communication with the same temporal resolution as the original signal.

Revised: Hence, we completely move away from windowed approaches, that quantify the coupling between brain regions within windows.

Revised: Secondly, we assume that brain regions are largely independent and can be modeled by a dynamical system with sparse inputs.

Revised: Lastly, we aim to learn a fully generative model of fMRI data, where the communications and initial state of each brain region’s dynamical system can be sampled from a simple distribution.

Revised: This also implies that our method is hard to compare with more common connectivity metrics, and we expect it to exhibit different behavior since it is modeled under different assumptions.

Revised: We propose this model because we believe it has tremendous value and complements many connectivity metrics in a meaningful way, not only in the neuroimaging community, but also in other scientific fields, such as sociology Barnes (1969), computational biology or chemistry Balaban (1985), and traffic prediction models Zhao et al.

Revised: The goal of our model is to get a better idea of communication dynamics in the brain.

Original: To evaluate whether our model is equipped to find the underlying dynamics of a known generative model, we  train it on simulated data.
Revised: To evaluate whether our model is equipped to find the underlying dynamics of a known generative model, we first train it on simulated data.

Original: After we show that our model finds the correct generative model from the simulated data, we apply our model to neuroimaging data with fairly well-established neural pathways.
Revised: After we show that our model finds the correct generative model from the simulated data, we apply our model to neuroimaging data with fairly well-established neural pathways.

Original: Although the pathways are well-established, we do not know the exact ground truth communication underlying fMRI data.
Revised: Although the pathways are well-established, we do not know the exact ground truth communication underlying fMRI data.

Original: We do know that communication dynamics depend on the task it is trying to perform, even outside of explicit windows in which these tasks are performed.
Revised: We do know that communication dynamics depend on the task it is trying to perform, even outside of explicit windows in which these tasks are performed.

Original: Hence, to evaluate our model on fMRI data, we train it on three different tasks and use logistic regression to predict what task a small window of timesteps is from using the communication our model predicts.
Revised: Hence, to evaluate our model on fMRI data, we train it on three different tasks and use logistic regression to predict what task a small window of timesteps is from using the communication our model predicts.

Original: We compare our model to dFNC and show that our method is more sensitive to the task, especially for small windows where no explicit task is performed.
Revised: We compare our model to dFNC and show that our method is more sensitive to the task, especially for small windows where no explicit task is performed.

Original: This has implications for both resting-state fMRI and task fMRI, since our model may be able to provide a deeper look into how tasks shape macroscale communications in the brain.
Revised: This has implications for both resting-state fMRI and task fMRI, since our model may be able to provide a deeper look into how tasks shape macroscale communications in the brain.

Original: Understanding task responses allows us to more accurately understand their correlates to behavior and how communication in the brain shapes certain behavior.
Revised: Understanding task responses allows us to more accurately understand their correlates to behavior and how communication in the brain shapes certain behavior.

Original: Since our results indicate that our model is more sensitive to communications even at resting-state that are still related to a task.
Revised: Since our results indicate that our model is more sensitive to communications even at resting-state that are still related to a task.

Original: Being sensitive even outside of external driving of fMRI activity is impactful for resting-state fMRI and psychiatric disorder research, where we expect communication dynamics to be different between controls and patients.
Revised: Being sensitive even outside of external driving of fMRI activity is impactful for resting-state fMRI and psychiatric disorder research, where we expect communication dynamics to be different between controls and patients.

Original: Furthermore, since we establish our method on simulated data and then extend it to real data, we provide an initial framework for scientists that may want to extend our method into their area of expertise.
Revised: Furthermore, since we establish our method on simulated data and then extend it to real data, we provide an initial framework for scientists that may want to extend our method into their area of expertise.

Original:  M ETHOD
Revised:  M ETHOD

Original: Generative communication models A generative communication model should meet the following requirements: it should not be prone to noise, model the directionality of the communication, and allow for instantaneous communication dynamics.

Original: Regarding noise, macroscale communication in the brain is often assessed using fMRI, which is prone to oscillatory sources of noise such as respiration Lund et al.

Original: (2006) among others.

Original: Connectivity metrics based on correlations are more prone to oscillatory noise since they are global and thus affect the synchrony between fMRI signals in different regions at the same time Birn (2012).

Original: There are generally, however, many reasons why fMRI signals can exhibit similar activation patterns in multiple regions that may or may not always be explained by communication Birn (2012).

Original: In our model, we model the more global signal of each region separately from the communication dynamics.

Original: The general trend of the signal is modeled by some initial inputs, and the sparsity regularization on the communication helps push our model towards solutions that model fewer spurious correlations.

Original: Secondly, the symmetry of many connectivity metrics has been mentioned previously, and we solve this issue by having each region explicitly send directional communication to each other region.

Original: Thirdly, connectivity metrics often estimate the connectivity over a temporal interval.

Original: For example, a correlation between two brain regions is calculated over several timesteps T .

Original: This leads to a trade-off between temporal resolution, and accuracy of the estimated correlation coefficient.

Original: In fact, the maximum temporal resolution is two timesteps, since it is impossible to calculate the variance over less than two samples.

Original: Our method bypasses this issue by explicitly sending communication at each timestep.

Original: Thus, the temporal resolution of the communication we estimate is the same as the temporal resolution of the input signal.

Original: Formal definition of our model The data structure we assume in this work is a simple graph
Revised: Formal definition of our model The data structure we assume in this work is a simple graph

Original: G =
Revised: G =

Original: ( V, E ) , with N vertices ( V ), and M edges ( E ), as shown in Figure 1A.
Revised: ( V, E ) , with N vertices ( V ), and M edges ( E ), as shown in Figure 1A.

Original: Generally, we can assume a fully-connected graph, since our proposed model can learn to not send any communication over an edge, meaning it essentially ’trims’ that edge.
Revised: Generally, we can assume a fully-connected graph, since our proposed model can learn to not send any communication over an edge, meaning it essentially ’trims’ that edge.

Original: Each vertex v i has a corresponding timeseries x i 1 ...T with T timesteps, this timeseries is modeled as the following dynamical system.
Revised: Each vertex v i has a corresponding timeseries x i 1 ...T with T timesteps, this timeseries is modeled as the following dynamical system.

Original: [Equation]
Revised: [Equation]

Original: One recent model that has been used to model this dynamical system is LFADs Pandarinath et al.
Revised: One recent model that has been used to model this dynamical system is LFADs Pandarinath et al.

Original: (2018), where they model the external inputs u ( t ) as samples from a normal distribution parameterized by a controller that has access to the hidden state of the encoder and decoder.
Revised: (2018), where they model the external inputs u ( t ) as samples from a normal distribution parameterized by a controller that has access to the hidden state of the encoder and decoder.

Revised: The controller can be fairly opaque in what it is concretely modeling, however, if we have access to timeseries for multiple nodes, we can replace the input from the controller with input coming from the neighbors of a node.

Original: We want to incorporate a more explicit inductive bias in our model, however.
Revised: This incorporates a more explicit inductive bias in our model.

Original: In our case, the external inputs for vertex v i are communications c
Revised: Formally, the external inputs for vertex v i are communications c

Original: → i 1 ...T and external events e ( t ) .
Revised: → i 1 ...T and external events e ( t ) .

Original: We use the notation → i to mean all incoming communication from neighbors of vertex v i , i → to mean all outgoing communication from vertex v i to its neighbors, and j → i to mean communication from vertex v j specifically to vertex v i .
Revised: We use the notation → i to mean all incoming communication from neighbors of vertex v i , i → to mean all outgoing communication from vertex v i to its neighbors, and j → i to mean communication from vertex v j specifically to vertex v i .

Original: The external events can be known external perturbations of the dynamical system, but are not modeled using a controller like in LFADs Pandarinath et al.
Revised: The external events are known external perturbations of the dynamical system, such as the block design of a task.

Original: We model the non-linear function F i for each vertex separately, using a recurrent neural network (RNN), with initial inputs z i 0 .
Revised: To model the dynamical system of each vertex separately, we train a recurrent neural network (RNN), which we call the decoder, to find the non-linear function ( F iθ ) that describes its dynamics. First, we consider the case where the model receives no inputs, this is a transient dynamical system, that only depends on its initial state z i 0 .

Original: We find the initial inputs to the dynamical systems using an encoder, with the following probability distributionlog p ( z i | x i 1 ...T ) =log N (cid:0)
Revised: We find the initial inputs to the dynamical systems using an encoder, with the following probability distribution p ( z i | x i 1 ...T ) =

Original:  z i ; µ i 0 , σ i 0 (cid:1) .
Revised: N (cid:0) z i ; µ i 0 , σ i 0 (cid:1) .

Original: This distribution is parameterized using a bi-directional RNN, and we sample the initial distributions from it z 0 ∼ p ( z i | x i 1 ...T ) .
Revised: This distribution is parameterized using a bi-directional RNN (the encoder), and we sample the initial state from that distribution z 0 ∼ p ( z i | x i 1 ...T ) .

Original: Together with the decoder p (ˆ x i 1 ...T | z i 0 , c → i 1 ...T , e 1 ...T ) , which is parameterized as an RNN as well, the encoderdecoder pair forms a sequential variational autoencoder (S-VAE)
Revised: The encoder and decoder together form a sequential variational autoencoder (S-VAE)

Original: Zhao et al.
Revised: Zhao et al.

Original: ; Kingma &
Revised: ; Kingma &

Original: Welling (2013), except we have added two extra inputs, namely the communication from other vertices, and external events.
Revised: Welling (2013).

Original: The S-VAE is visualized in Figure 1B.
Revised: The S-VAE is visualized in Figure 1B.

Revised: The decoder can be expressed as parameterizing the following distribution p

Revised: (ˆ x i 1 ...

Revised: T | z i 0 ) , which allows us to train it using the evidence lower bound (ELBO)

Revised: Kingma & Welling (2013).

Revised: We extend this model by adding two types of inputs, namely the communication from other vertices, and external events.

Original: Lastly, the communication from one vertex to another is also modeled as a recurrent neural network, with c i as a random variable.
Revised: The communication from one vertex to another is also modeled as a recurrent neural network, with c i as a sample from a Laplace distribution.

Original: log p ( c i  | x i 1 ...t  ) =
Revised: p ( c i toj t | x i 1 ...t − 1 ) =

Original: log N (cid:16) c i  ; µ i → j t , σ i → j t (cid:17) .
Revised: L (cid:16) c i → j t ; µ i → j t , σ i → j t (cid:17) .

Original: Each communication from vertex i to vertex j  is thus sampled from this distribution c i → j  ∼ p ( c i  | x i 1 ...t ) .
Revised: Communication from vertex i to vertex j at each timestep t is thus sampled from the following Laplace distribution c i → j t ∼ p ( c i → j t | x i 1 ...t ) .

Original: This allows us to regularize the  communicationdistribution between each vertex to be a zero-mean unit-variance multi-dimensional Gaussian, as with variational autoencoders Kingma & Welling (2013).
Revised: Modeling the communication timeseries between vertices as distributions for each timestep allows us to regularize the communication. To encourage sparse communication between brain regions, we minimize the KL-divergence between the communication timeseries and a zero-mean unit-variance multi-dimensional Laplace distribution during training. The communication network is visualized in Figure 1C. Note that communication from vertex i to vertex j computed at time t arrives at the decoder at time t + 1 to ensure the model is temporally causal. Thus, the decoder now models the following distribution p (ˆ x i 1 ...

Revised: T |

Original:  The communication network is visualized in Figure 1C.
Revised: z i 0 , c → i 1 ...T − 1 , e 1 ...T ) , where e 1 ...T are the external events. In our model, we assume that external events only affect one brain region, after which the signal propagates to other brain regions through communication. The external events are represented as one when the external event is active and zero when it is inactive. Our model trains a vector for each event type.

Revised: Each entry in the vector corresponds to the probability that an event should be inserted as an external event in that brain region.

Revised: The vector parameterizes a Gumbel-Softmax distribution Jang et al.

Revised: (2016), which we sample from to determine the index of the brain region that gets the external event as input.

Original: Since each communication network predicts communication to another brain region independently, wealso add a router network that works similarly to the forget gate in a long shortterm memory (LSTM) network.
Revised: So far, each communication network predicts communication to another brain region independently. To allow the model to suppress information that has already been sent by other brain regions, we add a router network that is similar to the forget gate in a long short-term memory (LSTM) network

Revised: Hochreiter & Schmidhuber (1997).

Original: The controller receives all the incoming communication c
Revised: The router network receives all the incoming communication c

Original: → i t at timestep t , these are fed through a multi-layer perceptron (MLP), which has the same number of outputs as the size of the communication vector c → i t .
Revised: → i t at timestep t and this vector is fed through a multi-layer perceptron (MLP), which has the same number of outputs as the size of the communication vector c → i t .

Original: The output is transformed through a sigmoid to be between [0, 1] and multiplied with the incoming communication vector.
Revised: The output is transformed through a sigmoid to be between [0, 1] and multiplied with the incoming communication vector.

Original: This allows the router to block information  fromcertain a certain vertex that is already captured by information sent from other vertices.
Revised: This allows the router to block information sent from a certain vertex that is already captured by information sent from other vertices.

Original: The controller is visually represented in Figure 1A.
Revised: The router is visually represented in Figure 1A.

Original: With simplified notation, we thus have the following equations for the S-VAE.
Revised: With simplified notation, we thus have the following equations for the S-VAE.

Original: [Equation]
Revised: [Equation]

Original: Then, for the communication network we obtain the following equations.
Revised: Then, for the communication network we obtain the following equations.

Original: [Equation]
Revised: [Equation]

Original: Where σ ( · ) is the sigmoid function, and MLP i Comm ,µ and MLP i Comm ,σ share every layer, except the last one.intense
Revised: Where σ ( · ) is a sigmoid function, and MLP i Comm ,µ and MLP i Comm ,σ share every layer, except the last one.

Original: We can then adapt the evidence lower-bound (ELBO) Kingma & Welling (2013) as follows, where q iϕ (cid:0) z i 0 , c i → 1 ...T | x i 1 ...T (cid:1) refers to the approximate posterior, and p iθ (cid:0) x i 1 ...T | z i 0 , c
Revised: We can then adapt the evidence lower-bound (ELBO) Kingma & Welling (2013) as follows, where q iϕ (cid:0) z i 0 , c i → 1 ...T | x i 1 ...T (cid:1) refers to the approximate posterior, and p iθ (cid:0) x i 1 ...T | z i 0 , c

Original: → i 1 ...T , e 1 ...T (cid:1) is the generative network, in our case the recurrent decoder.
Revised: → i 1 ...T , e 1 ...T (cid:1) is the generative network, in our case the recurrent decoder.

Original: [Equation]
Revised: [Equation]

Original: Where p ( z ) and p ( c ) are the priors for the initial inputs and the communications and λ is the sparsity multiplier, respectively.
Revised: Where p ( z ) and p ( c ) are the priors for the initial inputs and the communications and λ is the sparsity multiplier, respectively.

Original: In our work, we take a zero-mean unit-variance multi-dimensional Gaussian distribution for the initial inputs and  a zero-mean 0 .
Revised: Specifically, the first row of the equation 8 refers to the KL-divergence between the initial hidden states z i 0 , learned by the encoder ( q iϕ ) and the prior p ( z ) , which is a zeromean one standard deviation Gaussian. The second row of the equation refers to the KL-divergence, weighted by a sparsity parameter λ , between the communication timeseries ( c i → 1 ,...,T ) and a prior p ( c ) , which is a zero-mean 0 .

Original: 1 standard deviation Laplacian for the communications.
Revised: 1 standard deviation Laplace distribution.

Original: This ensures that the modelsends communication sparsely to other regions.
Revised: This encourages the model to learn sparse communication timeseries between the vertices.

Revised: The final row of the equation refers to the expected log-likelihood, or the reconstruction, of the signal that is modeled by the decoder ( p iθ ) .

Revised: We assume that the fMRI signal for each region is normally distributed with a constant standard deviation, which means that optimizing the expected log-likelihood is equivalent to minimizing the mean-squared error.

Revised: In our work, we decide to minimize the mean-squared error because we found it to be more stable.

Original: Intuition and synthetic data To gain some intuition about our model, we explain how our model works on the simulated data we use in this work.
Revised: Intuition and synthetic data To gain some intuition about our model, we explain how our model works on the simulated data we use in this work.

Original: Throughout its training process, each region’s communication model will try to find temporal patterns in its region that can help reconstruct another region’s timeseries.

Original: We assume that ifthis is the case, namely that part of the signal is explained by a pattern in another region at a previous timestep,then communication could have occurred.
Revised: We assume that if part of the signal is explained by a pattern in another region at a previous timestep, communication could have occurred.

Original: A simple example isa communication between two regions, where the timeseries for each region is uniform noise.
Revised: A simple example is communication between two regions, where the timeseries for each region is uniform noise.

Original: Let us assume there is a linearly decaying pulse in the first region at a time T  Uniform (0 , 70) , that decays until T 2  T 1 + Uniform (30 , 50) .
Revised: Let us assume there is a linearly decaying pulse in the first region at a time T 1 ∼ Uniform (0 , 70) , that decays until T 2 ∼ T 1 + 20 .

Original: Once the pulse has decayed in the first region at time T 2 , a pulse starts in the second region that linearly decays at time T 3  T 2 + Uniform (30 , 50) .
Revised: Once the pulse has decayed in the first region at time T 2 , a pulse starts in the second region that linearly decays at time T 3 ∼ T 2 + 20 .

Original: These are exactly the parameters we use to create our synthetic data.
Revised: These are exactly the parameters we use to create our synthetic data.

Original: Now, if we only observe one of the two timeseries, the pulses will seem sampled from a uniform distribution to the decoder of that region.
Revised: Now, if we only observe one of the two timeseries, the pulses will seem sampled from a uniform distribution to the decoder of that region.

Original: However, if we have access to both regions at the same time and enough training data, our model should be able to figure out that the pulse in the first region structurally precedes the pulse in the second region.
Revised: However, if we have access to both regions at the same time and enough training data, our model should be able to figure out that the pulse in the first region structurally precedes the pulse in the second region.

Original: In our synthetic data, we add a third region that models a sinusoid and has a pulse after the second region’s pulse that linearly decays at time T 4 T 3 + 
Revised: In our synthetic data, we add a third region that models a sinusoid and has a pulse after the second region’s pulse that linearly decays at time T 4 T 3 + 20 .

Original: Uniform (30 , 50) . This region also has a different underlying signal, namely a randomly shifted sinusoid (shifted between Uniform ( − 20 , 20) ) to show that our model can separate global signal from communication.
Revised: This region also has a different underlying signal, namely a randomly shifted sinusoid (shifted between Uniform ( − 20 , 20) ) to show that our model can separate global signal from communication.

Original: A visual example of our synthetic data is shown in Figure 2A.
Revised: A visual example of our synthetic data is shown in Figure 2A.

Original: We expect that the initial inputs z 0 models the global signal so that the communication our model predicts is not affected by the sinusoidal signal.
Revised: We expect that the initial inputs z 0 models the global signal so that the communication our model predicts is not affected by the sinusoidal signal.

Original: As mentioned previously, fMRI data also exhibits global signal that may affect connectivity metrics, such as physiological noise.
Revised: As mentioned previously, fMRI data also exhibits global signals that may affect connectivity metrics, such as physiological noise.

Original: Hence, we can infer whether our model is affected by the global signal in our simulation data to make it more realistic when transferred to biological data.
Revised: Hence, we can infer whether our model is affected by the global signal in our simulation data to make it more realistic when transferred to biological data.

Original: The communication we thus expect is shown in Figure 2B.
Revised: The communication we thus expect is shown in Figure 2B.

Original: Namely, the communication model in region 1 should learn to send a signal to region 2 when the pulse in its region has almost decayed, and region 2’s communication model should learn to send a pulse to region 3 once its pulse has almost decayed.
Revised: Namely, the communication model in region 1 should learn to send a signal to region 2 when the pulse in its region has almost decayed, and region 2’s communication model should learn to send a pulse to region 3 once its pulse has almost decayed.

Original: The expected communications in our synthetic data are shown in Figure 2B.
Revised: The expected communications in our synthetic data are shown in Figure 2B.

Original: To evaluate whether our model finds the expected communication, we center a normal distribution at the start of the pulse for regions 2 and 3 .

Original: We then correlate the communication from region 1 to region 2 and region 2 to region 3 with that normal distribution for multiple shifts between − 20 and 2 .

Original: We take the maximum correlation and subtract two times the L1 sparsity of the other communications from it to obtain a measure of similarity, where higher is better.

Original: Biological data
Revised: Biological data

Original: The fMRI data we use in this work is obtained from the open-access, under data usage terms, HCP-1200 dataset Van Essen et al.
Revised: The fMRI data we use in this work is obtained from the open-access, under data usage terms, HCP-1200 dataset Van Essen et al.

Original: for all subjects with cortical surface timeseries data.
Revised: for all subjects with cortical surface time- series data.

Original: We use  three tasks in our work, the Motor, Emotion Processing, and Language Processing tasks.
Revised: We use resting-state fMRI (rs-fMRI) and three tasks in our work, the motor, emotion processing, and language processing tasks.

Original: The data is registered using multimodal surface registration (MSM) Robinson et al.
Revised: The data is registered using multimodal surface registration (MSM) Robinson et al.

Original: (2014; 2018), and surfaces are constructed using Freesurfer Glasser et al.
Revised: (2014; 2018), and surfaces are constructed using Freesurfer Glasser et al.

Original: ; Fischl (2012).
Revised: ; Fischl (2012).

Original: We use the Yeo-17 atlas Yeo et al.
Revised: We use the Yeo-17 atlas Yeo et al.

Original: (2011) to separate the cortical surface data into 17 separate regions.
Revised: (2011) to separate the cortical surface data into 17 separate regions.

Original: The spatial information within each region is averaged, as is commonly done in communication analyses of the brain.
Revised: The spatial information within each region is averaged, as is commonly done in communication analyses of the brain.

Original: Each subject’s timeseries is then independently band-pass filtered independently ( 0 .
Revised: Each subject’s timeseries is then independently band-pass filtered independently ( 0 .

Original: 01 − 0 .
Revised: 01 − 0 .

Original: 15 Hz) and linearly detrended using  Nilearn package Abraham et al.
Revised: 15 Hz) and linearly detrended using the Nilearn package Abraham et al.

Original: Each task has a different length, so we cut off tasks at 176 timesteps, which is the smallest common size of the timeseries.
Revised: Each task has a different length, so we cut off tasks at 176 timesteps, which is the smallest common size of the timeseries.

Revised: For the rs-fMRI data we use the first 200 timesteps to make the length similar to the task data.

Original: The motor task consists of a visual cue, after which a subject has to start performing a motor task.
Revised: The motor task consists of a visual cue, after which a subject has to start performing a motor task.

Original: The motor tasks a subject performs are related to the left hand, right hand, left foot, right foot, and tongue.
Revised: The motor tasks a subject performs are related to the left hand, right hand, left foot, right foot, and tongue.

Original: For the emotion task, the subject sees a visual cue and then has to match two shapes shown at the bottom of the screen to a shape at the top.
Revised: For the emotion task, the subject sees a visual cue and then has to match two shapes shown at the bottom of the screen to a shape at the top.

Original: This sub-task is referred to as the neutral task, and the other task has the subject match one of two faces at the bottom of the screen with a face shown at the top.
Revised: This sub-task is referred to as the neutral task, and the other task has the subject match one of two faces at the bottom of the screen with a face shown at the top.

Original: The faces either look fearful or angry.
Revised: The faces either look fearful or angry.

Original: The language task consists of a sub-task where the participant has to perform arithmetic and push a button for the right answer.
Revised: The language task consists of a sub-task where the participant has to perform arithmetic and push a button for the right answer.

Original: The story sub-task presents the subjects with a short story of 5-9 sentences.
Revised: The story sub-task presents the subjects with a short story of 5sentences.

Original: The timing of each task is shown, after convolving it with SPM’s hemodynamic response function Penny et al.
Revised: The timing of each task is shown, after convolving it with SPM’s hemodynamic response function Penny et al.

Original: (2011), in Appendix B. We train one model on all three tasks such that the prediction of the task is not influenced by the fact that the model was specifically trained to model communication for that task.
Revised: (2011), in Appendix B. We train one model on all three tasks such that the prediction of the task is not influenced by the fact that the model was specifically trained to model communication for that task.

Original: Implementation Our model is implemented in Pytorch Paszke et al.
Revised: Our model is implemented in Pytorch Paszke et al.

Original: and trained on an internal cluster using single NVIDIA V100 and NVIDIA 2080 GPUs, with a batch size of 8 , the
Revised: and trained on an internal cluster using single NVIDIA V100 and NVIDIA 2080 GPUs, with a batch size of 8 , the

Original: Adam optimizerKingma & Ba (2014), a 1E-4 weight decay, a learning rate of 0.01, 0 .
Revised: Adam optimizer Kingma & Ba (2014), a 1E-4 weight decay, a learning rate of 0.01, 0 .

Original: 1 epsilon, and 0 . 9 , 0 .
Revised: 1 epsilon, and 0 . 9 , 0 .

Original: 999 as betas.
Revised: 999 as betas.

Original: We reduce the learning rate when it plateaus using a scheduler.
Revised: We reduce the learning rate when it plateaus using a scheduler.

Original: At each plateau, we reduce the learning rate by multiplying it with 0 .
Revised: At each plateau, we reduce the learning rate by multiplying it with 0 .

Original: 95 , with a patience of 6 epochs, and a minimum learning rate of 1 E − 5 .
Revised: 95 , with a patience of 6 epochs, and a minimum learning rate of 1 E − 5 .

Original: We train each model for 200  epochs, for the simulated data across 8 seeds: ( 42 , 1337 , 9999 , 50 , 100 , 500 , 1000 , 5000 ).
Revised: We train each model for 200 epochs and perform early stopping when the loss has not improved for 50 epochs, for the simulated data across 8 seeds: ( 42 , 1337 , 9999 , 50 , 100 , 500 , 1000 , 5000 ).

Original: The model is trained on the first three seeds for the biological data, this is due to computational limitations.
Revised: The model is trained on the first three seeds for the biological data, this is due to computational limitations.

Original: Since we use multiple sparsities, the number of runs quickly explodes, and each model on biological data takes about 5 − 6 hours to train.
Revised: Since we use multiple sparsities, the number of runs quickly explodes, and each model on biological data takes about 7 − 8 : 30 hours to train.

Original: All of the code will be made publicly available after the double-blind review has concluded on GitHub.
Revised: All of the code will be made publicly available after the double-blind review has concluded on GitHub.

Original: The hidden size we use for the encoder, decoder, and communication model are all 16, and the latent dimensions for z 0 is 32 .
Revised: The hidden size we use for the encoder, decoder, and communication model are all 16 .

Original: For the synthetic data we use sparsities: ( 0 . 001 , 0 .025 , 0 .
Revised: For the synthetic data we use sparsities: ( 0 . 001 , 0 .

Original: For the real data we use a smaller number of sparsities due to computation limitations, but also because we have narrowed the sparsity down at this point to: ( 0 . 001 , 0 . 005 , 0 . 01 , 0 .
Revised: For the real data we use a smaller number of sparsities due to computation limitations, but also because we have narrowed the sparsity down at this point to: ( 0 . 001 , 0 . 005 , 0 . 01 , 0 .

Original: In order to not mix subjects across the training, validation, and test set, we split them based on subject.
Revised: To not mix subjects across the training, validation, and test set, we split them based on the subjects.

Original: Hence, a subject occurs with every task in one of the sets it was assigned to and in no other sets.
Revised: Hence, a subject occurs with every task in one of the sets it was assigned to and in no other sets.

Revised: Data is z-scored across the temporal dimension of the data for each subject independently.

Original: Experiments To test whether our model is interesting as a complement to connectivity metrics, we evaluate our model in three different ways.
Revised: Experiments To test whether our model is interesting as a complement to connectivity metrics, we evaluate our model in three different ways.

Original: To get a better idea of how much the controller improves our model, we perform each experiment with both models.
Revised: To get a better idea of how much the controller improves our model, we perform each experiment with both models.

Original: We refer to the model without the controller as CommsVAE and the model with the controller as Meta-CommsVAE in the rest of the text.
Revised: We refer to the model without the controller as CommsVAE and the model with the controller as Meta-CommsVAE in the rest of the text.

Original: Firstly, we evaluate our model on the aforementioned synthetic data.
Revised: Firstly, we evaluate our model on the aforementioned synthetic data.

Original: We calculate the correspondence between the expected communications and the communication that our model predicts on a test set and look at how the sparsity parameter affects the average correspondence to the test set.
Revised: We calculate the correspondence between the expected communications and the communication that our model predicts on a test set and look at how the sparsity parameter affects the average correspondence to the test set.

Revised: The correspondence is calculated by convolving a normal distribution over an impulse response centered on the start of the event in that brain region.

Revised: Then we correlate the communication between the resulting signal and the communication to that brain region for different shifts of the communication signal.

Revised: Finally, we sum the maximum absolute correlations for R 1 − > R 2 and

Revised: R 2 − > R 3 , and subtract the sum of two times the average absolute communication for all other brain regions.

Revised: This corresponds to expecting zero communication in all other brain regions, and a spike right before the start of the actual event.

Original:  Note that when we usethe communication in these experiments, we take the mean of the Gaussian  distribution, since it is the most probably point.
Revised: By using shifts to calculate the correlation we do not make assumptions about the lag that is learned between the communication RNN and the decoder. Note that when we use communication in these experiments, we take the mean of the Gaussian or Laplace distribution, since it is the most probable sample under that distribution.

Original: Then, we train both our models on an fMRI dataset with three tasks, each model is trained on all three tasks conjointly not to induce any task-specific model noise that can affect the downstream prediction.
Revised: Then, we train both our models on an fMRI dataset with three tasks, each model is trained on all three tasks conjointly not to induce any task-specific model noise that can affect the downstream prediction.

Original: Since we do not have a ground truth for the communication of the fMRI signal, we determine the best model by looking at the reconstruction error on the validation set with respect to the sparsity, see
Revised: Since we do not have a ground truth for the communication of the fMRI signal, we determine the best model by looking at the reconstruction error on the validation set with respect to the sparsity, see

Original: Appendix A. We use the model that performs the best on average over all the seedsfor a certain sparsity parameter and uses it for task prediction.
Revised: Appendix A. We use the model that performs the best across the seeds and use it for task prediction.

Original: We use 4 different window sizes [ 5 , 10 , 20 , and 50 ] to compare our model to dFNC.
Revised: We use 4 different window sizes [ 5 , 10 , 20 , and 50 ] to compare our model to dFNC, Granger causality Granger (1969), transfer entropy Lobier et al. (2014), multiplication temporal derivatives Shine et al. (2015), and the full fMRI signal in the window for each region.

Original: This allows us to determine whether our model has better temporal resolution since the quality of the correlation will  get lower with a smaller dNFC window.
Revised: This allows us to determine whether our model has better temporal resolution since the quality of the correlation will likely get lower with a smaller dNFC window.

Original: For each window, we take the absolute value of our model’s communication (since it can be both negative and positive) and sum it across each window.
Revised: For each window, we take the absolute value of our model’s communication (since it can be both negative and positive) and sum it across each window.

Original: Then, we train a logistic regression classifierwith L1 sparsity on the training and validation set using the temporally summed communication to predict what task each subject in the test set’s window is from.
Revised: Then, we train a logistic regression classifier on the training and validation set using the temporally summed communication to predict what task each subject in the test set’s window is from.

Original: This results in a classification accuracy across time for the four windows, and we perform the same routine for the dFNC.
Revised: This results in a classification accuracy across time for the four windows, and we perform the same routine for the connectivity metric comparisons.

Original: Finally, we look at the absolute sum of the communication  for each task. We average the absolute temporal sum over the subjects in the test set and visualizea graph of the communication that exhibit more than 25% of the maximum sum for each task.
Revised: Finally, we look at the absolute correlation between the communication timeseries for each directed edge and the sub-tasks, we average the correlation across seeds and the subjects in the test set. We visualize the averaged correlations that are higher than 80% of the maximum value or higher than 0 . 2 , whichever value is higher.

Original: The reason we need to take the absolute sum is that the model can send both positive and negative communication.
Revised: This experiment allows us to see whether our method sends biologically realistic compared to a ground-truth.

Original:  R ESULTS
Revised:  R ESULTS

Original: Synthetic data
Revised: Synthetic data

Original: The results for the synthetic data are shown in Figure 3.
Revised: The results for the synthetic data are shown in Figure 3.

Original: Generally, we see a trend where higher sparsity parameters increase the correspondence to our expected communication on the test set, up to a point where increased sparsity suppresses any communication.
Revised: Generally, we see a trend where higher sparsity parameters increase the correspondence to our expected communication on the test set, up to a point where increased sparsity suppresses any communication.

Original: There is a peak around 0 .
Revised: There is a peak around 0 .

Original: 01 in Figure 3A for both models, after which the correspondence with the expected communication decreases.
Revised: 01 in Figure 3A for both models, after which the correspondence with the expected communication decreases.

Original: This exactly supports our hypothesis that encouraging sparse communication helps.
Revised: This exactly supports our hypothesis that encouraging sparse communication helps.

Original: To show what the communication looks like in the two models we test, we take the test set synthetic data, predict the communication for each, and use the synthetic generation parameters to align the communication timeseries.
Revised: To show what the communication looks like in the two models we test, we take the test set synthetic data, predict the communication for each, and use the synthetic generation parameters to align the communication timeseries.

Original: After aligning the communication timeseries, we take the average over the test set  and visualize what the communication looks like in Figure 3B.
Revised: After aligning the communication timeseries, we take the average over the test set for the best seeds for each model at 0 . 01 sparsity from Figure 3A and visualize what the communication looks like in Figure 3B.

Original: Right before the pulse ends in the region 1 , it communicates with region 2 before its pulse begins.
Revised: Right before the pulse ends in the region 1 , it communicates with region 2 before its pulse begins.

Original: The communication from region 2 to region 3 starts exactly when the pulse is almost linearly decayed, right before the pulse starts in region 3 .
Revised: The communication from region 2 to region 3 starts exactly when the pulse is almost linearly decayed, right before the pulse starts in region 3 .

Original: The main difference between our Meta-CommsVAE and CommsVAE model is that in Figure 3B, the CommsVAE’s communications between region 1 and region 2 are closer to the actual pulse.
Revised: The main difference between our Meta-CommsVAE and CommsVAE model is that in Figure 3B, the CommsVAE’s communications between region 1 and region 2 are closer to the actual pulse.

Original: Generally, we do not see a trend in terms of a difference between the two CommsVAE models, at least for the synthetic data.
Revised: Generally, we do not see a trend in terms of a difference between the two CommsVAE models, at least for the synthetic data.

Original: This is probably because the task itself is sparse and hence being able to increase sparsity using the controller is unnecessary.
Revised: This is probably because the task itself is sparse and hence being able to increase sparsity using the controller is unnecessary.

Original: An example of the reconstruction of validation data by our model is shown in Appendix C.  fMRI data
Revised: An example of the reconstruction of validation data by our model is shown in Appendix C.  fMRI data

Original: The results to test both the specificity of our generative model and how instantaneous the communication is are shown in Figure 4.
Revised: The results to test both the specificity of our generative model and how instantaneous the communication is are shown in Figure 4.

Original: We produce these results for the best model we find based on the reconstruction error on the validation set, see Appendix A.
Revised: We produce these results for the best model we find based on the reconstruction error on the validation set, see Appendix A.

Original: The best model is the MetaCommsVAE model with a sparsity of 0 . 
Revised: The best model is the MetaCommsVAE model with a sparsity of 0 . 05 .

Original: For thenext two results, we will use this model.
Revised: However, the standard deviation over the reconstruction is higher than for 0 . 01 , and since the difference in the average is negligible between the two sparsities, we select 0 . 01 for visualizations and the next two results because it seems more stable.

Original: Our results with this model clearly show that our method outperforms dynamic functional connectivity (dFNC) in terms of predicting what task each window is from.
Revised: Our results with this model clearly show that our method outperforms all of the connectivity metrics we compare it to, and even the fMRI data itself. This indicates that our model finds representations of the tasks in the rs-fMRI data that are intuitive and not linearly separable initially.

Original: This is true for every window, and every window size.
Revised: This is true for every window, and every window size.

Revised: Note that both Granger causality and transfer entropy perform so poorly in terms of classification accuracy, is likely because they are inaccurate when fit in small windows.

Original: Furthermore, our method only has a small decline in classification accuracy for smaller window sizes, which supports our hypothesis that our model can find instantaneous communication.
Revised: However, our method only has a small decline in classification accuracy for smaller window sizes, which supports our hypothesis that our model can find instantaneous communication.

Original: Our method already starts being task-specific early on in time, meaning that even before any of the tasks start, the communication our model finds in the brain is task-specific, which is likely to be true if we knew the ground truth of communication.
Revised: Our method already starts being task-specific early on in time, meaning that even before any of the tasks start, the communication our model finds in the brain is task-specific, which is likely to be true if we knew the ground truth of communication.

Original: One reason thatthe model increases in task accuracy  is that the communication modelis an RNN and only has access to points previously, so  it does not have enough information to start sending information earlier on in the timeseries.
Revised: One reason that task accuracy increases over time is that the communication model only has access to points previously, so early on in the timeseries, it does not have enough information to start sending high-quality communication.

Original: After a few windows, the accuracy of the task increases and is then relatively stable.
Revised: After a few windows, the accuracy of the task increases and is then relatively stable.

Original: This is different for the dFNC, which fluctuates, likely based on when the tasks start, but our model is specific to the task  essentially the same across all of the windows in the timeseries.
Revised: This is different for the dFNC, which fluctuates, likely based on when the tasks start, but our model is specific to the task and essentially the same across all of the windows in the timeseries.

Original: Thus, our model is even task-specific when there is relatively little task signal from either of the tasks in a certain window.
Revised: Thus, our model is even task-specific when there is relatively little task signal from either of the tasks in a certain window.

Original: Another result we can gather from Figure 4 is that the lower-triangular part of the matrix is worse at predicting the task than the upper triangular part of the communication matrix for each window.
Revised: Another result we can gather from Figure 4 is that the lower-triangular part of the matrix is worse at predicting the task than the upper-triangular part of the communication matrix for each window.

Original: The reason we include these results is that the dFNC is symmetric and the lower and upper triangular parts of the matrix are thus the same.
Revised: The reason we include these results is that the dFNC is symmetric and the lower and upper triangular parts of the matrix are thus the same.

Original: To make the comparison fair, we compare separately compare the triangular parts of our communication matrix with dFNC.
Revised: To make the comparison fair, we compare separately compare the triangular parts of our communication matrix with dFNC.

Original: Since our method is directed, we can also take the whole matrix, which outperforms both the lower and upper triangular parts of that matrix, as expected.
Revised: Since our method is directed, we can also take the whole matrix, which outperforms both the lower and upper triangular parts of that matrix, as expected.

Original: Our last experiment aims to evaluate whether our modelfinds explicitly directed communication  that is different across the tasks.
Revised: Our last experiment aims to evaluate whether our model communication between regions that aligns with the sub-tasks in the data.

Revised: We use naming conventions for these regions from Kahali et al.

Original: The hemodynamic response model over time for each sub-task is shown in Appendix B.
Revised: The hemodynamic response model over time for each sub-task is shown in Appendix B.

Original: The absolute sum over the timeseries for each task is shown in Figure 5. There are a few interesting results that align with known scientific processes. For example, in all models, the communication from the earlier visual areas (V-A) is stronger towards thelater visual areas (V-B).
Revised: The average correlation across subject and seeds is shown in each of the sub-tasks, and color of the title indicates whether it is a sub-task of the motor task (blue), emotion task (red), or language task (orange). Two of the most easily verifiable correlations to sub-tasks are the visual cues in the bottom left of Figure 5.

Original: Furthermore, the communication from  early visual areas (V-A) tothe somatomotor regions (SM-B) is the strongest in the motor task.
Revised: The edges from the early visual area (V-A) to somatomotor B (SM-B) and limbic A (L-A) are correlated to the hemodynamic response of the visual cue.

Original: Since the  motor sub-tasks are preceded by a visual cue, it is expected that theearly visual areas send information to the somatomotor areas, since those have to execute the task. Another interesting result is that one of the control regions (C-A) in the motor task is an important node in sending information to other brain regions.
Revised: This is a convincing result because of the nature of the task, namely motor execution after the visual cue. Motor execution has been shown to be related to both the somatomotor region as well as the limbic region Mogenson et al.

Original: Generally, most regions only send to other regions, without receiving any communications from those regions, meaning that our model finds explicitly directed communication.This aligns with our hypothesis that generative models should be directed.
Revised: Although the latter is relatively understudied, our results indicate a potentially fruitful direction when related to communication.

Original: This is especially obvious for the emotion task, where communication from the early visual areas (V-A) to other regions is the most dominant.
Revised: Furthermore, the visual cue for the emotion task is related to communication from the early visual area (V-A) to control A (C-A), which is a cognitive control area.

Original: The emotion task also contains a visual cue. The visual communication in the language task is less strong compared to the other regions, and there are many other communication dynamics, but since the math sub-task is partly visual, we do expect some of the visual communication that we observe in the figure. One of  thelarge differences between the motor and emotion, and language task is that the limbic system is much more involved in the language task. We would potentially expect some limbic communication in the emotion task as well since one of the sub-tasks pertains to seeing angry/fearful faces.
Revised: Given that the emotion task requires the participant to match a shape (neutral) or a face (fear) to a target in the top of a screen, the communication from the visual area to the cognitive control area also aligns with biological intuition.

Original: The limbic system composes the hippocampus and amygdala, which are involved in the storage of memories and regulating emotional memories, respectively. The reason that the language task could evoke communication from the limbic system is thus that the stories evoke some emotional memories in the subjects. The stories are from Aesop’s fables and may have modulated some emotional memories in the subjects.
Revised: Other notable communication occurs towards two default mode networks (D-A, D-B).

Original: However, inferences like these need to be studied more robustly in future work.
Revised: This aligns with previous research, which indicates sub-networks of the default mode network are coupled with language networks Gordon et al.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: This work proposes a generative model for macroscale communication in the brain that we show predicts directed, instantaneous, and task-specific communication dynamics.
Revised: This work proposes a generative model for macroscale communication in the brain that we show predicts directed, instantaneous, and task-specific communication dynamics.

Original: We compare our model on a synthetic task that tests the requirements we set out for our model and find that our model can accurately uncover the communication dynamics we incorporate into the synthetic data.
Revised: We compare our model on a synthetic task that tests the requirements we set out for our model and find that our model can accurately uncover the communication dynamics we incorporate into the synthetic data.

Original: Then, we train our two models on three fMRI tasks and find that reconstruction error decreases with higher sparsity parameters, indicating that the sparse prior on  brain communication dynamics fits our data.
Revised: Then, we train our two models on three fMRI tasks and find that reconstruction error decreases with higher sparsity parameters, indicating that the sparse prior on the brain communication dynamics fits our data.

Original: The sparsity of macroscale brain communication needs to be studied further but aligns with findings in computational neuroscience Olshausen & Field (2004) and the design principles of the brain Sterling & Laughlin (2015).
Revised: The sparsity of macroscale brain communication needs to be studied further but aligns with findings in computational neuroscience Olshausen & Field (2004) and the design principles of the brain Sterling & Laughlin (2015).

Original: We usea model with the lowest reconstruction error  to predict what task small time windows are from in a test set and compare the classification accuracy with a  commonly usedmetric, dynamic functional connectivity (dFNC).
Revised: We use the best model based on reconstruction error over seeds to predict what task small time windows are from in a test set and compare the classification accuracy with a multiple commonly used connectivity metrics, both directed, and undirected.

Original: Our model performs much better with smaller window sizes, supporting our claim that our model predicts instantaneous communication.
Revised: The performance of our model does not decrease as drastically with smaller window sizes, supporting our claim that our model predicts instantaneous communication.

Original: Then, to understand the added benefit of directed models, we look at the absolute sum of the communication for each task and find that almost all, especially strong, connections  are directed.
Revised: Then, to understand how communication is related to sub-tasks, we calculate the correlation between each sub-task and the communication timeseries of each edge. After averaging over test subjects, and seeds, we visualize the highest sub-task correlations. Almost all of the strong connections we find are directed, with correlations up to 0 .

Original: The directions and the strength of the communication also align with neuroscientific priors we have about how regions affect each other.
Revised: The directions and the strength of the communication also align with neuroscientific priors we have about how regions affect each other.

Original: Thus, our results confirm what we set out to incorporate into our model from the beginning.
Revised: Thus, our results confirm what we set out to incorporate into our model from the beginning.

Original: We want to expand on this model by applying it to resting-state fMRI data, and explore whether our model can be used to more deeply understand psychiatric disorders.
Revised: We want to expand on this model by applying it to resting-state fMRI data, and explore whether our model can be used to more deeply understand psychiatric disorders.

Revised: As an initial step, we have looked at how well our model can reconstruct 200 timesteps of resting-state fMRI data in Appendix

Original:  Furthermore, since this model is generative, we can start to probe the model and interpret the response functions when we synthetically increase communication from one region to another at certain time points.
Revised: G. Furthermore, since this model is generative, we can start to probe the model and interpret the response functions when we synthetically increase communication from one region to another at certain time points.

Original: We can design communication patterns to test how communication in different regions affects the signal in others.
Revised: We can design communication patterns to test how communication in different regions affects the signal in others.

Original:  R EPRODUCIBILITY
Revised:  R EPRODUCIBILITY

Original: By making our code available after the double-blind process has concluded, we hope that our work can be reproduced and inspected by other scientists.
Revised: By making our code available after the double-blind process has concluded we hope that our work can be reproduced and inspected by other scientists.

Original: Most of the parameters necessary for reproduction are mentioned in the experiments section, and a longer description of the formulas and math behind the model is provided in the methods section.
Revised: Most of the parameters necessary for reproduction are mentioned in the experiments section and a longer description of the formulas and math behind the model is provided in the methods section.

Original: We have explicitly added a section on the intuition behind the model so people can get a mental grasp of how the model works and that hopefully also helps people reproduce our work.
Revised: We have explicitly added a section on the intuition behind the model so people can get a mental grasp of how the model works and that hopefully also helps people reproduce our work

Original: A  F MRI DATA RECONSTRUCTION ERROR ACROSS SPARSITY
Revised: A T ASK F MRI DATA RECONSTRUCTION ERROR ACROSS SPARSITY

Original: Figure 6 shows that the Meta-CommsVAE outperforms the CommsVAE on all sparsities, but also has a clear dip around a certain sparsity.
Revised: Figure 6 shows that the Meta-CommsVAE outperforms the CommsVAE on all sparsities, but also has a clear dip around a certain sparsity.

Original: Since we do not know the ground truth of the communication in the fMRI dataset like we did with the synthetic data, the improvement in reconstruction error for higher sparsities shows promise for hyperparameter searches on other datasets.
Revised: Since we do not know the ground truth of the communication in the fMRI dataset as we did with the synthetic data, the improvement in reconstruction error for higher sparsities shows promise for hyperparameter searches on other datasets.

Original: Note the reconstruction error is a squared error, summed over time and averaged over the number of regions.
Revised: Note the reconstruction error is a squared error, summed over time and averaged over the number of regions.

Original: B T ASKS HEMODYNAMIC RESPONSE
Revised: B T ASKS HEMODYNAMIC RESPONSE

Original: Figure 7 shows the hemodynamic response for each of the tasks for the 176 timesteps we use in our model.
Revised: Figure 7 shows the hemodynamic response for each of the tasks for the 176 timesteps we use in our model.

Original: The legend indicates what sub-task the hemodynamic response corresponds to.
Revised: The legend indicates what sub-task the hemodynamic response corresponds to.

Original: C E XAMPLE RECONSTRUCTION OF THE SYNTHETIC DATA
Revised: C E XAMPLE RECONSTRUCTION OF THE SYNTHETIC DATA

Original: Figure 8 shows an example of the reconstruction of the synthetic data.
Revised: Figure 8 shows an example of the reconstruction of the synthetic data.

Original: The red line is the input signal, and the blue line is the reconstruction.
Revised: The red line is the input signal, and the blue line is the reconstruction.

Original: The first row is the first region, the second row  the second region, and the last row  the third region.
Revised: The first row is the first region, the second row is the second region, and the last row is the third region.

Original: The columns are four random timeseries from the validation set.
Revised: The columns are four random timeseries from the validation set.

Original: This shows that our model can still model the global signal, even for a shifted sinusoid, with almost perfect communication dynamics, see Figure 3, 
Revised: This shows that our model can still model the global signal, even for a shifted sinusoid, with almost perfect communication dynamics, see Figure 3, D S CALING

Revised: Our model in its current form is applied to data with less than twenty regions.

Revised: The scaling of the model to more regions is shown in Figure 9 and indicates fairly linear scaling for the inference time, and exponential scaling in the parameters.

Revised: E A BLATION : NO COMMUNICATION

Revised: As a way to further our understanding of the types of signal that are modeled using the communication, we compare a model trained without communication to a model trained with communication in Figure 10.

Revised: F I NDIVIDUAL TRIALS FOR SIMULATED DATA

Revised: The individual trials of our model (sparsity=0.01) on the test set.

Revised: Figure 11A shows a zoomed-in version for the first 50 test trials, and Figure 11B shows the overview of all trials.

Revised: Each trial exhibits the expected behavior, namely communication right at the end or atleast within the window between

Revised: T1 and T2, and T2 and T3, meaning between the event in the region that is sending and the region that is receiving.

Revised: G R ESTING - STATE F MRI

Revised: We train our model with a larger hidden size ( 128 ) on resting-state fMRI (rs-fMRI) data to ensure that sparsity also increases the reconstruction performance with higher sparsities on data that is not externally structured.

Revised: We find a similar pattern as with the task data, where higher sparsities improve the reconstruction error of the model in Figure 12 and 13.

Revised: To make sure the communication between brain regions is sparse, especially for rs-fMRI data, we also visualize the first four brain regions for two subjects in the validation set, see Figure 14.