Paper: Leveraging Low-Rank and Sparse Recurrent Connectivity for Robust Closed-Loop Control (ICLR-2024)

Reviewer comment (Theory):
The analysis focuses on the spectral properties of the connectivity matrix. While this is a reasonable choice for understanding the impact on low-rank and sparsity for a single architecture, it seems limited when it comes to comparing different architectures. Ideally, one would need to study the spectral properties of the recurrent Jacobian. The recurrent Jacobian would heavily depend on the recurrence connectivity matrix for all architectures, but important differences might still remain. The paper is currently missing a discussion of this point.

Author response:
We completely agree and note that Reviewer 59d7 shared a similar sentiment. To address this, we perform an analysis on the recurrent Jacobians with can be found in Section 4.2 in which we compute the norm of the gradients backwards through time in order to understand how the network’s recurrent attention evolves as a function of the amount of time that has passed since it observed a given input. We are able to show that the spectral radius is indeed a viable proxy for the rate at which the gradient evolves across time across models, recurrent ranks and recurrent sparsities, which is reassuring for the strength of the proposed parameterization as a modulator of recurrent memory.

Reviewer comment (Experiments):
It would be great to know which kind of distribution shift is used in the experiments. This information could not be found in the paper.

Author response:
This information can be found in Appendix A.11 in the evaluation metrics section. We employ canonical distribution shifts such as adding random noise and brightening/darkening/blurring input images.

Reviewer comment (Novelty):
The work of Herbert Jaeger or Wolfgang Maass is probably a better reference for echo state networks than the Deep Learning Book of Goodfellow et al.

Author response:
Agreed, this has been corrected.

Reviewer comment (Writing):
The paper is very difficult to read—while the paper provides some high-level intuition for particular quantities (e.g. low sparsity corresponding to the rate of vanishing gradient), there is a big gap between every subsection. It is often difficult to make the connections between the results. Furthermore, even with the large text of paragraphs, often the most important information is ambiguous.

Author response:
We tend to agree with the reviewer that the previous draft of the paper was difficult to parse given the gaps between sections. We have refactored the paper structure and rewritten sentences such that the flow of the story is more coherent and each of the analyses are more thoroughly motivated. Please refer to the comment addressed to all the reviewers to see both the structural and experimental changes we have made.

Reviewer comment (Theory):
Theorem 4.1 seems to relate the weight matrices based on particular initialization schemes. In figure 4, the paper appears to agree with theorem 4.1, but it is unclear why this theorem needs to be in the main paper as it does not explain any insight on the generalization/robustness directly.

Author response:
Theorem 1 proves properties (for some subset of cases presented in the paper) regarding how the spectral norm and radius of the recurrent weights change as a function of rank and sparsity. We motivate the relevance of these with respect to robustness more thoroughly in Sections 3 and A.5.2 but summarize them here. Spectral norm measures the magnitude of the expansion induced by a weight matrix and thus measured the extent to which a perturbation effects the hidden state (or input state in the case of the input weight matrix). Spectral radius acts as a proxy for the network's attention span across time and is thus a measure for how far into the future a perturbation will affect the system. Since we can modulate both via rank and sparsity, we maintain that this Theorem 1 is pertinent to the paper as it gives us provability regarding the efficacy of the parameterization.

Reviewer comment (Presentation):
The scale of the subplots should be consistent for easier comparison (e.g. figures 4, 5, and 6).

Author response:
The scale of the subplots is fixed in Figure 4a. In Figure 4b, we use the same scale horizontally but different scales vertically because we want the reader to be able to visualize the changes across ranks and sparsities in LSTMs (as well as see the SE bars) which would otherwise be squashed if we placed it on the same scale as the CfCs. Analogous logic was used for making the scales for Figure 5b. However, if the reviewer maintains that readability would be improved by standardizing the scales even in these cases, we are happy to make the modification.

Reviewer comment (Theory):
The proof of theorem 4.1 is unclear. While the paper uses random matrix theory to obtain results on the spectral radius and spectral norm based on the sparsity, it is not clear how the statements for rank \( r \) for both initializations are obtained.

Author response:
Apologies, this was a typo in the theorem statement that has been corrected. You are correct, we do not prove the trend regarding the spectral radius as a function of rank in the Glorot uniform case and have instead added an empirical analysis to A.6.

Reviewer comment (Evaluation):
In figure 5, the paper only indicates the variance of the data captured by the top 5 principal components (PCs), losing the information on the remaining variance captured by the proceeding PCs. As a result, using only the top 5 PCs to claim higher/lower effective dimensionality seems incorrect. For example, comparing two methods \( M_1 \) and \( M_2 \), if \( M_1 \)'s top 5 PCs capture 0.7 explained variance while \( M_2 \)'s capture 0.5, we cannot guarantee that \( M_1 \) is always capturing more variance as we increase the number of PCs.

Author response:
Using the top 5 PCs in particular was an arbitrary choice in order to analyze dimensionality. We agree that in order to be precise one should analyze the full set of PCs. We added a plot of this to Appendix A.8 (Figure 26) which shows that the conclusions made using only 5 PCs still hold: namely that CfCs have state space trajectories with lower dimensionality than LSTMs and RNNs.

Reviewer comment (Novelty):
Already existing analysis and tasks.

Author response:
We thank the reviewer for the thorough review of our work. We have incorporated many of your suggestions into the revised draft and believe that it has substantially improved the paper. For a summarization of the modifications we have made to the paper, please refer to the comment addressed to all of the reviewers. We respond to your specific points below.

1. Already existing analysis and tasks. A more comprehensive comparison to other networks trained on the tasks is missing.

We agree that the set of tasks is standard, however we do not intend to extend the task setting in the scope of this work beyond the closed-loop setting in which the ALE and Mujoco environments are canonical benchmarks. Furthermore, we believe that the strength of our work lies in the extensive analyses we perform on the network and thus makes up for any perceived limitations regarding the breadth of tasks explored. 

We disagree that the analyses presented in this work are unoriginal. As you noted, the decomposition of the state-space trajectory is not a canonical approach. Additionally, analyzing the singular value decay (which we agree was not properly emphasized in the previous draft, and was also an analysis on the eigenvalues previously which is less pertinent and has been modified) is also not a common mode of analysis, and is what we use to distinguish between pruning by inducing sparsity and pruning by reducing rank. 

We note in the related works “Alternative parameterization of RNNs” subsection as to why we restrict the scope of our analysis to this set of networks.

Reviewer comment (Evaluation):
Some existing analyses to measure robustness, for example as the loss in performance as a function of noise level, are missing.

Author response:
We examined 5 different, canonical distribution shifts and aggregated across them in order to measure robustness, (e.g., additive noise, blurring, darkening, etc.). Doing this as a function of the noise is computationally expensive, given that we train 20 sub-models for each architecture. Furthermore, our results comparing model performance are statistically significant, as can be verified in the tables in Appendix A.8.

We more thoroughly motivate the chosen proxies for robustness in Section 3 and detail them in depth in Appendix A.5.2. We summarize how we motivate them in the paper below.

* We motivate the analysis of the spectral norms more concretely as a measure of the magnitude of the transformation induced by the weights on a perturbation applied to the input (and thus also on the hidden state). We have also added in an analysis on the input spectral norms (as opposed to only the recurrent one) since the combination of the two better captures the robustness of the network. We also motivate the spectral norms via their relationship to the Lipschitz constant of the network (which is a canonical measure of robustness), but acknowledge the looseness of this approximation in the paper. 

* We motivate the decay of the singular value spectrum as a proxy for the effective number of directions the perturbation is transformed in. This is important since while the spectral norm captures a worst-case perspective of robustness, in practice also we care about how much the perturbation is expanded in each direction because if we imagine sampling distribution shifts from a uniform distribution, all directions are equally important.

* The dimensionality analysis serves to capture this notion of “effective number of directions transformed in” concretely by looking at the representations of the data across time via the state-space trajectory analysis. In particular, the higher dimensional the trajectory is, the more directions the state-space dynamics evolve in and hence the more directions the perturbation affects, which yields poor robustness. This has also been clarified in the paper.

Reviewer comment (Evaluation):
It is not entirely clear why some of the proxies used in the paper are sufficient to be used to assess robustness.

Author response:
We examined 5 different, canonical distribution shifts and aggregated across them in order to measure robustness, (e.g., additive noise, blurring, darkening, etc.). Doing this as a function of the noise is computationally expensive, given that we train 20 sub-models for each architecture. Furthermore, our results comparing model performance are statistically significant, as can be verified in the tables in Appendix A.8.

We more thoroughly motivate the chosen proxies for robustness in Section 3 and detail them in depth in Appendix A.5.2. We summarize how we motivate them in the paper below.

* We motivate the analysis of the spectral norms more concretely as a measure of the magnitude of the transformation induced by the weights on a perturbation applied to the input (and thus also on the hidden state). We have also added in an analysis on the input spectral norms (as opposed to only the recurrent one) since the combination of the two better captures the robustness of the network. We also motivate the spectral norms via their relationship to the Lipschitz constant of the network (which is a canonical measure of robustness), but acknowledge the looseness of this approximation in the paper. 

* We motivate the decay of the singular value spectrum as a proxy for the effective number of directions the perturbation is transformed in. This is important since while the spectral norm captures a worst-case perspective of robustness, in practice also we care about how much the perturbation is expanded in each direction because if we imagine sampling distribution shifts from a uniform distribution, all directions are equally important.

* The dimensionality analysis serves to capture this notion of “effective number of directions transformed in” concretely by looking at the representations of the data across time via the state-space trajectory analysis. In particular, the higher dimensional the trajectory is, the more directions the state-space dynamics evolve in and hence the more directions the perturbation affects, which yields poor robustness. This has also been clarified in the paper.

Reviewer comment (Evaluation):
There are many other measures for assessing the effect of perturbations on dynamics, for example Lyapunov exponents, and these should be calculated for a more complete analysis.

Author response:
4. There are many other measures for assessing the effect of perturbations on dynamics, for example Lyapunov exponents and these should be calculated for a more complete analysis.

As you stated, the Lyapunov exponent is a measure for both the recurrent memory and robustness. However, we would like to clarify that this measure of robustness is across the time dimension. In particular, it measures the effect of a perturbation applied to the hidden state into the future. We agree that this is a relevant measure of robustness and have clarified this in Section 4.2 when analyzing the recurrent memory via the recurrent Jacobian analysis (which is effectively capturing the same thing as the Lyapunov exponents). In contrast, the spectral norm and dimensionality analysis measure robustness in the input and hidden state representations at a given point in time. In particular, if we perturb the input $x_t$, the input weights spectral norm measures the worst-case expansion on the perturbation induced by $W_{inp}$ (and analogously for $h_t $ and the spectral norm of $W_{rec}$) which characterizes the magnitude change in the latent representation caused by the perturbation. The dimensionality analysis characterize the effective number of directions the perturbations can affect. We describe the intuition here in further detail in A.5.2.

Reviewer comment (Theory):
Because there is no assessment of the learning dynamics itself or the process of learning itself, the vanishing gradient analysis is a misnomer. The main point of these analyses seems to be some memory component or transient (convergent/divergent) dynamics itself instead of it being related to the question of whether the network architecture can support vanishing gradients.

Author response:
5. Because there is no assessment of the learning dynamics itself or the process of learning itself, the vanishing gradient analysis is a misnomer. I understand the main point of these analyses to be some memory component or transient (convergent/divergent) dynamics itself, instead of it being related to whether the network architecture can support vanishing gradients.

Agreed, we have renamed mentions of the vanishing gradient to recurrent memory horizon and instead motivate the analysis from the perspective of characterizing the attention profile of the network across time.

Reviewer comment (Theory):
For vanishing gradients, Lyapunov exponents are also a good method for analysis.

Author response:
3. That said, also for vanishing gradients Lyapunov exponents are a good method for analysis. 

We agree with the spirit of the Lyapunov exponent analysis in that making claims about the recurrent memory using only the spectral radius is not sufficient; rather, we need to analyze the recurrent Jacobians themselves to account for the contribution of the hidden state to the gradient (similar sentiment shared by R5dM). However, we note that Lyapunov exponents are computed in the infinite time limit (Vogt et. al 2020) and for our purposes we care more so about the finite-time horizon specified by the task. Thus, we instead computed the norm of the recurrent gradient as a function of time and added this analysis to the paper (Figure 4b). We find that this analysis aligns with the spectral radius (Figure 4a), which is reassuring of its viability as a proxy for recurrent memory.

Reviewer comment (Experiments):
It is unclear why a particular number of samples was considered to be sufficient. For example, why are 3 randoms sufficient to rule out that a favorable random sparse mask is not influencing the results?

Author response:
6. It is unclear why a particular number of samples were considered to be sufficient. For example, why are 3 randoms sufficient to rule out that a favorable random sparse mask is not influencing the results? Furthermore, for such high spaces for the task, averaging over 5 perturbations seems insufficient.

We determined 3 random masks to be sufficient given the stability in the spectral statistics, as noted by the standard error in all the plots (given by the shaded area). Regarding the perturbations, we considered a breadth of possible distribution shifts (i.e. noise, brightening, blurring, etc.) and note that within each model, we consider 20 sub models or (rank, sparsity) pairs. Given the fact that trends hold amongst groups of (rank, sparsity) pairs and not just single models, we maintain that our analysis is sufficiently exhaustive.

Reviewer comment (Experiments):
For such high spaces for the task, averaging over 5 perturbations seems insufficient.

Author response:
6. It is unclear why a particular number of samples were considered to be sufficient. For example, why are 3 randoms sufficient to rule out that a favorable random sparse mask is not influencing the results? Furthermore, for such high spaces for the task, averaging over 5 perturbations seems insufficient.

We determined 3 random masks to be sufficient given the stability in the spectral statistics, as noted by the standard error in all the plots (given by the shaded area). Regarding the perturbations, we considered a breadth of possible distribution shifts (i.e. noise, brightening, blurring, etc.) and note that within each model, we consider 20 sub models or (rank, sparsity) pairs. Given the fact that trends hold amongst groups of (rank, sparsity) pairs and not just single models, we maintain that our analysis is sufficiently exhaustive.

Reviewer comment (Presentation):
It is difficult to assess to what (good) performance the reward range in Figure 3 corresponds.

Author response:
8. \paragraph{Reward} It is difficult to assess to what (good) performance the reward range in Figure 3 corresponds to. Would it be possible to show what kind of actions the highest and lowest performing networks correspond to at least? The performance of the agents is a very relative concept for which a number is not sufficient to understand what the networks actually are doing when they perform the task (supposedly well). What is the maximum number of rewards?

We agree with the points you have brought up here, and have thus modified the reward results by normalizing by the performance of each of the models by the performance of the expert policy. These can be found in Figure 1. We find that the top performing imitation learning agents perform as well and in some cases slightly exceed the performance of the expert in-distribution.

Reviewer comment (Evaluation):
It would be useful to show what kind of actions the highest and lowest performing networks correspond to, to better understand performance.

Author response:
8. \paragraph{Reward} It is difficult to assess to what (good) performance the reward range in Figure 3 corresponds to. Would it be possible to show what kind of actions the highest and lowest performing networks correspond to at least? The performance of the agents is a very relative concept for which a number is not sufficient to understand what the networks actually are doing when they perform the task (supposedly well). What is the maximum number of rewards?

We agree with the points you have brought up here, and have thus modified the reward results by normalizing by the performance of each of the models by the performance of the expert policy. These can be found in Figure 1. We find that the top performing imitation learning agents perform as well and in some cases slightly exceed the performance of the expert in-distribution.

Reviewer comment (Evaluation):
The performance of the agents is a very relative concept for which a number is not sufficient to understand what the networks are actually doing when they perform the task (supposedly well).

Author response:
8. \paragraph{Reward} It is difficult to assess to what (good) performance the reward range in Figure 3 corresponds to. Would it be possible to show what kind of actions the highest and lowest performing networks correspond to at least? The performance of the agents is a very relative concept for which a number is not sufficient to understand what the networks actually are doing when they perform the task (supposedly well). What is the maximum number of rewards?

We agree with the points you have brought up here, and have thus modified the reward results by normalizing by the performance of each of the models by the performance of the expert policy. These can be found in Figure 1. We find that the top performing imitation learning agents perform as well and in some cases slightly exceed the performance of the expert in-distribution.

Reviewer comment (Reproducibility):
What is the maximum number of rewards?

Author response:
8. \paragraph{Reward} It is difficult to assess to what (good) performance the reward range in Figure 3 corresponds to. Would it be possible to show what kind of actions the highest and lowest performing networks correspond to at least? The performance of the agents is a very relative concept for which a number is not sufficient to understand what the networks actually are doing when they perform the task (supposedly well). What is the maximum number of rewards?

We agree with the points you have brought up here, and have thus modified the reward results by normalizing by the performance of each of the models by the performance of the expert policy. These can be found in Figure 1. We find that the top performing imitation learning agents perform as well and in some cases slightly exceed the performance of the expert in-distribution.

Reviewer comment (Evaluation):
The reward ranking appears to be a difficult-to-assess proxy for performance on the perturbed environments.

Author response:
9. \paragraph{Reward ranking} It seems that the reward ranking is a difficult to asses proxy for performance on the perturbed environments. For a full picture it would be better to show what amount of reward decrease the different shift cause. How do the distribution shift change the reward range for example? If all networks perform badly, but the Cfc slightly better, can we still claim that they are robust?

We also agree and now we normalize the performance of the models under distribution by the performance of the expert in the distribution-shifted environment and then take the average of the distribution shifts in this normalized space. These can be found in Figure 1. We find that the most robust imitation learning agents are more robust than the expert policy, which points to the importance of recurrent supervision as well as constructing a robust recurrent state in the distribution shift setting.

Reviewer comment (Evaluation):
For a full picture, it would be better to show what amount of reward decrease the different shifts cause.

Author response:
9. \paragraph{Reward ranking} It seems that the reward ranking is a difficult to asses proxy for performance on the perturbed environments. For a full picture it would be better to show what amount of reward decrease the different shift cause. How do the distribution shift change the reward range for example? If all networks perform badly, but the Cfc slightly better, can we still claim that they are robust?

We also agree and now we normalize the performance of the models under distribution by the performance of the expert in the distribution-shifted environment and then take the average of the distribution shifts in this normalized space. These can be found in Figure 1. We find that the most robust imitation learning agents are more robust than the expert policy, which points to the importance of recurrent supervision as well as constructing a robust recurrent state in the distribution shift setting.

Reviewer comment (Evaluation):
The best performing networks on the online performance measure seem not to be performing particularly well on distribution-shifted versions of the task. How does the ranking look if the performance of the in-distribution reward is also taken into account?

Author response:
10. The best performing networks on the online performance measure seem not to be performing particularly well on distribution shifted versions of the task. How does the ranking look like if the performance of the in-distribution reward is also taken into account?

* Yes, this is due to the disparity between pruning by inducing sparsity and pruning by reducing rank, which is one of the major points we discover in this work 
* A ranking that takes into account both in-distribution and distribution-shift performance is entirely a function of how much weight is placed on performance in each setting, so we refrain from computing a rank that takes into account both but enable the reader to do so via the tables containing all of the results in A.9.

Reviewer comment (Presentation):
In its current form, I found it a bit challenging to parse the main contributions of the paper. I think the reason might be that Section 3 (Parameterization of Connectivity) only details the form of the proposed connectivity and the theoretical ramifications on spectral radius and norm, without any mention of the main empirical findings in the paper. It is not until section 4 (Experiments), where the paper mentions the specific findings within each subsection. Even there, I found it difficult to get concrete takeaways from the experiments, because the results are usually described in great detail without a high-level point. For example, in Sec 4.1, it seems like low-sparsity, high-rank CfCs and LSTMs are good in online settings, and LSTMs tend to be better than CfCs and RNNs at high sparsities, and low-rank, sparsity CfCs tend to be good under distribution shift. From reading this section, it was not clear to me which configuration of sparsity, rank, and architecture was most effective? In general, I think it would be helpful to distill the main takeaways from the experiments and incorporate them into Section 3, before explaining the details of the experimental setup and results.

Author response:
We tend to agree with the reviewer (and note a similar sentiment shared by reviewer XFJy) that the previous version of the paper was challenging to parse. To remedy this, we have done the following:
* Added references to the section in which each of the bullet points in the Introduction are shown (the bullet points represent the main contributions of the paper) 
- Simplified the structure of the paper as follows
  - Section 3 motivates the parameterization of connectivity
  - Section 4.1 describes the performance of the models at a high-level and makes note of the particularly interesting trends
  - Section 4.2 focuses on the understanding the trends in performance of the in-distribution rewards
  - Section 4.3 focuses on the understanding the trends in performance of the  rewards under distribution shift and juxtaposes it to the findings in-distribution
  - Section 4.4 attempts to understand why the prior is most effective in CfCs
  - Placed many of the secondary details into the appendix to avoid distracting the reader (i.e time constant analysis in A.2 and task dimension commentary in A.3)
Added one/two sentences summaries of what the section aims to accomplish at the start of the section 
I.e. “Here, we aim to gain intuition for the results we observed in the online, in-distribution setting and in particular offer an explanation as to why the only effective form of pruning in-distribution was inducing sparsity in LSTMs.” (start of Section 4.2)

Reviewer comment (Experiments):
Right now, all of the neural network models used in the experiments use some kind of temporal connection. However, I believe this is not the standard architecture used when solving the tasks used in the experiments (Seaquest, halfcheetah, etc.). It would be helpful to include an additional baseline using standard architectures (conv net, fully connect, etc) to get a sense for the return that a "vanilla" approach can get, and to better appreciate the significance of the robustness gains made by the additional sparsity and low rank formulations.

Author response:
Our apologies for the confusion. We misstated that we used a TCN when in fact the architecture is actually a CNN applied to each point in time, hence there is no recurrent supervision. This has been corrected in the paper.