Paper:
Learning modular structures which reflect the dynamics of the environment can lead to better generalization and robustness to changes which only affect a few of the underlying causes. We propose Recurrent Independent Mechanisms (RIMs), a new recurrent architecture in which multiple groups of recurrent cells operate with nearly independent transition dynamics, communicate only sparingly through the bottleneck of attention, and are only updated at time steps where they are most relevant. We show that this leads to specialization amongst the RIMs, which in turn allows for dramatically improved generalization on tasks where some factors of variation differ systematically between training and evaluation.

Review:
This paper proposes a neural network architecture consisting of multiple independent recurrent modules that interact sparingly. These independent modules are not all used simultaneously, a subset of them is active at each time step. This subset of active modules is chosen through an attention mechanism. The idea behind this architecture is that it would allow the different modules to specialize in different mechanisms and that would allow compositionality. The empirical results suggest that the proposed approach is able to generalize better than traditional architectures (which all have the implicit assumption that all processes interact).
This paper is well-written and it provides a very thorough empirical analysis of the proposed idea. Because it is not in my area of expertise I’m not confident that I can assess its novelty or its relationship to other existing approaches.
In terms of presentation, I recommend the authors to enlarge some of the figures in the paper (e.g., I can’t read the small box in Figure 1) and to not use citations as nouns (e.g., “The mechanisms of this attention mechanism follow (Vaswani et al., 2017; Santoro et al., 2018), with the …”). I would also like to point out that although fairly different in how they tackle the problem, the work of Arjovsky et al. (2019) seems to be related to this one.
Three questions I believe were not answered in the paper are:
1) How is the performance related to the total number of subsystems (and the number of *active* ones). I can only see results related to that in Table 1, but the variation in the number of modules is pretty small (4-6). The results also don’t give any indication whether we want to have more modules active at each time step, if there’s a sweet spot, etc. It is said that the method seems to be robust to this choice but this claim is made because it performs similarly for the values 5 and 6 if I recall correctly.
2) Is there any incentive in this architecture for a module to not simply “give up”? I mean, the modules are not necessarily incentivized to be used as often as possible, so could it be the case that a module learns to set its weights to zero?
3) Would it make sense to present baseline results for an architecture that uses attention? It seems to me that LSTM was often the baseline of choice but RIMs have two important components: multiple LSTMs and an attention mechanism. Could the attention mechanism be explaining some of the results we are seeing?
Finally, despite the very long appendix, I feel there are important details missing with respect to the empirical setup, at least in the Atari experiments which I’m more familiar with. Was stochasticity used, that is, sticky actions (Machado et al., 2018)? Moreover, for how long was PPO (and RIMs-PPO) trained in terms of number of frames? Finally, I’d recommend the authors to include a table with the actual average (and standard deviation) performance in each Atari games. It is really hard to know how well a method is doing by just squinting at learning curves. It is hard to know if the results are significant without a notion of variance.
References:
Martín Arjovsky, Léon Bottou, Ishaan Gulrajani, David Lopez-Paz: Invariant Risk Minimization. CoRR abs/1907.02893 (2019)
Marlos C. Machado, Marc G. Bellemare, Erik Talvitie, Joel Veness, Matthew J. Hausknecht, Michael Bowling: Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents. J. Artif. Intell. Res. 61: 523-562 (2018)
------
>>> Update after rebuttal: I stand by my score after the rebuttal. 
Unfortunately I'm not an expert in this area and I don't feel confident in having a very strong opinion about this paper, willing to fight for its acceptance. I also agree with concerns raised by other reviewers. As I stated in the discussion with the authors, the clarifications and additional experiment does improve the paper a bit.