Paper:
Training agents to operate in one environment often yields overfitted models that are unable to generalize to the changes in that environment. However, due to the numerous variations that can occur in the real-world, the agent is often required to be robust in order to be useful. This has not been the case for agents trained with reinforcement learning (RL) algorithms. In this paper, we investigate the overfitting of RL agents to the training environments in visual navigation tasks. Our experiments show that deep RL agents can overfit even when trained on multiple environments simultaneously. We propose a regularization method which combines RL with supervised learning methods by adding a term to the RL objective that would encourage the invariance of a policy to variations in the observations that ought not to affect the action taken. The results of this method, called invariance regularization, show an improvement in the generalization of policies to environments not seen during training.

Review:
Summary:
The goal of the paper is to improve generalization of RL agents to a set of known transformations of the observation. 
The authors propose to explicitly include a term into the PPO loss function that incentivizes invariance to transformations of the environment which should not change the policy, in their case changing textures of walls. 
The idea itself is straightforward but worth exploring, although it does make the fairly strong assumption that one has access to or knowledge of the transformation function of the environment that can be applied to the observations. 
Overall, the paper is clearly written and easy to read. 
I'm currently recommending rejection based on the experimental evaluation as I don't believe that, in their current form, they sufficiently show the utility of the method (see detailed comments below).
However, I'm happy to change my rating if some or all of my comments and questions are addressed. 
My main concerns are with the experimental evaluation:
- I would encourage evaluation on a second environment. In particular, the CoinRun environment which is cited in the paper would in my opinion make an excellent second evaluation setting as it also allows randomization of the texture, so the setup is fairly similar to the setup here and, importantly, it allows comparison to published results. 
-I am surprised that training on _more_ environments reduces the performance for the baseline agent. My suspicion is that training on 100 or 500 task is, at first, much harder to learn for the agent as it sees individual levels much more rarely. With the number of training steps fixed, I think it is possible that those agents haven't finished training yet. It would be good to include the training curves for the agents or at the very least their final performance on the training set (for example in the appendix).
- Another question that was not clear to me from the text: When training the IR objective, is the transformation function restricted to produce observations from the set of limited environments? I.e. when training on 10 envs, does the transformation function produce only observations from those 10 or potentially from all 500 in the 'maximum' training set? Having access to all 500 would explain why the success rate for IR is constant across all number of training set sizes.
- I think figure 2 needs more random seeds and needs to show the standard deviation across them, as it might be fairly large. 
Edit:
Thank you for your response and your comments. 
Overall I think this is interesting and very promising work. Consequently, I am raising my score to "weak reject". Below, I discuss several points how I think the paper could be improved. 
Additional environment: I understand that Coinrun requires a significant amount of compute and that's not easily affordable for everyone. However, I do strongly believe that this work would profit a lot from additional experiments. For future versions of the paper, I do think Coinrun would be very interesting, but alternatively, maybe the Multiroom environment used in [1] might be an easier alternative? (With transformations being symmetry transformations?). Just an idea. 
More environments leading to a deterioration of performance: Unfortunately, I still don't understand how more data can lead to more overfitting. I believe it would be useful to investigate this further, either to avoid an unfair comparison to PPO if there's some training instability that can be easily avoided. Or alternatively, if this result holds, this could be a very useful insight as well if the authors can explain why this is the case. 
Good and constant performance of IR: I still find it surprising how constant the IR performance is independent of how many levels are being used. Providing more data/ablation studies to better support this result and provide further insight into IR would strengthen the paper a lot. Additionally/alternatively: Submitting the code for reviews to inspect would help here as well. 
>> "We hypothesize that adding this penalty encourages the model to find the invariance in the environment and the model is able to deduce the shared invariance within these environments and reach a certain performance with this added information only by observing a few .  
In particular, if you can show some further support for this claim, I think this would strengthen the paper a lot.