Repository: google-deepmind/meltingpot
Issue: An open question: what do you think about procedurally generated environment in MARL? 

Hi, as far as I know, Melting Pot aims to evaluate the generalization of agents of MARL when facing other agents that behave differently in the **same** environment. In RL, the evaluation of generalization of RL agents is carried out mainly in procedurally generated game-like environments, such as ProGen. 

What do you think about the procedurally generated environments in MARL? Is it another direction for generalization evaluation of Melting Pot?

Hi, great question!

We pitched Melting Pot as a way to probe "social environment generalization" as opposed to "physical environment generalization". It's about coping with social situations involving unfamiliar individuals (social env variation), not unfamiliar places (physical env variation).

Of course, in reality, social and physical environment generalization are deeply intertwined. Many situations require agents to handle both physical and social novelty simultaneously. Though, some situations do stress one more than the other. For instance, you can easily imagine a challenging social situation arising in a room that you know well.

So we assume that populations of agents being evaluated on Melting Pot can be familiar with the physical layout of their test environment, they will be meeting unfamiliar other players there. You can train on the same substrate where you will be tested. The test scenario will still differ substantially because of the presence of unfamiliar individuals.

That said, you aren't required to train on the same substrate where you will be tested. Melting Pot is entirely agnostic about what you do during training. You might want to train in some other substrate if you think doing that might produce policies that that would be more transferable to the test scenarios. One reason you might want to do this is that it's pretty computationally costly to train a new population for every single substrate in the suite. If you had a single procedurally generated substrate, that could create all the physical environment variation in the suite, then it might be a good idea to just train on that one "substrate to rule them all", and then test the single resulting population on all the test scenarios, from all the different test substrates. It's an open question whether this approach could end up being less computationally costly, since it might be hard to learn in such a crazy training substrate, but I think there's a reasonable chance that some version of this idea could work.

We're planning to try creating optional-to-use procedural training substrates eventually, but have no immediate plans to do so. Happy to chat more if you're interested.

> Hi, great question!
> 
> We pitched Melting Pot as a way to probe "social environment generalization" as opposed to "physical environment generalization". It's about coping with social situations involving unfamiliar individuals (social env variation), not unfamiliar places (physical env variation).
> 
> Of course, in reality, social and physical environment generalization are deeply intertwined. Many situations require agents to handle both physical and social novelty simultaneously. Though, some situations do stress one more than the other. For instance, you can easily imagine a challenging social situation arising in a room that you know well.
> 
> So we assume that populations of agents being evaluated on Melting Pot can be familiar with the physical layout of their test environment, they will be meeting unfamiliar other players there. You can train on the same substrate where you will be tested. The test scenario will still differ substantially because of the presence of unfamiliar individuals.
> 
> That said, you aren't required to train on the same substrate where you will be tested. Melting Pot is entirely agnostic about what you do during training. You might want to train in some other substrate if you think doing that might produce policies that that would be more transferable to the test scenarios. One reason you might want to do this is that it's pretty computationally costly to train a new population for every single substrate in the suite. If you had a single procedurally generated substrate, that could create all the physical environment variation in the suite, then it might be a good idea to just train on that one "substrate to rule them all", and then test the single resulting population on all the test scenarios, from all the different test substrates. It's an open question whether this approach could end up being less computationally costly, since it might be hard to learn in such a crazy training substrate, but I think there's a reasonable chance that some version of this idea could work.
> 
> We're planning to try creating optional-to-use procedural training substrates eventually, but have no immediate plans to do so. Happy to chat more if you're interested.

@jzleibo, Hi Dr Leibo, thank you for your generous and inspiring explanations about Melting Pot. I have been using Melting Pot as an evaluation tool in my research. It is great and easy to use. I am looking forward to discussing with you about the future research directions of Melting Pot.

Hi Dr Leibo, sorry I did not have much time to share my understanding then.

I think in Melting Pot, as the background agents can have different policies and the observation is partially observable, background agents act and change the inputs of the focal agents. It can be considered a partially procedurally generated environment. If the procedural generation scheme of ProGen is introduced in Melting Pot, I believe it also introduces challenges for training background policies.  That is my understanding. Melting Pot is a great tool; I benefit a lot from it since it frees researchers to create complex scenarios, and our research findings will be submitted to future conferences.