Repository: google-deepmind/meltingpot
Issue: Ablility to access observations of other agents in scenarios
Root cause: Documentation Deficiency

I'm trying to reproduce the results of OPRE paper. There the agents have been trained on meltingpot scenario-type settings (OPRE is trained on committed agents like rock, paper, scissors in "Running with scissors in the Matrix") along with self-play. During the training phase, the OPRE agent uses observations from other agents. But the scenario doesn't allow access to such information. Any possible way I could access that information in a clean manner rather than directly modifying the scenario creation code?

You should never need to add observations to a _Scenario_ (unless you are deliberately "cheating" at the test to get an upper bound or some other measure of interest).

In particular, OPRE is designed to learn a behavior policy that is not conditioned on opponent observations. That way it doesn't need them at test time.

However, it does use opponent observations at training-time to train the critic. We did this by training on a _Substrate_ wrapped with https://github.com/deepmind/meltingpot/blob/main/meltingpot/python/utils/scenarios/wrappers/all_observations_wrapper.py.

> You should never need to add observations to a Scenario (unless you are deliberately "cheating" at the test to get an upper bound or some other measure of interest).

Completely agree with this. I just want to use the bot's observations to validate that the algorithm is actually able to learn to exploit fixed opponents. That basically could also be used as an upper bound.

> In particular, OPRE is designed to learn a behavior policy that is not conditioned on opponent observations. That way it doesn't need them at test time.

I understand that.

OPRE in meltingpot paper is trained only using self-play.
However, in the original OPRE paper, the authors also train on specialised policies to test how effectively the agent learns to exploit fixed opponents (section 5.2 [link](https://arxiv.org/pdf/1906.01470.pdf)). In that case, I believe the trainer does use observations from bots as well. 

Also, I observe that even though the environment used in both meltingpot and OPRE is `Running with scissors` (there could be subtle implementation differences as I'm not sure if the original OPRE paper used some internal implementation of that environment or the same one that is open-sourced in meltingpot), the results are very different.


Hey Kinal,

The OPRE paper used an earlier version of what we have released in Melting Pot, @jzleibo will have a better idea what, if anything, has changed.

Section 5.2 in the OPRE paper shows evaluation of OPRE variants vs pure strategy bots, which shouldn't require access to opponent observations for either OPRE or the bot.

We have added these evaluations to Melting Pot as Scenarios: `running_with_scissors_2` (vs pure rock bot), `running_with_scissors_3` (vs pure paper bot), `running_with_scissors_4` (vs pure scissors bot). Averaging the results on those should allow you to reproduce Figure 4 of the OPRE paper.

J.

Hi John,

Thank you for these details.

It is mentioned that `section 5.2` evaluates performance when **trained directly against pure strategies** (Pink highlight in the screenshot). And as OPRE uses other agents' observations to estimate $q(z|x'_t)$ during training time, I believe the observations from bots might be required.

Whereas `section 5.3` evaluates agents **trained using self-play** against pure strategies.


![image](https://user-images.githubusercontent.com/19221028/185940772-08a735d3-7e28-41cf-96f2-5b5b1ee7eb1b.png)

Thank You
Kinal

Ah sorry, I should have read more carefully :-)

To train on Scenarios with wrappers you can add a `substrate_transform` after getting the config: https://github.com/deepmind/meltingpot/blob/e05c70464b1a0e1fb9b42859d623fcac3347f082/meltingpot/python/scenario.py#L79

Something like this should be what you need for OPRE: https://github.com/deepmind/meltingpot/blob/e05c70464b1a0e1fb9b42859d623fcac3347f082/meltingpot/python/utils/scenarios/substrate_transforms.py#L53

[Edit for other people finding this comment in the future: note that this is for training exploiters, upper bounds etc. Agents being tested on the benchmark should not use any wrappers.]


Thank you so much, @jagapiou.
I'll try this soon and get back if I have any more queries.

I'll close this out but please reopen if it doesn't work.