zhiminy/EvalEng huggingface dataset:
Repository: google-deepmind/meltingpot
Issue: Training prosocial agents

Dear Authors,

Prosocial agents perform well and I found there is not much text on training prosocial agents in the paper.  In your paper, 

> Since the prosocial agents were explicitly trained to maximize the reward of other players, it is unsurprising that they would most benefit the background population.

Here, how is the loss function defined?  I also found

> We also trained prosocial variants of all three algorithms, which directly optimized the per-capita return (rather than individual return), by sharing reward between players during training.

Is it similar to the training paradigm of MADDPG and QMIX?

Thanks in advance. 

The only difference between the vanilla and prosocial variants was that the prosocial variants used `np.mean(timestep.reward)` instead of `timestep.reward[player_index]` as the reward during training (e.g. to train the critic).

We did this by modifying the *Substrate* observations by adding a `substrate_transform` to add coplayer rewards (via `all_observations_wrapper`). This allowed the training step to calculate the mean from the observations in the generated trajectories.

Note that _none of our agents' policies were conditioned on reward_, so we didn't use this information at behavior time (nor at test time).

This is important because since the Scenario is a test of the trained policy, we consider it "cheating" to modify the *Scenario* observations (e.g. things like explicit coordination directives). The timestep reward at test time can therefore only be `timestep.reward[player_index]`, and you shouldn't train the policies under test to expect anything else.


> The only difference between the vanilla and prosocial variants was that the prosocial variants used `np.mean(timestep.reward)` instead of `timestep.reward[player_index]` as the reward during training (e.g. to train the critic).
> 
> We did this by modifying the _Substrate_ observations by adding a `substrate_transform` to add coplayer rewards (via `all_observations_wrapper`). This allowed the training step to calculate the mean from the observations in the generated trajectories.
> 
> Note that _none of our agents' policies were conditioned on reward_, so we didn't use this information at behavior time (nor at test time).
> 
> This is important because since the Scenario is a test of the trained policy, we consider it "cheating" to modify the _Scenario_ observations (e.g. things like explicit coordination directives). The timestep reward at test time can therefore only be `timestep.reward[player_index]`, and you shouldn't train the policies under test to expect anything else.

@jagapiou  Dear John, thanks for the clarification. In my code, I use `timestep.reward` from the substrate for MARL training. I think that is what you mentioned.

```python 
reward = np.sum(timestep.reward)/n_agents
```

Thank you!
Repository: google-deepmind/meltingpot
Issue: meltingpot_env.py not deleted
Root cause: Validation Gap

Hi, in my last PR I renamed examples/pettingzoo/meltingpot_env.py to utils.py. However, the merge has created examples/pettingzoo/utils.py without deleting examples/pettingzoo/meltingpot_env.py.

Thanks for the heads up, not sure if it's an issue with our pipeline or if I accidentally didn't include the delete in the manual stage of things. I've made the change internally and it should be submitted soon.
Repository: google-deepmind/meltingpot
Issue: Second episode using same seed as the first
Root cause: Algorithmic Error

https://github.com/deepmind/meltingpot/commit/449ac8b083c5bbb0d338404e7094d63c00beb5e3 introduced a bug where the random seed was not updated after the first episode. This means that there was incorrect randomization and second episode was the same as the first episode.

Fixed in 976465ae0473051cf4bd63965708328da51c6863
Repository: google-deepmind/meltingpot
Issue: Pylint

Hi - what's the standard practice for pylintrc files with Deep Mind projects?

Could we add the standard Google one to the repo, and perhaps also add a linting step the CD? Happy to commit a PR if useful.

Hi Alan, I'll add a .pylintrc and push that out. Our internal tooling means that i don't have a great idea what's best for the external workflow; so if you could point me at something (or do a PR once the .pylintrc is in) that would be great.

Thank you :-)

Thanks John!

Something like https://github.com/marketplace/actions/python-code-quality-and-lint should work

Should the .style.yapf not use equals instead of colons? i.e.
```
[style]
based_on_style = chromium
indent_width = 2 
split_before_named_assigns = True
column_limit = 80
```

Actually, I am still getting errors using this file. Would you be able to advise me please? I believe this is because chromium is not a recognised style.

Yapf says that you can choose a predefined style https://github.com/google/yapf
... based in the file style.py https://github.com/google/yapf/blob/main/yapf/yapflib/style.py
But chromium is not one of them.

Update: I was wrong about the equals signs, colons are fine. But I think you want 
`based_on_style: yapf`
That appears to be what the chromium style is now called
https://github.com/google/yapf/pull/789

Thanks for the heads-up, I'd just copy-pasted it from elsewhere. I think it should just be:

```
[style]
based_on_style: yapf
```

Change incoming soon.
Repository: google-deepmind/meltingpot
Issue: Installation frozen
Root cause: Environment Incompatibility

Hello,

I am trying to install meltingpot on my ubuntu 20.04 machine, these are my conda, python and bazel versions:

- conda: 4.12.0
- Python 3.9.11
- bazel 5.2.0
- (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0

The installation freezes as shown in the screenshot. I left the installation overnight and it still does not progress.

![image](https://user-images.githubusercontent.com/102532937/176475950-fbf5268c-b4ce-488e-b7fc-03954228709c.png)


It froze on the tests (the final stage after installation). Hopefully those would have passed, but it is installed.
Repository: google-deepmind/meltingpot
Issue: Training Details for A3C

Hi,

It is mentioned in the MeltingPot paper that "Every agent participated in every training episode, with each agent playing as exactly one player (**selected randomly on each episode**)." - Section 7 Experiments
But in A3C/IMPALA, the experience collection is de-coupled from the training loop. So how exactly the information about which agent was used to generate the experience for which player, is shared between the experience collection loop and training loop?
E.g. Let the agent-player mapping be as follows for a 2-player environment be 
- {"agent_1":"player_1", "agent_2":"player_2"} for first episode
- {"agent_2":"player_1", "agent_2":"player_1"} for second episode

As the experience will be stored in a replay buffer, and multiple episodes can be sampled from the buffer together(depending on batch size), the agent-player mapping can vary for each example in the batch. How exactly is this being handled?

### Possible options:
- It doesn't matter which agent was used to collect the experience as we are anyways, passing the log_prob and applying v-trace, so the agent-player mapping in the experience collection and training loop is handled completely independently
- Sample only a single episode at once, and batch it accordingly so that the agent-player mapping is consistent for that specific training step

Thanks 
Kinal


At learning-time the asynchronous actor-critic baseline agent ("A3C/IMPALA") processes trajectories of it's own observations/actions only. So it doesn't need to know which player slot it was in when that data was generated.

Only OPRE saw coplayer observations during learning (but not during behavior), and it received a batch of one-hot vectors to tell it which slot was it's own perspective.

Thank you for your answer.

So if I understand correctly, even if mapping is `{"agent_1":"player_1", "agent_2":"player_2"} ` during the experience collection time, the policy-gradients can be calculated with a different mapping say `{"agent_2":"player_1", "agent_2":"player_1"}`?
(Note: here `agent_*` refers to the network being used to select an action or whose parameters are being updated using PG)

Is there any form of parameter-sharing being adopted? E.g., for CNN feature extraction.

I'm not sure what you mean about using a different mapping in the policy-gradient update. The mapping of agent to player slot is a behavior-time detail that is always irrelevant to the A3C/IMPALA agent:

1. Policies are conditioned only on their own observations. They have to infer the existence of co-players from their observations. (Decentralized execution).
2. In the A3C/IMPALA agent, the critic is conditioned only on the agent's own behavior-time observations so the exact behavior-time slot is irrelevant. It must also infer the existence of co-players. (Decentralized learning).

We did no parameter-sharing between agents (or data-sharing), so they were only coupled by correlated experience arising from being in the same episodes (from different perspectives).
Repository: google-deepmind/meltingpot
Issue: macOS Apple M1 Chip Support
Root cause: External Dependency Breakage

Hi, while running test scripts on macOS Apple M1 Chip. An error occurred:

```
$ python3 setup.py test
running test
WARNING: Testing via this command is deprecated and will be removed in a future version. Users looking for a generic test entry point independent of test runner are encouraged to use tox.
running egg_info
creating dm_meltingpot.egg-info
writing dm_meltingpot.egg-info/PKG-INFO
writing dependency_links to dm_meltingpot.egg-info/dependency_links.txt
writing requirements to dm_meltingpot.egg-info/requires.txt
writing top-level names to dm_meltingpot.egg-info/top_level.txt
writing manifest file 'dm_meltingpot.egg-info/SOURCES.txt'
reading manifest file 'dm_meltingpot.egg-info/SOURCES.txt'
writing manifest file 'dm_meltingpot.egg-info/SOURCES.txt'
running build_ext
[1]    96188 illegal hardware instruction  python3 setup.py test
```

Does it mean MeltingPot cannot run on macOS Apple M1 Chip?


Looking at `dmlab2d` I see only x86 wheels: https://github.com/deepmind/lab2d/releases/tag/release_candidate_2021-07-13 so I guess the issue is you used one of those.

I've spoken to the `dmlab2d` team about adding M1 wheels, and we're looking into it.

For now you will have to build your own wheel for `dmlab2d` using:
```sh
bazel build -c opt --define=lua=5_1 //dmlab2d:dmlab2d_wheel
```
See [install.sh](https://github.com/deepmind/meltingpot/blob/main/install.sh) for how this can be done.

> Looking at `dmlab2d` I see only x86 wheels: https://github.com/deepmind/lab2d/releases/tag/release_candidate_2021-07-13 so I guess the issue is you used one of those.
> 
> I've spoken to the `dmlab2d` team about adding M1 wheels, and we're looking into it.
> 
> For now you will have to build your own wheel for `dmlab2d` using:
> 
> ```shell
> bazel build -c opt --define=lua=5_1 //dmlab2d:dmlab2d_wheel
> ```
> 
> See [install.sh](https://github.com/deepmind/meltingpot/blob/main/install.sh) for how this can be done.

Thank you very much. I will build the `dmlab2d`.
Repository: google-deepmind/meltingpot
Issue: Fixing dependencies in `setup.py`
Root cause: External Dependency Breakage

I have installed Melting Pot via the Dockerfile, and while it passes the tests, I get issues with the RLLib and PettingZoo examples that seem to stem from the versions of a number of packages, and their respective dependencies. I have been able to get the `self_play_train.py` RLLib example to run using the following combination of packages

```
pettingzoo==1.18.1
supersuit==3.3.1
stable_baselines3==1.2.0
gym==0.21.0
```

but the `sb3_train.py` script currently yields the following error:

```
Traceback (most recent call last):
  File "/home/samuelab/software/meltingpot/examples/pettingzoo/sb3_train.py", line 185, in <module>
    main()
  File "/home/samuelab/software/meltingpot/examples/pettingzoo/sb3_train.py", line 122, in main
    env = ss.concat_vec_envs_v1(
  File "/home/samuelab/.local/lib/python3.9/site-packages/supersuit/vector/vector_constructors.py", line 49, in concat_vec_envs_v1
    vec_env = MakeCPUAsyncConstructor(num_cpus)(*vec_env_args(vec_env, num_vec_envs))
  File "/home/samuelab/.local/lib/python3.9/site-packages/supersuit/vector/concat_vec_env.py", line 12, in __init__
    self.vec_envs = vec_envs = [vec_env_fn() for vec_env_fn in vec_env_fns]
  File "/home/samuelab/.local/lib/python3.9/site-packages/supersuit/vector/concat_vec_env.py", line 12, in <listcomp>
    self.vec_envs = vec_envs = [vec_env_fn() for vec_env_fn in vec_env_fns]
  File "/home/samuelab/.local/lib/python3.9/site-packages/supersuit/vector/vector_constructors.py", line 11, in env_fn
    env.seed(None)
  File "/home/samuelab/.local/lib/python3.9/site-packages/supersuit/vector/markov_vector_wrapper.py", line 32, in seed
    self.par_env.seed(seed)
  File "/home/samuelab/.local/lib/python3.9/site-packages/supersuit/generic_wrappers/utils/shared_wrapper_util.py", line 89, in seed
    super().seed(seed)
  File "/home/samuelab/.local/lib/python3.9/site-packages/pettingzoo/utils/env.py", line 250, in seed
    raise NotImplementedError(
NotImplementedError: Calling seed externally is deprecated; call reset(seed=seed) instead
```

Looking into it, this seems to be an issue with `supersuit==3.3.1` which is resolved in `3.4.0`. But then this version of supersuit requires `gym==0.22.0`, while `stable_baselines3` needs `gym<=0.21.0` up until its latest version. 

Is there some combination of versions for these packages so that all of the constraints are satisfied and I can get all of the Melting Pot example scripts to run?

@duenez I have updated Melting Pot to `v1.0.4` but this still does not seem to work.

Not sure that you want to bump to Ray 2.0.0, the rllib examples were written for ray 1.13.0, and there could be breaking changes.

I think there is some conflict between the sb3 and rllib examples, and the user should only install *one* of the dependencies.

I've fixed the RLLib example, and updated it to 2.2.0. It should pass at head.
Repository: google-deepmind/meltingpot
Issue: Puppets share state in Scenarios
Root cause: Resource Mishandling

Certain puppeteer functions (e.g. `GrimTwoResourceInTheMatrix`) are not pure functions, but are `Callable` objects that maintain their own state between `__call__` and update it as a side-effect. This means that puppeteer state is not exposed in the `PuppetPolicy` state returned by `initial_state` and transformed by `step`.

1. Since puppeteer functions are shared between `PuppetPolicy`, multiple puppets running in the same Scenario will erroneously share a single puppeteer state between multiple puppets.
2. Puppeteer state is not reset at the start of a new episode.

Both of these issues will cause undefined behavior when running the affected scenarios.

Affected bots:
* chicken_puppet_grim
* cleanup_puppet_reciprocator_threshold_low
* cleanup_puppet_reciprocator_threshold_mid
* prisoners_dilemma_puppet_grim_threshold_high
* prisoners_dilemma_puppet_grim_threshold_low
* stag_hunt_puppet_grim

Affected scenarios:
* chicken_in_the_matrix_4
* clean_up_4
* clean_up_5
* clean_up_6
* prisoners_dilemma_in_the_matrix_4
* prisoners_dilemma_in_the_matrix_5
* stag_hunt_in_the_matrix_2
No other scenarios were affected by this bug.

NOTE: This bug did not affect the evaluations run on these scenarios in the Melting Pot paper.

I'll add some additional interpretation of how to understand this issue, and what has changed now that it is fixed.

The puppet bots for `prisoners_dillemma_in_the_matrix`, `chicken_in_the_matrix`, and `stag_hunt_in_the_matrix` implement [grim trigger](https://en.wikipedia.org/wiki/Grim_trigger) strategies. So once any player defects on one of these bots they then "trigger" and start defecting in all future interactions. Note: once triggered, they defect on everyone, not just the player that defected on them (these are 8 player games).

The bug was that multiple puppet bots playing in the same episode would share state with one another instead of maintaining their own state.  So, while it was in effect, when any player defected on any puppet bot then ALL puppet bots would share that state change and consequently start defecting indiscriminately.  So, these situations were even more grim than we had intended. 

As a result, we expect scores will likely increase on the affected `*_in_the_matrix` scenarios now that the fix is in.

For `clean_up`, the situation is different. The affected bots are supposed to clean the river whenever a specified number of other players are also cleaning the river. Since this was a global signal in the first place, it didn't matter that it was erroneously shared between puppet bots. They were all meant to clean at the same time, and under the same circumstances. So fixing the bug should not have any effect on `clean_up` results.
Repository: google-deepmind/meltingpot
Issue: Cannot run the example code of coding a new substrate for Melting Pot
Root cause: Documentation Deficiency

I tried to run the full working example of the tutorial: https://github.com/deepmind/meltingpot/blob/main/docs/substrate_tutorial/index.md#fully-working-example. However, it return the following error


```bash
python -m examples.tutorial.harvest.play_harvest

....

  File "/Users/me/meltingpot/meltingpot/python/utils/substrates/wrappers/reset_wrapper.py", line 33, in __init__
    env = build_environment()
  File "/Users/me/meltingpot/meltingpot/python/utils/substrates/builder.py", line 184, in build_environment
    env_raw = dmlab2d.Lab2d(_DMLAB2D_ROOT, lab2d_settings_dict)
ValueError: 
ERROR: [tensor.ByteTensor.fill] - Must call with number or an array that matches last dimension received: (table)
stack trace-back:
ERROR: [C]: in function 'fill'
ERROR: dmlab2d/lib/game_scripts/common/image_helpers.lua:130: in function 'func'
ERROR: dmlab2d/lib/game_scripts/common/image_helpers.lua:71: in function '_visitText'
ERROR: dmlab2d/lib/game_scripts/common/image_helpers.lua:126: in function 'textToSprite'
ERROR: dmlab2d/lib/game_scripts/common/tile_set.lua:110: in function 'addShape'
ERROR: @./meltingpot/lua/modules/component_library.lua:593: in function '_addShapesToTileSet'
ERROR: @./meltingpot/lua/modules/component_library.lua:600: in function 'addSprites'
ERROR: @./meltingpot/lua/modules/game_object.lua:237: in function 'func'
ERROR: @./meltingpot/lua/modules/game_object.lua:160: in function '_doOnAllComponents'
ERROR: @./meltingpot/lua/modules/game_object.lua:234: in function 'addSprites'
ERROR: @./meltingpot/lua/modules/base_simulation.lua:330: in function 'addSprites'
ERROR: @./meltingpot/lua/modules/api_factory.lua:49: in function '_createSprites'
ERROR: @./meltingpot/lua/modules/api_factory.lua:65:
```

The OS is macOS and Python's version is 3.9 with the latest Lab2D and Melting Pot. Since it is an issue in the Lab2D, could you please point out how to solve it?




Ah, I believe that's probably my fault. It should work if you change 'shapes.APPLE' to 'shapes.LEGACY_APPLE'.  I will fix it in place shortly. Thank you for letting us know about this

> Ah, I believe that's probably my fault. It should work if you change 'shapes.APPLE' to 'shapes.LEGACY_APPLE'. I will fix it in place shortly. Thank you for letting us know about this

@jzleibo Thank you Dr Leibo.
Repository: google-deepmind/meltingpot
Issue: Calculating the reward in * in the matrix substrates.
Root cause: Documentation Deficiency

Dear authors, 

I have a question on calculating the reward of each timesteps of * in the matrix substrates. For example chicken_in_the_matrix, the matrix for the interaction is (copied from the paper):

$$
A_{row} = A^{T}_{col}= \left[\begin{array}{cc}
3 & 2\\
5 & 0
\end{array}\right]
$$

Give inventories $v_1=$`[1 1]` for agent 1 and $v_2=$`[1 3]` for agent 2 in an interaction. How can I calculate the reward $r_1$ for agent 1  and reward $r_2$ for agent 2? I check the OPRE paper and found the following equation:

$$
r_0 = \frac{v_0}{||v_0||}M(\frac{v_1}{||v_1||})^{T}=-r_1
$$

where $M=A_{row} = A^{T}_{col}$.

Can it be applied for chicken_in_the_matrix?

Thank you very much!


Yes, you can use the same formula, except that, since this is not zero sum, $r_0$ is not the same as $-r_1$. Instead, $r_1 = \frac{v_1}{||v_1||} A_{col} \left( \frac{v_0^T}{||v_0||} \right)$.
Repository: google-deepmind/meltingpot
Issue: Melting Pot Version 2.0.0
GitHub labels: enhancement

We are currently preparing the codebase for the next Melting Pot release, in which we will release new substrates and scenarios.

All this work will be done at `HEAD` on `main`, but we will keep the v1 substrates and scenarios working and unchanged up until we release v2.0.0.

However, there will likely be some disruption during this process (e.g. API changes, file moves). To avoid this please use tagged release [v1.0.4](https://github.com/deepmind/meltingpot/releases/tag/v1.0.4).

> We are currently preparing the codebase for the next Melting Pot release, in which we will release new substrates and scenarios.
> 
> All this work will be done at `HEAD` on `main`, but we will keep the v1 substrates and scenarios working and unchanged up until we release v2.0.0.
> 
> However, there will likely be some disruption during this process (e.g. API changes, file moves). To avoid this please use tagged release [v1.0.4](https://github.com/deepmind/meltingpot/releases/tag/v1.0.4).

@jagapiou Thank you John. Thanks for the great efforts made by your team to make the new Melting Pot.

[v2.0.0](https://github.com/deepmind/meltingpot/releases/tag/v2.0.0) is released!

There will likely be a few maintenance patches over the next few weeks to deal with things like https://github.com/deepmind/meltingpot/issues/84.

In addition, we intend to add a few more substrates, an evaluation tool and some raw data.

Hi,

The changes for v2.0 include "Removed all v1 scenarios, bots, and substrates and replaced with new versions." as one of the points. Is there any comparison of the changes between v1 and v2 of meltingpot as most of the environments seem to be quite similar to those on v1 (e.g. Clean-up, Running with scissors in the matrix, etc)

Thanks
Kinal Mehta

Hi Kinal,

We don't have a point-by-point comparison for the entire suite. Most of the changes are additions of new substrates and scenarios. 

For some of the older substrates the changes are purely cosmetic, e.g. clean_up, territory, and allelopathic_harvest. For commons_harvest we improved the graphics and also made the maps smaller and used less players (7 versus 12). For *_in_the_matrix substrates we added the "ready to interact" mechanic, which resolves some issues with the old version and makes things more unified across all substrates in the *_in_the_matrix family.

We did retrain all the background population bots for all the scenarios, so their performance may have shifted somewhat. You should expect all scenarios that still have the same description to work the same way though.

But please ask here if there are more detailed questions about specific substrates or scenarios, I can dig out any such info you might need.

Thanks for these details. I don't have any specific questions as of now.
I was asking this to understand how it would affect results in my codebase if I directly swap in v1 environments with v2. I believe it doesn't affect the self-play training in the scenarios you explained. 

Thanks again for this awesome environment suite.