zhiminy/EvalEng huggingface dataset:
Repository: google-deepmind/meltingpot
Issue: Implementing extrinsic rewards using MeltingPot
Root cause: Documentation Deficiency

Hello everybody, 

I am trying to replicate the results of the paper _Inequity aversion improves cooperation in
intertemporal social dilemmas_ using MeltingPot. According with the tutorial that I have been following the reward that the agent receives is specified by the "Edible" component. Aditionally,  in [self_play.py](https://github.com/deepmind/meltingpot/blob/407598aefdc5d42ffadef1a751b3b01a58830187/examples/rllib/self_play_train.py) I notice that Rllib handles the training completely in the `trainer.train()` method. Therefore, I am not sure which is the best way to implement the _advantageous and disadvantageous inequity aversion_  since they need to punish or reward agents on each time step taking into account the reward that the other agents have received. Should I implement it using lua ? in a new substrate ?

Aditionally, When I run  `python3 meltingpot/python/human_players/play_clean_up.py --observation WORLD.RGB` I notice that there are no apples in the field. Is this normal ? Will they appear as the agents clean the river? 

Thanks in advance for your help and for providing this library!

Best regards.

I would implement the inequity aversion reward function as part of the agent, not the environment. If you use self_play.py that would then mean using RLLib to implement it.

As for clean_up, yes it's normal that there are no apples present at the start. You have to clean the river to get them to appear. The growth settings were chosen so that it's easiest to get lots of apples to appear when two players clean at the same time.  But if you move quickly with the human player you can still get it to work alone.

Good luck! Let us know how it goes. We're always happy to discuss.



Thank you @jzleibo, 

I will let you guys know when I have the code ready!

Once again thank you.
Repository: google-deepmind/meltingpot
Issue: Verbose flag in play_ scripts depends on inaccessible flag
Root cause: Validation Gap

Trying to run a play script with `--verbose` as follows: `python -m meltingpot.human_players.play_collaborative_cooking --verbose True --print_events True`

This results in the following error: 
```
  File "/dev/meltingpot/meltingpot/human_players/play_collaborative_cooking.py", line 65, in verbose_fn
    if env_timestep.observation[f'{lua_index}.{obs}']:
       ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
KeyError: '1.ADDED_INGREDIENT_TO_COOKING_POT'
```

From a quick look through the source code, it seems like susbtrates have a default `_ENABLE_DEBUG_OBSERVATIONS = False` flag that needs to be manually enabled before the CLI works as intended. 

Happy to work on a patch if I can get a suggested fix from a maintainer. Thanks!
Repository: google-deepmind/meltingpot
Issue: Building problems in Ubuntu 18.04
Root cause: Configuration Error

Hello, 

I am trying to install meltingpot in my Ubuntu18.04 machine.

When I run `./install.sh` I got the following error message 
`
ERROR: dmlab2d-1.0-cp38-cp38-manylinux_2_27_x86_64.whl is not a supported wheel on this platform.`

I am using a conda environment with python version: `Python 3.7.0`
gcc version: `gcc (Ubuntu 8.4.0-1ubuntu1~18.04) 8.4.0`
bazel: `bazel 5.1.1`

How can I fix this ?

That seems like the wrong wheel (3.8) for your 3.7 platform.

Are you using the latest version of `install.sh`?

Yes,  I am using the latest version, 
I noticed that after lab2d is populated the path `meltingpot/lab2d/bazel-bin/dmlab2d` contains multiple wheels.

```
dmlab2d-1.0-cp37-cp37m-manylinux_2_27_x86_64.whl
dmlab2d-1.0-cp37-cp37m-manylinux_2_27_x86_64.whlname
dmlab2d-1.0-cp38-cp38-manylinux_2_27_x86_64.whl
dmlab2d-1.0-cp38-cp38-manylinux_2_27_x86_64.whlname
```

Since install.sh uses `pip install lab2d/bazel-bin/dmlab2d/dmlab2d-*.whl` it installs everything which causes the issue
I changed the line to `pip install lab2d/bazel-bin/dmlab2d/dmlab2d-1.0-cp37*.whl` and it solved the issue


Glad it got sorted :-)

For me it only created the one wheel. Is it possible you ran the dmlab2d wheel builder previously but for Python 3.7? I should probably add some kind of tempfile in there to ensure that repeated runs of install.sh are independent.

I think `pip install --find-links=lab2d/bazel-bin/dmlab2d dmlab2d` should ensure `pip` selects the wheel itself, I'll update `install.sh`.

Updated in 383321d78a8b2b540456f98c4352492022f46050
Repository: google-deepmind/meltingpot
Issue: Coop-mining environment bug
Root cause: Algorithmic Error

I have been using the cooperative mining environment and training agents to perform the task. I encountered two potential issues related to multiple agents shooting at gold:

1. Limited Shot Registration:

When more than two agents attempt to shoot at gold simultaneously, only the first two agents receive rewards. Although multiple agents can execute the shooting action at the same time, only two shots are registered and counted as a mining event ([registering mining event in lua code).](https://github.com/google-deepmind/meltingpot/blob/b56ad1a5a53cc63c1e5c84b504562492e03ebddd/meltingpot/lua/levels/coop_mining/components.lua#L123)

2. Gold Persistence and Excess Rewards:

When more than two agents shoot at the gold, the gold does not disappear immediately, even though two agents receive rewards. This results in agents seemingly receiving double or triple rewards for mining the same gold.

 I have recorded a video with three agents, where I display each agent's actions and rewards at each time step. In the video, action index 7 corresponds to the shooting action. Reward for iron mining grants 0.9, and gold mining grants 5.9 (after applying a -0.1 penalty per shooting action).

Video Link: https://drive.google.com/file/d/1PgsrAodoNAD5wrQrW6d9IjnGyIXkWDm7/view?usp=sharing 
The issues occur frequently in the last 30 seconds of the video.

I appreciate any insights on why this might happen, thank you! 


I am experiencing the same issue, agents are sometimes able to exploit this bug to gain unreasonably high rewards from a single gold ore. It would be greatly appreciated if this could be addressed. Thank you!

This is a bug. I'll see what we can do

@linfangu check my pull request.

**File modified:** `meltingpot/lua/levels/coop_mining/components.lua`

**What changed:**

Updated the Ore component to de-duplicate mining attempts per miner within a single ore interaction.
Ensured ore transitions to the waitState immediately after a successful extraction to prevent duplicate rewards.
Added a fast-fail guard when too many distinct miners attempt the same ore before the required count is met.
Preserved the original immediate-extraction behavior for iron (1 miner) and gold (2 miners).
Repository: google-deepmind/meltingpot
Issue: Unable to install requirements for examples
Root cause: External Dependency Breakage

I'm trying to run `pip install -r examples/requirements.txt`,  which is `pip install -r examples/requirements.in` now but keep getting the following error

```
Collecting gym==0.21 (from stable_baselines3->-r requirements.in (line 12))
  Using cached gym-0.21.0.tar.gz (1.5 MB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [3 lines of output]
      /home/alvaro/anaconda3/envs/meltingPot/lib/python3.11/site-packages/setuptools/_distutils/dist.py:270: UserWarning: Unknown distribution option: 'tests_require'
        warnings.warn(msg)
      error in gym setup command: 'extras_require' must be a dictionary whose values are strings or lists of strings containing valid project/version requirement specifiers.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
```

I can't run `python self_play_train.py` or ` python sb3_train.py`.

Hey @albo437 

I ran into a bit of a snag trying to get the examples running with pip install -r examples/requirements.in. It looks like gym==0.21 throws a fit on Python 3.11 because the newer version of setuptools is way stricter about how extras_require is formatted in setup.py.

Basically, gym is using an old format that modern setuptools doesn't like anymore, which is why the metadata generation fails.

The quick fix that worked for me: Just downgrade setuptools for a second before running the requirements install. It makes the installer a bit more "forgiving" with the old Gym code:

Bash
```

pip install "setuptools<65.0"
pip install -r examples/requirements.in
```

Once I did that, `self_play_train.py `and `sb3_train.py` both fired up perfectly. Hope this helps anyone else hitting the same wall!
Repository: google-deepmind/meltingpot
Issue: Bots receive incorrect timesteps in Scenario
Root cause: Algorithmic Error

Due to an errant `-` bots were sent timesteps `(1, ..., num_bots + 1)` rather than `(num_bots, ...)` as intended. This was fixed in https://github.com/deepmind/meltingpot/commit/09b69a53ca42d3d39bcc034d47c71b5b3796c10a.

This will completely mess up the behavior of the bots on the scenarios as they don't see their own observations, and so any evaluation run using before this fix will be incorrect and should be rerun.

(NOTE: this didn't affect our experiments run for the Melting Pot paper since these weren't run using `Scenario`.)
Repository: google-deepmind/meltingpot
Issue: Required test-examples CI check fails for RLlib examples due to missing optional dependencies
Root cause: Validation Gap

### Summary

The required `test-examples` CI check currently fails for PRs that touch
RLlib example-related files because the example tests import optional
dependencies (`ray`, `gymnasium`) that are not installed in CI.

As a result, the `test-examples` job fails and **blocks merging**, even when
the PR itself does not modify test logic.

This issue builds on the installation problems discussed in #299, but focuses
specifically on the CI behavior and merge blocking.

---

### Observed CI behavior

- `test-examples` is a required check for merging PRs.
- CI does not install `examples/requirements.in`.
- RLlib example tests under `examples/rllib/` import Ray and Gymnasium at
  module import time.
- During CI:
  - tests fail with `ModuleNotFoundError`, or
  - the test job hangs or errors during collection.
- This causes `test-examples` to fail and **prevents PRs from being merged**.

Example failure from CI:

```
ERROR examples/rllib/self_play_train_test.py
ModuleNotFoundError: No module named 'ray'
```

---

### Expected behavior

One of the following should be true:

- Example tests explicitly skip when optional dependencies are not installed, or
- CI installs the example dependencies before running `test-examples`, or
- Example tests are excluded from required CI checks.

In all cases, example-related tests should not cause mandatory CI failures
when optional dependencies are intentionally absent.

---

### Additional context

PR #315 pins the RLlib example dependencies to a known-working set, addressing
runtime incompatibilities reported in #299. However, the PR is currently
blocked because `test-examples` fails due to the behavior described above.

This issue is intended to document the CI failure and discuss the preferred
long-term solution.
Repository: google-deepmind/meltingpot
Issue: Visualize environment for substrate as well as scenario
Root cause: Documentation Deficiency

For substrate using WORLD.RGB from any one of the agents seems to do the job.
**BUT**
I can't seem to find a way to visualize the environment for a scenario except by adding WORLD.RBG to the permitted keys. Is there any other possible way to do that?

In general, we want to be strict about scenarios. Since those are the actual tests, we do to not have any extra information there that you are not supposed to use at test-time. That's why we tend to hide everything there. If you want to debug or visualise what's going on, feel free to edit the permitted keys. Does that make sense?

Alternatively, the proper way to access privileged information from the Scenario (or Substrate for that matter) is to use the observables. That is:

```python
def my_render_function(timestep: dm_env.TimeStep):
  ...

scenario = scenario_factory.build(...)
scenario.observables().substrate.timestep.subscribe(my_render_function)
# Run your action / observation loop as desired.
```

Thanks @duenez. 
I was doing exactly as you suggested first. But the second approach worked perfectly and looks like a more neat way to approach it without making any changes to the internal library.
Thanks a lot :D
Repository: google-deepmind/meltingpot
Issue: RLlib example utils.py error

Hi, I think https://github.com/deepmind/meltingpot/blob/main/examples/rllib/utils.py#L28

```python
from meltingpot.python import bot
```

should be changed to the following line

```python
from meltingpot.python.utils.bots import policy as bot
```

The current code reports error.

You are right, we did a bit of refactoring of these files, and this example got left behind. Thanks for the catch!
Repository: google-deepmind/meltingpot
Issue: Simulation speedup disappears with Python 3.9
Root cause: Environment Incompatibility

Hi, previously, there is a big improvement in the simulation speed with Python 3.7 (https://github.com/deepmind/meltingpot/issues/7#issuecomment-1111847462). However, recently, with Python 3.9, the speedup disappears. It would be great if you can check the code! Thank you!

Sorry, my bad, it is a false alarm. I installed the old lab2d.
Repository: google-deepmind/meltingpot
Issue: Suppress warning

Hey, dear authors, I saw many warnings, shown below. It would be great if we can suppress these warnings.

```
W @/home/aiops/user/Projects/Github/DeepMind/meltingpot/meltingpot/lua/modules/base_simulation.lua:225] GameObject 'scene' did not have a Transform component, but explicitly specifying one is strongly preferr
ed. Using a default.
```

This is probably happening because you have created some custom game objects and forgotten to list the transform component. You can get rid of the warning message without changing anything about the behavior of your environment. All you need to do is make sure all your game objects contain the following in their components dictionary `{"component": "Transform",}`.

So it will look like:

```
NEW_GAME_OBJECT = {
    "name": "new_game_object_name",
    "components": [
          ...,
          {"component": "Transform",},
          ...,
    ]
}
```




In particular, we don't require scene objects to be explicitly created for a level. When that happens, we just create a default one, which then triggers the warning as you see it, because it didn't have a transform specified.

> In particular, we don't require scene objects to be explicitly created for a level. When that happens, we just create a default one, which then triggers the warning as you see it, because it didn't have a transform specified.

Hi, actually I did not change the code. I  just ran the code in the repo and got the warnings.

There is no such warning in the latest version of Melting Pot. I am closing this issue.
Repository: google-deepmind/meltingpot
Issue: Dear authors, will you plan to release the training curves?

Dear authors, 

Thanks for the great contribution to the MARL community. Melting Pot provides a great set of evaluation scenarios for MARL research. In your paper, we can find many convincing results both in the main text and the appendix. It would be great if you can also put training curves as well as evaluation curves (like many Atari/mujoco RL papers) since it can help researchers to check how many training timesteps we need to get some specific level of performance.  It is my humble suggestion, thank you!

Best regards,
A Melting Pot user

This is a great idea, and you have good timing :). I don't see any reason why we couldn't do it for the next release, which we are preparing now. We are re-running all the baselines anyway.

> This is a great idea, and you have good timing :). I don't see any reason why we couldn't do it for the next release, which we are preparing now. We are re-running all the baselines anyway.

Cool. Thank you!
Repository: google-deepmind/meltingpot
Issue: [Feature] Explicitly force bots in scenarios to use CPUs
Root cause: Configuration Error

Hi, all. For the saved TF models, when using GPUs, it will raise errors:

```
2 root error(s) found.
  (0) UNIMPLEMENTED:  The Conv2D op currently only supports the NHWC tensor format on the CPU. The op was given the format: NCHW
	 [[{{node learner_agent/step/encoder/pixels_encoder/conv_2d_0/convolution}}]]
	 [[StatefulPartitionedCall/StatefulPartitionedCall/learner_agent/step/encoder/pixels_encoder/Relu_1/_6]]
  (1) UNIMPLEMENTED:  The Conv2D op currently only supports the NHWC tensor format on the CPU. The op was given the format: NCHW
	 [[{{node learner_agent/step/encoder/pixels_encoder/conv_2d_0/convolution}}]]
0 successful operations.
0 derived errors ignored.
	 [[StatefulPartitionedCall]]
```

My suggestion is when creating scenarios, we can forcely set CPU for TensorFlow. 
```python
tf.config.set_visible_devices([], 'GPU')
```

It works for TF 2.1+. Thank you!

We recently changed the inference to use GPU is available. Internally this didn't fail. Might be a version incompatibility? John could you check?

In 9b922e9ddfda2eb447449f7e38fba515255402c0 I specified a `DistributionStrategy` for bots that defaults to CPU. Does that not work?

> In [9b922e9](https://github.com/deepmind/meltingpot/commit/9b922e9ddfda2eb447449f7e38fba515255402c0) I specified a `DistributionStrategy` for bots that defaults to CPU. Does that not work?

Thanks. Maybe there was an implicit behaviour that caused the issue. I will check it and try to find the reason.
Repository: google-deepmind/meltingpot
Issue: Training procedure in Collaborative Cooking.

Hi, 
I was wondering if you used any intrinsic rewards/exploration methods besides an entropy bonus for training agents in collaborative cooking environments? I am using A3C and I'm having trouble getting any rewards in those environments due to sparsity. 

Hi!

In order to make the baseline results reported in the overview paper as simple and pure as possible, we didn't use any pseuodrewards, intrinsic motivations, or exploration bonuses for any of the environments.

You are correct that the collaborative cooking environments are very difficult without the domain-specific pseudorewards. The baseline algorithms we tested all got scores of 0.0, exactly as you are reporting. You can see this number in the table in Appendix F of the paper. They get scores of 0.0 in self play, i.e. in training. They get higher scores in some of the scenarios because in those cases they are paired with a background population that is already skilled in the game.

I believe most research on collaborative cooking environments uses domain-specific pseudorewards to get initial learning off the ground. Our implementation also supports these. Check the original papers for the exact details, but I believe that by far the most important pseudoreward is the one for putting tomatoes in cooking pots. You can enable it by editing [this line](https://github.com/deepmind/meltingpot/blob/main/meltingpot/python/configs/substrates/collaborative_cooking.py#L733) to set reward = 1.

You may also be interested to know that in the next release -- which is coming very soon! -- we will be adding to the suite several more collaborative cooking environments, which will more closely correspond to the maps used in other papers.

Thank you very much, that answers all my questions.

I also found training Collaborative Cooking is not easy with shared parameters of agent networks. The rewards are zeros during the training of the substrate. 

@jzleibo By the way, in Melting Pot, are there any substrates that are easier to train?  I found training MARL methods that have experience replay need much larger RAM than Atari games since there are many agents. I think on-policy methods cost less RAM.
Repository: google-deepmind/meltingpot
Issue: Python seed is not forwarded to Lua
Root cause: Configuration Error

https://github.com/deepmind/meltingpot/commit/449ac8b083c5bbb0d338404e7094d63c00beb5e3 introduced a bug where the random seed was no longer forwarded from Python to Lua, meaning that the Lua-controlled parts of the environment were no longer randomized.

Fixed in c009d0a601670c51107994e4bf63db6ed2f71c0a
Repository: google-deepmind/meltingpot
Issue: Substrate tutorial
Root cause: Validation Gap

Hello, 
I think that there is a minor issue in the substrate tutorial when I run:
`python3 meltingpot/examples/tutorial/harvest/play_harvest.py --observation WORLD.RGB --display_text`
I got an index out of range in:

```
line 317, in run_episode
    player_prefix = player_prefixes[player_index]
```
I fresh installed meltingpot  yesterday.

Anyway, the finished example works perfectly, is just the blank project



Thanks for letting us know! We have an internal fix that will be getting pushed here shortly.