Repository: google-deepmind/meltingpot
Issue: Cannot load bots trained with self_play_train.py
Root cause: Interface Contract Mismatch

Thank you for your efforts in releasing this exciting library. I am trying to train an agent, and then load that agent to view its behaviour rendered in a pygame window. However, the agents that I train are cannot be loaded into a PermissiveModel because they do not have 'function_tables' (or signatures for that matter).

I add the following to examples/rllib/self_play_train.py
```
   # 7. Train once
   trainer.train()

   # Additions
   trainer.get_policy("av").export_model("./some_path")
   from meltingpot.python import bot
   agent = bot.SavedModelPolicy("./some_path")
```

Running self_play_train.py results in the following error:

```
Traceback (most recent call last):
  File "<repo_path>/meltingpot/examples/rllib/self_play_train.py", line 117, in <module>
    main()
  File "<repo_path>/meltingpot/examples/rllib/self_play_train.py", line 114, in main
    agent = bot.SavedModelPolicy("./some_path")
  File "<repo_path>/meltingpot/meltingpot/python/bot.py", line 103, in __init__
    self._model = permissive_model.PermissiveModel(model)
  File "<repo_path>/meltingpot/meltingpot/python/utils/bots/permissive_model.py", line 51, in __init__
    self._tables = self.model.function_tables()
AttributeError: 'AutoTrackable' object has no attribute 'function_tables'
```

Would it please be possible for you to extend this example to include the saving and loading of bots?

Hello,

   Thank you for your comment. Unfortunately we don't use RLLib internally, and we are far from experts on it. We don't really know how to save models as a TF SavedModel / PermissiveModel. The self_play_train.py script is just a very bare bones example to get some agents training. If you know how to do this, we'd be happy to take a contribution. However, you probably don't require saving/loading your trained agents as SavedModel at all. You could just plug your agent directly into a Scenario (or Substrate).

Hope this helps.

@Muff2n I guess you need to install the right TF. On my side, I can load the pretrained bot of MeltingPot with `tensorflow==2.8.0`. 

You can create a `scenario_creator` which is similar to `substrate_creator` like this (most of the code is copied and modified from the source code of MeltingPot):

```python
import numpy as np

from ml_collections import config_dict
from meltingpot.python import bot as bot_factory
from meltingpot.python import substrate, scenario
from meltingpot.python import substrate as substrate_factory
from meltingpot.python.utils.scenarios.wrappers import agent_slot_wrapper
from meltingpot.python.utils.scenarios.wrappers import all_observations_wrapper
from meltingpot.python.utils.scenarios.wrappers import default_observation_wrapper

from environments.multiagent_wrapper import MeltingPotEnv


def substrate_creator(env_config):
    env = substrate.build(config_dict.ConfigDict(env_config))
    env = MeltingPotEnv(env)
    return env


def scenario_creator(env_config):
    env = build_scenario(config_dict.ConfigDict(env_config))
    env = MeltingPotEnv(env)
    return env


def get_config_scenario(scenario_name, use_individual_observation_names_of_substrate=True):
    config = scenario.get_config(scenario_name)

    if use_individual_observation_names_of_substrate:
        _substrate = substrate_factory.build(config.substrate)
        # Add observations needed by some bots. These are removed for focal players.
        _substrate_observations = set(_substrate.observation_spec()[0])
        config = config.unlock()
        config.update(
            permitted_observations=_substrate_observations,
            use_individual_observation_names_of_substrate=use_individual_observation_names_of_substrate,
        )
    return config.lock()


def build_scenario(config: config_dict.ConfigDict) -> scenario.Scenario:
    """Builds a scenario for the given config.

    Args:
      config: config resulting from `get_config`.
      use_individual_observation_names_of_substrate: scenario uses the same obs of substrate

    Returns:
      The test scenario.
    """
    _substrate = substrate_factory.build(config.substrate)
    _bots = {
        bot_name: bot_factory.build(bot_config)
        for bot_name, bot_config in config.bots.items()
    }

    # Add observations needed by some bots. These are removed for focal players.
    _substrate_observations = set(_substrate.observation_spec()[0])
    _substrate = all_observations_wrapper.Wrapper(
        _substrate, observations_to_share=['POSITION'], share_actions=True)
    _substrate = agent_slot_wrapper.Wrapper(_substrate)
    if 'INVENTORY' not in _substrate_observations:
        _substrate = default_observation_wrapper.Wrapper(
            _substrate, key='INVENTORY', default_value=np.zeros([1]))
    
    if config.use_individual_observation_names_of_substrate:
        permitted_observations = config.permitted_observations
    else:
        permitted_observations = scenario.PERMITTED_OBSERVATIONS & _substrate_observations
    
    return scenario.Scenario(
        substrate=_substrate,
        bots=_bots,
        is_focal=config.is_focal,
        permitted_observations=permitted_observations)
```

and then register the env in RLlib

```python
register_env("meltingpot_scenario", scenario_creator)
```

> Hello,
> 
> Thank you for your comment. Unfortunately we don't use RLLib internally, and we are far from experts on it. We don't really know how to save models as a TF SavedModel / PermissiveModel. The self_play_train.py script is just a very bare bones example to get some agents training. If you know how to do this, we'd be happy to take a contribution. However, you probably don't require saving/loading your trained agents as SavedModel at all. You could just plug your agent directly into a Scenario (or Substrate).
> 
> Hope this helps.

Hi, I agree that the RLLib code provided is a bares bone example because I tried to run the code of A3C with RLlib and only got 1/3 performance of the MeltingPot paper.  I guess to get the performance in the paper, much work should be done in RLlib.

We are definitely interested in better baselines for the repo. If anyone ends up creating one with RLLib, we'd welcome the contribution!

@GoingMyWay Thank you for your response. I am using tensorflow 2.8.0 and raylib 1.11.0. Are you suggesting that you the lines of code that I wrote work for you? I think the problem is that the saved_model protobuf files do not contain the attributes that the PermissiveModel expects.

I was thinking that the easiest fix would be to write a new RayModelPolicy class that inherits from meltingpot.bot.Policy to wrap my trained rllib.policy.Policy. This class would (hopefully) allow me to use the existing scenario code as is.

It appears that you have instead wrapped the scenario by altering the observations. Unfortunately I am not familiar with dm.env.Environment, so I do not understand the observation_spec() method. Would you be able to talk a little bit more about your approach and why it would be better than what I have suggested?

Something to note about the RLLib code is that it is using a different network to the released paper. It seems to be larger than that reported in Appendix C, where the fully connected layers and LSTM had 64 and 128 neurons respectively. Perhaps this is partly the cause of the reduced performance?

> @GoingMyWay Thank you for your response. I am using tensorflow 2.8.0 and raylib 1.11.0. Are you suggesting that you the lines of code that I wrote work for you? I think the problem is that the saved_model protobuf files do not contain the attributes that the PermissiveModel expects.
> 
> I was thinking that the easiest fix would be to write a new RayModelPolicy class that inherits from meltingpot.bot.Policy to wrap my trained rllib.policy.Policy. This class would (hopefully) allow me to use the existing scenario code as is.
> 
> It appears that you have instead wrapped the scenario by altering the observations. Unfortunately I am not familiar with dm.env.Environment, so I do not understand the observation_spec() method. Would you be able to talk a little bit more about your approach and why it would be better than what I have suggested?
> 
> Something to note about the RLLib code is that it is using a different network to the released paper. It seems to be larger than that reported in Appendix C, where the fully connected layers and LSTM had 64 and 128 neurons respectively. Perhaps this is partly the cause of the reduced performance?

@Muff2n Hi

> Are you suggesting that you the lines of code that I wrote work for you? 

I will try to run your code and give you feedback.

> Would you be able to talk a little bit more about your approach and why it would be better than what I have suggested?

The code was modified from the suggested source code for loading scenarios in MeltingPot. Using such a wrapper is much easier and more convenient for evaluation. 

> Perhaps this is partly the cause of the reduced performance?

I am not sure. I am also conducting more experiments to reproduce the results. The A3C used in the paper has many advanced components and the A3C in RLlib is the vanilla A3C.



> We are definitely interested in better baselines for the repo. If anyone ends up creating one with RLLib, we'd welcome the contribution!

Thank you!!!

@[duenez](https://github.com/duenez)

I have a pull request to submit which resolves this issue. How might I go about this? I do not have permission to push my branch to the repo.

Edit: I now realise that I need to fork the repo.

Hello,

Thank you for the contribution. Please ensure you have agreed to the contributor license and are happy with the terms therein. We will get back to you on the pull request itself. Thank!

Fixed by https://github.com/deepmind/meltingpot/pull/30 Thank you!

@duenez that PR #30 fixed issue #26, not this one. The confusion has come because I removed my first PR and split it into two. PR #30 was the first part, I will submit the second part that addresses this issue next week.


This should be fixed now.