Repository: google-deepmind/meltingpot
Issue: Help Needed to Access Model Architectures and Weights in Meltingpot Assets
Root cause: Documentation Deficiency

I'm currently working with the pretrained models provided in the assets from setup.py, available at this URL (http://storage.googleapis.com/dm-meltingpot/meltingpot-assets-2.1.0.tar.gz). My goal is to utilize these models as a baseline to code an analysis pipeline for further research and comparison.

**Challenges Encountered**
Despite my background predominantly in PyTorch, I have invested time in learning TensorFlow to understand how to effectively work with TensorFlow models. However, I've encountered significant challenges:

Accessing Model Attributes: The architecture and weights of the pretrained models seem to be deeply encapsulated within the provided assets. Standard methods to extract these details like accessing graph and trainable_variables, as suggested by TensorFlow tutorials, have been unsuccessful.
Interacting with the Model: I've tried accessing the "graph" and "node" fields while exploring the model through the PermissiveModel and TF2SavedModelPolicy wrapper but failed to understand the printouts. 
Performance Issues: A single rollout of 200 timesteps for the predator_prey__alley_hunt took about half an hour, indicating that I might not be interacting with the models in the intended manner.

Could you provide any guidance or methods to:
1. Access these model properties more effectively?
2. Understand the model layers and retrieve the trained weights?

Any assistance, pointers, or documentation would be greatly appreciated!

@jagapiou is the expert on this since I believe he wrote PermissiveModel. He is probably the one best placed to answer this. I do know why the structure is so confusing though.. It's because of the history of these saved models. We trained these agents in Jax and initially saved them using a library that was never released in open source. So our release process involved converting the files we saved in our format into TF2 SavedModels since TF2 SavedModel is available in open source. 

On performance, 200 timesteps taking 30 minutes is much too slow, by several orders of magnitude I think.

I didn't write permissive model I'm afraid: it's a piece of internal code we were given permission to open source.

We've not used these bots for any purpose other than executing the SavedModel. For this we definitely get faster than 0.1 steps per second. If all you want to do is run the bots on a Scenario, there's code that does that here:
https://github.com/google-deepmind/meltingpot/blob/957ce59e6f463f7720c6164f71fe2eb890cfc242/meltingpot/utils/evaluation/evaluation.py#L340C14-L340C37

Post-training, we don't access the weights or the architecture ourselves, so not sure how you'd go about doing that.

I  just asked someone who was involved in creating some of the tools we used to produce these models, and he told me that one reason it's hard to extract the weights and architecture is that the saving process converts the weights to constants in the tfgraph. So there's no distinction between the constants that are weights and the constants that are not weights (like architecture parameters).

Thank you all for your assistance in addressing my issues! I now have a clearer understanding of the difficulties involved in retrieving model parameters from the published assets.
I'm glad to report that I've made progress on the rollout speed. It now takes only 1 second per episode of 200 steps. 

I would continue to explore other resources of pretrained models like those from the meltingpot challenge 2023. Do you have any recommendations for additional meltingpot study resources?

Additionally, I would like to address a new challenge I'm facing. I'm currently working on retrieving a Unicode-style format out of the RGB format field of video. The goal is to easily identify the position of opponents, walls, apples, and other elements. Is there any guidance or documentation to help me approach this task effectively?

Once again, I deeply appreciate your timely guidance and explanations, which have proven immensely valuable to me. 

You can add custom observations that surface all that information. Take a look at the [tutorial](https://github.com/google-deepmind/meltingpot/blob/main/docs/substrate_tutorial/index.md) and [documentation](https://github.com/google-deepmind/meltingpot/blob/main/docs/concepts.md) around how to create and modify substrates.

Sounds like your problem is solved @QLs-Learning-Bot so gonna close this out. Feel free to reopen.