Repository: google-deepmind/meltingpot
Issue: RLLib example not using conv as configured
Root cause: Documentation Deficiency

While convolutions are specified in the RLLib example the actual models that are spawned are LSTM+FC only.

The configuration in https://github.com/deepmind/meltingpot/blob/main/examples/rllib/self_play_train.py:
```
    config["model"]["conv_filters"] = [[16, [8, 8], 8], [128, [11, 11], 1]]
    config["model"]["conv_activation"] = "relu"

```

The log of the architecture of one agent:
```
(RolloutWorker pid=287877) Model: "model_9"
(RolloutWorker pid=287877) __________________________________________________________________________________________________
(RolloutWorker pid=287877)  Layer (type)                   Output Shape         Param #     Connected to                     
(RolloutWorker pid=287877) ==================================================================================================
(RolloutWorker pid=287877)  seq_in (InputLayer)            [(None,)]            0           []                               
(RolloutWorker pid=287877)                                                                                                   
(RolloutWorker pid=287877)  tf_op_layer_av_wk1/SequenceMas  [()]                0           ['seq_in[0][0]']                 
(RolloutWorker pid=287877)  k/Max (TensorFlowOpLayer)                                                                        
(RolloutWorker pid=287877)                                                                                                   
(RolloutWorker pid=287877)  tf_op_layer_av_wk1/SequenceMas  [()]                0           ['tf_op_layer_av_wk1/SequenceMask
(RolloutWorker pid=287877)  k/Maximum (TensorFlowOpLayer)                                   /Max[0][0]']                     
(RolloutWorker pid=287877)                                                                                                   
(RolloutWorker pid=287877)  tf_op_layer_av_wk1/SequenceMas  [(None, 1)]         0           ['seq_in[0][0]']                 
(RolloutWorker pid=287877)  k/ExpandDims (TensorFlowOpLaye                                                                   
(RolloutWorker pid=287877)  r)                                                                                               
(RolloutWorker pid=287877)                                                                                                   
(RolloutWorker pid=287877)  tf_op_layer_av_wk1/SequenceMas  [(None,)]           0           ['tf_op_layer_av_wk1/SequenceMask
(RolloutWorker pid=287877)  k/Range (TensorFlowOpLayer)                                     /Maximum[0][0]']                 
(RolloutWorker pid=287877)                                                                                                   
(RolloutWorker pid=287877)  tf_op_layer_av_wk1/SequenceMas  [(None, 1)]         0           ['tf_op_layer_av_wk1/SequenceMask
(RolloutWorker pid=287877)  k/Cast (TensorFlowOpLayer)                                      /ExpandDims[0][0]']              
(RolloutWorker pid=287877)                                                                                                   
(RolloutWorker pid=287877)  inputs (InputLayer)            [(None, None, 267)]  0           []                               
(RolloutWorker pid=287877)                                                                                                   
(RolloutWorker pid=287877)  h (InputLayer)                 [(None, 256)]        0           []                               
(RolloutWorker pid=287877)                                                                                                   
(RolloutWorker pid=287877)  c (InputLayer)                 [(None, 256)]        0           []                               
(RolloutWorker pid=287877)                                                                                                   
(RolloutWorker pid=287877)  tf_op_layer_av_wk1/SequenceMas  [(None, None)]      0           ['tf_op_layer_av_wk1/SequenceMask
(RolloutWorker pid=287877)  k/Less (TensorFlowOpLayer)                                      /Range[0][0]',                   
(RolloutWorker pid=287877)                                                                   'tf_op_layer_av_wk1/SequenceMask
(RolloutWorker pid=287877)                                                                  /Cast[0][0]']                    
(RolloutWorker pid=287877)                                                                                                   
(RolloutWorker pid=287877)  lstm (LSTM)                    [(None, None, 256),  536576      ['inputs[0][0]',                 
(RolloutWorker pid=287877)                                  (None, 256),                     'h[0][0]',                      
(RolloutWorker pid=287877)                                  (None, 256)]                     'c[0][0]',                      
(RolloutWorker pid=287877)                                                                   'tf_op_layer_av_wk1/SequenceMask
(RolloutWorker pid=287877)                                                                  /Less[0][0]']                    
(RolloutWorker pid=287877)                                                                                                   
(RolloutWorker pid=287877)  logits (Dense)                 (None, None, 11)     2827        ['lstm[0][0]']                   
(RolloutWorker pid=287877)                                                                                                   
(RolloutWorker pid=287877)  values (Dense)                 (None, None, 1)      257         ['lstm[0][0]']                   
(RolloutWorker pid=287877)                                                                                                   
(RolloutWorker pid=287877) ==================================================================================================
(RolloutWorker pid=287877) Total params: 539,660
(RolloutWorker pid=287877) Trainable params: 539,660
(RolloutWorker pid=287877) Non-trainable params: 0
(RolloutWorker pid=287877) __________________________________________________________________________________________________
(RolloutWorker pid=287877)   ...

```

I think that this is caused by how the observation spaces are flattened in RLLib.


```
pip freeze
absl-py==1.0.0
aiohttp==3.8.1
aiohttp-cors==0.7.0
aioredis==1.3.1
aiosignal==1.2.0
astunparse==1.6.3
async-timeout==4.0.2
attrs==21.4.0
blessed==1.19.1
cachetools==5.0.0
certifi==2021.10.8
charset-normalizer==2.0.12
chex==0.1.1
click==8.0.4
cloudpickle==2.0.0
colorful==0.5.4
contextlib2==21.6.0
cycler==0.11.0
Deprecated==1.2.13
dm-env==1.5
-e git+https://github.com/deepmind/meltingpot@79f8756389d590c2b965de37c9b54cdf8679f7a7#egg=dm_meltingpot
dm-tree==0.1.6
dmlab2d @ https://github.com/deepmind/lab2d/releases/download/release_candidate_2021-07-13/dmlab2d-1.0-cp39-cp39-manylinux_2_31_x86_64.whl
filelock==3.6.0
flatbuffers==2.0
fonttools==4.30.0
frozenlist==1.3.0
gast==0.5.3
google-api-core==2.7.1
google-auth==2.6.0
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
googleapis-common-protos==1.55.0
gpustat==1.0.0b1
grpcio==1.43.0
gym==0.21.0
h5py==3.6.0
hiredis==2.0.0
idna==3.3
imageio==2.16.1
immutabledict==2.2.1
importlib-metadata==4.11.3
jax==0.3.1
jaxlib==0.3.0
jsonschema==4.4.0
keras==2.8.0
Keras-Preprocessing==1.1.2
kiwisolver==1.4.0
libclang==13.0.0
lz4==4.0.0
Markdown==3.3.6
matplotlib==3.5.1
ml-collections==0.1.1
msgpack==1.0.3
multidict==6.0.2
networkx==2.7.1
nose==1.3.7
numpy==1.22.3
nvidia-ml-py3==7.352.0
oauthlib==3.2.0
opencensus==0.8.0
opencensus-context==0.1.2
opt-einsum==3.3.0
packaging==21.3
pandas==1.4.1
Pillow==9.0.1
prometheus-client==0.13.1
protobuf==3.19.4
psutil==5.9.0
py-spy==0.3.11
pyasn1==0.4.8
pyasn1-modules==0.2.8
pygame==2.1.2
pyparsing==3.0.7
pyrsistent==0.18.1
python-dateutil==2.8.2
pytz==2021.3
PyWavelets==1.3.0
PyYAML==6.0
ray==1.11.0
redis==4.1.4
requests==2.27.1
requests-oauthlib==1.3.1
rsa==4.8
Rx==3.2.0
scikit-image==0.19.2
scipy==1.8.0
six==1.16.0
smart-open==5.2.1
tabulate==0.8.9
tensorboard==2.8.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorboardX==2.5
tensorflow==2.8.0
tensorflow-io-gcs-filesystem==0.24.0
termcolor==1.1.0
tf-estimator-nightly==2.8.0.dev2021122109
tifffile==2022.2.9
toolz==0.11.2
typing_extensions==4.1.1
urllib3==1.26.8
wcwidth==0.2.5
Werkzeug==2.0.3
wrapt==1.14.0
yarl==1.7.2
zipp==3.7.0
```


Good catch! Do you know how to fix it? Could you show us how? None of us are experts on RLLib...

I was wrong, convolutions are in fact used. After investigation, the issue is that the architecture is not properly logged. There are 7 networks that are used to pre-process the inputs before that they are flatten and these 7 networks are not logged.

The parameter count confirms this observation, only the parameters of the LSTM are counted here.

And looking at the memory consumption also confirms that convolutions are used!

Sorry for the false alarm :)  

Hello @Manuscrit , 
May I ask how did you find that out ?
I would like to visualize the entire architecture but I am not sure how. 

I understand that the convolutional layer transforms the input image into a feature vector and then concatenates the other observations to finally use this new feature vector as the input to the LSTM network that outputs the agent actions. 

however the network description is as simple as:

```
config["model"]["conv_filters"] = [[16, [8, 8], 8], [128, [11, 11], 1]]
config["model"]["conv_activation"] = "relu"
config["model"]["post_fcnet_hiddens"] = [256]
config["model"]["post_fcnet_activation"] = "relu"
config["model"]["use_lstm"] = True
config["model"]["lstm_use_prev_action"] = True
config["model"]["lstm_use_prev_reward"] = False
config["model"]["lstm_cell_size"] = 256
```
Does RLlib automatically infer that it must do that concatenation and feed the LSTM network as describes above ?
Maybe this is a super dumb question. 

Thanks!



If I remember correctly, before the concatenation that will lead to the LSTM, last FC and the "decision head", before this concatenation, you have a FC for each of the inputs that is not an image. (So the frames go through the Conv +  post_fcnet_hiddens and the other inputs only go through post_fcnet_hiddens before that all of that is given to the LSTM and the last FC)
The problem is that when you print the architecture, all the "post_fcnet_hiddens" applied on the other inputs are not displayed.
You can see the logic for that in the [complex_input_net file](https://github.com/ray-project/ray/blob/master/rllib/models/torch/complex_input_net.py)

> Does RLlib automatically infer that it must do that concatenation and feed the LSTM network as describes above ?

Yes