Repository: pytorch
Issue: Implicit module registration permits silent programming errors in torch.nn.Module
Labels: module: nn,triaged

## 🐛 Bug

`torch.nn.Module` implements an implicit module and parameter registration via `__setattr__`:
https://github.com/pytorch/pytorch/blob/8af6281201d19e2be084d54de7a6c4b6656355d3/torch/nn/modules/module.py#L1136-L1181

Currently, Module *requires* that each of its constituent sub-modules is assigned to the Module instance as an attribute, but this requirement is not spelled out in the documentation, nor is it programmatically enforced. This implicit module registration logic permits difficult to debug silent programming errors, especially for novice pytorch users. When some constituent modules are unintentionally not registered, downstream Module uses may produce incorrect behavior: optimizers will silently fail to update parameters of those modules, calls to `train()` and `eval()` will fail to set training state of the modules, etc.

Ideally, module registration should be explicit rather than implicit, or the setattr automagic method  should warn the user when the type of the attribute value is not recognized (the warning could be disabled if desired by an optional flag to the super's constructor).

## To Reproduce

Consider this code:
```
class Net(torch.nn.Module):

  def __init__(self):
    super(Net, self).__init__()
    self.layers = [torch.nn.Linear(4, 4) for i in range(2)]

  def forward(self, x):
    for layer in self.layers:
      x = layer(x)
    return(x)

net = Net()
print(list(net.parameters())) # prints an empty list
```

One would naively expect that Net's parameters consist of parameters of `layers`, but this is not the case because `layers` is a list rather than a `Module`, and so `__setattr__` failed to register its parameters. 

In this very case, one could use `torch.nn.Sequential` or `torch.nn.ModuleList` and get the expected behavior. The point of this issue is that in order to do that, one needs to first know and then remember that this is the blessed way, and that failure to do so will not result in a failing error.

Consider another example which illustrates how a user may be deceived that everything behaves as expected. In this example, a user experiments with network depth, expecting different results depending on the depth.

```
class Net(torch.nn.Module):

  def __init__(self, dim_input, dim_output, dim_bulk, depth):
    super(Net, self).__init__()
    self.layer_in = torch.nn.Linear(dim_input, dim_bulk) # this sub-module is registered properly
    self.layers = [torch.nn.Linear(dim_bulk, dim_bulk) for i in range(depth)] # these sub-modules are not registered!!!
    self.layer_out = torch.nn.Linear(dim_bulk, dim_output) # this sub-module is registered properly

  def forward(self, x):
    f = torch.nn.functional.relu
    x = f(self.layer_in(x))
    for layer in self.layers:
      x = f(layer(x))
    x = self.layer_out(x)
    return(x)

torch.manual_seed(0)
for depth in range(1, 10):
  net = Net(dim_input=1, dim_output=1, dim_bulk=128, depth=depth)
  train_and_evaluate(net)
```

The user fails to register parameters of the middle stack of layers, and as a result they're not optimized as part of training. Yet, all basic sanity expectations of the user are met because the unregistered parameters are still used in the forward pass: the networks produce different results for different values of `depth`, the computational cost is higher for higher `depth`, etc. The user is falsely led to believe that everything works as intended.



cc @albanD @mruberry @jbschlosser