Repository: pytorch
Issue: circular module reference raises RecursionError
Labels: module: nn,triaged

## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->

## To Reproduce

Steps to reproduce the behavior:

1. Run the following code snippet
```py
import torch
import torch.nn as nn

class ModuleA(nn.Module):
    def __init__(self):
        super().__init__()
        self.bs = nn.ModuleList()

class ModuleB(nn.Module):
    def __init__(self, a):
        super().__init__()
        self.a = a

a = ModuleA()
a.bs.append(ModuleB(a))
a.to("cpu")
```
<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

## Expected behavior

Running the code snippet raises `RecursionError` due to the circular reference. In this particular case, `a.to("cpu")` should have no effect because there are no tensors to move, but in general, it should move all tensors to the corresponding device without raising any error. See `additional context` for an explanation of why a circular reference like this might be useful in practice.

## Environment

```
PyTorch version: 0.4.1
Is debug build: No
CUDA used to build PyTorch: 9.0.176

OS: Ubuntu 16.04.3 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
CMake version: version 3.5.1

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 7.5.17
GPU models and configuration: GPU 0: GeForce GTX 1080
Nvidia driver version: 384.130
cuDNN version: Could not collect

Versions of relevant libraries:
[pip] Could not collect
[conda] pytorch                   0.4.1            py37ha74772b_0  
```

## Additional context

A particular case where a circular reference is useful is when `ModuleB.forward` computes something based on attributes of `ModuleA`, and `ModuleA.forward` computes the sum of `ModuleB.forward` for every module in its `ModuleList`. A workaround to avoid the `RecursionError` is to pass all the attributes explicitly to the constructor of `ModuleB`, but it becomes impractical when many parameters are involved and it's particularly problematic for mutable shared state like numbers that get passed by value. For example, consider the following code snippet:

```py
import torch
import torch.nn as nn

class ModuleA(nn.Module):
    def __init__(self):
        super().__init__()
        self.n = 123
        self.bs = nn.ModuleList()

    def forward(self, x):
        output = 0.0
        for b in self.bs:
            output += b(x)
        return output

class ModuleB(nn.Module):
    def __init__(self, a):
        super().__init__()
        self.a = a

    def forward(self, x):
        if self.a.n > 100:
            return x
        else:
            return x + 1

a = ModuleA()
a.bs.append(ModuleB(a))
x = torch.zeros(1)
print(a(x)) # 0
a.n = 99
print(a(x)) # 1
```
This code snippet has a circular module reference and works without problems, but `a.to(device)` will result in infinite recursion. A workaround to avoid infinite recursion when calling `a.to(device)` is to make the constructor of `ModuleB` accept `n` as an argument rather than `a`, and store it as an attribute, but changes in `a.n` won't have the desired effect since it's no longer shared.

cc @albanD @mruberry @jbschlosser