Repository: pytorch
Issue: Pytorch JIT Compilation Does Not Finish (Infinite Loop?) for Deeper Models
Labels: oncall: jit,triaged

## 🐛 Bug
The Torchscript JIT compilation does not finish for a deep model (~4K layers). Is this a fundamental limitation? If so, what are alternative techniques to optimize the performance of PyTorch with deep models (hundreds of thousands of layers)?

<!-- A clear and concise description of what the bug is. -->

## To Reproduce

Steps to reproduce the behavior:

1. Run the below minimal file that reproduces the behavior. 
```
import sys
sys.setrecursionlimit(100000)
import torch
class TensorConstant(torch.nn.Module):
    def __init__(self, value):
        # type: (Tensor) -> None
        super(TensorConstant, self).__init__()
        self.val = torch.tensor(value)

    def forward(self, xs):
        # type: (Tuple[Dict[str,Tensor], Dict[str, Tensor]]) -> Tensor
        reals, bools = xs
        return self.val

class Real(torch.nn.Module):
    def __init__(self, name):
        super(Real, self).__init__()
        self.name = name

    def forward(self, xs):
        # type: (Tuple[Dict[str,Tensor], Dict[str, Tensor]]) -> Tensor
        reals, bools = xs
        return reals[self.name]

class Minus(torch.nn.Module):
    def __init__(self, f1, f2):
        super(Minus, self).__init__()
        self.f1 = f1
        self.f2 = f2

    def forward(self, xs):
        # type: (Tuple[Dict[str,Tensor], Dict[str, Tensor]]) -> Tensor
        val = self.f1.forward(xs) - self.f2.forward(xs)
        return val

xs = ({'foo': torch.tensor(1.0, requires_grad=True)}, {})
f1 = TensorConstant(3.0)
f2 = Real("foo")
model = Minus(f1, f2)
for i in range(4_000):
    model = Minus(f2, model)

print(model(xs)) # works and finishes very quickly

scripted_model = torch.jit.script(model) # doesn't finish
```
2. If you press `Ctrl-C` after 30 minutes, you get the following error message, which suggests the Pytorch JIT compilation is stuck in an infinite loop. 
```
 File "/data/user/miniconda3/envs/env/lib/python3.7/site-packages/torch/jit/_recursive.py", line 121, in infer_concrete_type_builder
    sub_concrete_type = concrete_type_store.get_or_create_concrete_type(item)
  File "/data/user/miniconda3/envs/env/lib/python3.7/site-packages/torch/jit/_recursive.py", line 236, in get_or_create_concrete_type
    concrete_type_builder = infer_concrete_type_builder(nn_module)
  File "/data/user/miniconda3/envs/env/lib/python3.7/site-packages/torch/jit/_recursive.py", line 121, in infer_concrete_type_builder
    sub_concrete_type = concrete_type_store.get_or_create_concrete_type(item)
  File "/data/user/miniconda3/envs/env/lib/python3.7/site-packages/torch/jit/_recursive.py", line 245, in get_or_create_concrete_type
    if known_type.equals(concrete_type_builder):
KeyboardInterrupt
```

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

## Expected behavior
I expect either an error message or the JIT compilation to finish. 
<!-- A clear and concise description of what you expected to happen. -->

## Environment

```
Collecting environment information...
PyTorch version: 1.4.0
Is debug build: No
CUDA used to build PyTorch: 10.1

OS: Ubuntu 18.04.3 LTS
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
CMake version: Could not collect

Python version: 3.7
Is CUDA available: No
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA

Versions of relevant libraries:
[pip] numpy==1.18.1
[pip] pytorch-memlab==0.0.4
[pip] torch==1.4.0
[conda] pytorch-memlab            0.0.4                    pypi_0    pypi
[conda] torch                     1.4.0                    pypi_0    pypi
```

## Additional context
I added the `sys.setrecursionlimit` to avoid a `maximum recursion depth exceeded` error in Python. Also, I faced this issue on Pytorch 1.2.0 and 1.3.1, so I don't think it was just introduced when switching to 1.4.0. Thanks for your time!
<!-- Add any other context about the problem here. -->


cc @suo