Repository: pytorch
Issue: jit.trace checker fails for LSTM 
Labels: oncall: jit

## 🐛 Bug

It is not possible to trace an LSTM without `check_trace=False` - i.e the checker is unusable. 

May be related to #23993?

## To Reproduce
A minimal example is as follows:
```python
input_size = 2
hidden_size = 3
num_layers = 2
batch = 1
seq_len = 2

lstm = torch.nn.LSTM(input_size, hidden_size, num_layers)

# args
zeros = torch.zeros(num_layers * 1, batch, hidden_size)
a, b =  torch.randn(seq_len, batch, input_size), (zeros, zeros)
torch.jit.trace(lstm, (a, b))
```

throws
```shell
---------------------------------------------------------------------------
TracingCheckError                         Traceback (most recent call last)
<ipython-input-13-74cf3bd6b360> in <module>
     10 zeros = torch.zeros(num_layers * 1, batch, hidden_size)
     11 a, b =  torch.randn(seq_len, batch, input_size), (zeros, zeros)
---> 12 torch.jit.trace(lstm, (a, b))

~/miniconda3/envs/ml_ci/lib/python3.7/site-packages/torch/jit/__init__.py in trace(func, example_inputs, optimize, check_trace, check_inputs, check_tolerance, _force_outplace, _module_class, _compilation_unit)
    880         return trace_module(func, {'forward': example_inputs}, None,
    881                             check_trace, wrap_check_inputs(check_inputs),
--> 882                             check_tolerance, _force_outplace, _module_class)
    883 
    884     if (hasattr(func, '__self__') and isinstance(func.__self__, torch.nn.Module) and

~/miniconda3/envs/ml_ci/lib/python3.7/site-packages/torch/jit/__init__.py in trace_module(mod, inputs, optimize, check_trace, check_inputs, check_tolerance, _force_outplace, _module_class, _compilation_unit)
   1042                 else:
   1043                     _check_trace([inputs], func, check_trace_method,
-> 1044                                  check_tolerance, _force_outplace, True, _module_class)
   1045     finally:
   1046         torch.jit._trace_module_map = old_module_map

~/miniconda3/envs/ml_ci/lib/python3.7/site-packages/torch/autograd/grad_mode.py in decorate_no_grad(*args, **kwargs)
     47         def decorate_no_grad(*args, **kwargs):
     48             with self:
---> 49                 return func(*args, **kwargs)
     50         return decorate_no_grad
     51 

~/miniconda3/envs/ml_ci/lib/python3.7/site-packages/torch/jit/__init__.py in _check_trace(check_inputs, func, traced_func, check_tolerance, force_outplace, is_trace_module, _module_class)
    683         diag_info = graph_diagnostic_info()
    684         if any(info is not None for info in diag_info):
--> 685             raise TracingCheckError(*diag_info)
    686 
    687 

TracingCheckError: Tracing failed sanity checks!
ERROR: Graphs differed across invocations!
	Graph diff:
		  graph(%self : __torch__.torch.nn.modules.module.Module,
		        %input : Tensor,
		        %2 : (Tensor, Tensor)):
		    %3 : Tensor = prim::GetAttr[name="bias_hh_l1"](%self)
		    %4 : Tensor = prim::GetAttr[name="bias_ih_l1"](%self)
		    %5 : Tensor = prim::GetAttr[name="weight_hh_l1"](%self)
		    %6 : Tensor = prim::GetAttr[name="weight_ih_l1"](%self)
		    %7 : Tensor = prim::GetAttr[name="bias_hh_l0"](%self)
		    %8 : Tensor = prim::GetAttr[name="bias_ih_l0"](%self)
		    %9 : Tensor = prim::GetAttr[name="weight_hh_l0"](%self)
		    %10 : Tensor = prim::GetAttr[name="weight_ih_l0"](%self)
		-   %hx : Tensor, %12 : Tensor = prim::TupleUnpack(%2)
		?                  ^^
		+   %hx.1 : Tensor, %hx : Tensor = prim::TupleUnpack(%2)
		?      ++            ^^
		-   %13 : Tensor[] = prim::ListConstruct(%hx, %hx)
		+   %13 : Tensor[] = prim::ListConstruct(%hx.1, %hx)
		?                                           ++
		    %14 : Tensor[] = prim::ListConstruct(%10, %9, %8, %7, %6, %5, %4, %3)
		    %15 : bool = prim::Constant[value=1]() # /home/julian/miniconda3/envs/ml_ci/lib/python3.7/site-packages/torch/nn/modules/rnn.py:559:0
		    %16 : int = prim::Constant[value=2]() # /home/julian/miniconda3/envs/ml_ci/lib/python3.7/site-packages/torch/nn/modules/rnn.py:559:0
		    %17 : float = prim::Constant[value=0]() # /home/julian/miniconda3/envs/ml_ci/lib/python3.7/site-packages/torch/nn/modules/rnn.py:559:0
		    %18 : bool = prim::Constant[value=1]() # /home/julian/miniconda3/envs/ml_ci/lib/python3.7/site-packages/torch/nn/modules/rnn.py:559:0
		    %19 : bool = prim::Constant[value=0]() # /home/julian/miniconda3/envs/ml_ci/lib/python3.7/site-packages/torch/nn/modules/rnn.py:559:0
		    %20 : bool = prim::Constant[value=0]() # /home/julian/miniconda3/envs/ml_ci/lib/python3.7/site-packages/torch/nn/modules/rnn.py:559:0
		    %21 : Tensor, %22 : Tensor, %23 : Tensor = aten::lstm(%input, %13, %14, %15, %16, %17, %18, %19, %20) # /home/julian/miniconda3/envs/ml_ci/lib/python3.7/site-packages/torch/nn/modules/rnn.py:559:0
		    %24 : (Tensor, Tensor) = prim::TupleConstruct(%22, %23)
		    %25 : (Tensor, (Tensor, Tensor)) = prim::TupleConstruct(%21, %24)
		    return (%25)
	First diverging operator:
	Node diff:
		- %hx : Tensor, %12 : Tensor = prim::TupleUnpack(%2)
		?                ^^
		+ %hx.1 : Tensor, %hx : Tensor = prim::TupleUnpack(%2)
		?    ++            ^^
```

By the looks of it the compiler renames `hx`. 

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

## Expected behaviour

This to pass fine
<!-- A clear and concise description of what you expected to happen. -->

## Environment

Collecting environment information...
PyTorch version: 1.4.0
Is debug build: No
CUDA used to build PyTorch: 10.0

OS: Ubuntu 18.04.3 LTS
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
CMake version: version 3.10.2

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: GeForce GTX 1050 Ti with Max-Q Design
Nvidia driver version: 430.64
cuDNN version: Could not collect

Versions of relevant libraries:
[pip3] numpy==1.13.3
[pip3] torch-stft==0.1.4
[conda] _pytorch_select           0.1                       cpu_0    anaconda
[conda] blas                      1.0                         mkl  
[conda] mkl                       2019.4                      243    anaconda
[conda] mkl-service               2.3.0            py37he904b0f_0    anaconda
[conda] mkl_fft                   1.0.12           py37ha843d7b_0    anaconda
[conda] mkl_random                1.0.2            py37hd81dba3_0    anaconda
[conda] pytorch                   1.4.0           py3.7_cuda10.0.130_cudnn7.6.3_0    pytorch
[conda] torchaudio                0.4.0                    pypi_0    pypi
[conda] torchvision               0.2.1                    py37_0

## Additional context

<!-- Add any other context about the problem here. -->


cc @suo