Repository: pytorch
Issue: [jit] set up views for Autodiff and autograd hooks 
Labels: oncall: jit,triaged

Derived from the bug finding in #19769 , Here is what we found is missing in our Autodiff infrastructure. In Autodiff, we don't set up the views for inputs/outputs after we execute the forward graph(see this [TODO](https://github.com/pytorch/pytorch/blob/be33434d85953fe6f237d475cc8ae2e9a822ed34/torch/csrc/jit/graph_executor.cpp#L350)). This disables us from adding complicated Ad formulas (such as `linear` and `layer_norm`, etc.), because when we calculate higher level gradients, we usually ended up capturing multiple outputs, and some of the output is just a view of another output, and we want the gradient to flow to the original output, which we could not do in the current autodiff infrastructure. 

Taking a example from a graph after we added `linear` in AD, the first order backward graph of a simple model might (second order gradient forward graph) looks like the blow:

```
graph(%1 : int[],
      %4 : Float(*, *),
      %8 : int[],
      %11 : Float(*, *),
      %15 : int[],
      %17 : Float(*, *),
      %19 : Float(*, *),
      %21 : Float(*, *),
      %22 : Tensor):
  %28 : int = prim::Constant[value=1]()
  %146 : int[] = aten::size(%22)
  %24 : Tensor = aten::add(%21, %22, %28)
  %grad_input.2 : Float(*, *) = aten::mm(%24, %19)
  %grad_input.3 : Float(*, *) = aten::mm(%grad_input.2, %17)
  %16 : Tensor = aten::reshape(%grad_input.3, %15)
  %self_size.6 : int[] = aten::size(%grad_input.3)
  %97 : Float(*, *) = aten::t(%grad_input.2)
  %grad_weight.3 : Float(*, *) = aten::mm(%97, %11)
  %self_size.4 : int[] = aten::size(%grad_input.2)
  %72 : Tensor = aten::_grad_sum_to_size(%grad_input.2, %8)
  %63 : Tensor = aten::t(%24)
  %grad_weight.1 : Float(*, *) = aten::mm(%63, %4)
  %self_size.2 : int[] = aten::size(%24)
  %38 : Tensor = aten::_grad_sum_to_size(%24, %1)
  return (%38, %grad_weight.1, %72, %grad_weight.3, %16, %146, %24, %grad_input.2, %self_size.6, %97, %self_size.4, %63, %self_size.2)
```
The reason why we have so much outputs is that the second order backward need to know the intermediate outputs of this graph, and autodiff captures them during the reverse of the forward graph, which can bee seen as below:
```
%grad_bias.4 : Tensor, %grad_weight.1 : Float(*, *), %grad_bias.1 : Tensor, %grad_weight.3 : Float(*, *), %102 : Tensor = prim::DifferentiableGraph[Subgraph=<Graph>, ReverseSubgraph=<Graph>, f_real_outputs=5, df_input_vjps=[0, 1, 2, 3, 4, 6, 7], df_input_captured_inputs=[1, 3, 5, 6]
, df_input_captured_outputs=[5, 6, 7, 8, 9, 10, 11, 12], df_output_vjps=[1, 3, 5, 6, 8]](%22, %input.2, %34, %input.1, %self_size.1, %6, %4, %1, %14)
```

We can see that in the above forward graph, `%97` is just a view variable of `%grad_input.2`, but our Autodiff will create a new variable (instead of a view variable) for `%97` which will make this variable be a leaf that trying to flow the gradients back to it, which is wrong. This could be fixed if we correctly set up the views for `df_input_vjps` and `caputuredOutputs`. 

Because we allow tensor lists to show up in the symbolic differentiable graph in https://github.com/pytorch/pytorch/pull/16784, we could not scan the lists to do the view discovery otherwise we will make the compiler slow on execution. It is possible after the Tensor/Variable merge, we can refactor the autograd to make it safe for tensors to be constructed and automatically track the viewing relationship. 

Creating this issue to keep track of the content discovered and potential solutions. 

CC @zdevito @apaszke @ailzhang @yf225 @bwasti @bddppq