Repository: pytorch
Issue: [TorchScript] Document TorchScript IR
Labels: oncall: jit

## 📚 Documentation

The language specification and the internal IR representation are the two backbones of any language and its implementation. The effort of revamping TorchScript language specification is underway [here](https://github.com/pytorch/pytorch/issues/50434). This issue describes the scope of internal IR documentation.

### Motivation

Internal IR is the the interface between different components of TorchScript implementations, such as 
- Frontend
  - the frontend parse incoming Python ASTs and converts them to TorchScript IR (or sometimes called JIT IR);
- Graph-level transformations
  - graph transformation passes transforms incoming TorchScript IRs into outgoing TorchScript IRs
- TE-level transformations
  - `TensorExpression` pass _lowers_ some TorchScript IR to TE IR
  - `TensorExpression` transformation passes transforms incoming TE+TorchScript IRs to outgoing TE+TorchScript IRs
- IR Execution
  - `Interpreter` implements the functional behaviors of TorchScript IRs
  - `SimpleIREvaluator` implements the functional behaviors of TE IRs
  - `GraphExecutor` takes TorchScript + TE IRs as input and dispatches it to the appropriate downstream execution (?)
- Code-gen
  - `LLVMCodeGen` takes TE+TorchScript IR as inputs and generates LLVM IRs that implement the semantics of TE IR
  - `CudaCodeGen` takes TE + TorchScript IR as inputs and generates Cuda codes that implement the semantics of TE IR
 - Static runtime IR
   - And yes, there is yet another (slightly different IR or slightly different interpretation of IRs ) for the static runtime 

This is just a rough outline of the life-time of IRs inside TorchScript JIT. There are at least two levels of IRs (because there is an explicit lowering from one to another) and perhaps a 3rd one for the static runtime. And the IRs serve multiple purposes, e.g., as an internal transformation IR, an internal execution IR, sometimes as a serialization IR (e.g., to interface w/ mobile's lite interpreter), or as an external interface IR (e.g., to interface 3rd party graph compiler from other hardware vendors). 

### Scope and steps

#### Step 1: Understanding IR architecture
  
Build a holistic view of TorchScript JIT architecture from the IR's point of view
    - clear classification of different IRs are in TorchScript JIT
    - classification of different roles that different IRs serve
    - clear workflow of how different IRs are manipulated inside the JIT
    - identifying any external interfaces that depend on the semantics of the IRs (which may pose backward compatibility issues)

Please refer to [JIT overview](https://github.com/pytorch/pytorch/blob/master/torch/csrc/jit/OVERVIEW.md) for existing information on JIT architecture.

Deliverables
    - Produce (hopefully) a single figure to capture the overall TorchScript JIT architecture through the lens of the lifetime of different IRs
    - Update [JIT overview](https://github.com/pytorch/pytorch/blob/master/torch/csrc/jit/OVERVIEW.md) with the latest information

Timeline: two weeks

#### Step 2: Documenting IR semantics 
 
The actual documentation work happens in this step

- Precise definitions of the type constraints of each IR and some operational semantics of each IR (here is a [FB internal example](https://www.internalfb.com/intern/wiki/Users/pengwu/PyTorch_IR_Specifications/))

- Document any external interfaces that rely on the IRs

 **Deliverables**
       - TorchScript IR specification published to `pytorch.org/docs/jit/`

Timeline: two weeks

#### Step 3: Evaluating current IR design
   
Now that we have built a model of TorchScript IRs, we can evaluate current IR implementations to identify rough edges in the design and inconsistency or missing constraint checking in the implementation. 

Deliverables
     - Create an umbrella issue for all the rough edges and inconsistencies discovered during the documentation process;
     - Organize a design meeting to go through all the uncovered issues and collectively determine the next steps
     - Propose new IR design or improvements (if any)

Timeline: two weeks

cc @gmagogsfm