Repository: pytorch
Issue: [JIT / NNC] Incomplete Ops Coverage
Labels: oncall: jit

## 🐛 Bug

We have run into internal and external models which are missing fusion opportunities because we a) don't support all activations in autodiff and b) don't support all activations in NNC. These are very easy to add and important for generating fusions. 

## Symbolic Script Support (necessary for fusing training):
[Pointwise Activations](https://pytorch.org/docs/stable/nn.html#non-linear-activations-weighted-sum-nonlinearity)

- [ ] ELU
- [x] Hardshrink (Nick)
- [ ] Hardsigmoid
- [x] Hardtanh (Nick)
- [x] Hardswish
- [x] LeakyRelu (edvard)
- [ ] LogSigmoid
- [ ] PReLU
- [x] ReLU
- [ ] ReLU6 (Bin)
- [ ] RReLU
- [ ] SELU
- [ ] CELU
- [x] GELU
- [x] Sigmoid
- [ ] SiLU
- [ ] Softplus
- [ ] Softshrink (Nick)
- [ ] Softsign
- [x] Tanh
- [ ] Tanhshrink
- [x] Threshold


[Normalization Layers](https://pytorch.org/docs/stable/nn.html#normalization-layers)

- [ ] BatchNorm1d
- [ ] BatchNorm2d
- [ ] BatchNorm3d
- [ ] GroupNorm
- [ ] SyncBatchNorm
- [ ] InstanceNorm1d
- [ ] InstanceNorm2d
- [ ] InstanceNorm3d
- [ ] LayerNorm
- [ ] LocalResponseNorm

[Pooling layers](https://pytorch.org/docs/stable/nn.html#pooling-layers)

 - [ ] MaxPool1d
 - [ ] MaxPool2d
 - [ ] MaxPool3d
 - [ ] MaxUnpool1d
 - [ ] MaxUnpool2d
 - [ ] MaxUnpool3d
 - [ ] AvgPool1d
 - [ ] AvgPool2d
 - [ ] AvgPool3d
 - [ ] FractionalMaxPool2d
 - [ ] LPPool1d
 - [ ] LPPool2d
 - [ ] AdaptiveMaxPool1d
 - [ ] AdaptiveMaxPool2d
 - [ ] AdaptiveMaxPool3d
 - [ ] AdaptiveAvgPool1d
 - [ ] AdaptiveAvgPool2d
 - [ ] AdaptiveAvgPool3d

[Padding Layers](https://pytorch.org/docs/stable/nn.html#padding-layers)

 - [ ] ReflectionPad1d
 - [ ] ReflectionPad2d
 - [ ] ReplicationPad1d
 - [ ] ReplicationPad2d
 - [ ] ReplicationPad3d
 - [ ] ZeroPad2d
 - [ ] ConstantPad1d
 - [ ] ConstantPad2d
 - [ ] ConstantPad3d

**Other Ops**

 - [ ] nan_to_num
 - [ ] t (@Chillee )
 - [ ] transpose (@Chillee )
 - [ ] flatten (@Chillee )
 - [ ] mv


## NNC Support (necessary for fusion)
[Pointwise Activations](https://pytorch.org/docs/stable/nn.html#non-linear-activations-weighted-sum-nonlinearity)

- [ ] ELU
- [ ] Hardshrink
- [x] Hardsigmoid
- [x] Hardtanh
- [x] Hardswish
- [ ] LeakyRelu
- [ ] LogSigmoid
- [ ] PReLU
- [x] ReLU
- [ ] ReLU6 (Bin)
- [ ] RReLU (Nick)
- [ ] SELU
- [ ] CELU
- [ ] GELU (@eellison )
- [x] Sigmoid
- [ ] SiLU
- [ ] Softplus
- [ ] Softshrink
- [ ] Softsign
- [x] Tanh
- [ ] Tanhshrink
- [x] Threshold

[Normalization Layers](https://pytorch.org/docs/stable/nn.html#normalization-layers)

- [ ] BatchNorm1d
- [ ] BatchNorm2d
- [ ] BatchNorm3d
- [ ] GroupNorm
- [ ] SyncBatchNorm
- [ ] InstanceNorm1d
- [ ] InstanceNorm2d
- [ ] InstanceNorm3d
- [ ] LayerNorm
- [ ] LocalResponseNorm

[Pooling layers](https://pytorch.org/docs/stable/nn.html#pooling-layers)

 - [ ] MaxPool1d
 - [ ] MaxPool2d
 - [ ] MaxPool3d
 - [ ] MaxUnpool1d
 - [ ] MaxUnpool2d
 - [ ] MaxUnpool3d
 - [ ] AvgPool1d
 - [ ] AvgPool2d
 - [ ] AvgPool3d
 - [ ] FractionalMaxPool2d
 - [ ] LPPool1d
 - [ ] LPPool2d
 - [ ] AdaptiveMaxPool1d
 - [ ] AdaptiveMaxPool2d
 - [ ] AdaptiveMaxPool3d
 - [ ] AdaptiveAvgPool1d
 - [ ] AdaptiveAvgPool2d
 - [ ] AdaptiveAvgPool3d

[Padding Layers](https://pytorch.org/docs/stable/nn.html#padding-layers)

 - [ ] ReflectionPad1d
 - [ ] ReflectionPad2d
 - [ ] ReplicationPad1d
 - [ ] ReplicationPad2d
 - [ ] ReplicationPad3d
 - [ ] ZeroPad2d
 - [ ] ConstantPad1d
 - [ ] ConstantPad2d
 - [ ] ConstantPad3d

**Other Ops**

 - [ ] nan_to_num
 - [x] t (@Chillee )
 - [x] transpose (@Chillee )
 - [ ] flatten (@Chillee )
 - [ ] mv

cc @gmagogsfm