Repository: pytorch
Issue: [LTC] Code-gen IRs and TorchScript lowerings needed by running full TorchBench
Labels: oncall: jit,triaged,module: lazy

Using the prototype #64104 to code-gen all IRs and TorchScript lowerings needed by running full TorchBench. This issue will also keep tracking improvement needed/made over the prototype.

### Tasks (grouped by models)
hf_Bert:
- [x] _copy_from (N.A., see doc)
- [x] add (#66912)
- [x] addcdiv_ (#66895)
- [x] bmm (@wconstab #66998)
- [x] div ((#67044))
- [x] embedding_dense_backward (#67044)
- [ ] expand (@alanwaketan,  non-trivial, save for later)
- [ ] fill_ (@alanwaketan, non-trivial, save for later)
- [x] index_select (#67409)
- [ ] mul (@alanwaketan, non-trivial, save for later)
- [ ] native_batch_norm (@wconstab #66995)
- [ ] native_batch_norm_backward (@wconstab #66995)
- [x] nll_loss_backward (#67238)
- [x] nll_loss_forward (#67238)
- [x] permute (N.A., see doc)
- [x] slice (N.A., see doc)
- [x] sqrt (#67531)
- [ ] sub (@alanwaketan, non-trivial, save for later)
- [x] sum (@wconstab #66984)
- [x] t (N.A., see doc)
- [x] transpose (N.A., see doc)
- [x] unsqueeze (N.A., see doc)
- [x] view (N.A., see doc)
- [x] zero_ (#67126)

resnet50:
- [x] _copy_from_and_resize (N.A., see doc)
- [x] addmm (@wconstab)
- [x] mean.out (#67174)
- [x] random_.to (#67296)
- [x] relu_ (#67126)
- [x] threshold_backward (#67532)

misc:
- [ ] constant_pad_nd (@alanwaketan, non-trivial, save for late)
- [x] leaky_relu (#67572)
- [x] leaky_relu_backward (#67572)
- [ ] repeat (@alanwaketan, non-trivial, save for later)
- [ ] stack (@alanwaketan, non-trivial, save for later)
- [x] threshold (#67579)
- [x] ne (#67584)
- [x] eq (#67584)
- [x] ge (#67584)
- [x] le (#67584)
- [x] gt (#67584)
- [x] lt (#67584)
- [x] masked_fill_.Scalar (bert_pytorch) (#68119)
- [x] std.correction (bert_pytorch) @wconstab (#68106)
- [x] nll_loss2d_forward (bert_pytorch) @davidberard98 

### Ways to contribute
So far, we have received a number of pull requests on code-generating IR, and we do appreciate the community's interests and contributions. Here we list some guidelines so that we can cooperate better and keep the hype!
1. Claim a task before starting coding such that we can avoid potential duplicated efforts. Either ping me in the conversation to add you (and your new task) to the above task list or edit the list yourself.
2. Easy patches that just adding operators to the full_codegen list are always welcomed! Here is an example, #66106.
3. Complicated patches that touch the code-gen mechanism might need more attentions as we internally are actively working on improving the mechanism itself. A community effort like that might well end up with a collision with an internal effort. Moreover, we would love to keep a consistent design. Here is an example, #66192.

### Pro-tips for contributions
1. Use `lazy_tensor_core/scripts/generate_code.sh` to generate the code without building.
2. Generated files are: LazyNativeFunctions.cpp, LazyShapeDtype.h, LazyLazyIr.h, LazyTsLowering.cpp and RegisterLazy.cpp.
3. Be sure to test your newly generated code with AtenLtcTsTensorTest (test_aten_ltc_ts_tensor.cpp), you should be able to find an existing test. If not, please add one. Finally, don't forget to add a test plan in the commit message of your pull request, see #66106.
4. If you happen to modify the code-gen mechanism, it's important to show your generated code in the conversation of the pull request too, see #66192.

### Docs
During the process, we have summarized a [spreadsheet](https://docs.google.com/spreadsheets/d/1EL2cVodD25FCOiMs4LYJPIVoIpXH1iE-2S0uWNlZn5o/edit?usp=sharing) to keep track of some findings and the progress. If you would like to view it, feel free to request accesses.

CC @wconstab, @desertfire, @JackCaoG