RNN-Transducer Support (Beta)
State: closed | Pull request: yes

# Changelog
- Adds support for RNN-T based models
- Adds support for RNN-T BPE based models
- Adds basic configs (prototypes) for RNN-T based models.
- Adds examples/asr/speech_to_text_rnnt.py and examples/asr/speech_to_text_rnnt_bpe.py training scripts.
- Adds support for RNN-T loss based on https://github.com/HawkAaron/warp-transducer
- Add support for [Greedy] - `greedy`, `greedy_batch` and [Beam] - `beam`, `tsd` and `alsd` decoding support for RNNT models
- Detach *Decoding from WER computation so it should be easier to update to PTL 1.x
- Add experimental support for fused joint-loss-wer step in order to save memory by trading off compute speed.

# Todo
- [x] Model infrastructure
- [x] Basic config prototypes
- [x] Unittests
  - [x] Save restore RNNT and RNNTBPE models
  - [x] Change vocab for RNNT and RNNTBPE models
  - [x] Basic functionality test for RNNTJoint and RNNTDecoder
- [x] Cleanup commented code
- [x] Detach `RNNTDecoding` and `RNNTBPEDecoding` from PTL 1.x TensorMetric replacement.
- [x] Wrap the output of *Decoding to PTL 1.x TensorMetric replacement
- [x] Update configs to support PTL 1.x
- [ ] ~(Maybe) Factorize common elements of CTC and RNNT out into ASRCTCModel and ASRRNNTModel.~
- [ ] ~(Maybe) Refactor `rnn.py`~ 
- [ ] CI pipeline
- [x] Documentation
  - [x] RNNT Models
  - [x] RNNT BPE Models
  - [x] RNNT Loss
  - [x] RNNT Decoding + RNNT WER
  - [x] RNNT WPE Decoding + RNNT BPE WER
  - [x] RNNT Modules
  - [x] `rnn.py`
  - [x] GreedyRNNTInfer
  - [x] BeamRNNTInfer
  - [x] `rnnt_utils.py`


This pull request **introduces 21 alerts** when merging e66d1e17c5d326bd82cd24399827a5eb7865f242 into fd98a89adf80012987851a2cd3c3f4dc63bb8db6 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-374ed711bd66f7baf09c4deb09b2dacc56617b95)

**new alerts:**

* 15 for Unused import
* 3 for Unused local variable
* 3 for Nested loops with same variable

This pull request **introduces 2 alerts** when merging 2c9758adfd8daac523693e925bbcbbe3903dfc4d into 87206f7d160e8bec3d6cf5fa23e479588aebd371 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-fd5d10ac364efccba34ac584b9474f593135b7ca)

**new alerts:**

* 2 for Unused import

This pull request **introduces 2 alerts** when merging 00db8c1a243e9da271f14c18e31dd6202dba9669 into 87206f7d160e8bec3d6cf5fa23e479588aebd371 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-1b417017f4b183cd7c4e211ccb7816a2467c9f24)

**new alerts:**

* 2 for Unused import

This pull request **introduces 2 alerts** when merging e8995a8651d157cb031e1dfd353492d807cb8bdd into 87206f7d160e8bec3d6cf5fa23e479588aebd371 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-25ac57d3c26d8d8da8c2e3fc027a32a02e2564a6)

**new alerts:**

* 2 for Unused import

This pull request **introduces 4 alerts** when merging eb4699bb075c579b565f6ef4f363ce187e54ff51 into 9983329bbb87ac0d03cac17abd8342c46e36cbe6 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-e4cec13ef6c93c6de0b45d54b5fe60a553989b31)

**new alerts:**

* 2 for Unused import
* 2 for &#39;import \*&#39; may pollute namespace

This pull request **introduces 2 alerts** when merging a9202225518c4fc4c01dd3d68bd5e1129178dccf into 9983329bbb87ac0d03cac17abd8342c46e36cbe6 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-b2717831db6174b1bff5ca30126d159bdc7fd18b)

**new alerts:**

* 2 for Unused import

This pull request **introduces 3 alerts** when merging 440ed13231b8945df314771395aa84add6371249 into 4561fbfc91c2ffe80be55fdbe9edc9611bd0b2c0 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-7d3d2ef1f6675f40b104a484083c0a46bbfdbc2c)

**new alerts:**

* 3 for Unused import

This pull request **introduces 3 alerts** when merging 12b95fcdae58577b2c7c016a80aa8d603ff4c1cd into 4561fbfc91c2ffe80be55fdbe9edc9611bd0b2c0 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-1f344ec8112d5510718b77b33d113e7b883b5cff)

**new alerts:**

* 3 for Unused import

This pull request **introduces 3 alerts** when merging db3fb01cebb27f4d908862ee17d454e0af09697d into 5cf042856bf718d27233a7538e0b094ce576d5c4 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-683a3eca810d3a290cb894deaa103adfee0ad22f)

**new alerts:**

* 3 for Unused import

This pull request **introduces 3 alerts** when merging 5120ded2d2357e5157ef830a1b248377e921fbd6 into c09012b6018243861a59a51401393bc8d64c163b - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-cbd08adf18f03c0fe57a0ff91aad3a95cb5203fb)

**new alerts:**

* 3 for Unused import

This pull request **introduces 3 alerts** when merging a929d3c34b6def5d605c7828afef52a122445403 into c09012b6018243861a59a51401393bc8d64c163b - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-342f3946490d55c7e08925330716c290bfc7212b)

**new alerts:**

* 3 for Unused import

This pull request **introduces 3 alerts** when merging 29efc45f300313b57db38aa50ad4b98853682741 into cd55594e0f60a4bcbaf6d0e4196c1404591e0076 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-89464041e99078d0bcb9150d6cdbdf1797951b2d)

**new alerts:**

* 3 for Unused import

This pull request **introduces 3 alerts** when merging dedc3f3a0224baedd3dbea08ba3d1168eefe3bd3 into cd55594e0f60a4bcbaf6d0e4196c1404591e0076 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-6216057de85d892b19c663e4192e7e0bb698fc72)

**new alerts:**

* 3 for Unused import

This pull request **introduces 3 alerts** when merging c7924749ca9ebf448adea1e346796f69bc1e968b into 16a94758d2dafe7af8aa5bef504cf0584ed7d905 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-927c63ee894aeb98c251a8c3823bffca0384a7a7)

**new alerts:**

* 3 for Unused import

This pull request **introduces 4 alerts** when merging 83579321c8d9bb39c6cda565ffbf347e248c1957 into 94b64d32c0c54fd680b34c3b7dfb664b1035481b - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-1f88ef0364d442f79d770df62da8dc85f105a0b7)

**new alerts:**

* 4 for Unused import

This pull request **introduces 3 alerts** and **fixes 1** when merging 7f5611f81afde3caede022e495fd7d3ee39eaf9e into d061dfd7970b95790d713b24f7874511129833e5 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-679105686bb673f2de61d00a235eb2f84c8cecb7)

**new alerts:**

* 3 for Unused import

**fixed alerts:**

* 1 for &#39;import \*&#39; may pollute namespace

@jbalam-nv could you please have a look?

@okuchaiev sure, will review this.

This pull request **introduces 2 alerts** and **fixes 1** when merging f12e0f0d05feb2fc60c90ecae2ca2e5ce15a007d into 17fc2541172dd3d781bfa0b5a1fa41576d4bfe21 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-06785615343a05ebdfa722b4ce7633b90bc06859)

**new alerts:**

* 2 for Unused import

**fixed alerts:**

* 1 for &#39;import \*&#39; may pollute namespace

This pull request **introduces 2 alerts** and **fixes 1** when merging 5129edbaf1ed9e5a4fc7719383c5767be3b83b04 into 610756628b188d0eac44dd384cec4cfce61ee304 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-2e44ecaf4b597a6a1ab531e67fe4e54fd8bbafd0)

**new alerts:**

* 2 for Unused import

**fixed alerts:**

* 1 for &#39;import \*&#39; may pollute namespace

This pull request **introduces 7 alerts** and **fixes 1** when merging 5cc3ab4049bd1a51bc0c424caa90a1c9ce99ef0d into ece60b02f10494d73b3ca109d2c4fa6ef556ec47 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-d51f2239ea14a1a71ff104bad3a3a2e58e25b664)

**new alerts:**

* 7 for Unused import

**fixed alerts:**

* 1 for &#39;import \*&#39; may pollute namespace

This pull request **introduces 8 alerts** when merging 36d7b2e4de97c106ab214833d81d290863dc4f96 into 61ab90646df5ce81928a68bcd5ba2a10c3c648fe - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-a64c9a6f41de392aba8c6b580b73e2ac5200851d)

**new alerts:**

* 8 for Unused import

@redoctopus Thinking about it, the validation of configs is (kinda) part of my other PR here - https://github.com/NVIDIA/NeMo/pull/1328. Maybe lets tackle it in a separate PR?

This pull request **introduces 8 alerts** when merging ebf1968a9e0a310882b8c484ae7a402a26ba13af into 61ab90646df5ce81928a68bcd5ba2a10c3c648fe - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-9e921249edd5cc69f4d40ba090ff00d9f0a5b0d8)

**new alerts:**

* 8 for Unused import

> @redoctopus Thinking about it, the validation of configs is (kinda) part of my other PR here - #1328. Maybe lets tackle it in a separate PR?

Yeah, having it in that PR sounds fine to me! Makes more sense, too; it'll be good to have it as something more inherent to NeMo rather than just ASR-specific.

This pull request **introduces 7 alerts** when merging 6492b3952f5f9987bf3ec0c6fc84c74fcfcc33f1 into 99ed9c2ae5c840a8723f236530ccb1abe8c75377 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-8fc3eb36cbd9c21c439179bdfe90e55567c907ba)

**new alerts:**

* 7 for Unused import

This pull request **introduces 7 alerts** when merging 4eb3866d7cd42fab01696ad34a6c2216ae8f6ff7 into 4de7bd8394fb7e809e04ee8bcadcf9a1711ef497 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-1cd18e101a7619d2a9155348668fa5c866e57e4a)

**new alerts:**

* 7 for Unused import

This pull request **introduces 7 alerts** when merging bf94190301d8a8b541333dfea8c8f795ac96394f into 4de7bd8394fb7e809e04ee8bcadcf9a1711ef497 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-64c7f1f3ebdf514d2ad9cff589f6476c84d3e6fc)

**new alerts:**

* 7 for Unused import

This pull request **introduces 7 alerts** when merging 0aa22c3d09c182747dfcedbf8c12f7397bf08dc8 into 1ac828df423fbcec1b34c650b3a20266bb133dde - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-9be4d213c3009847f4e3d47bab86e62a829c2eb8)

**new alerts:**

* 7 for Unused import

This pull request **introduces 7 alerts** when merging 0c40c6d0f2fb497e1a22667657eaa2f443f39483 into 1ac828df423fbcec1b34c650b3a20266bb133dde - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-27e8d869b68de009a44b113c9b464bafaa04cf7b)

**new alerts:**

* 7 for Unused import