Wer fix
State: closed | Pull request: yes
Labels: name: TTS

name: ASR

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.
- Reformats `collections.asr.metrics` wer classes into a single class for model modularity.


**Collection**: [Note which collection this PR will affect]
ASR TTS Multimodal

# Changelog 
- Refactored all `wer` metrics into a single metric
- Moved `ctc_decoding` and `rnnt_decoding` (plus configs) into `parts.submodules`
- Updated ASR tests to reflect new metric classes
- Updated ASR tutorials to reflect new metric classes and decoder locations
- Updated tts/g2p to reflect new metric classes.
- Updates speech_cv to reflect new metric classes and decoder locations.
- Added `wer` getter and setters to all ASR, Speech_cv, and G2P models to allow access to WER metric
- Flyby update of `transformer_bpe` to address translation issue.


# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.
jenkins

# Before your PR is "Ready for review"
**Pre checks**:
- [Y] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [Y] Did you write any new necessary tests?
- [N] Did you add or update any necessary documentation?
- [N] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [Y] New Feature
- [Y ] Bugfix
- [ ] Documentation



## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


@titu1994 <- `wer` metric changes as discussed in thread. (gonna just do the refactoring first before deciding on what to do with bleu)

@AlexGrinch <- fly by change for `transformer_bpe`

> Wait what exactly is going on with the removal of decoding logic?? Revert that

Which, moving the decoding into `parts` or aliasing the `prediction_tensors`?

If the former: there's no reason to keep the decoding in metrics since they're not reflecting respective models anymore. We also don't put any other decoding in metrics, only WER has that, and it's really confusing to have to look in a metrics folder for decoding classes. Given we keep the other decoding stuff in `parts.submodules`, it seems like they're better off over there. 

If the latter: yeah I don't like it, but it's either there or we perform a check in the forward loop. Better solution would be renaming `{rnnt/ctc}_predictions_tensor` to just `predictions_tensor`, but I think people are just accustomed to that name scheme at this point.

@sam1373 can you review just the metrics files (wer.py, wer_bpe.py) to make sure stuff seems correct for SSL ?

jenkins 

Aight, added back in the `fold_consecutive` flag. Still trying to keep the `update` pass model agnostic so had to be a bit hacky with the `__init__`. If that's not gonna work, I may have to either add a conditional in the `update` pass or edit the `ctc_decoding_predictions_tensor` to allow setting the `fold_consecutive` flag. 

jenkins

CI seems to have failed ?

> CI seems to have failed ?

Yeah someone must have added another test over the break. Lemme check

> CI seems to have failed ?

Aight, seems like I messed up a variable assignment in the lambda call. All my pytests passing on local now (exceping that weird tokenization bug in `test_asr_rnnt_encodec_model`

jenkins

jenkins

jenkins

@titu1994 I'm getting through unittests but Jenkins seems to be breaking down around when running Speaker Diarization. Don't really have access to the backend but none of it seems related to my PR.

You can click the link in Jenkins. I'll discuss offline 

jenkins 

jenkins

jenkins

jenkins

jenkins

jenkins

@titu1994 Jenkins is happy. Could I get a final review one last time? (Before this extends into yet more modules.)

jenkins