Lids
State: closed | Pull request: yes

# What does this PR do ?

Enable optional output of language IDs during AASR transcription / inference

**Collection**: [Note which collection this PR will affect]
ASR 

# Changelog 
- Aggregate tokenizer modified so that it can output token and sample LIDs
- rnnt_wer.py modified to handle the compute_langs flag and produce output
- rnnt_wer_bpe.py modified to handle the compute_langs flag and produce output

# Usage

See the sample code below. This works similar to timestamp output.
```
# Run this:
# python langs.py

# Sample output
Transcription:   o que es demasiado ruidosa o que tiene palabras que no deberían de ponerlas en canciones porque es como

Langs:  es

Langs chars [{'char': ' o', 'lang': 'es'}, {'char': ' que', 'lang': 'es'}, {'char': ' es', 'lang': 'es'}, {'char': ' de', 'lang': 'es'}, {'char': 'ma', 'lang': 'es'}, {'char': 's', 'lang': 'es'}, {'char': 'i', 'lang': 'es'}, {'char': 'ado', 'lang': 'es'}, {'char': ' ru', 'lang': 'es'}, {'char': 'ido', 'lang': 'es'}, {'char': 's', 'lang': 'es'}, {'char': 'a', 'lang': 'es'}, {'char': ' o', 'lang': 'es'}, {'char': ' que', 'lang': 'es'}, {'char': ' tiene', 'lang': 'es'}, {'char': ' palabra', 'lang': 'es'}, {'char': 's', 'lang': 'es'}, {'char': ' que', 'lang': 'es'}, {'char': ' no', 'lang': 'es'}, {'char': ' debe', 'lang': 'es'}, {'char': 'ría', 'lang': 'es'}, {'char': 'n', 'lang': 'es'}, {'char': ' de', 'lang': 'es'}, {'char': ' poner', 'lang': 'es'}, {'char': 'las', 'lang': 'es'}, {'char': ' en', 'lang': 'es'}, {'char': ' can', 'lang': 'es'}, {'char': 'ciones', 'lang': 'es'}, {'char': ' porque', 'lang': 'es'}, {'char': ' es', 'lang': 'es'}, {'char': ' como', 'lang': 'es'}]

Langs words  [{'word': ' o', 'lang': 'es'}, {'word': ' que', 'lang': 'es'}, {'word': ' es', 'lang': 'es'}, {'word': ' demasiado', 'lang': 'es'}, {'word': ' ruidosa', 'lang': 'es'}, {'word': ' o', 'lang': 'es'}, {'word': ' que', 'lang': 'es'}, {'word': ' tiene', 'lang': 'es'}, {'word': ' palabras', 'lang': 'es'}, {'word': ' que', 'lang': 'es'}, {'word': ' no', 'lang': 'es'}, {'word': ' deberían', 'lang': 'es'}, {'word': ' de', 'lang': 'es'}, {'word': ' ponerlas', 'lang': 'es'}, {'word': ' en', 'lang': 'es'}, {'word': ' canciones', 'lang': 'es'}, {'word': ' porque', 'lang': 'es'}, {'word': ' es', 'lang': 'es'}, {'word': ' como', 'lang': 'es'}]
```

```python
import pytorch_lightning as pl
from omegaconf import OmegaConf, open_dict

from nemo.collections.asr.models import ASRModel


def main():

    trainer = pl.Trainer(gpus=[0], precision=32, logger=False)

    # Char model
    # path = "Path to char model"

    # Multilingual model
    path = "/data3/models/enes_agg_277e.nemo"

    asr_model = ASRModel.restore_from(path, trainer=trainer, map_location='cpu')  

    asr_model.eval()
    asr_model.freeze()

    asr_model.preprocessor.featurizer.dither = 0.0
    asr_model.preprocessor.featurizer.pad_to = 0

    decoding_cfg = OmegaConf.create({})

    with open_dict(decoding_cfg):
        decoding_cfg.strategy = "greedy_batch"
        decoding_cfg.preserve_alignments = True
        decoding_cfg.compute_langs = True

    asr_model.change_decoding_strategy(decoding_cfg)

    # speech sample 
    WAV = 'es.wav'

    hyp, _ = asr_model.transcribe([WAV],
                   batch_size=1, return_hypotheses=True)

    print("Evaluating ---")
    print()
    print("Transcription: ", hyp[0].text)
    print()
    print("Langs: ", hyp[0].langs)
    print()
    print("Langs chars", hyp[0].langs_chars)
    print()
    print("Langs words ", hyp[0].langs_words)
    print()


if __name__ == '__main__':
    main()
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ X] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)