TDT compute timestamps option and Extra Whitespace handling for SPE
State: closed | Pull request: yes
Labels: name: ASR

name: common

name: Run CICD

# What does this PR do ?

This PR includes option for getting timestamps from TDT Decoder head, including on a segment level. This option is enabled for RNNT and CTC decodings as well. And also introduces workaround for getting tokens and id_s for extra whitespaces (even if the tokenizer was not trained in this way.) This is crucial for getting alignments in the future.

**Collection**: asr, common

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* To get timestamps from TDT decoding head, just change the decoding config of the model in the following way:

```python
import nemo.collections.asr.models as nemo_asr
import omegaconf

tdt_asr_model = nemo_asr.EncDecRNNTBPEModel.from_pretrained(model_name='nvidia/parakeet-tdt_ctc-1.1b')

cfg = model.cfg.decoding
with omegaconf.open_dict(cfg):
    cfg['compute_timestamps']=True

tdt_asr_model.change_decoding_strategy(decoding_cfg=cfg)
```

* To control timestamp level manipulate the following field of decoding config:

```python

cfg = model.cfg.decoding
with omegaconf.open_dict(cfg):
    cfg['compute_timestamps']=True
    cfg['rnnt_timestamp_type'] = 'all' # can be "char", "word", "segment" or "all"

tdt_asr_model.change_decoding_strategy(decoding_cfg=cfg)
```



* To make the SPE tokenizer (already trained) encode the text not ignoring occurring extra whitespaces (which is the default behaviour), modify the tokenizer in the following way:

```python
model.tokenizer.ignore_extra_whitespaces = False
```

* To train the SPE tokenizer to make it ignore extra whitespaces both during training and further encoding, use this flag in the training script:

```shell
python scripts/tokenizers/process_asr_text_tokenizer.py --tokenizer=spe --spe_remove_extra_whitespaces
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


[🤖]: Hi @monica-sekoyan 👋,

We wanted to let you know that a [CICD pipeline](https://github.com/NVIDIA/NeMo/actions/runs/11498295768) for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g