Bugfixes for decoding with Flashlight decoder
State: closed | Pull request: yes
Labels: name: stale

name: ASR

# What does this PR do ?

Bugfixes for CTC beam search decoding with Flashlight.

1. Fix the maximum decoding length for CTC beam search decoding with Flashlight. The info about the maximum sequence length has to passed to Flashlight decoder otherwise junk/masked frames are decoded (results in insertion errors etc.). This info is passed properly to OpenSeq2Seq decoders and Pyctcdecoders by restricting the sequence length.

2. Fix the dictionary creation for lexicon-free decoding. The Dictionary has to contain a mapping from the ASR model's output IDs to the token in the vocabulary, i.e., exactly the same as the vocabulary in the tokeniser. The order of this mapping matters.

**Collection**: ASR

# Changelog 
- Fix the maximum decoding length for CTC beam search decoding with Flashlight
- Fix the dictionary creation for lexicon-free decoding

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


The PR is ready to merge, however we require all contributors to sign their commits. Could you follow the instructions in the dco bot to resolve this - https://github.com/NVIDIA/NeMo/pull/8856/checks?check_run_id=23589977417

Ok, will do so. But there is one more commit that I want to make which fixes the lexicon-free decoding. Or should I make a new PR for it? It's also only a few lines of code that will change and I think it's better to have it in a single PR.

@titu1994 I hope everything is fine now.

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

@titu1994 @trias702 

I had some time to test the fix with an English model. The model is the same as the baseline from [here](https://arxiv.org/abs/2401.11700) (CTC Conformer) and the LM is also the same (sentencepiece 6gram).

The tokeniser has an `<unk>` token, but the results clearly improve.

## Without bug fix
- beam width: 10, alpha: 0.7, beta: 0.0

| Task                   |   WER(%) |  Ins(%) |   Del(%) |   Sub(%) | 
|:-----------------------|---------:|---------:|---------:|---------:|
| Librispeech_dev_clean  |     8.86 |  4.32 |     0.57 |     3.97 | 
| Librispeech_dev_other  |    14.98 |   4.78 |     1.94 |     8.26 |
| Librispeech_test_clean |     9.15 |    4.39 |     0.66 |     4.10 |
| Librispeech_test_other |    15.06 |    4.96 |     1.88 |     8.23 |

## With bug fix
- beam width: 10, alpha: 0.7, beta: 0.0

| Task                   |   WER(%) |  Ins(%) |   Del(%) |   Sub(%) |
|:-----------------------|---------:|---------:|--------:|---------:|
| Librispeech_dev_clean  |     2.54 |   0.22 |     0.24 |     2.08 |
| Librispeech_dev_other  |     6.64 |  0.52 |     0.87 |     5.25 |
| Librispeech_test_clean |     2.90 |   0.29 |     0.30 |     2.30 |
| Librispeech_test_other |     6.64 |    0.53 |     0.86 |     5.25 |


## For reference
- OS2S beam search decoder results from above paper:
  - dev clean: 2.53
  - test clean: 2.79
  - dev other: 6.50
  - test other: 6.56
- Note: these results used slightly different settings for alpha and beta

> @titu1994 @trias702
> 
> I had some time to test the fix with an English model. The model is the same as the baseline from [here](https://arxiv.org/abs/2401.11700) (CTC Conformer) and the LM is also the same (sentencepiece 6gram).
> 
> The tokeniser has an `<unk>` token, but the results clearly improve.
> ## Without bug fix
> 
>     * beam width: 10, alpha: 0.7, beta: 0.0
> 
> 
> Task 	WER(%) 	Ins(%) 	Del(%) 	Sub(%)
> Librispeech_dev_clean 	8.86 	4.32 	0.57 	3.97
> Librispeech_dev_other 	14.98 	4.78 	1.94 	8.26
> Librispeech_test_clean 	9.15 	4.39 	0.66 	4.10
> Librispeech_test_other 	15.06 	4.96 	1.88 	8.23
> ## With bug fix
> 
>     * beam width: 10, alpha: 0.7, beta: 0.0
> 
> 
> Task 	WER(%) 	Ins(%) 	Del(%) 	Sub(%)
> Librispeech_dev_clean 	2.54 	0.22 	0.24 	2.08
> Librispeech_dev_other 	6.64 	0.52 	0.87 	5.25
> Librispeech_test_clean 	2.90 	0.29 	0.30 	2.30
> Librispeech_test_other 	6.64 	0.53 	0.86 	5.25
> ## For reference
> 
>     * OS2S beam search decoder results from above paper:
>       
>       * dev clean: 2.53
>       * test clean: 2.79
>       * dev other: 6.50
>       * test other: 6.56
> 
>     * Note: these results used slightly different settings for alpha and beta

Sounds good, I appreciate you running these, LGTM, I'll approve

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.