TypeError: AggregateTokenizer.token_to_id() missing 1 required positional argument: 'lang_id'
State: closed | Pull request: no
Labels: name: bug

name: stale

**TypeError: AggregateTokenizer.token_to_id() missing 1 required positional argument: 'lang_id'**

I am getting `TypeError: AggregateTokenizer.token_to_id() missing 1 required positional argument: 'lang_id'` when doing an inference on a multilingual ASR model trained using multiple tokenizers with `land_id` added in the manifest.

This happens when I change the `decoding strategy to "flashlight"`.

```python
asr_model = nemo_asr.models.EncDecCTCModelBPE.restore_from(nemo_model_file, map_location=device)
decoding_cfg = CTCDecodingConfig()
decoding_cfg.strategy = "flashlight"
.
.
.
asr_model.change_decoding_strategy(decoding_cfg)
```

**Environment overview (please complete the following information)**

 - Environment location: LambdaLabs
 - Method of NeMo install: from source. Installed via ./reinstall

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version: `Ubuntu`
- PyTorch version: `2.3.1+cu121`
- Nemo version: `2.0.0rc2`
- Python version: `Python 3.10.12`
- GPU model: A100


@titu1994 Your help would be greatly appreciated. 

@KunalDhawan can you add the flag to the tokenizer ?

Hi @titu1994, after some investigation into the above issue, I don't think AggregateTokenizer is well suited for the current implementation of flashlight decoder/ word-boosting due to the following reasons:

- With aggregate tokenizer, the tokenizer for each language might have it's own \<unk\> token, due to which we would have to maintain a list for unk_id and self.silence parameters (https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/asr/modules/flashlight_decoder.py#L135) and significantly modify the FlashLightKenLMBeamSearchDecoder algo. 
- In the AggregateTokenizer, the id -> token and id -> lang mapping is one-to-one, but to go from token to id (used [here](https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/asr/modules/flashlight_decoder.py#L167)) we would also need to provide the language corresponding to each token, again because we might have multiple ids corresponding to the same token appearing in different constituent tokenizers of the AggregateTokenizer

A possible way to address this could be - when using Flashlight decoder with Aggregate tokenizer, we can ask the users to input the target language to use. AggregateTokenizer with lang specified == standard tokenizer, which we can use as it with the current Flashlight implementation. We can also have the default lang as 'en' and error out if 1) target lang is not provided and 2) AggregateTokenizer doesn't have a tokenizer corresponding to 'en'. Would love to hear your thoughts on this.



I created a quick patch to FlashLightKenLMBeamSearchDecoder with the above idea and verified that it works with an AggregateTokenizer within the assumptions highlighted in the previous comment: [https://github.com/KunalDhawan/NeMo/commit/0e8d6c53caa59cdd7d6ce3aeb61ecd8483c859e3](https://github.com/KunalDhawan/NeMo/commit/0e8d6c53caa59cdd7d6ce3aeb61ecd8483c859e3). I can create a PR to main with this if everything looks good! 

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.