Nemo ASR: TypeError: ConfidenceConfig.__init__() got an unexpected keyword argument 'tdt_include_duration'
State: closed | Pull request: no

Dear Team,

I met a strange issue.
After training of new model I am unable to rut it on other machines.
I am getting next error, meanwhile I can run it in the original machine without problem.
Could you advise me? I failed to find any useful information in Internet.
Thank you in advance!

```
[NeMo E 2024-09-01 06:45:53 common:517] Model instantiation failed!
    Target class:	nemo.collections.asr.models.ctc_bpe_models.EncDecCTCModelBPE
    Error(s):	ConfidenceConfig.__init__() got an unexpected keyword argument 'tdt_include_duration'
    Traceback (most recent call last):
      File "/home/deep/.local/lib/python3.10/site-packages/nemo/core/classes/common.py", line 496, in from_config_dict
        instance = imported_cls(cfg=config, trainer=trainer)
      File "/home/deep/.local/lib/python3.10/site-packages/nemo/collections/asr/models/ctc_bpe_models.py", line 73, in __init__
        super().__init__(cfg=cfg, trainer=trainer)
      File "/home/deep/.local/lib/python3.10/site-packages/nemo/collections/asr/models/ctc_models.py", line 97, in __init__
        self.decoding = CTCDecoding(self.cfg.decoding, vocabulary=OmegaConf.to_container(self.decoder.vocabulary))
      File "/home/deep/.local/lib/python3.10/site-packages/nemo/collections/asr/parts/submodules/ctc_decoding.py", line 935, in __init__
        super().__init__(decoding_cfg=decoding_cfg, blank_id=blank_id)
      File "/home/deep/.local/lib/python3.10/site-packages/nemo/collections/asr/parts/submodules/ctc_decoding.py", line 193, in __init__
        self._init_confidence(self.cfg.get('confidence_cfg', None))
      File "/home/deep/.local/lib/python3.10/site-packages/nemo/collections/asr/parts/utils/asr_confidence_utils.py", line 339, in _init_confidence
        ConfidenceConfig() if confidence_cfg is None else ConfidenceConfig(**confidence_cfg)
    TypeError: ConfidenceConfig.__init__() got an unexpected keyword argument 'tdt_include_duration'
    
Traceback (most recent call last):
  File "/maroc/nemoasr/eval.py", line 27, in <module>
    asr_model = nemo_asr.models.EncDecCTCModel.restore_from(restore_path="checkpoints/ASR-Char-Model-Language-hy.nemo",map_location="cpu")
  File "/home/deep/.local/lib/python3.10/site-packages/nemo/core/classes/modelPT.py", line 449, in restore_from
    instance = cls._save_restore_connector.restore_from(
  File "/home/deep/.local/lib/python3.10/site-packages/nemo/core/connectors/save_restore_connector.py", line 241, in restore_from
    loaded_params = self.load_config_and_state_dict(
  File "/home/deep/.local/lib/python3.10/site-packages/nemo/core/connectors/save_restore_connector.py", line 164, in load_config_and_state_dict
    instance = calling_cls.from_config_dict(config=conf, trainer=trainer)
  File "/home/deep/.local/lib/python3.10/site-packages/nemo/core/classes/common.py", line 518, in from_config_dict
    raise e
  File "/home/deep/.local/lib/python3.10/site-packages/nemo/core/classes/common.py", line 510, in from_config_dict
    instance = cls(cfg=config, trainer=trainer)
  File "/home/deep/.local/lib/python3.10/site-packages/nemo/collections/asr/models/ctc_models.py", line 97, in __init__
    self.decoding = CTCDecoding(self.cfg.decoding, vocabulary=OmegaConf.to_container(self.decoder.vocabulary))
  File "/home/deep/.local/lib/python3.10/site-packages/nemo/collections/asr/parts/submodules/ctc_decoding.py", line 935, in __init__
    super().__init__(decoding_cfg=decoding_cfg, blank_id=blank_id)
  File "/home/deep/.local/lib/python3.10/site-packages/nemo/collections/asr/parts/submodules/ctc_decoding.py", line 193, in __init__
    self._init_confidence(self.cfg.get('confidence_cfg', None))
  File "/home/deep/.local/lib/python3.10/site-packages/nemo/collections/asr/parts/utils/asr_confidence_utils.py", line 339, in _init_confidence
    ConfidenceConfig() if confidence_cfg is None else ConfidenceConfig(**confidence_cfg)
TypeError: ConfidenceConfig.__init__() got an unexpected keyword argument 'tdt_include_duration'

```
I am using very plain code

```
import nemo.collections.asr as nemo_asr
wave_file = "test.wav"
#asr_model = nemo_asr.models.EncDecCTCModel.load_from_checkpoint("checkpoints/ASR-Char-Model-Language-hy--val_wer=0.0292-epoch=4.ckpt",map_location="cpu") 
asr_model = nemo_asr.models.EncDecCTCModel.restore_from(restore_path="checkpoints/ASR-Char-Model-Language-hy.nemo",map_location="cpu")
print(asr_model.transcribe(wave_file,batch_size=8))

```

I got the same bug 

When I comment 339-342 lines in 
.local/lib/python3.10/site-packages/nemo/collections/asr/parts/utils/asr_confidence_utils.py
the script works.
```
        # OmegaConf.structured ensures that post_init check is always executed
        #confidence_cfg = OmegaConf.structured(
        #    ConfidenceConfig() if confidence_cfg is None else ConfidenceConfig(**confidence_cfg)
        #)
        #self.confidence_method_cfg = confidence_cfg.method_cfg
```


This could happen when cfg defers from from the restored model config. Can you post the version you were using? 

> This could happen when cfg defers from from the restored model config. Can you post the version you were using?

The version is 
nemo_toolkit             1.23.0
Frankly I am not remember that I touch model config.
Always I prefer to use default configuration. 

I fixed this problem by override my model, removing 'tdt_include_duration' in config of my model

This problem occurs because conflicting between nemo versions