Unable to load pertained titanet_large for EncDecSpeakerLabelModel
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

Unable to load pre-trained speaker label model

**Steps/Code to reproduce bug**
```
import nemo.collections.asr as nemo_asr
speaker_model = nemo_asr.models.EncDecSpeakerLabelModel.from_pretrained(model_name='titanet_large')
```

**Expected behavior**
Just loading a pertained model should not throw an error

**Environment overview (please complete the following information)**

 - Environment location: AWS g5.xlarge
 - Method of NeMo install: pip install with the steps in the guide on Python3.9, not from source

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version ubuntu 20
- PyTorch version: '1.13.0'
- Python version: 3.9

**Additional context**

```
[NeMo W 2022-12-11 06:24:03 nemo_logging:349] /opt/conda/envs/classify-audio/lib/python3.9/site-packages/torch/jit/annotations.py:299: UserWarning: TorchScript will treat type annotations of Tensor dtype-specific subtypes as if they are normal Tensors. dtype constraints are not enforced in compilation either.
      warnings.warn("TorchScript will treat type annotations of Tensor "

[NeMo I 2022-12-11 06:24:03 cloud:56] Found existing object /root/.cache/torch/NeMo/NeMo_1.14.0rc0/titanet-l/492c0ab8416139171dc18c21879a9e45/titanet-l.nemo.
[NeMo I 2022-12-11 06:24:03 cloud:62] Re-using file from: /root/.cache/torch/NeMo/NeMo_1.14.0rc0/titanet-l/492c0ab8416139171dc18c21879a9e45/titanet-l.nemo
[NeMo I 2022-12-11 06:24:03 common:912] Instantiating model from pre-trained checkpoint
[NeMo W 2022-12-11 06:24:04 modelPT:156] If you intend to do training or fine-tuning, please call the ModelPT.setup_training_data() method and provide a valid configuration file to setup the train data loader.
    Train config :
    manifest_filepath: /manifests/combined_fisher_swbd_voxceleb12_librispeech/train.json
    sample_rate: 16000
    labels: null
    batch_size: 64
    shuffle: true
    time_length: 3
    is_tarred: false
    tarred_audio_filepaths: null
    tarred_shard_strategy: scatter
    augmentor:
      noise:
        manifest_path: /manifests/noise/rir_noise_manifest.json
        prob: 0.5
        min_snr_db: 0
        max_snr_db: 15
      speed:
        prob: 0.5
        sr: 16000
        resample_type: kaiser_fast
        min_speed_rate: 0.95
        max_speed_rate: 1.05
    num_workers: 15
    pin_memory: true

[NeMo W 2022-12-11 06:24:04 modelPT:163] If you intend to do validation, please call the ModelPT.setup_validation_data() or ModelPT.setup_multiple_validation_data() method and provide a valid configuration file to setup the validation data loader(s).
    Validation config :
    manifest_filepath: /manifests/combined_fisher_swbd_voxceleb12_librispeech/dev.json
    sample_rate: 16000
    labels: null
    batch_size: 128
    shuffle: false
    time_length: 3
    num_workers: 15
    pin_memory: true

[NeMo I 2022-12-11 06:24:04 label_models:126] Setting angular: true/false in decoder is deprecated and will be removed in 1.13 version, use specific loss with _target_
[NeMo I 2022-12-11 06:24:04 features:267] PADDING: 16
[NeMo W 2022-12-11 06:24:04 modelPT:156] If you intend to do training or fine-tuning, please call the ModelPT.setup_training_data() method and provide a valid configuration file to setup the train data loader.
    Train config :
    manifest_filepath: /manifests/combined_fisher_swbd_voxceleb12_librispeech/train.json
    sample_rate: 16000
    labels: null
    batch_size: 64
    shuffle: true
    time_length: 3
    is_tarred: false
    tarred_audio_filepaths: null
    tarred_shard_strategy: scatter
    augmentor:
      noise:
        manifest_path: /manifests/noise/rir_noise_manifest.json
        prob: 0.5
        min_snr_db: 0
        max_snr_db: 15
      speed:
        prob: 0.5
        sr: 16000
        resample_type: kaiser_fast
        min_speed_rate: 0.95
        max_speed_rate: 1.05
    num_workers: 15
    pin_memory: true

[NeMo W 2022-12-11 06:24:04 modelPT:163] If you intend to do validation, please call the ModelPT.setup_validation_data() or ModelPT.setup_multiple_validation_data() method and provide a valid configuration file to setup the validation data loader(s).
    Validation config :
    manifest_filepath: /manifests/combined_fisher_swbd_voxceleb12_librispeech/dev.json
    sample_rate: 16000
    labels: null
    batch_size: 128
    shuffle: false
    time_length: 3
    num_workers: 15
    pin_memory: true

[NeMo I 2022-12-11 06:24:04 features:267] PADDING: 16
[NeMo E 2022-12-11 06:24:05 common:505] Model instantiation failed!
    Target class:	nemo.collections.asr.models.label_models.EncDecSpeakerLabelModel
    Error(s):
    Traceback (most recent call last):
      File "/opt/conda/envs/classify-audio/lib/python3.9/site-packages/nemo/core/classes/common.py", line 484, in from_config_dict
        instance = imported_cls(cfg=config, trainer=trainer)
      File "/opt/conda/envs/classify-audio/lib/python3.9/site-packages/nemo/collections/asr/models/label_models.py", line 168, in __init__
        self._macro_accuracy = Accuracy(num_classes=num_classes, average='macro', task='multiclass')
      File "/opt/conda/envs/classify-audio/lib/python3.9/site-packages/torchmetrics/classification/accuracy.py", line 490, in __new__
        assert isinstance(top_k, int)
    AssertionError

Traceback (most recent call last):
  File "/home/ubuntu/classify-audio/test.py", line 2, in <module>
    speaker_model = nemo_asr.models.EncDecSpeakerLabelModel.from_pretrained(model_name='titanet_large')
  File "/opt/conda/envs/classify-audio/lib/python3.9/site-packages/nemo/core/classes/common.py", line 851, in from_pretrained
    instance = class_.restore_from(
  File "/opt/conda/envs/classify-audio/lib/python3.9/site-packages/nemo/core/classes/modelPT.py", line 330, in restore_from
    instance = cls._save_restore_connector.restore_from(
  File "/opt/conda/envs/classify-audio/lib/python3.9/site-packages/nemo/core/connectors/save_restore_connector.py", line 235, in restore_from
    loaded_params = self.load_config_and_state_dict(
  File "/opt/conda/envs/classify-audio/lib/python3.9/site-packages/nemo/core/connectors/save_restore_connector.py", line 158, in load_config_and_state_dict
    instance = calling_cls.from_config_dict(config=conf, trainer=trainer)
  File "/opt/conda/envs/classify-audio/lib/python3.9/site-packages/nemo/core/classes/common.py", line 506, in from_config_dict
    raise e
  File "/opt/conda/envs/classify-audio/lib/python3.9/site-packages/nemo/core/classes/common.py", line 498, in from_config_dict
    instance = cls(cfg=config, trainer=trainer)
  File "/opt/conda/envs/classify-audio/lib/python3.9/site-packages/nemo/collections/asr/models/label_models.py", line 168, in __init__
    self._macro_accuracy = Accuracy(num_classes=num_classes, average='macro', task='multiclass')
  File "/opt/conda/envs/classify-audio/lib/python3.9/site-packages/torchmetrics/classification/accuracy.py", line 490, in __new__
    assert isinstance(top_k, int)
AssertionError
```