Trying to use wav2vec model with nemo
State: closed | Pull request: no
Labels: name: bug

Hello everyone,
I'm trying to use wav2vec model that I downloaded from the following link ( https://dl.fbaipublicfiles.com/fairseq/wav2vec/xlsr_53_56k.pt ) which corresponds to the multilingual pre-trained wav2vec. I tried to load it with the following command: 
```
from nemo.collections.asr.models.wav2vec.wav2vec_model import Wav2VecEncoderModel
model = Wav2VecEncoderModel.restore_from('xlsr_53_56k.pt')
```
I got this error message : 
```
Traceback (most recent call last):                                                                                                                                                                          
  File "/home/lsam/anaconda3/envs/nemo_100b4/lib/python3.7/tarfile.py", line 1646, in gzopen                                                                                                                
    t = cls.taropen(name, mode, fileobj, **kwargs)                                                                                                                                                          
  File "/home/lsam/anaconda3/envs/nemo_100b4/lib/python3.7/tarfile.py", line 1623, in taropen                                                                                                               
    return cls(name, mode, fileobj, **kwargs)                                                                                                                                                               
  File "/home/lsam/anaconda3/envs/nemo_100b4/lib/python3.7/tarfile.py", line 1486, in __init__                                                                                                              
    self.firstmember = self.next()                                                                                                                                                                          
  File "/home/lsam/anaconda3/envs/nemo_100b4/lib/python3.7/tarfile.py", line 2289, in next                                                                                                                  
    tarinfo = self.tarinfo.fromtarfile(self)                                                                                                                                                                
  File "/home/lsam/anaconda3/envs/nemo_100b4/lib/python3.7/tarfile.py", line 1094, in fromtarfile                                                                                                           
    buf = tarfile.fileobj.read(BLOCKSIZE)                                                                                                                                                                   
  File "/home/lsam/anaconda3/envs/nemo_100b4/lib/python3.7/gzip.py", line 287, in read                                                                                                                      
    return self._buffer.read(size)                                                                                                                                                                          
  File "/home/lsam/anaconda3/envs/nemo_100b4/lib/python3.7/_compression.py", line 68, in readinto                                                                                                           
    data = self.read(len(byte_view))                                                                                                                                                                        
  File "/home/lsam/anaconda3/envs/nemo_100b4/lib/python3.7/gzip.py", line 474, in read                                                                                                                      
    if not self._read_gzip_header():                                                                                                                                                                        
  File "/home/lsam/anaconda3/envs/nemo_100b4/lib/python3.7/gzip.py", line 422, in _read_gzip_header                                                                                                         
    raise OSError('Not a gzipped file (%r)' % magic)
OSError: Not a gzipped file (b'PK')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/lsam/Nemo_100b4/NeMo/nemo/core/classes/modelPT.py", line 509, in restore_from
    return cls._default_restore_from(restore_path, override_config_path, map_location, strict, return_config)
  File "/home/lsam/Nemo_100b4/NeMo/nemo/core/classes/modelPT.py", line 385, in _default_restore_from
    cls.__unpack_nemo_file(path2file=restore_path, out_folder=tmpdir)
  File "/home/lsam/Nemo_100b4/NeMo/nemo/core/classes/modelPT.py", line 1298, in __unpack_nemo_file
    tar = tarfile.open(path2file, "r:gz")
  File "/home/lsam/anaconda3/envs/nemo_100b4/lib/python3.7/tarfile.py", line 1593, in open
    return func(name, filemode, fileobj, **kwargs)
  File "/home/lsam/anaconda3/envs/nemo_100b4/lib/python3.7/tarfile.py", line 1650, in gzopen
    raise ReadError("not a gzip file")
tarfile.ReadError: not a gzip file
```
Is there anywone who have an idea about how does it work or how to fix this bug. Thanks in advance.
With kind regards,


``.restore_from`` only works with ``.nemo`` files. You can try loading regular Pytorch weights via regular Pytorch load_state_dict methods because any NeMo model is a Pytorch module

Thank you very much ! 

this error occurs when ```restore_from``` with ```.nemo ``` files.
```sh
asr_model = nemo_asr.models.EncDecCTCModelBPE.restore_from(
            restore_path=os.path.join(MODELS_URL, "stt_en_conformer_ctc_xlarge.nemo"),
            map_location=device,
)
```
I tried restore_path with this code. But this occurs this error:
 ```sh
not a gzip file
```
and
```sh
Could not load English CTC model
[NeMo I 2024-01-03 15:43:46 tokenizer_utils:125] Getting HuggingFace AutoTokenizer with pretrained_model_name: bert-base-uncased, vocab_file: /tmp/tmpy2yvwun8/tokenizer.vocab_file, special_tokens_dict: {}, and use_fast: False
[NeMo W 2024-01-03 15:43:47 modelPT:135] If you intend to do training or fine-tuning, please call the ModelPT.setup_training_data() method and provide a valid configuration file to setup the train data loader.
    Train config : 
    use_tarred_dataset: false
    label_info_save_dir: null
    text_file: text_train.txt
    labels_file: labels_train.txt
    tokens_in_batch: null
    max_seq_length: 128
    num_samples: -1
    use_cache: true
    cache_dir: null
    get_label_frequences: false
    verbose: true
    n_jobs: 0
    tar_metadata_file: null
    tar_shuffle_n: 1
    shuffle: true
    drop_last: false
    pin_memory: true
    num_workers: 8
    persistent_workers: true
    ds_item: punct_dataset_complete
    
[NeMo W 2024-01-03 15:43:47 modelPT:142] If you intend to do validation, please call the ModelPT.setup_validation_data() or ModelPT.setup_multiple_validation_data() method and provide a valid configuration file to setup the validation data loader(s). 
    Validation config : 
    use_tarred_dataset: false
    label_info_save_dir: null
    text_file: text_dev.txt
    labels_file: labels_dev.txt
    tokens_in_batch: null
    max_seq_length: 128
    num_samples: -1
    use_cache: true
    cache_dir: null
    get_label_frequences: false
    verbose: true
    n_jobs: 0
    tar_metadata_file: null
    tar_shuffle_n: 1
    shuffle: true
    drop_last: false
    pin_memory: true
    num_workers: 8
    persistent_workers: true
    ds_item: punct_dataset_complete
    
[NeMo W 2024-01-03 15:43:47 modelPT:148] Please call the ModelPT.setup_test_data() or ModelPT.setup_multiple_test_data() method and provide a valid configuration file to setup the test data loader(s).
    Test config : 
    use_tarred_dataset: false
    label_info_save_dir: null
    text_file: text_dev.txt
    labels_file: labels_dev.txt
    tokens_in_batch: null
    max_seq_length: 128
    num_samples: -1
    use_cache: true
    cache_dir: null
    get_label_frequences: false
    verbose: true
    n_jobs: 0
    tar_metadata_file: null
    tar_shuffle_n: 1
    shuffle: true
    drop_last: false
    pin_memory: true
    num_workers: 8
    persistent_workers: true
    ds_item: punct_dataset_complete
    
[NeMo W 2024-01-03 15:43:47 nlp_overrides:208] Apex was not found. Please see the NeMo README for installation instructions: https://github.com/NVIDIA/NeMo#megatron-gpt.
[NeMo W 2024-01-03 15:43:47 modelPT:1088] World size can only be set by PyTorch Lightning Trainer.
[NeMo W 2024-01-03 15:43:47 nlp_model:63] Apex was not found. Please see the NeMo README for installation instructions: https://github.com/NVIDIA/NeMo#megatron-gpt.
[NeMo W 2024-01-03 15:43:47 punctuation_capitalization_model:681] The artifact `class_labels.punct_labels_file` was not found in checkpoint. Will rely on `punct_label_ids` parameter
[NeMo W 2024-01-03 15:43:47 punctuation_capitalization_model:703] The artifact `class_labels.capit_labels_file` was not found in checkpoint. Will rely on `capit_label_ids` parameter
[NeMo W 2024-01-03 15:43:47 modelPT:202] You tried to register an artifact under config key=tokenizer.vocab_file but an artifact for it has already been registered.
Error(s) in loading state_dict for PunctuationCapitalizationModel:
        Unexpected key(s) in state_dict: "bert_model.embeddings.position_ids". 
Could not load English NLP model
not a gzip file
Could not load Russian CTC model
not a gzip file
Could not load Russian NLP model
System check identified no issues (0 silenced).
```
I re-download .nemo models from official website [Nemo Models](https://catalog.ngc.nvidia.com/models)
But it still throw that error,
 
how to fix this error ???

Please own a new thread instead of responding to a closed one. 

---

This is an error not in asr but in the Punctuation capitalization model - 

Error(s) in loading state_dict for PunctuationCapitalizationModel:
        Unexpected key(s) in state_dict: "bert_model.embeddings.position_ids". 

```
[NeMo W 2024-01-03 15:43:47 nlp_model:63] Apex was not found. Please see the NeMo README for installation instructions: https://github.com/NVIDIA/NeMo#megatron-gpt.
[NeMo W 2024-01-03 15:43:47 punctuation_capitalization_model:681] The artifact `class_labels.punct_labels_file` was not found in checkpoint. Will rely on `punct_label_ids` parameter
[NeMo W 2024-01-03 15:43:47 punctuation_capitalization_model:703] The artifact `class_labels.capit_labels_file` was not found in checkpoint. Will rely on `capit_label_ids` parameter
```