Omegaconf error training Nemo ASR on Sagemaker Studio
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

When attempting to set the training data information for ASR training I am getting an error from omegaconf 

AttributeError: 'dict' object has no attribute '_get_node_flag'

**Steps/Code to reproduce bug**

This is a minimal way to reproduce the bug

import nemo.collections.asr as nemo_asr
quartznet = nemo_asr.models.EncDecCTCModel.from_pretrained(model_name='stt_en_jasper10x5dr') 
quartznet.setup_training_data(train_data_config={'manifest_filepath': './training/training_samples.json', 'sample_rate': 16000, 'labels': [' ', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', "'"], 'batch_size': 16, 'trim_silence': True, 'max_duration': 16.7, 'shuffle': True, 'is_tarred': False, 'tarred_audio_filepaths': None})

**Expected behavior**

The location of the training data should be updated

**Environment overview (please complete the following information)**

 - AWS SageMaker, Python 3 (Base Python), 4vCPU + 16GiB +  1 GPU.
 - Method of NeMo install: %pip install nemo_toolkit['all']

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version - Debian GNU/Linux 10 (buster)
- PyTorch version - 1.9.0
- Python version - 3.6.13

**Additional context**

The full stack trace 

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-2-27fbbcbb2846> in <module>
      1 import nemo.collections.asr as nemo_asr
      2 quartznet = nemo_asr.models.EncDecCTCModel.from_pretrained(model_name='stt_en_jasper10x5dr')
----> 3 quartznet.setup_training_data(train_data_config={'manifest_filepath': './training/training_samples.json', 'sample_rate': 16000, 'labels': [' ', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', "'"], 'batch_size': 16, 'trim_silence': True, 'max_duration': 16.7, 'shuffle': True, 'is_tarred': False, 'tarred_audio_filepaths': None})

/usr/local/lib/python3.6/site-packages/nemo/collections/asr/models/ctc_models.py in setup_training_data(self, train_data_config)
    421         self._update_dataset_config(dataset_name='train', config=train_data_config)
    422 
--> 423         self._train_dl = self._setup_dataloader_from_config(config=train_data_config)
    424 
    425         # Need to set this because if using an IterableDataset, the length of the dataloader is the total number

/usr/local/lib/python3.6/site-packages/nemo/collections/asr/models/ctc_models.py in _setup_dataloader_from_config(self, config)
    346 
    347         # Automatically inject args from model config to dataloader config
--> 348         audio_to_text_dataset.inject_dataloader_value_from_model_config(self.cfg, config, key='sample_rate')
    349         audio_to_text_dataset.inject_dataloader_value_from_model_config(self.cfg, config, key='labels')
    350 

/usr/local/lib/python3.6/site-packages/nemo/collections/asr/data/audio_to_text_dataset.py in inject_dataloader_value_from_model_config(model_cfg, dataloader_cfg, key)
     53             # Dataloader `key` is None or values match
     54             # Propagate from model level `key` (even if they match)
---> 55             with open_dict(dataloader_cfg):
     56                 dataloader_cfg[key] = model_cfg[key]
     57 

/usr/local/lib/python3.6/contextlib.py in __enter__(self)
     79     def __enter__(self):
     80         try:
---> 81             return next(self.gen)
     82         except StopIteration:
     83             raise RuntimeError("generator didn't yield") from None

/usr/local/lib/python3.6/site-packages/omegaconf/omegaconf.py in open_dict(config)
    935 @contextmanager
    936 def open_dict(config: Container) -> Generator[Container, None, None]:
--> 937     prev_state = config._get_node_flag("struct")
    938     try:
    939         OmegaConf.set_struct(config, False)

AttributeError: 'dict' object has no attribute '_get_node_flag'