[NMT] Unable to run `megatron_nmt_training.py`
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

Trying to run `megatron_nmt_training.py`, with `conf/aayn_base_megatron.yml` initialization crashes with the following trace:
```python
Traceback (most recent call last):
  File "examples/nlp/machine_translation/megatron_nmt_training.py", line 95, in <module>
    main()
  File "/workspace/nemo/nemo/core/config/hydra_runner.py", line 104, in wrapper
    _run_hydra(
  File "/ceph/hpc/home/ilb/.local/lib/python3.8/site-packages/hydra/_internal/utils.py", line 377, in _run_hydra
    run_and_report(
  File "/ceph/hpc/home/ilb/.local/lib/python3.8/site-packages/hydra/_internal/utils.py", line 214, in run_and_report
    raise ex
  File "/ceph/hpc/home/ilb/.local/lib/python3.8/site-packages/hydra/_internal/utils.py", line 211, in run_and_report
    return func()
  File "/ceph/hpc/home/ilb/.local/lib/python3.8/site-packages/hydra/_internal/utils.py", line 378, in <lambda>
    lambda: hydra.run(
  File "/ceph/hpc/home/ilb/.local/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 111, in run
    _ = ret.return_value
  File "/ceph/hpc/home/ilb/.local/lib/python3.8/site-packages/hydra/core/utils.py", line 233, in return_value
    raise self._return_value
  File "/ceph/hpc/home/ilb/.local/lib/python3.8/site-packages/hydra/core/utils.py", line 160, in run_job
    ret.return_value = task_function(task_cfg)
  File "examples/nlp/machine_translation/megatron_nmt_training.py", line 68, in main
    MTDataPreproc(cfg=cfg.model, trainer=trainer)
  File "/workspace/nemo/nemo/collections/nlp/data/machine_translation/preproc_mt_data.py", line 172, in __init__
    encoder_model_name=cfg.encoder.get('model_name'),
  File "/ceph/hpc/home/ilb/.local/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 357, in __getattr__
    self._format_and_raise(key=key, value=None, cause=e)
  File "/ceph/hpc/home/ilb/.local/lib/python3.8/site-packages/omegaconf/base.py", line 190, in _format_and_raise
    format_and_raise(
  File "/ceph/hpc/home/ilb/.local/lib/python3.8/site-packages/omegaconf/_utils.py", line 738, in format_and_raise
    _raise(ex, cause)
  File "/ceph/hpc/home/ilb/.local/lib/python3.8/site-packages/omegaconf/_utils.py", line 716, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set end OC_CAUSE=1 for full backtrace
  File "/ceph/hpc/home/ilb/.local/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 351, in __getattr__
    return self._get_impl(key=key, default_value=_DEFAULT_MARKER_)
  File "/ceph/hpc/home/ilb/.local/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 438, in _get_impl
    node = self._get_node(key=key, throw_on_missing_key=True)
  File "/ceph/hpc/home/ilb/.local/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 465, in _get_node
    self._validate_get(key)
  File "/ceph/hpc/home/ilb/.local/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 166, in _validate_get
    self._format_and_raise(
  File "/ceph/hpc/home/ilb/.local/lib/python3.8/site-packages/omegaconf/base.py", line 190, in _format_and_raise
    format_and_raise(
  File "/ceph/hpc/home/ilb/.local/lib/python3.8/site-packages/omegaconf/_utils.py", line 818, in format_and_raise
    _raise(ex, cause)
  File "/ceph/hpc/home/ilb/.local/lib/python3.8/site-packages/omegaconf/_utils.py", line 716, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set end OC_CAUSE=1 for full backtrace
omegaconf.errors.ConfigAttributeError: Key 'encoder' is not in struct
    full_key: model.encoder
    object_type=dict
``` 

@MaximumEntropy Tracking down the source, the offending line seems to be https://github.com/NVIDIA/NeMo/blob/09be885e3f96ccded893dabe5985ffa22736f111/nemo/collections/nlp/data/machine_translation/preproc_mt_data.py#L172

as the `encoder` property is defined in all other NMT yaml files, but not in `aayn_base_megatron.yml`. This is true also for `decoder`. Hance the following lines will lead to a crash
https://github.com/NVIDIA/NeMo/blob/09be885e3f96ccded893dabe5985ffa22736f111/nemo/collections/nlp/data/machine_translation/preproc_mt_data.py#L172
https://github.com/NVIDIA/NeMo/blob/09be885e3f96ccded893dabe5985ffa22736f111/nemo/collections/nlp/data/machine_translation/preproc_mt_data.py#L227
https://github.com/NVIDIA/NeMo/blob/09be885e3f96ccded893dabe5985ffa22736f111/nemo/collections/nlp/data/machine_translation/preproc_mt_data.py#L177
https://github.com/NVIDIA/NeMo/blob/09be885e3f96ccded893dabe5985ffa22736f111/nemo/collections/nlp/data/machine_translation/preproc_mt_data.py#L232


**Environment details**

- `nemo:1.9.0` built on `pytorch:22.05-py3` with `./reinstall.sh`