Pre-trained HuggingFace decoder fails with NotImplementedError
State: closed | Pull request: no
Labels: name: bug

Hi 👋 I'm trying to load pre-trained HuggingFace model ([mBART](https://huggingface.co/facebook/mbart-large-cc25)) for machine translation as described in [this doc](https://github.com/NVIDIA/NeMo/blob/2ce45369f7ab6cd20c376d1ed393160f5e54be0c/docs/source/nlp/machine_translation.rst). 
I'm using [this script](https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/machine_translation/enc_dec_nmt.py) as a base. However, I was only able to load pre-trained encoder but not decoder. I see there are corresponding fields in this [huggingface config file](https://github.com/NVIDIA/NeMo/blob/stable/examples/nlp/machine_translation/conf/huggingface.yaml):
```
 decoder:
    library: nemo
    model_name: null
    pretrained: false
```
But when I try to load pre-trained decoder and pass corresponding parameters it fails with `NotImplementedError`:
```
Traceback (most recent call last):
  File "enc_dec_nmt.py", line 125, in main
    mt_model = MTEncDecModel(cfg.model, trainer=trainer)
  File "/home/user/anaconda3/envs/oMeN/lib/python3.8/site-packages/nemo/collections/nlp/models/machine_translation/mt_enc_dec_model.py", line 193, in __init__
    embedding=self.decoder.embedding,
  File "/home/user/anaconda3/envs/oMeN/lib/python3.8/site-packages/nemo/collections/nlp/modules/common/decoder_module.py", line 51, in embedding
    raise NotImplementedError
NotImplementedError
```
Parameters:
```
python3 enc_dec_nmt.py \
--config-path=conf \
--config-name=huggingface \
model.encoder.pretrained=true \
model.encoder.model_name=facebook/mbart-large-cc25 \
model.decoder.library=huggingface \
model.decoder.hidden_size=1024 \
model.decoder.inner_size=4096 \
model.decoder.model_name=facebook/mbart-large-cc25 \
model.decoder.pretrained=true \
model.decoder_tokenizer.tokenizer_model=facebook/mbart-large-cc25 \
model.decoder_tokenizer.library=huggingface \
model.train_ds.src_file_name=/nas/vkriger/bart_nemo/train.correct.short.txt \
model.train_ds.tgt_file_name=/nas/vkriger/bart_nemo/train.noisy.short.txt \
model.validation_ds.src_file_name=/nas/vkriger/bart_nemo/val.correct.short.txt \
model.validation_ds.tgt_file_name=/nas/vkriger/bart_nemo/val.noisy.short.txt \
model.test_ds.src_file_name=/nas/vkriger/bart_nemo/val.correct.short.txt \
model.test_ds.tgt_file_name=/nas/vkriger/bart_nemo/val.noisy.short.txt \
model.encoder_tokenizer.tokenizer_model=facebook/mbart-large-cc25  \
trainer.devices=2
```

I've tried to implement omitting fields and methods by hand in [HuggingFaceDecoderModule](https://github.com/NVIDIA/NeMo/blob/c4ec081c2a6d718f8917155d55a006b5f6fcbb27/nemo/collections/nlp/modules/common/huggingface/huggingface_decoder.py#L25) (these are `self._embedding`, `self._decoder`, `self._max_sequence_length` and `forward` method similarly to what's in [this file](https://github.com/NVIDIA/NeMo/blob/c4ec081c2a6d718f8917155d55a006b5f6fcbb27/nemo/collections/nlp/modules/common/transformer/transformer.py#L164)) and the `NotImplementedError` was gone but I am not sure if I did it the right way. As far as I see in the logs the decoder remains plain `TransformerDecoder` but not `MBartDecoder`. I attached the modified `HuggingFaceDecoderModule` file below.

![image](https://user-images.githubusercontent.com/55043240/159219346-e7223df7-99cd-4964-98e6-cd110b206c7a.png)

Is it possible to load pre-trained HuggingFace decoder at the moment? If no, are there any workarounds? Lastly, can I use my modified `HuggingFaceDecoderModule` file as a workaround? 

**Additional files**

Modified HuggingFaceDecoderModule: https://github.com/zzomg/for_issues/blob/main/huggingface_decoder.py
Config file: https://github.com/zzomg/for_issues/blob/main/huggingface.yaml
Test files: [train.correct.short.txt](https://github.com/zzomg/for_issues/blob/main/train.correct.short.txt) [train.noisy.short.txt](https://github.com/zzomg/for_issues/blob/main/train.noisy.short.txt) [val.correct.short.txt](https://github.com/zzomg/for_issues/blob/main/val.correct.short.txt) [val.noisy.short.txt](https://github.com/zzomg/for_issues/blob/main/val.noisy.short.txt)

Thank you!

**Environment details**

NeMo ver.: 1.7.0rc0