Distributed Validation Error in Nemo `MTEncDecModel` with PyTorch 1.13.0a0+d321be6
State: closed | Pull request: no
Labels: name: bug

### Description: 
I am encountering an issue while attempting to evaluate a pre-trained machine translation model using the `MTEncDecModel` class from the `nemo.collections.nlp.models.machine_translation` module. The model is being evaluated on a node with four GPUs and I am using PyTorch version 1.13.0a0+d321be6.
The error occurs when I try to run the validation process using the PyTorch Lightning Trainer with the "ddp" strategy:
```
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 687, in validate
    return call._call_and_handle_interrupt(
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 36, in _call_and_handle_interrupt
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 90, in launch
    return function(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 736, in _validate_impl
    results = self._run(model, ckpt_path=self.ckpt_path)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1098, in _run
    results = self._run_stage()
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1174, in _run_stage
    return self._run_evaluate()
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1214, in _run_evaluate
    eval_loop_results = self._evaluation_loop.run()
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 206, in run
    output = self.on_run_end()
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 180, in on_run_end
    self._evaluation_epoch_end(self._outputs)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 288, in _evaluation_epoch_end
    self.trainer._call_lightning_module_hook(hook_name, output_or_outputs)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1342, in _call_lightning_module_hook
    output = fn(*args, **kwargs)
  File "/NeMo/nemo/collections/nlp/models/machine_translation/mt_enc_dec_model.py", line 496, in validation_epoch_end
    self.eval_epoch_end(outputs, 'val', self.global_rank)
  File "/neural/nemo_transformer/adapted_transformer.py", line 175, in eval_epoch_end
    super().eval_epoch_end(outputs, mode, global_rank)
  File "/NeMo/nemo/collections/nlp/models/machine_translation/mt_enc_dec_model.py", line 430, in eval_epoch_end
    dist.all_gather_object(
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 1682, in all_gather_object
    object_list[i] = _tensor_to_object(tensor, tensor_size)
IndexError: list assignment index out of range
```

### Steps to Reproduce:
```
from pytorch_lightning import Trainer
from nemo.collections.nlp.models.machine_translation import MTEncDecModel

# loading a pre-trained model
cfg_model = MTEncDecModel(restore_path=model_path, return_config=True)
mt_model = MTEncDecModel(restore_path=model_path, map_location=torch.device('cuda:0'))
trainer = Trainer(strategy="ddp")
mt_model.setup_multiple_validation_data(val_data_config=cfg_model.validation_ds)

trainer.validate(mt_model)
```
With the validation config:
```
validation_ds:
    src_file_name: [<11 source files>]
    tgt_file_name: [<11 target files>]
    tokens_in_batch: 8192
    clean: false
    max_seq_length: 512
    shuffle: false
    num_samples: -1
    drop_last: true
    pin_memory: false
    num_workers: 8
```

### Expected Result:
The model should be evaluated successfully.

### Software Configuration:
PyTorch version: 1.13.0a0+d321be6
Operating System: Ubuntu 20.04.4 LTS
GPU: 4xA100 with 40G VRAM
Python Version: 3.8.13
NeMo version: 1.14
CUDA version: 11.7