Error following the Megatron GPT pre-training example
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

Following the example from here: https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/nlp/nemo_megatron/gpt/gpt_training.html

I downloaded and processed the data to run GPT pretraining. But I encountered the following error. 
I am using the latest nemo docker container (docker pull nvcr.io/nvidia/nemo:23.01) and did everything from within the container. 

```
  File "/usr/local/lib/python3.8/dist-packages/apex/transformer/pipeline_parallel/schedules/fwd_bwd_no_pipelining.py", line 80, in forward_backward_no_pipelining
    cur_micro_batch = get_kth_microbatch(batch, i)
  File "/usr/local/lib/python3.8/dist-packages/apex/transformer/pipeline_parallel/utils.py", line 135, in get_kth_microbatch
    size = x.size(0)
AttributeError: 'dict' object has no attribute 'size'
```


**Steps/Code to reproduce bug**

I followed the steps outlined in the above link, trying both GPT2 or SPM options.

**Expected behavior**

training on a single GPU

**Environment overview (please complete the following information)**

docker pull nvcr.io/nvidia/nemo:23.01

**Additional context**

* Files: 

root@bd75e03a74c9:/workspace/data# ls
enwiki-latest-pages-articles.xml.bz2                                                    hfbpe_gpt_training_data_text_document_test_indexmap_9648ns_1024sl_1234s_doc_idx.npy
gpt2-merges.txt                                                                         hfbpe_gpt_training_data_text_document_test_indexmap_9648ns_1024sl_1234s_sample_idx.npy
gpt2-vocab.json                                                                         hfbpe_gpt_training_data_text_document_test_indexmap_9648ns_1024sl_1234s_shuffle_idx.npy
gpt_training_data_text_document.bin                                                     hfbpe_gpt_training_data_text_document_train_indexmap_57888000ns_1024sl_1234s_doc_idx.npy
gpt_training_data_text_document.idx                                                     hfbpe_gpt_training_data_text_document_train_indexmap_57888000ns_1024sl_1234s_sample_idx.npy
gpt_training_data_text_document_test_indexmap_9648ns_1024sl_1234s_doc_idx.npy           hfbpe_gpt_training_data_text_document_train_indexmap_57888000ns_1024sl_1234s_shuffle_idx.npy
gpt_training_data_text_document_test_indexmap_9648ns_1024sl_1234s_sample_idx.npy        hfbpe_gpt_training_data_text_document_valid_indexmap_0ns_1024sl_1234s_doc_idx.npy
gpt_training_data_text_document_test_indexmap_9648ns_1024sl_1234s_shuffle_idx.npy       hfbpe_gpt_training_data_text_document_valid_indexmap_0ns_1024sl_1234s_sample_idx.npy
gpt_training_data_text_document_train_indexmap_57888000ns_1024sl_1234s_doc_idx.npy      hfbpe_gpt_training_data_text_document_valid_indexmap_0ns_1024sl_1234s_shuffle_idx.npy
gpt_training_data_text_document_train_indexmap_57888000ns_1024sl_1234s_sample_idx.npy   hfbpe_gpt_training_data_text_document_valid_indexmap_9657648ns_1024sl_1234s_doc_idx.npy
gpt_training_data_text_document_train_indexmap_57888000ns_1024sl_1234s_shuffle_idx.npy  hfbpe_gpt_training_data_text_document_valid_indexmap_9657648ns_1024sl_1234s_sample_idx.npy
gpt_training_data_text_document_valid_indexmap_9657648ns_1024sl_1234s_doc_idx.npy       hfbpe_gpt_training_data_text_document_valid_indexmap_9657648ns_1024sl_1234s_shuffle_idx.npy
gpt_training_data_text_document_valid_indexmap_9657648ns_1024sl_1234s_sample_idx.npy    nemo_experiments
gpt_training_data_text_document_valid_indexmap_9657648ns_1024sl_1234s_shuffle_idx.npy   sentencepiece
hfbpe_gpt_training_data_text_document.bin                                               spm_32k_wiki.model
hfbpe_gpt_training_data_text_document.idx                                               spm_32k_wiki.vocab
hfbpe_gpt_training_data_text_document_test_indexmap_193ns_1024sl_1234s_doc_idx.npy      text
hfbpe_gpt_training_data_text_document_test_indexmap_193ns_1024sl_1234s_sample_idx.npy   text_for_tokenizer.txt
hfbpe_gpt_training_data_text_document_test_indexmap_193ns_1024sl_1234s_shuffle_idx.npy  train_data.jsonl

* Detailed command line and outputlog:

Epoch 0:   0%|                                                                                                                                                                                                                                     | 0/351750 [00:00<?, ?it/s][NeMo W 2023-04-19 21:40:25 experimental:27] Module <class 'nemo.collections.asr.modules.audio_modules.SpectrogramToMultichannelFeatures'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2023-04-19 21:40:27 experimental:27] Module <class 'nemo.collections.nlp.data.language_modeling.megatron.megatron_batch_samplers.MegatronPretrainingRandomBatchSampler'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2023-04-19 21:40:27 experimental:27] Module <class 'nemo.collections.nlp.models.text_normalization_as_tagging.thutmose_tagger.ThutmoseTaggerModel'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo I 2023-04-19 21:40:27 indexed_dataset:454]     reading sizes...
[NeMo I 2023-04-19 21:40:27 indexed_dataset:456]     reading pointers...
[NeMo I 2023-04-19 21:40:27 indexed_dataset:460]     reading document index...
[NeMo I 2023-04-19 21:40:27 indexed_dataset:513]     creating numpy buffer of mmap...
[NeMo I 2023-04-19 21:40:27 indexed_dataset:515]     creating memory view of numpy buffer...
[NeMo W 2023-04-19 21:40:33 experimental:27] Module <class 'nemo.collections.asr.modules.audio_modules.SpectrogramToMultichannelFeatures'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2023-04-19 21:40:35 experimental:27] Module <class 'nemo.collections.nlp.data.language_modeling.megatron.megatron_batch_samplers.MegatronPretrainingRandomBatchSampler'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2023-04-19 21:40:35 experimental:27] Module <class 'nemo.collections.nlp.models.text_normalization_as_tagging.thutmose_tagger.ThutmoseTaggerModel'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo I 2023-04-19 21:40:35 indexed_dataset:454]     reading sizes...
[NeMo I 2023-04-19 21:40:35 indexed_dataset:456]     reading pointers...
[NeMo I 2023-04-19 21:40:35 indexed_dataset:460]     reading document index...
[NeMo I 2023-04-19 21:40:35 indexed_dataset:513]     creating numpy buffer of mmap...
[NeMo I 2023-04-19 21:40:35 indexed_dataset:515]     creating memory view of numpy buffer...
Error executing job with overrides: ['trainer.devices=1', 'trainer.num_nodes=1', 'trainer.max_epochs=null', 'trainer.max_steps=300000', 'trainer.val_check_interval=300', 'trainer.log_every_n_steps=50', 'trainer.limit_val_batches=50', 'trainer.limit_test_batches=50', 'trainer.accumulate_grad_batches=1', 'trainer.precision=16', 'model.micro_batch_size=6', 'model.global_batch_size=192', 'model.tensor_model_parallel_size=1', 'model.pipeline_model_parallel_size=1', 'model.max_position_embeddings=1024', 'model.encoder_seq_length=1024', 'model.hidden_size=768', 'model.ffn_hidden_size=3072', 'model.num_layers=12', 'model.num_attention_heads=12', 'model.init_method_std=0.021', 'model.hidden_dropout=0.1', 'model.layernorm_epsilon=1e-5', 'model.tokenizer.vocab_file=gpt2-vocab.json', 'model.tokenizer.merge_file=gpt2-merges.txt', 'model.data.data_prefix=[1.0,hfbpe_gpt_training_data_text_document]', 'model.data.num_workers=2', 'model.data.seq_length=1024', "model.data.splits_string='980,10,10'", 'model.optim.name=fused_adam', 'model.optim.lr=6e-4', 'model.optim.betas=[0.9,0.95]', 'model.optim.weight_decay=0.1', 'model.optim.sched.name=CosineAnnealing', 'model.optim.sched.warmup_steps=750', 'model.optim.sched.constant_steps=80000', 'model.optim.sched.min_lr=6e-5', 'exp_manager.resume_if_exists=True', 'exp_manager.resume_ignore_no_checkpoint=True', 'exp_manager.create_checkpoint_callback=True', 'exp_manager.checkpoint_callback_params.monitor=val_loss', 'exp_manager.checkpoint_callback_params.save_top_k=3', 'exp_manager.checkpoint_callback_params.mode=min', 'exp_manager.checkpoint_callback_params.always_save_nemo=False']
Traceback (most recent call last):
  File "/workspace/nemo/examples/nlp/language_modeling/megatron_gpt_pretraining.py", line 86, in main
    trainer.fit(model)
  File "/usr/local/lib/python3.8/dist-packages/pytorch_lightning/trainer/trainer.py", line 608, in fit
    call._call_and_handle_interrupt(
  File "/usr/local/lib/python3.8/dist-packages/pytorch_lightning/trainer/call.py", line 36, in _call_and_handle_interrupt
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 88, in launch
    return function(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pytorch_lightning/trainer/trainer.py", line 650, in _fit_impl
    self._run(model, ckpt_path=self.ckpt_path)
  File "/usr/local/lib/python3.8/dist-packages/pytorch_lightning/trainer/trainer.py", line 1103, in _run
    results = self._run_stage()
  File "/usr/local/lib/python3.8/dist-packages/pytorch_lightning/trainer/trainer.py", line 1182, in _run_stage
    self._run_train()
  File "/usr/local/lib/python3.8/dist-packages/pytorch_lightning/trainer/trainer.py", line 1205, in _run_train
    self.fit_loop.run()
  File "/usr/local/lib/python3.8/dist-packages/pytorch_lightning/loops/loop.py", line 199, in run
    self.advance(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pytorch_lightning/loops/fit_loop.py", line 267, in advance
    self._outputs = self.epoch_loop.run(self._data_fetcher)
  File "/usr/local/lib/python3.8/dist-packages/pytorch_lightning/loops/loop.py", line 199, in run
    self.advance(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 213, in advance
    batch_output = self.batch_loop.run(kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pytorch_lightning/loops/loop.py", line 199, in run
    self.advance(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 88, in advance
    outputs = self.optimizer_loop.run(optimizers, kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pytorch_lightning/loops/loop.py", line 199, in run
    self.advance(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 202, in advance
    result = self._run_optimization(kwargs, self._optimizers[self.optim_progress.optimizer_position])
  File "/usr/local/lib/python3.8/dist-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 249, in _run_optimization
    self._optimizer_step(optimizer, opt_idx, kwargs.get("batch_idx", 0), closure)
  File "/usr/local/lib/python3.8/dist-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 370, in _optimizer_step
    self.trainer._call_lightning_module_hook(
  File "/usr/local/lib/python3.8/dist-packages/pytorch_lightning/trainer/trainer.py", line 1347, in _call_lightning_module_hook
    output = fn(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pytorch_lightning/core/module.py", line 1744, in optimizer_step
    optimizer.step(closure=optimizer_closure)
  File "/usr/local/lib/python3.8/dist-packages/pytorch_lightning/core/optimizer.py", line 169, in step
    step_output = self._strategy.optimizer_step(self._optimizer, self._optimizer_idx, closure, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pytorch_lightning/strategies/ddp.py", line 280, in optimizer_step
    optimizer_output = super().optimizer_step(optimizer, opt_idx, closure, model, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pytorch_lightning/strategies/strategy.py", line 234, in optimizer_step
    return self.precision_plugin.optimizer_step(
  File "/usr/local/lib/python3.8/dist-packages/pytorch_lightning/plugins/precision/native_amp.py", line 75, in optimizer_step
    closure_result = closure()
  File "/usr/local/lib/python3.8/dist-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 149, in __call__
    self._result = self.closure(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 135, in closure
    step_output = self._step_fn()
  File "/usr/local/lib/python3.8/dist-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 419, in _training_step
    training_step_output = self.trainer._call_strategy_hook("training_step", *kwargs.values())
  File "/usr/local/lib/python3.8/dist-packages/pytorch_lightning/trainer/trainer.py", line 1485, in _call_strategy_hook
    output = fn(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pytorch_lightning/strategies/ddp.py", line 351, in training_step
    return self.model(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1480, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/parallel/distributed.py", line 1098, in forward
    output = self._run_ddp_forward(*inputs, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/parallel/distributed.py", line 1054, in _run_ddp_forward
    return module_to_run(*inputs, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1480, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pytorch_lightning/overrides/base.py", line 98, in forward
    output = self._forward_module.training_step(*inputs, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/nemo/utils/model_utils.py", line 362, in wrap_training_step
    output_dict = wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py", line 367, in training_step
    losses_reduced_per_micro_batch = fwd_bwd_function(
  File "/usr/local/lib/python3.8/dist-packages/apex/transformer/pipeline_parallel/schedules/fwd_bwd_no_pipelining.py", line 80, in forward_backward_no_pipelining
    cur_micro_batch = get_kth_microbatch(batch, i)
  File "/usr/local/lib/python3.8/dist-packages/apex/transformer/pipeline_parallel/utils.py", line 135, in get_kth_microbatch
    size = x.size(0)
AttributeError: 'dict' object has no attribute 'size'

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.


* GPU is H100.