Unable to process dataset for Neo-megatron-GPT-20 Billion model 
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**
```
home/mayanksharma/nemo_gpt_fintuning.py:261 in <module>                                         │
│                                                                                                  │
│   258                                                                                            │
│   259                                                                                            │
│   260 if __name__ == '__main__':                                                                 │
│ ❱ 261 │   main()                                                                                 │
│                                                                                                  │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/nemo/core/config/hydra_runner.py:104 in      │
│ wrapper                                                                                          │
│                                                                                                  │
│   101 │   │   │   │                                                                              │
│   102 │   │   │   │   # no return value from run_hydra() as it may sometime actually run the t   │
│   103 │   │   │   │   # multiple times (--multirun)                                              │
│ ❱ 104 │   │   │   │   _run_hydra(                                                                │
│   105 │   │   │   │   │   args_parser=_argparse_wrapper(args),                                   │
│   106 │   │   │   │   │   task_function=task_function,                                           │
│   107 │   │   │   │   │   config_path=config_path,                                               │
│                                                                                                  │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/hydra/_internal/utils.py:377 in _run_hydra   │
│                                                                                                  │
│   374 │   │   if num_commands == 0:                                                              │
│   375 │   │   │   args.run = True                                                                │
│   376 │   │   if args.run:                                                                       │
│ ❱ 377 │   │   │   run_and_report(                                                                │
│   378 │   │   │   │   lambda: hydra.run(                                                         │
│   379 │   │   │   │   │   config_name=config_name,                                               │
│   380 │   │   │   │   │   task_function=task_function,                                           │
│                                                                                                  │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/hydra/_internal/utils.py:214 in              │
│ run_and_report                                                                                   │
│                                                                                                  │
│   211 │   │   return func()                                                                      │
│   212 │   except Exception as ex:                                                                │
│   213 │   │   if _is_env_set("HYDRA_FULL_ERROR") or is_under_debugger():                         │
│ ❱ 214 │   │   │   raise ex                                                                       │
│   215 │   │   else:                                                                              │
│   216 │   │   │   try:                                                                           │
│   217 │   │   │   │   if isinstance(ex, CompactHydraException):                                  │
│                                                                                                  │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/hydra/_internal/utils.py:211 in              │
│ run_and_report                                                                                   │
│                                                                                                  │
│   208                                                                                            │
│   209 def run_and_report(func: Any) -> Any:                                                      │
│   210 │   try:                                                                                   │
│ ❱ 211 │   │   return func()                                                                      │
│   212 │   except Exception as ex:                                                                │
│   213 │   │   if _is_env_set("HYDRA_FULL_ERROR") or is_under_debugger():                         │
│   214 │   │   │   raise ex                                                                       │
│                                                                                                  │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/hydra/_internal/utils.py:378 in <lambda>     │
│                                                                                                  │
│   375 │   │   │   args.run = True                                                                │
│   376 │   │   if args.run:                                                                       │
│   377 │   │   │   run_and_report(                                                                │
│ ❱ 378 │   │   │   │   lambda: hydra.run(                                                         │
│   379 │   │   │   │   │   config_name=config_name,                                               │
│   380 │   │   │   │   │   task_function=task_function,                                           │
│   381 │   │   │   │   │   overrides=args.overrides,                                              │
│                                                                                                  │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/hydra/_internal/hydra.py:111 in run          │
│                                                                                                  │
│   108 │   │   callbacks.on_run_end(config=cfg, config_name=config_name, job_return=ret)          │
│   109 │   │                                                                                      │
│   110 │   │   # access the result to trigger an exception in case the job failed.                │
│ ❱ 111 │   │   _ = ret.return_value                                                               │
│   112 │   │                                                                                      │
│   113 │   │   return ret                                                                         │
│   114                                                                                            │
│                                                                                                  │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/hydra/core/utils.py:233 in return_value      │
│                                                                                                  │
│   230 │   │   │   sys.stderr.write(                                                              │
│   231 │   │   │   │   f"Error executing job with overrides: {self.overrides}" + os.linesep       │
│   232 │   │   │   )                                                                              │
│ ❱ 233 │   │   │   raise self._return_value                                                       │
│   234 │                                                                                          │
│   235 │   @return_value.setter                                                                   │
│   236 │   def return_value(self, value: Any) -> None:                                            │
│                                                                                                  │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/hydra/core/utils.py:160 in run_job           │
│                                                                                                  │
│   157 │   │   with env_override(hydra_cfg.hydra.job.env_set):                                    │
│   158 │   │   │   callbacks.on_job_start(config=config)                                          │
│   159 │   │   │   try:                                                                           │
│ ❱ 160 │   │   │   │   ret.return_value = task_function(task_cfg)                                 │
│   161 │   │   │   │   ret.status = JobStatus.COMPLETED                                           │
│   162 │   │   │   except Exception as e:                                                         │
│   163 │   │   │   │   ret.return_value = e                                                       │
│                                                                                                  │
│ /home/mayanksharma/nemo_gpt_fintuning.py:256 in main                                             │
│                                                                                                  │
│   253 │                                                                                          │
│   254 │   model = MegatronGPTModel(cfg.model, trainer)                                           │
│   255 │                                                                                          │
│ ❱ 256 │   trainer.fit(model)                                                                     │
│   257 │   # trainer.fit(model, train_dataloaders=train_dataloader, val_dataloaders=valid_datal   │
│   258                                                                                            │
│   259                                                                                            │
│                                                                                                  │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:696 in  │
│ fit                                                                                              │
│                                                                                                  │
│    693 │   │   │   datamodule: An instance of :class:`~pytorch_lightning.core.datamodule.Lightn  │
│    694 │   │   """                                                                               │
│    695 │   │   self.strategy.model = model                                                       │
│ ❱  696 │   │   self._call_and_handle_interrupt(                                                  │
│    697 │   │   │   self._fit_impl, model, train_dataloaders, val_dataloaders, datamodule, ckpt_  │
│    698 │   │   )                                                                                 │
│    699                                                                                           │
│                                                                                                  │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:648 in  │
│ _call_and_handle_interrupt                                                                       │
│                                                                                                  │
│    645 │   │   """                                                                               │
│    646 │   │   try:                                                                              │
│    647 │   │   │   if self.strategy.launcher is not None:                                        │
│ ❱  648 │   │   │   │   return self.strategy.launcher.launch(trainer_fn, *args, trainer=self, **  │
│    649 │   │   │   else:                                                                         │
│    650 │   │   │   │   return trainer_fn(*args, **kwargs)                                        │
│    651 │   │   # TODO(awaelchli): Unify both exceptions below, where `KeyboardError` doesn't re  │
│                                                                                                  │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/pytorch_lightning/strategies/launchers/subpr │
│ ocess_script.py:93 in launch                                                                     │
│                                                                                                  │
│    90 │   │   """                                                                                │
│    91 │   │   if not self.cluster_environment.creates_processes_externally:                      │
│    92 │   │   │   self._call_children_scripts()                                                  │
│ ❱  93 │   │   return function(*args, **kwargs)                                                   │
│    94 │                                                                                          │
│    95 │   def _call_children_scripts(self) -> None:                                              │
│    96 │   │   # bookkeeping of spawned processes                                                 │
│                                                                                                  │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:735 in  │
│ _fit_impl                                                                                        │
│                                                                                                  │
│    732 │   │   self._ckpt_path = self.__set_ckpt_path(                                           │
│    733 │   │   │   ckpt_path, model_provided=True, model_connected=self.lightning_module is not  │
│    734 │   │   )                                                                                 │
│ ❱  735 │   │   results = self._run(model, ckpt_path=self.ckpt_path)                              │
│    736 │   │                                                                                     │
│    737 │   │   assert self.state.stopped                                                         │
│    738 │   │   self.training = False                                                             │
│                                                                                                  │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:1105 in │
│ _run                                                                                             │
│                                                                                                  │
│   1102 │   │   self.strategy.setup_environment()                                                 │
│   1103 │   │   self.__setup_profiler()                                                           │
│   1104 │   │                                                                                     │
│ ❱ 1105 │   │   self._call_setup_hook()  # allow user to setup lightning_module in accelerator e  │
│   1106 │   │                                                                                     │
│   1107 │   │   # check if we should delay restoring checkpoint till later                        │
│   1108 │   │   if not self.strategy.restore_checkpoint_after_setup:                              │
│                                                                                                  │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:1449 in │
│ _call_setup_hook                                                                                 │
│                                                                                                  │
│   1446 │   │   if self.datamodule is not None:                                                   │
│   1447 │   │   │   self._call_lightning_datamodule_hook("setup", stage=fn)                       │
│   1448 │   │   self._call_callback_hooks("setup", stage=fn)                                      │
│ ❱ 1449 │   │   self._call_lightning_module_hook("setup", stage=fn)                               │
│   1450 │   │                                                                                     │
│   1451 │   │   self.strategy.barrier("post_setup")                                               │
│   1452                                                                                           │
│                                                                                                  │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:1550 in │
│ _call_lightning_module_hook                                                                      │
│                                                                                                  │
│   1547 │   │   pl_module._current_fx_name = hook_name                                            │
│   1548 │   │                                                                                     │
│   1549 │   │   with self.profiler.profile(f"[LightningModule]{pl_module.__class__.__name__}.{ho  │
│ ❱ 1550 │   │   │   output = fn(*args, **kwargs)                                                  │
│   1551 │   │                                                                                     │
│   1552 │   │   # restore current_fx when nested context                                          │
│   1553 │   │   pl_module._current_fx_name = prev_fx_name                                         │
│                                                                                                  │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/nemo/collections/nlp/models/language_modelin │
│ g/megatron_gpt_model.py:604 in setup                                                             │
│                                                                                                  │
│   601 │   │   else:                                                                              │
│   602 │   │   │   # TODO: consider adding a ModelPT guard to check if model is being restored.   │
│   603 │   │   │   # allowing restored models to optionally setup datasets                        │
│ ❱ 604 │   │   │   self.build_train_valid_test_datasets()                                         │
│   605 │   │   │   self.setup_training_data(self.cfg.data)                                        │
│   606 │   │   │   self.setup_validation_data(self.cfg.data)                                      │
│   607 │   │   │   self.setup_test_data(self.cfg.data)                                            │
│                                                                                                  │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/nemo/collections/nlp/models/language_modelin │
│ g/megatron_gpt_model.py:519 in build_train_valid_test_datasets                                   │
│                                                                                                  │
│   516 │   │   │   eval_iters * global_batch_size,                                                │
│   517 │   │   │   test_iters * global_batch_size,                                                │
│   518 │   │   ]                                                                                  │
│ ❱ 519 │   │   self._train_ds, self._validation_ds, self._test_ds = build_train_valid_test_data   │
│   520 │   │   │   cfg=self.cfg,                                                                  │
│   521 │   │   │   trainer=self.trainer,                                                          │
│   522 │   │   │   data_prefix=self.cfg.data.data_prefix,                                         │
│                                                                                                  │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/nemo/collections/nlp/data/language_modeling/ │
│ megatron/gpt_dataset.py:81 in build_train_valid_test_datasets                                    │
│                                                                                                  │
│    78 │   valid_datasets = []                                                                    │
│    79 │   test_datasets = []                                                                     │
│    80 │   for i in range(len(prefixes)):                                                         │
│ ❱  81 │   │   train_ds, valid_ds, test_ds = _build_train_valid_test_datasets(                    │
│    82 │   │   │   cfg,                                                                           │
│    83 │   │   │   trainer,                                                                       │
│    84 │   │   │   prefixes[i],                                                                   │
│                                                                                                  │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/nemo/collections/nlp/data/language_modeling/ │
│ megatron/gpt_dataset.py:131 in _build_train_valid_test_datasets                                  │
│                                                                                                  │
│   128 │   """Build train, valid, and test datasets."""                                           │
│   129 │                                                                                          │
│   130 │   # Indexed dataset.                                                                     │
│ ❱ 131 │   indexed_dataset = get_indexed_dataset_(data_prefix, data_impl, skip_warmup)            │
│   132 │                                                                                          │
│   133 │   total_num_of_documents = indexed_dataset.sizes.shape[0]                                │
│   134 │   splits = get_train_valid_test_split_(splits_string, total_num_of_documents)            │
│                                                                                                  │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/nemo/collections/nlp/data/language_modeling/ │
│ megatron/gpt_dataset.py:183 in get_indexed_dataset_                                              │
│                                                                                                  │
│   180 │   indexed_dataset = make_indexed_dataset(data_prefix, data_impl, skip_warmup)            │
│   181 │   logging.info(f"indexed_dataset: {indexed_dataset.__dict__}")                           │
│   182 │   logging.info(' > finished creating indexed dataset in {:4f} ' 'seconds'.format(time.   │
│ ❱ 183 │   logging.info('    number of documents: {}'.format(indexed_dataset.sizes.shape[0]))     │
│   184 │                                                                                          │
│   185 │   return indexed_dataset                                                                 │
│   186                                                                                            │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
AttributeError: 'CSVMemMapDataset' object has no attribute 'sizes'
```

**Steps/Code to reproduce bug**
I am using scrip borrowed from - [https://github.com/NVIDIA/NeMo/blob/v1.12.0/examples/nlp/language_modeling/megatron_gpt_pretraining.py]()
```
@hydra_runner(config_path="conf", config_name="megatron_gpt_config.yaml")
def main(cfg) -> None:
    logging.info("\n\n************** Experiment configuration ***********")
    logging.info(f'\n{OmegaConf.to_yaml(cfg)}')
    
    cfg.model.data.data_prefix=[0.5,"/home/mayanksharma/nemo_data/small_train", 0.5, "/home/mayanksharma/nemo_data/small_valid", 0.5, "/home/mayanksharma/nemo_data/small_test"]
    cfg.model.data.seq_length=128
    cfg.model.data.data_impl="csv_mmap"
    cfg.trainer.devices=1
    
    cfg.model.sequence_parallel=True
    cfg.model.global_batch_size=16

    megatron_amp_o2 = cfg.model.get('megatron_amp_O2', False)
    with_distributed_adam = cfg.model.optim.get('name') == 'distributed_fused_adam'

    plugins = []
    strategy = NLPDDPStrategy(
        no_ddp_communication_hook=True,  # we don't use DDP for async grad allreduce
        gradient_as_bucket_view=cfg.model.gradient_as_bucket_view,
        find_unused_parameters=False,
    )
    if cfg.trainer.precision in [16, 'bf16']:
        scaler = None
        if cfg.trainer.precision == 16:
            scaler = GradScaler(
                init_scale=cfg.model.get('native_amp_init_scale', 2 ** 32),
                growth_interval=cfg.model.get('native_amp_growth_interval', 1000),
                hysteresis=cfg.model.get('hysteresis', 2),
            )
        if megatron_amp_o2 and not with_distributed_adam:
            plugins.append(MegatronHalfPrecisionPlugin(precision=cfg.trainer.precision, device='cuda', scaler=scaler))
        else:
            plugins.append(PipelineMixedPrecisionPlugin(precision=cfg.trainer.precision, device='cuda', scaler=scaler))

    if cfg.get('cluster_type', None) == 'BCP':
        plugins.append(TorchElasticEnvironment())

    trainer = Trainer(plugins=plugins, strategy=strategy, **cfg.trainer)

    exp_manager(trainer, cfg.exp_manager)

    # update resume from checkpoint found by exp_manager
    if cfg.model.resume_from_checkpoint is not None:
        resume_from_checkpoint = cfg.model.resume_from_checkpoint
    else:
        resume_from_checkpoint = trainer._checkpoint_connector.resume_from_checkpoint_fit_path

    logging.info(f'Resuming training from checkpoint: {resume_from_checkpoint}')

    trainer._checkpoint_connector = CheckpointConnector(trainer, resume_from_checkpoint=resume_from_checkpoint)
    # Override timer callback to a stateless one
    for idx, callback in enumerate(trainer.callbacks):
        if isinstance(callback, Timer):
            trainer.callbacks[idx] = StatelessTimer(cfg.trainer.max_time,)

    # hydra interpolation does not work here as the interpolation key is lost when PTL saves hparams
    with open_dict(cfg):
        cfg.model.precision = cfg.trainer.precision

    model = MegatronGPTModel(cfg.model, trainer)

    trainer.fit(model)


if __name__ == '__main__':
    main()
```

**Config File That I am Using**
I have borrowed config from here [https://github.com/NVIDIA/NeMo/blob/v1.12.0/examples/nlp/language_modeling/conf/megatron_gpt_config.yaml]()
```
name: megatron_gpt_20B
restore_from_path: "nemo_model/nemo-megatron-gpt-20B/nemo_gpt20B_bf16_tp4.nemo" # used when starting from a .nemo file

trainer:
  devices: 2
  num_nodes: 1
  accelerator: gpu
  precision: 16
  logger: False # logger provided by exp_manager
  enable_checkpointing: False
  replace_sampler_ddp: False
  max_epochs: -1 # PTL default. In practice, max_steps will be reached first. 
  max_steps: 100000 # consumed_samples = global_step * micro_batch_size * data_parallel_size * accumulate_grad_batches
  log_every_n_steps: 10
  val_check_interval: 100
  limit_val_batches: 50
  limit_test_batches: 500
  accumulate_grad_batches: 1 # do not modify, grad acc is automatic for training megatron models
  gradient_clip_val: 1.0
  benchmark: False

exp_manager:
  explicit_log_dir: null
  exp_dir: null
  name: megatron_gpt
  create_wandb_logger: False
  wandb_logger_kwargs:
    project: null
    name: null
  resume_if_exists: True
  resume_ignore_no_checkpoint: True
  create_checkpoint_callback: True
  checkpoint_callback_params:
    monitor: val_loss
    save_top_k: 10
    mode: min
    always_save_nemo: False # saves nemo file during validation, not implemented for model parallel
    save_nemo_on_train_end: False # not recommended when training large models on clusters with short time limits
    filename: 'megatron_gpt--{val_loss:.2f}-{step}-{consumed_samples}'
    model_parallel_size: ${multiply:${model.tensor_model_parallel_size}, ${model.pipeline_model_parallel_size}}


model:
  # specify micro_batch_size, global_batch_size, and model parallelism
  # gradient accumulation will be done automatically based on data_parallel_size
  micro_batch_size: 4 # limited by GPU memory
  global_batch_size: 8 # will use more micro batches to reach global batch size
  tensor_model_parallel_size: 1 # intra-layer model parallelism
  pipeline_model_parallel_size: 1 # inter-layer model parallelism
  resume_from_checkpoint: null # manually set the checkpoint file to load from

  # model architecture
  encoder_seq_length: 512
  max_position_embeddings: 1024 #${.encoder_seq_length} mayank changed
  num_layers: 12
  hidden_size: 768
  ffn_hidden_size: 3072 # Transformer FFN hidden size. Usually 4 * hidden_size.
  num_attention_heads: 12
  init_method_std: 0.02 # Standard deviation of the zero mean normal distribution used for weight initialization.')
  use_scaled_init_method: True # use scaled residuals initialization
  hidden_dropout: 0.1 # Dropout probability for hidden state transformer.
  kv_channels: null # Projection weights dimension in multi-head attention. Set to hidden_size // num_attention_heads if null
  apply_query_key_layer_scaling: True # scale Q * K^T by 1 / layer-number.
  normalization: layernorm # Type of normalization layers
  layernorm_epsilon: 1e-5
  do_layer_norm_weight_decay: False # True means weight decay on all params
  make_vocab_size_divisible_by: 128 # Pad the vocab size to be divisible by this value for computation efficiency.
  pre_process: True # add embedding
  post_process: True # add pooler
  persist_layer_norm: True # Use of persistent fused layer norm kernel.
  bert_binary_head: 2 #used for classification

  tokenizer:
    library: 'huggingface'
    type: 'gpt2'
    model: null
    vocab_file: null
    merge_file: null 
    delimiter: null # only used for tabular tokenizer
    sentencepiece_legacy: True # Legacy=True allows you to add special tokens to sentencepiece tokenizers.

  # precision
  native_amp_init_scale: 4294967296 # 2 ** 32
  native_amp_growth_interval: 1000
  hysteresis: 2 # Gradient scale hysteresis
  fp32_residual_connection: False # Move residual connections to fp32
  fp16_lm_cross_entropy: False # Move the cross entropy unreduced loss calculation for lm head to fp16

  # Megatron O2-style half-precision
  megatron_amp_O2: False # Enable O2-level automatic mixed precision using main parameters
  grad_allreduce_chunk_size_mb: 125
  grad_div_ar_fusion: True # Fuse grad division into torch.distributed.all_reduce

  # miscellaneous
  seed: 1234
  use_cpu_initialization: False # Init weights on the CPU (slow for large models)
  onnx_safe: False # Use work-arounds for known problems with Torch ONNX exporter.
  apex_transformer_log_level: 30 # Python logging level displays logs with severity greater than or equal to this
  gradient_as_bucket_view: True # PyTorch DDP argument. Allocate gradients in a contiguous bucket to save memory (less fragmentation and buffer memory)
  gradient_accumulation_fusion: False # Fuse weight gradient accumulation to GEMMs. Only used with pipeline parallelism.

  ## Activation Checkpointing
  # NeMo Megatron supports 'selective' activation checkpointing where only the memory intensive part of attention is checkpointed.
  # These memory intensive activations are also less compute intensive which makes activation checkpointing more efficient for LLMs (20B+).
  # See Reducing Activation Recomputation in Large Transformer Models: https://arxiv.org/abs/2205.05198 for more details.
  # 'full' will checkpoint the entire transformer layer.
  activations_checkpoint_granularity: null # 'selective' or 'full' 
  activations_checkpoint_method: null # 'uniform', 'block', not used with 'selective'
  # 'uniform' divides the total number of transformer layers and checkpoints the input activation
  # of each chunk at the specified granularity
  # 'block' checkpoints the specified number of layers per pipeline stage at the specified granularity
  activations_checkpoint_num_layers: null # not used with 'selective'
  # when using 'uniform' this creates groups of transformer layers to checkpoint. Usually set to 1. Increase to save more memory.
  # when using 'block' this this will checkpoint the first activations_checkpoint_num_layers per pipeline stage.

  ## Sequence Parallelism
  # Makes tensor parallelism more memory efficient for LLMs (20B+) by parallelizing layer norms and dropout sequentially
  # See Reducing Activation Recomputation in Large Transformer Models: https://arxiv.org/abs/2205.05198 for more details.
  sequence_parallel: True

  data:
    # Path to data must be specified by the user.
    # can override from the CLI: "model.data.data_prefix=[.5,/raid/data/pile/my-gpt3_00_text_document,.5,/raid/data/pile/my-gpt3_01_text_document]",
    # Or see example below: 
    # data_prefix: 
    #   - .5
    #   - /raid/data/pile/my-gpt3_00_text_document
    #   - .5
    #   - /raid/data/pile/my-gpt3_01_text_document
    data_prefix: ???
    index_mapping_dir: null # path to save index mapping .npy files, by default will save in the same location as data_prefix
    data_impl: csv_map
    splits_string: 900,50,50
    seq_length: ${model.encoder_seq_length}
    skip_warmup: True
    num_workers: 2
    dataloader_type: single # cyclic
    reset_position_ids: False # Reset position ids after end-of-document token
    reset_attention_mask: False # Reset attention mask after end-of-document token
    eod_mask_loss: False # Mask loss for the end of document tokens
    # masked_lm_prob: 0.15
    # short_seq_prob: 0.1

  # Nsys profiling options
  nsys_profile:
    enabled: False
    start_step: 10  # Global batch to start profiling
    end_step: 10 # Global batch to end profiling
    ranks: [0] # Global rank IDs to profile
    gen_shape: False # Generate model and kernel details including input shapes
  
  optim:
    name: fused_adam
    lr: 2e-4
    weight_decay: 0.01 
    betas: 
    - 0.9
    - 0.98
    sched:
      name: CosineAnnealing
      warmup_steps: 500
      constant_steps: 50000
      min_lr: 2e-5
```

**Inpu train,valid, test files I am Using**
Input small_train, small_valid, small_test files look like
```
0,|<startoftext>|sentence[LABEL]:Target|<endoftext>|
1,|<startoftext>|sentence[LABEL]:Target|<endoftext>|
2,|<startoftext>|sentence[LABEL]:Target|<endoftext>|
```


**Expected behaviour**
The train, valid and test files for gpt model should be processed correctly, as a similar file properly formatted for megatron-bert got correctly processed. for the megatron-bert model, the input train file looked something like this. It was a binary classification problem.

```
0,sentence,label1
1,sentence,label2
2,sentence,label1
```

**Environment overview (please complete the following information)**

 - I am using GCP machine with 4 A100 Nvidia gpus `Linux nemo-training 4.19.0-22-cloud-amd64 #1 SMP Debian 4.19.260-1 (2022-09-29) x86_64 GNU/Linux`
 - Environment location: `Cloud(specify cloud provider - GCP)`
 - Method of NeMo install: [pip install or from source]. `pip install nemo-toolkit["all"]` and `nemo-version: 1.12.0`.

**Environment details**

If an NVIDIA docker image is used, you don't need to specify these.
Otherwise, please provide:
- OS version - `Debian 4.19.260-1 (2022-09-29) x86_64 GNU/Linux`
- PyTorch version - `1.13.0`
- PyTorch Lightning version - `1.7.6`
- Python version - `python3.8`

**Additional context**
*Please correct me if my input is formatted incorrectly for a gpt model. I could not find any suitable sample example for fine-tuning/pretraining the neo-megatron-20 billion models. Neither could I find any sample input files, or how they should be formatted to process correctly for GPT*