nlp language modeling transformer lm raises value error with tarred dataset
State: closed | Pull request: no

I followed the steps in this[ language modeling documentation ](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/nlp/language_modeling.html#sentence-dataset-construction) but when I run the[ training script ](https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/language_modeling/transformer_lm.py) it raises a value error which seemingly is related to using a tarred dataset instead of a normal SentenceDataset in the config file (it works with plain txt but since I have a large dataset I should be using the tarred format)

**my config file:**

name: TransformerLanguageModel
do_training: true # set to False if only preprocessing data
is_tarred: true

model:
  label_smoothing: 0.0
  preproc_out_dir: "" # path to store data preprocessing outputs
  
  train_ds:
    use_tarred_dataset: true
    tar_files: "tarred/trainTarred/lm-batches.tokens.2048._OP_1..1866_CL_."
    metadata_file: "tarred/trainTarred/metadata.json" 
    shard_strategy: scatter
    tar_shuffle_n: 256

  
  validation_ds:
    use_tarred_dataset: true
    tar_files: "tarred/validTarred/lm-batches.tokens.2048._OP_1..455_CL_."
    metadata_file: "tarred/validTarred/metadata.json" 
    shard_strategy: scatter
    tar_shuffle_n: 256

    
  test_ds:
    use_tarred_dataset: true
    tar_files: "tarred/testTarred/lm-batches.tokens.2048._OP_1..204_CL_."
    metadata_file: "tarred/testTarred/metadata.json"
    shard_strategy: scatter
    tar_shuffle_n: 256

    
  optim:
    name: adam
    lr: 0.001
    betas:
      - 0.9
      - 0.98
    weight_decay: 0.0
    sched:
      name: InverseSquareRootAnnealing
      min_lr: 0.0
      last_epoch: -1
      warmup_ratio: 0.1

  tokenizer:
    tokenizer_name: yttm
    tokenizer_model: "tokenizer.model"
    vocab_file: "persian_tokenizer.vocab"
    special_tokens:
      - "<bos>"
      - "<eos>"
    # training_sample_size: null # valid for sentencepiece tokenizer

  encoder:
    library: nemo
    model_name: null
    pretrained: false
    max_sequence_length: 512
    num_token_types: 0
    embedding_dropout: 0.1
    learn_positional_encodings: false
    hidden_size: 512
    num_layers: 6
    inner_size: 2048
    num_attention_heads: 8
    ffn_dropout: 0.1
    attn_score_dropout: 0.1
    attn_layer_dropout: 0.1
    hidden_act: relu
    mask_future: true
    pre_ln: false

  head:
    num_layers: 1
    use_transformer_init: true
    activation: relu
    log_softmax: true
    dropout: 0.0

trainer:
  devices: 3
  num_nodes: 1
  max_epochs: 100
  precision: 16-mixed 
  accelerator: gpu
  strategy: ddp
  enable_checkpointing: false
  logger: false
  log_every_n_steps: 50 
  check_val_every_n_epoch: 1
  benchmark: false

exp_manager:
  name: TransformerLM
  files_to_copy: []
  
  
  
  
**Error traceback:**

    
Error executing job with overrides: []
Traceback (most recent call last):
  File "/model_training.py", line 36, in <module>
    main() 
  File "-/python3.10/site-packages/nemo/core/config/hydra_runner.py", line 126, in wrapper
    _run_hydra(
  File "-/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "-/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "-/python3.10/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "-/python3.10/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "-/python3.10/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
  File "-/python3.10/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "-/python3.10/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "-/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/home/roshan/ariana/model_training.py", line 32, in main
    trainer.fit(transformer_lm)
  File "-/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 529, in fit
    call._call_and_handle_interrupt(
  File "-/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 41, in _call_and_handle_interrupt
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
  File "-/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 91, in launch
    return function(*args, **kwargs)
  File "-/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 568, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "-/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 973, in _run
    results = self._run_stage()
  File "-/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1014, in _run_stage
    self._run_sanity_check()
  File "-/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1043, in _run_sanity_check
    val_loop.run()
  File "-/python3.10/site-packages/pytorch_lightning/loops/utilities.py", line 177, in _decorator
    return loop_run(self, *args, **kwargs)
  File "-/python3.10/site-packages/pytorch_lightning/loops/evaluation_loop.py", line 108, in run
    batch, batch_idx, dataloader_idx = next(data_fetcher)
  File "-/python3.10/site-packages/pytorch_lightning/loops/fetchers.py", line 136, in __next__
    self._fetch_next_batch(self.dataloader_iter)
  File "-/python3.10/site-packages/pytorch_lightning/loops/fetchers.py", line 150, in _fetch_next_batch
    batch = next(iterator)
  File "-/python3.10/site-packages/pytorch_lightning/utilities/combined_loader.py", line 284, in __next__
    out = next(self._iterator)
  File "-/python3.10/site-packages/pytorch_lightning/utilities/combined_loader.py", line 123, in __next__
    out = next(self.iterators[0])
  File "-/python3.10/site-packages/torch/utils/data/dataloader.py", line 633, in __next__
    data = self._next_data()
  File "-/python3.10/site-packages/torch/utils/data/dataloader.py", line 1345, in _next_data
    return self._process_data(data)
  File "-/python3.10/site-packages/torch/utils/data/dataloader.py", line 1371, in _process_data
    data.reraise()
  File "-/python3.10/site-packages/torch/_utils.py", line 644, in reraise
    raise exception
ValueError: Caught ValueError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "-/python3.10/site-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
    data = fetcher.fetch(index)
  File "-/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 32, in fetch
    data.append(next(self.dataset_iter))
  File "-/python3.10/site-packages/webdataset/pipeline.py", line 64, in iterator
    for sample in self.iterator1():
  File "-/python3.10/site-packages/webdataset/pipeline.py", line 58, in iterator1
    source = self.invoke(step, source)
  File "-/python3.10/site-packages/webdataset/pipeline.py", line 52, in invoke
    raise ValueError(f"{f}: not a valid pipeline stage")
ValueError: None: not a valid pipeline stage

Error executing job with overrides: []
Traceback (most recent call last):
  File "/home/roshan/ariana/model_training.py", line 36, in <module>
    main() 
  File "-/python3.10/site-packages/nemo/core/config/hydra_runner.py", line 126, in wrapper
    _run_hydra(
  File "-/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "-/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "-/python3.10/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "-/python3.10/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "-/python3.10/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
  File "-/python3.10/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "-/python3.10/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "-/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/home/roshan/ariana/model_training.py", line 32, in main
    trainer.fit(transformer_lm)
  File "-/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 529, in fit
    call._call_and_handle_interrupt(
  File "-/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 41, in _call_and_handle_interrupt
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
  File "-/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 91, in launch
    return function(*args, **kwargs)
  File "-/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 568, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "-/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 973, in _run
    results = self._run_stage()
  File "-/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1014, in _run_stage
    self._run_sanity_check()
  File "-/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1043, in _run_sanity_check
    val_loop.run()
  File "-/python3.10/site-packages/pytorch_lightning/loops/utilities.py", line 177, in _decorator
    return loop_run(self, *args, **kwargs)
  File "-/python3.10/site-packages/pytorch_lightning/loops/evaluation_loop.py", line 108, in run
    batch, batch_idx, dataloader_idx = next(data_fetcher)
  File "-/python3.10/site-packages/pytorch_lightning/loops/fetchers.py", line 136, in __next__
    self._fetch_next_batch(self.dataloader_iter)
  File "-/python3.10/site-packages/pytorch_lightning/loops/fetchers.py", line 150, in _fetch_next_batch
    batch = next(iterator)
  File "-/python3.10/site-packages/pytorch_lightning/utilities/combined_loader.py", line 284, in __next__
    out = next(self._iterator)
  File "-/python3.10/site-packages/pytorch_lightning/utilities/combined_loader.py", line 123, in __next__
    out = next(self.iterators[0])
  File "-/python3.10/site-packages/torch/utils/data/dataloader.py", line 633, in __next__
    data = self._next_data()
  File "-/python3.10/site-packages/torch/utils/data/dataloader.py", line 1345, in _next_data
    return self._process_data(data)
  File "-/python3.10/site-packages/torch/utils/data/dataloader.py", line 1371, in _process_data
    data.reraise()
  File "-/python3.10/site-packages/torch/_utils.py", line 644, in reraise
    raise exception
ValueError: Caught ValueError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "-/python3.10/site-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
    data = fetcher.fetch(index)
  File "-/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 32, in fetch
    data.append(next(self.dataset_iter))
  File "-/python3.10/site-packages/webdataset/pipeline.py", line 64, in iterator
    for sample in self.iterator1():
  File "-/python3.10/site-packages/webdataset/pipeline.py", line 58, in iterator1
    source = self.invoke(step, source)
  File "-/python3.10/site-packages/webdataset/pipeline.py", line 52, in invoke
    raise ValueError(f"{f}: not a valid pipeline stage")
ValueError: None: not a valid pipeline stage

Error executing job with overrides: []
Traceback (most recent call last):
  File "/home/roshan/ariana/model_training.py", line 36, in <module>
    main() 
  File "-/python3.10/site-packages/nemo/core/config/hydra_runner.py", line 126, in wrapper
    _run_hydra(
  File "-/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "-/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "-/python3.10/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "-/python3.10/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "-/python3.10/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
  File "-/python3.10/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "-/python3.10/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "-/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/home/roshan/ariana/model_training.py", line 32, in main
    trainer.fit(transformer_lm)
  File "-/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 529, in fit
    call._call_and_handle_interrupt(
  File "-/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 41, in _call_and_handle_interrupt
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
  File "-/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 91, in launch
    return function(*args, **kwargs)
  File "-/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 568, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "-/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 973, in _run
    results = self._run_stage()
  File "-/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1014, in _run_stage
    self._run_sanity_check()
  File "-/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1043, in _run_sanity_check
    val_loop.run()
  File "-/python3.10/site-packages/pytorch_lightning/loops/utilities.py", line 177, in _decorator
    return loop_run(self, *args, **kwargs)
  File "-/python3.10/site-packages/pytorch_lightning/loops/evaluation_loop.py", line 108, in run
    batch, batch_idx, dataloader_idx = next(data_fetcher)
  File "-/python3.10/site-packages/pytorch_lightning/loops/fetchers.py", line 136, in __next__
    self._fetch_next_batch(self.dataloader_iter)
  File "-/python3.10/site-packages/pytorch_lightning/loops/fetchers.py", line 150, in _fetch_next_batch
    batch = next(iterator)
  File "-/python3.10/site-packages/pytorch_lightning/utilities/combined_loader.py", line 284, in __next__
    out = next(self._iterator)
  File "-/python3.10/site-packages/pytorch_lightning/utilities/combined_loader.py", line 123, in __next__
    out = next(self.iterators[0])
  File "-/python3.10/site-packages/torch/utils/data/dataloader.py", line 633, in __next__
    data = self._next_data()
  File "-/python3.10/site-packages/torch/utils/data/dataloader.py", line 1345, in _next_data
    return self._process_data(data)
  File "-/python3.10/site-packages/torch/utils/data/dataloader.py", line 1371, in _process_data
    data.reraise()
  File "-/python3.10/site-packages/torch/_utils.py", line 644, in reraise
    raise exception
ValueError: Caught ValueError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "-/python3.10/site-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
    data = fetcher.fetch(index)
  File "-/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 32, in fetch
    data.append(next(self.dataset_iter))
  File "-/python3.10/site-packages/webdataset/pipeline.py", line 64, in iterator
    for sample in self.iterator1():
  File "-/python3.10/site-packages/webdataset/pipeline.py", line 58, in iterator1
    source = self.invoke(step, source)
  File "-/python3.10/site-packages/webdataset/pipeline.py", line 52, in invoke
    raise ValueError(f"{f}: not a valid pipeline stage")
ValueError: None: not a valid pipeline stage

I'd appreciate it if anyone would take a look into this.