BERT Pre-training
State: closed | Pull request: no
Labels: name: bug

Hi

I am trying to run BERT Pre- training with sample data with reference to https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/zh/latest/nlp/bert_pretraining.html 

Steps followed :
1. examples/nlp/language_modeling/get_wkt2.sh download_dir. 
2. Changed the config file NeMo/examples/nlp/language_modeling/conf/bert_pretraining_from_text_config.yaml. 
3. after running the python bert_pretraining.py
Getting this error

[NeMo W 2021-06-02 17:24:59 optimizers:46] Apex was not found. Using the lamb optimizer will error out.
[NeMo W 2021-06-02 17:25:01 nemo_logging:349] /home/kruthika/NeMo/venv/lib/python3.6/site-packages/omegaconf/basecontainer.py:232: UserWarning: cfg.pretty() is deprecated and will be removed in a future version.
    Use OmegaConf.to_yaml(cfg)
    
      category=UserWarning,
    
[NeMo I 2021-06-02 17:25:01 bert_pretraining:28] Config:
     name: PretrainingBERTFromText
    trainer:
      gpus: 1
      num_nodes: 1
      max_epochs: 2
      max_steps: null
      accumulate_grad_batches: 1
      precision: 16
      amp_level: O1
      accelerator: ddp
      gradient_clip_val: 0.0
      log_every_n_steps: 1
      val_check_interval: 1.0
      checkpoint_callback: false
      logger: false
    model:
      nemo_path: null
      only_mlm_loss: false
      num_tok_classification_layers: 1
      num_seq_classification_layers: 2
      max_seq_length: 128
      mask_prob: 0.15
      short_seq_prob: 0.1
      language_model:
        pretrained_model_name: bert-base-uncased
        lm_checkpoint: null
        config:
          attention_probs_dropout_prob: 0.1
          hidden_act: gelu
          hidden_dropout_prob: 0.1
          hidden_size: 768
          initializer_range: 0.02
          intermediate_size: 3072
          max_position_embeddings: 512
          num_attention_heads: 12
          num_hidden_layers: 12
          type_vocab_size: 2
          vocab_size: 30522
        config_file: null
      tokenizer:
        tokenizer_name: ${model.language_model.pretrained_model_name}
        vocab_file: null
        tokenizer_model: null
        special_tokens:
          unk_token: '[UNK]'
          sep_token: '[SEP]'
          pad_token: '[PAD]'
          bos_token: '[CLS]'
          mask_token: '[MASK]'
          eos_token: '[SEP]'
          cls_token: '[CLS]'
      train_ds:
        data_file: /home/kruthika/NeMo/download_dir/wikitext-2/train.txt
        max_seq_length: ${model.max_seq_length}
        mask_prob: ${model.mask_prob}
        short_seq_prob: ${model.short_seq_prob}
        batch_size: 16
        shuffle: true
        num_samples: -1
        num_workers: 2
        drop_last: false
        pin_memory: false
      validation_ds:
        data_file: /home/kruthika/NeMo/download_dir/wikitext-2/valid.txt
        max_seq_length: ${model.max_seq_length}
        mask_prob: ${model.mask_prob}
        short_seq_prob: ${model.short_seq_prob}
        batch_size: 16
        shuffle: false
        num_samples: -1
        num_workers: 2
        drop_last: false
        pin_memory: false
      optim:
        name: adamw
        lr: 3.0e-05
        weight_decay: 0.0
        sched:
          name: CosineAnnealing
          warmup_steps: null
          warmup_ratio: 0.1
          min_lr: 0.0
          last_epoch: -1
    exp_manager:
      exp_dir: null
      name: PretrainingBERTFromText
      create_tensorboard_logger: true
      create_checkpoint_callback: true
    
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
Using native 16bit precision.
[NeMo I 2021-06-02 17:25:01 exp_manager:210] Experiments will be logged at /home/kruthika/NeMo/examples/nlp/language_modeling/nemo_experiments/PretrainingBERTFromText/2021-06-02_17-25-01
[NeMo I 2021-06-02 17:25:01 exp_manager:550] TensorboardLogger has been set up
Traceback (most recent call last):
  File "bert_pretraining.py", line 30, in main
    exp_manager(trainer, cfg.get("exp_manager", None))
  File "/home/kruthika/NeMo/nemo/utils/exp_manager.py", line 232, in exp_manager
    configure_checkpointing(trainer, log_dir, checkpoint_name, cfg.checkpoint_callback_params)
  File "/home/kruthika/NeMo/nemo/utils/exp_manager.py", line 679, in configure_checkpointing
    checkpoint_callback = NeMoModelCheckpoint(**params)
  File "/home/kruthika/NeMo/nemo/utils/exp_manager.py", line 580, in __init__
    super().__init__(**kwargs)
TypeError: __init__() got an unexpected keyword argument 'prefix'

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.


Please check your pytorch lightning version - RC1 required 1.1.5, main requires 1.3+

I have nemo-toolkit-1.0.0rc2 .. with pytorch lightning version  1.1.5..getting error as 
    from pytorch_lightning.plugins import DDPPlugin
ImportError: cannot import name 'DDPPlugin'


And with 1.3+ same error as above

@kruthikakr I wasn't able to reproduce it. Could you share full stack trace that shows where the DDPPlugin occurs? It looks like an environment setup issue. As an alternative option you could run inside NeMo container. 

python bert_pretraining.py
Traceback (most recent call last):
  File "bert_pretraining.py", line 18, in <module>
    from pytorch_lightning.plugins import DDPPlugin
ImportError: cannot import name 'DDPPlugin'

Nemo Installation fro Source
git clone https://github.com/NVIDIA/NeMo
cd NeMo
./reinstall.sh
pytorch lightning version 1.1.5.


This looks like an environmental problem. After running ./reinstall.sh on the main branch, you should have PyTorch Lightning >= 1.3.

Make sure that source code that you're running matches the version that you installed.

@kruthikakr  did it work? can this issue be closed?

I am facing similar kind of issue here.
While running the bert pretraining file

[NeMo I 2021-07-05 05:26:45 bert_pretraining:28] Config:
     name: PretrainingBERTFromText
    trainer:
      gpus: 1
      num_nodes: 1
      max_epochs: 5
      max_steps: null
      accumulate_grad_batches: 1
      precision: 16
      amp_level: O1
      accelerator: ddp
      gradient_clip_val: 0.0
      log_every_n_steps: 1
      val_check_interval: 1.0
      checkpoint_callback: false
      logger: false
    model:
      nemo_path: null
      only_mlm_loss: false
      num_tok_classification_layers: 1
      num_seq_classification_layers: 2
      max_seq_length: 128
      mask_prob: 0.15
      short_seq_prob: 0.1
      language_model:
        pretrained_model_name: bert-base-uncased
        lm_checkpoint: null
        config:
          attention_probs_dropout_prob: 0.1
          hidden_act: gelu
          hidden_dropout_prob: 0.1
          hidden_size: 768
          initializer_range: 0.02
          intermediate_size: 3072
          max_position_embeddings: 512
          num_attention_heads: 12
          num_hidden_layers: 12
          type_vocab_size: 2
          vocab_size: 30522
        config_file: null
      tokenizer:
        tokenizer_name: ${model.language_model.pretrained_model_name}
        vocab_file: null
        tokenizer_model: null
        special_tokens:
          unk_token: '[UNK]'
          sep_token: '[SEP]'
          pad_token: '[PAD]'
          bos_token: '[CLS]'
          mask_token: '[MASK]'
          eos_token: '[SEP]'
          cls_token: '[CLS]'
      train_ds:
        data_file: /media/nlp_drive/syed/BERT-NeMo/DeepLearningExamples/PyTorch/LanguageModeling/BERT/NeMo/examples/nlp/language_modeling/download_dir/wikitext-2/train.txt
        max_seq_length: ${model.max_seq_length}
        mask_prob: ${model.mask_prob}
        short_seq_prob: ${model.short_seq_prob}
        batch_size: 16
        shuffle: true
        num_samples: -1
        num_workers: 2
        drop_last: false
        pin_memory: false
      validation_ds:
        data_file: null
        max_seq_length: ${model.max_seq_length}
        mask_prob: ${model.mask_prob}
        short_seq_prob: ${model.short_seq_prob}
        batch_size: 16
        shuffle: false
        num_samples: -1
        num_workers: 2
        drop_last: false
        pin_memory: false
      optim:
        name: adamw
        lr: 3.0e-05
        weight_decay: 0.0
        sched:
          name: CosineAnnealing
          warmup_steps: null
          warmup_ratio: 0.1
          min_lr: 0.0
          last_epoch: -1
    exp_manager:
      exp_dir: null
      name: PretrainingBERTFromText
      create_tensorboard_logger: true
      create_checkpoint_callback: true
    
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
Using native 16bit precision.
[NeMo I 2021-07-05 05:26:45 exp_manager:216] Experiments will be logged at /media/nlp_drive/syed/BERT-NeMo/DeepLearningExamples/PyTorch/LanguageModeling/BERT/NeMo/examples/nlp/language_modeling/nemo_experiments/PretrainingBERTFromText/2021-07-05_05-26-45
[NeMo I 2021-07-05 05:26:45 exp_manager:563] TensorboardLogger has been set up
Traceback (most recent call last):
  File "bert_pretraining.py", line 31, in main
    bert_model = BERTLMModel(cfg.model, trainer=trainer)
  File "/anaconda/envs/py38_default/lib/python3.8/site-packages/nemo/collections/nlp/models/language_modeling/bert_lm_model.py", line 63, in __init__
    super().__init__(cfg=cfg, trainer=trainer)
  File "/anaconda/envs/py38_default/lib/python3.8/site-packages/nemo/core/classes/modelPT.py", line 130, in __init__
    self.setup_multiple_validation_data(val_data_config=None)
  File "/anaconda/envs/py38_default/lib/python3.8/site-packages/nemo/core/classes/modelPT.py", line 564, in setup_multiple_validation_data
    model_utils.resolve_validation_dataloaders(model=self)
  File "/anaconda/envs/py38_default/lib/python3.8/site-packages/nemo/utils/model_utils.py", line 242, in resolve_validation_dataloaders
    model.setup_validation_data(cfg.validation_ds)
  File "/anaconda/envs/py38_default/lib/python3.8/site-packages/nemo/collections/nlp/models/language_modeling/bert_lm_model.py", line 199, in setup_validation_data
    else self._setup_dataloader(val_data_config)
  File "/anaconda/envs/py38_default/lib/python3.8/site-packages/nemo/collections/nlp/models/language_modeling/bert_lm_model.py", line 230, in _setup_dataloader
    dataset = BertPretrainingDataset(
  File "/anaconda/envs/py38_default/lib/python3.8/site-packages/nemo/collections/nlp/data/language_modeling/lm_bert_dataset.py", line 70, in __init__
    sentence_indices, sentence_idx_file, data_dir = load_data_indices(
  File "/anaconda/envs/py38_default/lib/python3.8/site-packages/nemo/collections/nlp/data/data_utils/data_preprocessing.py", line 468, in load_data_indices
    data_dir = data_file[: data_file.rfind('/')]
AttributeError: 'NoneType' object has no attribute 'rfind'


Can you suggest some small dataset which is similar to wikicorpus, so that I can run the bert_pretraining.py file

@BakingBrains you need to specify validation file too, changes those arguments to mandatory args 
https://github.com/NVIDIA/NeMo/pull/2449