Repository: pytorch
Issue: RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one.
Labels: oncall: distributed,triaged,module: data parallel

## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->

## To Reproduce
Epoch: 1, iter 0: loss = 10.099
  0%|                                    | 1/144967 [00:02<116:54:31,  2.90s/it]
Traceback (most recent call last):
  File "train.py", line 99, in <module>
    solver.train()
  File "/home/yckj2453/nlp_space/jd_multimodal_dialogue/multi-modal-dialogue-transformer_bart/utils/time_track.py", line 18, in timed
    result = method(*args, **kwargs)
  File "/home/yckj2453/nlp_space/jd_multimodal_dialogue/multi-modal-dialogue-transformer_bart/solver.py", line 284, in train
    decoder_input_ids=decoder_input_ids)
  File "/root/anaconda3/envs/jddc_mddr/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/root/anaconda3/envs/jddc_mddr/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 473, in forward
    self.reducer.prepare_for_backward(list(_find_tensors(output)))
RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by (1) passing the keyword argument `find_unused_parameters=True` to `torch.nn.parallel.DistributedDataParallel`; (2) making sure all `forward` function outputs participate in calculating loss. If you already have done the above two steps, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's `forward` function. Please include the loss function and the structure of the return value of `forward` of your module when reporting this issue (e.g. list, dict, iterable).
Traceback (most recent call last):
  File "/root/anaconda3/envs/jddc_mddr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/root/anaconda3/envs/jddc_mddr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/root/anaconda3/envs/jddc_mddr/lib/python3.7/site-packages/torch/distributed/launch.py", line 263, in <module>
    main()
  File "/root/anaconda3/envs/jddc_mddr/lib/python3.7/site-packages/torch/distributed/launch.py", line 259, in main
    cmd=cmd)
subprocess.CalledProcessError: Command '['/root/anaconda3/envs/jddc_mddr/bin/python', '-u', 'train.py', '--local_rank=0']' returned non-zero exit status 1.

Steps to reproduce the behavior:

1.
1.
1.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

Please copy and paste the output from our
[environment collection script](https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py)
(or fill out the checklist below manually).

You can get the script and run it with:
```
wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
```

 - PyTorch Version (1.5.1):
 - OS (e.g., Linux):
 - How you installed PyTorch (`conda`, `pip`, source):
 - Build command you used (if compiling from source):
 - Python version: 3.7.5
 - CUDA/cuDNN version: 10.1/7.6
 - GPU models and configuration:
 - Any other relevant information:
   pip transformers==2.11.0 
   pip numpy==1.19.0

## Additional context

<!-- Add any other context about the problem here. -->
## here is my code:
`    def train(self):
        epoch_loss_history = []
        best_eval_loss = float('inf')  # 记录最佳损失

        # 设置并行计算
        if self.config.n_gpu > 1:
            print("use torch.nn.DataParallel for the parallel operations.")
            self.model = nn.DataParallel(self.model)
        if self.config.local_rank != -1:
            print("use torch.nn.parallel.DistributedDataParallel for the parallel operations.")
            self.model = nn.parallel.DistributedDataParallel(self.model,
                                                             device_ids=[self.config.local_rank],
                                                             output_device=self.config.local_rank,
                                                             find_unused_parameters=True)

        for epoch_i in range(self.epoch_i, self.config.n_epoch):
            # self.epoch_i = epoch_i
            batch_loss_history = []
            loss_history = []
            num_batch = 0
            self.model.train()
            n_total_words = 0

            # 每个batch开始之前, 先进行梯度清空
            # self.optimizer.zero_grad()
            self.model.zero_grad()  # 更加安全的清理梯度

            # epoch_iterator = tqdm(self.train_data_loader, desc="Iteration",
            #                       disable=self.config.local_rank not in [-1, 0])
            for batch_i, (input_ids, label_ids,
                          images, img_char_positions) in enumerate(tqdm(self.train_data_loader, ncols=80)):
                # input_ids: [batch, sentence_length]
                num_batch = batch_i

                # flatten input and target conversations
                # 去除PAD_ID列表长度, -1为了去除起始字符的长度
                label_origin_length = [len(single_label) - single_label.count(PAD_ID) - 1 for single_label in label_ids]
                if self.config.is_images_embedding:
                    # 将一个batch内的所有image都压缩到一个列表之中
                    input_images = [image for sentence_images in images for image in sentence_images]
                    # image在sentence中对应的索引号
                    input_image_indexes = [i for sentence_images_index in img_char_positions
                                           for i in sentence_images_index]
                    # 计算每个句子包含多少张图片
                    input_images_length = [len(sentence_images_index) for sentence_images_index in img_char_positions]

                    # 确保输入图片数量与图像索引号数量相同
                    assert len(input_images) == sum(input_images_length)
                    assert len(input_image_indexes) == sum(input_images_length)

                input_sentences = to_var(torch.LongTensor(input_ids))
                target_sentences = to_var(torch.LongTensor(label_ids))
                target_sentence_length = to_var(torch.LongTensor(label_origin_length))
                if self.config.is_images_embedding:
                    input_images = to_var(torch.stack(input_images))
                    input_images_length = to_var(torch.LongTensor(input_images_length))
                    input_image_indexes = to_var(torch.LongTensor(input_image_indexes))
                else:
                    input_images = None
                    input_images_length = None
                    input_image_indexes = None

                # if self.config.gradient_accumulation_step == 1:
                #     # reset gradient
                #     self.optimizer.zero_grad()
                #     self.model.zero_grad()

                attention_mask = input_sentences.ne(0).long()
                # decoder_input_ids = target_sentences[:, :-1]  # GPT解码输入, 去除末尾的结束字符
                decoder_input_ids = self.shift_tokens_right_custom(target_sentences, PAD_ID)  # 删除EOS_ID
                outputs = self.model(input_ids=input_sentences,
                                     input_images=input_images,
                                     input_images_length=input_images_length,
                                     input_image_indexes=input_image_indexes,
                                     attention_mask=attention_mask,  # input_sentences.eq(0)
                                     # lm_labels=target_sentences,
                                     decoder_input_ids=decoder_input_ids)

                # sentence_logits = self.model(
                #    input_sentences,
                #    input_sentence_length,
                #    input_conversation_length,
                #    target_sentences,
                #    input_images,
                #    input_images_length=input_images_length,
                #    input_image_indexes=input_image_indexes)

                decoder_target_label_ids = target_sentences[:, 1:]  # GPT解码Label, 去除首部的起始字符
                sentence_logits = outputs[0]  # 获取Bart的logits
                batch_loss, n_words = masked_cross_entropy(
                    sentence_logits,
                    decoder_target_label_ids,
                    target_sentence_length)

                if self.config.n_gpu > 1:
                    # mean() to average on multi-gpu parallel (not distributed) training
                    batch_loss = batch_loss.mean()
                    n_words = n_words.mean()
                if self.config.gradient_accumulation_step > 1:
                    batch_loss = batch_loss / self.config.gradient_accumulation_step
                    n_words = n_words / self.config.gradient_accumulation_step

                # assert not isnan(batch_loss.item())
                batch_loss_history.append(batch_loss.item())
                n_total_words += n_words.item()
                # loss_history.append(loss)

                if batch_i % self.config.print_every == 0:
                    # tqdm.write(
                    #    f'Epoch: {epoch_i+1}, iter {batch_i}: loss = {loss.item():.3f}')
                    tqdm.write(
                        f'Epoch: {epoch_i+1}, iter {batch_i}: loss = {batch_loss.item()/ n_words.item():.3f}')

                # Back-propagation
                # loss.backward()
                batch_loss.backward()

                # Gradient cliping
                torch.nn.utils.clip_grad_norm_(self.model.parameters(), self.config.clip)

                # 进行梯度累积
                if (batch_i + 1) % self.config.gradient_accumulation_step == 0:
                    # Run optimizer & scheduler
                    self.optimizer.step()
                    self.scheduler.step()
                    # self.optimizer.zero_grad()  # 清空梯度
                    self.model.zero_grad()

            torch.cuda.empty_cache()
            gc.collect()

            # epoch_loss = np.sum(loss_history) / (num_batch + 1)
            epoch_loss = np.sum(batch_loss_history) / n_total_words
            epoch_loss_history.append(epoch_loss)
            self.epoch_loss = epoch_loss

            print_str = f'Epoch {epoch_i+1} loss average: {epoch_loss:.3f}'
            print(print_str)

            if epoch_i % self.config.save_every_epoch == 0:
                self.save_model(epoch_i + 1)

            # Only evaluate when single GPU otherwise metrics may not average well
            if self.config.local_rank == -1:
                # print('\n<BLEU score>...')
                # self.calculate_bleu()

                print('\n<Validation>...')
                self.validation_loss = self.evaluate()

                # 保存最佳validation los model
                if self.validation_loss < best_eval_loss:
                   self.save_model(epoch_i, best='best_model')
                   # 更新最佳验证损失
                   best_eval_loss = self.validation_loss
            #
            # if epoch_i % self.config.plot_every_epoch == 0:
            #     self.write_summary(epoch_i)

        self.save_model(self.config.n_epoch)

        return epoch_loss_history`

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @xush6528 @osalpekar @jiayisuse @agolynski