Repository: pytorch
Issue: Enhance DDP doc to have a complete example including data loading
Labels: oncall: distributed

## 📚 Documentation

<!-- A clear and concise description of what content in https://pytorch.org/docs is an issue. If this has to do with the general https://pytorch.org website, please file an issue at https://github.com/pytorch/pytorch.github.io/issues/new/choose instead. If this has to do with https://pytorch.org/tutorials, please file an issue at https://github.com/pytorch/tutorials/issues/new -->
hi,
this is about the doc of ddp (DistributedDataParallel). i may be wrong, but here are two related points, 
1. i couldnt find a `complete` tutorial on using ddp, in particular examples that use dataloader with multiple workers. i found these pages: [p1](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html), [p2](https://pytorch.org/docs/stable/notes/ddp.html), [p3](https://pytorch.org/tutorials/beginner/dist_overview.html), [p4](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel), [p5](https://pytorch.org/tutorials/intermediate/dist_tuto.html), [p6](https://pytorch.org/docs/stable/distributed.html#distributed-basics), [p7](https://github.com/ml-edu/examples/blob/master/imagenet/main.py), [p8](https://github.com/pytorch/examples/blob/master/distributed/ddp/README.md). while all these pages provide very helpful information, they often discuss a specific part of ddp or they sem to use a different way to kick start processes. including dataloaders with multilple works, access to model attributes for a model tha has been wrapped within ddp could be very helpful to quickly start using this module.
2. the second point is about the difference between all the links about on starting multiprocessing. there may be different ways to start multiprocess in ddp (here, process is the one that will run on different gpu). it seems like some pages using `torch.multiprocessing`, then expliclty call `spawn` to create multiple process combined with `torch.distributed.init_process_group`. other, simply call `torch.distrbuted.init_process_group` without `torch.multiprocessing.spawn`. for example, this [page](https://github.com/pytorch/examples/blob/master/distributed/ddp/README.md) uses the last method . so, it is not clear about the `spawn` part. i may be wrong, but probably this is most dated way to use ddp. the last method seems to use the pytorch [launcher](https://pytorch.org/docs/stable/distributed.html) that automatically spawns. now after reading again the launcher part, it seems as a second option to run distributed computations that is helpful in clusters with multiples nodes. this brings me to the last part that is related to spawn, dataloader and multiple workers. 
3. i found this warning in the [doc](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel) of ddp `If you plan on using this module with a nccl backend or a gloo backend (that uses Infiniband), together with a DataLoader that uses multiple workers, please change the multiprocessing start method to forkserver (Python 3 only) or spawn. Unfortunately Gloo (that uses Infiniband) and NCCL2 are not fork safe, and you will likely experience deadlocks if you don’t change this setting.`. because ddp can be used in a transparent way without calling spawn, and because when using dataloader with multiple workers we dont specify the forking method, it is not clear where we should set this forking method. the method reference here `set_start_method` is in `torch.multiprocessing`, since when using ddp, it does not seem necessary to use `torch.multiprocessing` unless you intentionally want to create furthermore process other the ones created with `torch.distributed` to run on each gpu. 
 i guess if i use the launcher, i shouldnt worry about the forking method because it is `torch.multiprocessing.spawn` that has an argument named `start_method` which refers to start_processes() to use a different method other than `spawn` but it did not mention `forkserver`. setting the start method could have something to do with `multiprocessing_context` in [dataloader](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader) which is by the way undocumented. from what i understood from examples, using multiple workers with dataloader is transparent when using ddp. for example, without ddp, asking to use `n` workers will create `n` processes to load the data. the same thing is expected to happen when using ddp but for each `ddp process`. so far i am not sure about this because of the above warning. also, now that dataloader is combined with a distributed [sampler](https://pytorch.org/docs/stable/data.html). 

in short, it would be very helpful to illustrate a complete way to use ddps.
sorry for the long text.
hope this will improve the doc.
thanks

-- just hit another vague aspect: evaluation with ddp. how to? and how to ask to gather the output of all forwards into a single machine (master for example) to measure the performance only one time but allow other process to forward the splited mibibatch between gpus? the doc seems to focus mostly on synch gradients, but what about output for evaluation. i may have missed something in the doc about this. doing evaluation across gpus will help speedup the evaluation if every gpu deals with a chunk of the minibatch. then, one of them gather all the output and measure the performance taking advantage of distributed computation even in evaluation. reading about this pytorch [lightning](https://github.com/PyTorchLightning/pytorch-lightning/issues/702) also [here](https://github.com/PyTorchLightning/pytorch-lightning/discussions/3225).  also, i just realized that the computed loss during training is the loss of ONLY the chunk processed at that gpu. in my case, i allow only the master to report the loss. this means it is the wrong loss. losses over other chunks is missing. i am still going through doc, but i hope there is/will be a way to synch other stuff (output, loss, ...) on demand to not saturate the communiation. a [workaround](https://discuss.pytorch.org/t/ddp-evaluation-gather-output-loss-and-stuff-how-to/130593/2?u=sbelharbi).  thanks


cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @agolynski @SciPioneer @H-Huang @mrzzd @cbalioglu @gcramer23