Repository: pytorch
Issue: when distribute training  load pretrain model error
Labels: oncall: distributed,module: serialization

### 🐛 Describe the bug

```python
def resnet50(pretrained=False, **kwargs):
    """Constructs a ResNet-50 model.

    Args:
        pretrained (bool): If True, returns a model pre-trained on ImageNet
    """
    model = ResNet(Bottleneck, [3, 4, 6, 3], **kwargs)
    local_rank = kwargs.get("local_rank", -1)
    if pretrained:

        pretrained_model = model_zoo.load_url(model_urls['resnet50'])
        state = model.state_dict()
        for key in state.keys():
            if key in pretrained_model.keys():
                state[key] = pretrained_model[key]
        model.load_state_dict(state)
    return model
```

when training with multi Card,and one host, i try to load pretrain model of resnet50 by function  (resnet50) as above:

like this:
```python
 model = models.resnet50(pretrained=True, num_classes=kernel_num,local_rank=args.local_rank)
```

here is errors:
```
Traceback (most recent call last):
  File "./train.py", line 443, in <module>
    main(args)
  File "./train.py", line 289, in main
    model = models.resnet50(pretrained=True, num_classes=kernel_num,local_rank=args.local_rank)
  File "/home/siit/ccprjnet/.GitNet/models/fpn_resnet.py", line 291, in resnet50
    pretrained_model = model_zoo.load_url(model_urls['resnet50'])
  File "/home/siit/.local/lib/python3.6/site-packages/torch/hub.py", line 509, in load_state_dict_from_url
    return torch.load(cached_file, map_location=map_location)
  File "/home/siit/.local/lib/python3.6/site-packages/torch/serialization.py", line 593, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "/home/siit/.local/lib/python3.6/site-packages/torch/serialization.py", line 763, in _legacy_load
    magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: unpickling stack underflow
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/siit/.local/lib/python3.6/site-packages/torch/distributed/launch.py", line 263, in <module>
    main()
  File "/home/siit/.local/lib/python3.6/site-packages/torch/distributed/launch.py", line 259, in main
    cmd=cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python3.6', '-u', './train.py', '--local_rank=1']' returned non-zero exit status 1.

work = _default_pg.barrier() RuntimeError: Connection reset by peer
```

### Versions

```python
def resnet50(pretrained=False, **kwargs):


    model = ResNet(Bottleneck, [3, 4, 6, 3], **kwargs)

    if pretrained:

        pretrained_model = model_zoo.load_url(model_urls['resnet50'])
        state = model.state_dict()
        for key in state.keys():
            if key in pretrained_model.keys():
                state[key] = pretrained_model[key]
        model.load_state_dict(state)
    return model
```

when training with multi Card,and one host, i try to load pretrain model of resnet50 by function  (resnet50) as above:

like this:

```
 model = models.resnet50(pretrained=True,)
```

here is errors:
```
Traceback (most recent call last):
  File "./train.py", line 443, in <module>
    main(args)
  File "./train.py", line 289, in main
    model = models.resnet50(pretrained=True, num_classes=kernel_num,local_rank=args.local_rank)
  File "/models/fpn_resnet.py", line 291, in resnet50
    pretrained_model = model_zoo.load_url(model_urls['resnet50'])
  File "local/lib/python3.6/site-packages/torch/hub.py", line 509, in load_state_dict_from_url
    return torch.load(cached_file, map_location=map_location)
  File ".local/lib/python3.6/site-packages/torch/serialization.py", line 593, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "/.local/lib/python3.6/site-packages/torch/serialization.py", line 763, in _legacy_load
    magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: unpickling stack underflow
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File ".local/lib/python3.6/site-packages/torch/distributed/launch.py", line 263, in <module>
    main()
  File "/.local/lib/python3.6/site-packages/torch/distributed/launch.py", line 259, in main
    cmd=cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python3.6', '-u', './train.py', '--local_rank=1']' returned non-zero exit status 1.
```

did it can't not load pretrain model when distribute training?
python:3.6
pytorch version:1.5.1
nccl version:nccl-local-repo-ubuntu1804-2.11.4-cuda10.2_1.0-1_amd64


cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501 @mruberry