Repository: pytorch
Issue: Deepcopy fails with nn.parallel.replicate
Labels: triaged,module: data parallel

## 🐛 Bug

I have a use case where I need to copy an entire model using copy.deepcopy. If I want to run this on multiple GPUs, ideally I would use `DataParallel` but since my input to my model is not a tensor, I am trying to parallelize it myself using `nn.parallel.replicate` and `nn.parallel.parallel_apply`. However, when I try to do the forward pass using `nn.parallel.parallel_apply` after the models were replicated on all devices using `nn.parallel.replicate`, I get an error saying the deepcopy is not supported. What is the idea behind not allowing this? Is the replicate method doing a different kind of copy of the model that prevents performing a deepcopy later? Is there a workaround in this case?

## To Reproduce

Code to reproduce the behavior:

```
import copy
from torch import nn, optim
import torch
from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence


class Base(nn.Module):
    def __init__(self, hidden=10):
        super(Base, self).__init__()
        self.hidden = hidden
        self.gru = nn.GRU(input_size=50,
                          hidden_size=self.hidden,
                          num_layers=1,
                          batch_first=True,
                          bidirectional=True
                          )
        self.linear = nn.Linear(2 * self.hidden, 10)
        self.tanh = nn.Tanh()

    def forward(self, input, input_len):
        packed = pack_padded_sequence(input, input_len, batch_first=True, enforce_sorted=False)
        hidden, _ = self.gru(packed)
        hidden, _ = pad_packed_sequence(hidden, batch_first=True)
        hidden = self.tanh(hidden)
        d = self.tanh(self.linear(hidden))
        return d


class MetaModel(nn.Module):
    def __init__(self):
        super(MetaModel, self).__init__()
        self.learner = Base()
        self.loss = nn.CrossEntropyLoss()
        self.lr = 0.1

    def forward(self, support_x, support_y, support_len, query_x, query_y, query_len):
        learner = copy.deepcopy(self.learner)
        params = [p for p in learner.parameters() if p.requires_grad]
        optimizer = optim.SGD(params, lr=self.lr)
        output = learner(support_x, support_len)
        loss = self.loss(output.view(-1, 10), support_y.view(-1))
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        output = learner(query_x, query_len)
        loss = self.loss(output.view(-1, 10), query_y.view(-1))
        grads = torch.autograd.grad(loss, learner.parameters())
        return grads


if __name__ == '__main__':
    support_x = torch.randn(size=(5, 10, 50))
    support_y = torch.randint(0, 10, size=(5, 10))
    support_len = torch.ones(5) * 10
    query_x = torch.randn(size=(5, 10, 50))
    query_y = torch.randint(0, 10, size=(5, 10))
    query_len = torch.ones(5) * 10

    model = MetaModel()
    grads = model(support_x, support_y, support_len, query_x, query_y, query_len)

    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    if torch.cuda.is_available() and torch.cuda.device_count() > 1:
        model = model.to(device)
        device_ids = list(range(torch.cuda.device_count()))
        replica_models = nn.parallel.replicate(model, device_ids)
        inputs = ([support_x, support_y, support_len, query_x, query_y, query_len], ) * len(device_ids)
        print(inputs)
        print(len(inputs))
        outputs = nn.parallel.parallel_apply(replica_models, inputs)
        print(outputs)
```

## Expected behavior

Expected to work fine without any errors but I get this:
``` outputs = nn.parallel.parallel_apply(replica_models, inputs)
  File "/home/nithinh/.local/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 85, in parallel_apply
    output.reraise()
  File "/home/nithinh/.local/lib/python3.6/site-packages/torch/_utils.py", line 394, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/home/nithinh/.local/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
    output = module(*input, **kwargs)
  File "/home/nithinh/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "dummy.py", line 37, in forward
    learner = copy.deepcopy(self.learner)
  File "/hpc/eb/Debian9/Python/3.6.3-foss-2017b/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/hpc/eb/Debian9/Python/3.6.3-foss-2017b/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/hpc/eb/Debian9/Python/3.6.3-foss-2017b/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/hpc/eb/Debian9/Python/3.6.3-foss-2017b/lib/python3.6/copy.py", line 240, in _deepcopy_dict           y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/hpc/eb/Debian9/Python/3.6.3-foss-2017b/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/hpc/eb/Debian9/Python/3.6.3-foss-2017b/lib/python3.6/copy.py", line 306, in _reconstruct
    value = deepcopy(value, memo)
  File "/hpc/eb/Debian9/Python/3.6.3-foss-2017b/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/hpc/eb/Debian9/Python/3.6.3-foss-2017b/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/hpc/eb/Debian9/Python/3.6.3-foss-2017b/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/hpc/eb/Debian9/Python/3.6.3-foss-2017b/lib/python3.6/copy.py", line 240, in _deepcopy_dict           y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/hpc/eb/Debian9/Python/3.6.3-foss-2017b/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/hpc/eb/Debian9/Python/3.6.3-foss-2017b/lib/python3.6/copy.py", line 306, in _reconstruct
    value = deepcopy(value, memo)
  File "/hpc/eb/Debian9/Python/3.6.3-foss-2017b/lib/python3.6/copy.py", line 161, in deepcopy
    y = copier(memo)
  File "/home/nithinh/.local/lib/python3.6/site-packages/torch/tensor.py", line 44, in __deepcopy__           raise RuntimeError("Only Tensors created explicitly by the user "
RuntimeError: Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment
```

## Environment

PyTorch version: 1.4.0
Is debug build: No
CUDA used to build PyTorch: 10.1

OS: Debian GNU/Linux 9.12 (stretch)
GCC version: (GCC) 6.4.0
CMake version: version 3.7.2

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: GeForce GTX 1080 Ti
GPU 1: GeForce GTX 1080 Ti
GPU 2: GeForce GTX 1080 Ti
GPU 3: GeForce GTX 1080 Ti

Nvidia driver version: 418.56
cuDNN version: Could not collect

Versions of relevant libraries:
[pip3] numpy==1.18.1
[pip3] numpydoc==0.9.1
[pip3] pytorch-pretrained-bert==0.6.2
[pip3] pytorch-transformers==1.1.0
[pip3] torch==1.4.0
[pip3] torch-nightly==1.2.0.dev20190722
[pip3] torchtext==0.5.0
[pip3] torchvision==0.5.0
[conda] Could not collect

## Additional context

NA