Repository: pytorch
Issue: Second-order gradient cause segfault over time
Labels: needs reproduction,triaged

## 🐛 Bug

Hello, I’m running a GAN model with some second-order regularization. The training went well at the beginning and the loss decreased. However, after a random period of time, ranging from 3000-8000 updates, the segfault occurs and the training was terminated automatically. 

## To Reproduce

run the following script and wait, the error will finally occur. In my case it took 2 days to occur (total number fo iteration = 443298). 

```
import torch
from torch.utils.data import Dataset, DataLoader
import torchvision.models as models
import torch.nn as nn
import torch.autograd as autograd
import torch.nn.functional as F

class data_gpu_prefetcher():
    def __init__(self, loader, stop=False):
        self.original_loader = loader
        self.loader = iter(loader)
        self.stream = torch.cuda.Stream()
        self.stop = stop
        self._count = 0
        self.preload()

    def preload(self):
        self.next_input = next(self.loader, None)
        if self.next_input is None and not self.stop:
            self.loader = iter(self.original_loader)
            self.next_input = next(self.loader)
        with torch.cuda.stream(self.stream):
            if isinstance(self.next_input, list):
                self.next_input = [item.cuda(non_blocking=True) for item in self.next_input]
            elif self.next_input is not None:
                self.next_input = self.next_input.cuda(non_blocking=True)

    def next(self):
        self._count += 1
        torch.cuda.current_stream().wait_stream(self.stream)
        input = self.next_input
        self.preload()
        return input

    def reset_count(self):
        self._count = 0

    def get_count(self):
        return self._count

class RandomDataset(Dataset):

    def __init__(self, size, length):
        self.len = length
        self.data = torch.randn(length, 3, size, size)
        self.label = torch.randint(0, 10, (length,))

    def __getitem__(self, index):
        return self.data[index], self.label[index]

    def __len__(self):
        return self.len

class resnet_cls(nn.Module):
    def __init__(self):
        super(resnet_cls, self).__init__()
        self.model = models.resnet50()
        self.cls = nn.Linear(1000, 10)
        self._count = 0

    def forward(self, im, label, r1_reg=False):
        im.requires_grad_(r1_reg)
        score = self.model(im)
        loss = F.nll_loss(F.log_softmax(self.cls(score), dim=1), target=label, reduction='none')
        if r1_reg:
            gradients = autograd.grad(
                outputs=score.sum(),
                inputs=im,
                grad_outputs=None,
                create_graph=True,
                retain_graph=True,
                only_inputs=True,
            )[0]
            gradients = gradients.view(gradients.size(0), -1)
            gradient_norms = gradients.pow(2).sum(1)
            loss += gradient_norms * 1
        return loss

rand_loader = DataLoader(dataset=RandomDataset(224, 1000),
                         batch_size=32, shuffle=True)
train_loader_iterator = data_gpu_prefetcher(rand_loader, stop=False)
model = resnet_cls()
model.cuda()
model_para = nn.DataParallel(model)
opt = torch.optim.Adam(model.parameters(), lr=0.001, betas=(0., 0.99))
count = 0
while True:
    r1_reg = count % 2 == 0
    print(r1_reg)
    count += 1
    imgs, labels = train_loader_iterator.next()
    loss = model_para(imgs, labels, r1_reg).mean()
    opt.zero_grad()
    loss.backward()
    opt.step()
    print(loss.item(), flush=True)
```

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

## Expected behavior

After some random number of updates, in my case 443298 iterations, the problem occurs and training was terminated. 

In PyTorch 1.3 it shows: 

> Segmentation fault (core dumped)

It prints "True" for the value of "r1_reg" for this iteration, but the loss for the forward pass didn't print out. No other information is shown. 

<!-- A clear and concise description of what you expected to happen. -->

## Environment

I'm using the official PyTorch Docker image to run my training. I used the official image 1.3-cuda10.1-cudnn7-devel.

I’m using 4 Titan x (pascal) for data-parallel training.

## Additional context

<!-- Add any other context about the problem here. -->