Repository: pytorch
Issue: Model Evaluation Returns Nan Values Sometimes for the Same Input
Labels: triaged,module: NaNs and Infs

## 🐛 Bug

My model returns nan values while evaluating. If I run a specific code the model goes crazy and returns some part of the output NaN. It sometimes fixes itself after feeding some input images, sometimes it may not. I can't find where my problem is.

## To Reproduce

Steps to reproduce the behavior:

1. Implemented PSM-Net by myself. Loaded the pretrained model.
2. Run `t_utils.show_model_error(e1.model, pretraining_train_dataloader, device, count=5, interpolate_gt=True)` and the model goes crazy after the first output. There is no error or anything. Just some outputs are NaN.
3. Run this code but the model still returns NaN sometimes.
```
l,r,gt = next(iter(pretraining_train_dataloader))
for s in range(15):
    with torch.no_grad():
        out = e1.model(l.to(device), r.to(device))
    plt.imshow(out.detach().cpu().numpy().squeeze(0).squeeze(0))
    plt.show()
```
4. Restart the kernel run `t_utils.show_model_error(e1.model, pretraining_train_dataloader, device, count=5, interpolate_gt=False)` everything runs as supposed to be. No problem so far.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

```
def show_model_error(model, dataloader, device, count=1, cmap="plasma", interpolate_err=False, interpolate_gt=True, interpolate_method="cubic", pixel_error=3, axis_off=False, save_as_svg=False, unnormalize=True):
    if "nearest" in interpolate_method and (interpolate_err or interpolate_gt):
        print("Warn: Kitti dataset has no data points on the upper region the image. Results may be highly inaccurate!")

    model.eval()
    dataloader_iter = iter(dataloader)
    for i in range(count):
        with torch.no_grad():
            l_,r_,gt = next(dataloader_iter)
            out = model(l_.to(device),r_.to(device))
            out_ = out.detach().cpu()

        if unnormalize:
            unnormalizer_std = torchvision.transforms.Normalize(mean=[0, 0, 0], std=[1/0.229, 1/0.224, 1/0.225])
            unnormalizer_mean = torchvision.transforms.Normalize(mean=[-0.485, -0.456, -0.406], std=[1., 1., 1.])
            l = unnormalizer_mean(unnormalizer_std(l_))
            r = unnormalizer_mean(unnormalizer_std(r_))
        else:
            l = l_
            r = r_

        gt = gt[0].squeeze(dim=0)
        out_ = out_[0].squeeze(dim=0)

        # Keep the same normalization for both images
        vmin = np.min([torch.min(out_), torch.min(gt)])
        vmax = np.max([torch.max(out_), torch.max(gt)])

        # remove points from model output that are not located in gt for error calculation
        out_masked = out_.clone() # out_.contiguous()
        mask = gt <= 0.0
        out_masked[mask] = 0.0
        err = (out_masked - gt).abs()

        if interpolate_err:
            err,_ = interpolate_zerofilled_image(err, method=interpolate_method)

        if interpolate_gt:
            gt,_ = interpolate_zerofilled_image(gt, method=interpolate_method)

        fig, axs = plt.subplots(2,2, figsize=(12,5))
        axs[0,0].grid(False)
        axs[0,0].imshow(l[0].squeeze(dim=0).moveaxis(0,2))
        axs[0,0].set_title("Image input (Left)")

        axs[0,1].grid(False)
        axs[0,1].imshow(out_, cmap=cmap, vmin=vmin, vmax=vmax)
        axs[0,1].set_title("Model Output")
        
        axs[1,0].grid(False)
        axs[1,0].imshow(err, cmap=cmap, vmin=0, vmax=pixel_error)
        axs[1,0].set_title("Error (Yellow: {}PE)".format(pixel_error))
        if interpolate_err:
            axs[1,0].set_title("Error (Interpolation: {}, Yellow: {}PE)".format(interpolate_method, pixel_error))
        for _, spine in axs[1,0].spines.items():
            spine.set_visible(True)

        axs[1,1].grid(False)
        axs[1,1].imshow(gt, cmap=cmap, vmin=vmin, vmax=vmax)
        axs[1,1].set_title("Groundtruth")
        if interpolate_gt:
            axs[1,1].set_title("Groundtruth (Interpolation: {})".format(interpolate_method))

        if axis_off:
            axs[0,0].set_axis_off()
            axs[0,1].set_axis_off()
            axs[1,0].set_axis_off()
            axs[1,1].set_axis_off()

        fig.tight_layout()

        if save_as_svg:
            plt.savefig("show_model_error{}.svg".format(i), format="svg")

# img.shape = [height, width]
# method = 'nearest', 'linear', or 'cubic'
# see: https://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.griddata.html
def interpolate_zerofilled_image(img, method='cubic', interpolate_on_depth=False, max_neighbor_dist=20, invalid_fill_val=0.0):
    img = img.clone()
    points = []
    values = []
    for i in range(img.shape[0]):
        for j in range(img.shape[1]):
            if img[i,j] > 0:
                points.append([i,j])
                values.append(img[i,j])

    if interpolate_on_depth:
        for i,v in enumerate(values):
                values[i] = 1.0 / v

    grid_x, grid_y = np.mgrid[0:img.shape[0],0:img.shape[1]]
    interpolated_img = griddata(points, values, (grid_x, grid_y), method=method, fill_value=invalid_fill_val)

    if interpolate_on_depth:
        mask = interpolated_img > 0
        interpolated_img[mask] = 1.0 / interpolated_img[mask]

    tree = cKDTree(points)
    q = np.array((grid_x, grid_y)).reshape(2,-1).T
    dist, _ = tree.query(q, k=1)
    dist = dist.reshape(interpolated_img.shape)

    if max_neighbor_dist > 0:
        interpolated_img[dist > max_neighbor_dist] = invalid_fill_val

    return interpolated_img, dist
```

## Expected behavior

I would expect all outputs to be without nan values and the same.

## Environment

PyTorch version: 1.10.0.dev20210814
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.3 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: 10.0.0-4ubuntu1 
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.9.6 (default, Jul 30 2021, 16:35:19)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.13.19-xanmod1-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3080 Laptop GPU
Nvidia driver version: 470.74
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.2.4
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.2.4
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.2.4
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.2.4
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.2.4
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.2.4
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.2.4
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.20.3
[pip3] pytorch-model-summary==0.1.1
[pip3] torch==1.10.0.dev20210814
[pip3] torchaudio==0.10.0.dev20210814
[pip3] torchvision==0.11.0.dev20210814
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.1.74              h6bb024c_0    nvidia
[conda] libblas                   3.9.0            11_linux64_mkl    conda-forge
[conda] libcblas                  3.9.0            11_linux64_mkl    conda-forge
[conda] liblapack                 3.9.0            11_linux64_mkl    conda-forge
[conda] mkl                       2021.3.0           h06a4308_520  
[conda] mkl-service               2.4.0            py39h7f8727e_0  
[conda] mkl_fft                   1.3.0            py39h42c9631_2  
[conda] mkl_random                1.2.2            py39h51133e4_0  
[conda] numpy                     1.20.3           py39hf144106_0  
[conda] numpy-base                1.20.3           py39h74d4b33_0  
[conda] pytorch                   1.10.0.dev20210814 py3.9_cuda11.1_cudnn8.0.5_0    pytorch-nightly
[conda] pytorch-model-summary     0.1.1                      py_0    conda-forge
[conda] torchaudio                0.10.0.dev20210814            py39    pytorch-nightly
[conda] torchvision               0.11.0.dev20210814      py39_cu111    pytorch-nightly

## Additional context

Here some screenshots obtained with running 

![Screenshot from 2021-09-26 08-06-30](https://user-images.githubusercontent.com/12991962/134795888-3f53d94a-e3ab-4f8d-be53-b1f10ade1dea.png)
![Screenshot from 2021-09-26 08-03-46](https://user-images.githubusercontent.com/12991962/134795892-e87a358b-4ad0-44c3-8c40-df47ce8e3e5b.png)
![Screenshot from 2021-09-26 08-02-15](https://user-images.githubusercontent.com/12991962/134795895-6082218f-deef-4318-9246-671113835af4.png)
![Screenshot from 2021-09-26 07-33-24](https://user-images.githubusercontent.com/12991962/134795896-798d41a7-3250-4ef9-87dd-fb2a0e705cba.png)