Repository: pytorch
Issue: The use of epsilon in torch.nn.functional.pairwise_distance leads to incorrect minima of the distances
Labels: module: nn,triaged,module: correctness (silent),module: distance functions

## 🐛 Bug

The epsilon parameter in `torch.nn.functional.pairwise_distance` is used in a strange way, which leads to wrong minima of the distance function.
While the documentation (https://pytorch.org/docs/stable/generated/torch.nn.PairwiseDistance.html) mentions that epsilon is used to avoid division by zero, it is both not what is expected or implemented.
Conceptually, adding an epsilon to the sum of powers before the root makes senses to avoid computing the derivative of the root at 0 (which would be undefined), but the implementation adds it before computing the sum of powers :
https://github.com/pytorch/pytorch/blob/151ec56311ecc93dcb8703c2a1075ca1d8643e5f/aten/src/ATen/native/Distance.cpp#L20-L22
It also is not added to avoid division by zero, since it would be added to `p` instead where it would be useful when computing the l0 norm.

This also leads to a wrong minimum in practice (although the difference to the expected one is small).

This is what is currently computed :
![image](https://user-images.githubusercontent.com/26287517/125320760-4def3f80-e33c-11eb-9085-52343e000c29.png)

Whereas this is what I would expect to avoid numerical issues:
![image](https://user-images.githubusercontent.com/26287517/125320841-62333c80-e33c-11eb-8d13-e13c2c0281ba.png)

(Sorry for the lack of contrast if using a dark theme)

## To Reproduce

Sample code that shows how to obtain a distance of 0, which is both unexpected since x1 != x2 and gives a wrong minimum to the distance between x1 and x2 :

```python
import torch
import torch.nn.functional as F

eps = 1e-6

a = torch.ones((2,3)).float()
b = (torch.ones((2,3)) + eps).float()

dists = F.pairwise_distance(a,b,eps=eps).mean()
dists_id = F.pairwise_distance(a,a,eps=eps).mean()
print("Average distance float :", dists)
print("Average distance between a and itself float :", dists_id)

c = torch.ones((2,3)).half()
d = (torch.ones((2,3)) + eps).half()

dists2 = F.pairwise_distance(c,d,eps=eps).mean()
dists2_id = F.pairwise_distance(c,c,eps=eps).mean()
print("Average distance half :", dists2)
print("Average distance between c and itself half :", dists2_id)
```
Running this gives the following output:
```
Average distance float : tensor(8.0238e-08)
Average distance between a and itself float : tensor(1.7321e-06)
Average distance half : tensor(0., dtype=torch.float16)
Average distance between c and itself half : tensor(0., dtype=torch.float16)
```

For 32bits floats the behaviour is rather erroneous, as the minimum of the distance is not between a vector and itself. 
In 16 bits while we do get the erroneous value of 0 when comparing `c` and `c + 1e-6`, the correct minimum is present, but we have a larger set of correct ones.

## Expected behavior

Due to the precision of floats we probably cannot expect to have exactly 0 distance between a vector and itself, but this should still be a minimum of the distance, which is not the case here.
In the current implementation, the use of epsilon also does not help against numerical issues, which I think is its intended goal.

While this may not have a huge impact in practice, the current implementation is either not completely correct, or should be described with more clarity in the documentation.


## Environment
```
PyTorch version: 1.8.1
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.1 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.9.5 (default, Jun  4 2021, 12:28:51)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.4.0-52-generic-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: 11.0.221
GPU models and configuration: 
GPU 0: Quadro GP100
GPU 1: Quadro GP100

Nvidia driver version: 450.80.02
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.20.2
[pip3] torch==1.8.1
[pip3] torchaudio==0.8.0a0+e4e171a
[pip3] torchvision==0.9.1
[pip3] torchviz==0.0.2
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.1.1               h6406543_8    conda-forge
[conda] mkl                       2021.2.0           h06a4308_296  
[conda] mkl-service               2.3.0            py39h27cfd23_1  
[conda] mkl_fft                   1.3.0            py39h42c9631_2  
[conda] mkl_random                1.2.1            py39ha9443f7_2  
[conda] numpy                     1.20.2           py39h2d18471_0  
[conda] numpy-base                1.20.2           py39hfae3a4d_0  
[conda] pytorch                   1.8.1           py3.9_cuda11.1_cudnn8.0.5_0    pytorch
[conda] torchaudio                0.8.1                      py39    pytorch
[conda] torchvision               0.9.1                py39_cu111    pytorch
[conda] torchviz                  0.0.2                    pypi_0    pypi
```

cc @ezyang @gchanan @zou3519 @bdhirsh @jbschlosser @anjali411 @albanD @mruberry