Repository: pytorch
Issue: Incorrect results for `torch.distributed.gather` for tensor created from permuted NumPy array
Labels: oncall: distributed,module: c10d

### 🐛 Describe the bug

Hi,

When creating a tensor from a non-contiguous NumPy array, e.g.
```python
my_array = np.reshape(np.arange(10), (2, 5))
my_array = np.transpose(my_array, (1, 0))
my_tensor = torch.from_numpy(my_array)
```
and then using `torch.distributed.gather`, the tensor on the destination rank is not correct.

**Expected behaviour**: either the tensor on the destination rank is equal to the tensor on the source rank, **or**, `torch.distributed.gather` throws an exception to let the user know that the tensor must be contiguous. This silent error is very hard to be aware of.


## Minimal example

The following code
```python
import os
import time
import torch
import numpy as np
import torch.distributed as dist
import torch.multiprocessing as mp 


def worker_fn(rank):
    print(f"Hello from rank {rank}")
    os.environ['MASTER_ADDR'] = '127.0.0.1'
    os.environ['MASTER_PORT'] = '29500'
    dist.init_process_group(backend='gloo', rank=rank, world_size=2)
    dist.barrier()

    if rank == 0:
        # Source rank
        my_array = np.reshape(np.arange(10), (2, 5))
        my_array = np.transpose(my_array, (1, 0))
        my_tensor = torch.from_numpy(my_array).to(torch.float32)
        dist.gather(my_tensor, dst=1)
        print("Sent tensor     = " + str(my_tensor.tolist()))
    elif rank == 1:
        # Destination rank
        not_used = torch.zeros((5, 2))
        results = [torch.zeros((5, 2)) for _ in range(2)]
        dist.gather(not_used, gather_list=results, dst=1)
        # Printing tensor from rank 0
        print("Received tensor = " + str(results[0].tolist()))


if __name__ == "__main__":
    processes = []
    mp.set_start_method("spawn")
    for rank in range(2):
        p = mp.Process(target=worker_fn, args=(rank,))
        p.start()
        processes.append(p)
    for p in processes:
        p.join()
```
leads to the following CLI output under torch 1.11.0, tested on two different systems:

```
Hello from rank 1
Hello from rank 0
Sent tensor     = [[0.0, 5.0], [1.0, 6.0], [2.0, 7.0], [3.0, 8.0], [4.0, 9.0]]
Received tensor = [[0.0, 1.0], [2.0, 3.0], [4.0, 5.0], [6.0, 7.0], [8.0, 9.0]]
```

The received tensor does not have the same permutation as the sent tensor.

### Versions

Collecting environment information...
PyTorch version: 1.11.0+cu102
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.9.10 | packaged by conda-forge | (main, Feb  1 2022, 21:24:37)  [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.13.0-30-generic-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3090
Nvidia driver version: 495.29.05
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.22.2
[pip3] pytorch-lightning==1.5.9
[pip3] torch==1.11.0
[pip3] torchaudio==0.10.2
[pip3] torchmetrics==0.7.1
[pip3] torchvision==0.11.3
[conda] blas                      2.113                       mkl    conda-forge
[conda] blas-devel                3.9.0            13_linux64_mkl    conda-forge
[conda] cudatoolkit               11.3.1              ha36c431_10    conda-forge
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] libblas                   3.9.0            13_linux64_mkl    conda-forge
[conda] libcblas                  3.9.0            13_linux64_mkl    conda-forge
[conda] liblapack                 3.9.0            13_linux64_mkl    conda-forge
[conda] liblapacke                3.9.0            13_linux64_mkl    conda-forge
[conda] mkl                       2022.0.1           h8d4b97c_803    conda-forge
[conda] mkl-devel                 2022.0.1           ha770c72_804    conda-forge
[conda] mkl-include               2022.0.1           h8d4b97c_803    conda-forge
[conda] mypy-extensions           0.4.3                    pypi_0    pypi
[conda] numpy                     1.22.2           py39h91f2184_0    conda-forge
[conda] pytorch-lightning         1.5.9                    pypi_0    pypi
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torch                     1.11.0                   pypi_0    pypi
[conda] torchaudio                0.10.2               py39_cu113    pytorch
[conda] torchmetrics              0.7.1                    pypi_0    pypi
[conda] torchvision               0.11.3               py39_cu113    pytorch

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang