Repository: pytorch
Issue: c10d and discontiguous tensors with mismatch strides
Labels: oncall: distributed,module: c10d

## 🐛 Bug

I am running the following tests:

script:
```python
import torch
import torch.distributed as dist
import argparse

parser = argparse.ArgumentParser()
parser.add_argument('backend')
parser.add_argument('--cuda', action='store_true')
args = parser.parse_args()

dist.init_process_group(args.backend)
rank = dist.get_rank()
if args.cuda:
  device = f'cuda:{rank}'
else:
  device = 'cpu'

if rank == 0:
  t = torch.randint(10, [4, 4], dtype=torch.int, device=device).T
  print("send: ", t, sep='\n')
  dist.send(t, 1, tag=128)

if rank == 1:
  t = torch.zeros([4, 4], dtype=torch.int, device=device)
  print("recv before: ", t, sep='\n')
  dist.recv(t, 0, tag=128)
  print("recv after: ", t, sep='\n')

```

with different c10d backends. And here is what I get:

NCCL gets silent wrong result:
```
$ python -m torch.distributed.run --nproc_per_node 2 torch_sendrecv_test.py nccl --cuda
send: 
tensor([[7, 5, 1, 1],
        [0, 0, 6, 9],
        [5, 2, 2, 1],
        [0, 4, 0, 2]], device='cuda:0', dtype=torch.int32)
recv before: 
tensor([[0, 0, 0, 0],
        [0, 0, 0, 0],
        [0, 0, 0, 0],
        [0, 0, 0, 0]], device='cuda:1', dtype=torch.int32)
recv after: 
tensor([[7, 0, 5, 0],
        [5, 0, 2, 4],
        [1, 6, 2, 0],
        [1, 9, 1, 2]], device='cuda:1', dtype=torch.int32)
```

gloo gets an error:
```
$ python -m torch.distributed.run --nproc_per_node 2 torch_sendrecv_test.py gloo
Traceback (most recent call last):
  File "/home/gaoxiang/torch_ucc/test/torch_sendrecv_test.py", line 20, in <module>
    dist.send(t, 1, tag=128)
  File "/home/gaoxiang/.local/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 929, in send
    default_pg.send([tensor], dst, tag).wait()
RuntimeError: input tensor has to be contiguous

$ python -m torch.distributed.run --nproc_per_node 2 torch_sendrecv_test.py gloo --cuda
Traceback (most recent call last):
  File "/home/gaoxiang/torch_ucc/test/torch_sendrecv_test.py", line 20, in <module>
    dist.send(t, 1, tag=128)
  File "/home/gaoxiang/.local/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 929, in send
    default_pg.send([tensor], dst, tag).wait()
RuntimeError: input tensor has to be contiguous
```

mpi gets an error:
```
$ python -m torch.distributed.run --nproc_per_node 2 torch_sendrecv_test.py mpi
Traceback (most recent call last):
  File "/home/gaoxiang/torch_ucc/test/torch_sendrecv_test.py", line 20, in <module>
    dist.send(t, 1, tag=128)
  File "/home/gaoxiang/.local/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 929, in send
    default_pg.send([tensor], dst, tag).wait()
RuntimeError: input tensor has to be contiguous
```

I am wondering what should be the correct behavior: On the one hand, distributed training on channels-last models should be supported. But on the other hand, I can not think of any good solution about how to avoid generating silent wrong results in such cases. And I didn't see any documentation saying anything about discontiguous tensors.

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @agolynski @SciPioneer @H-Huang @mrzzd @cbalioglu @gcramer23