Repository: pytorch
Issue: cat out= variant memory_format inconsistent for cpu and cuda
Labels: module: cuda,triaged,module: memory format

## 🐛 Bug
The `memory_format` behavior of `cat_out` is different on `cpu` and `cuda`.

* [`cat_out_cpu`](https://github.com/pytorch/pytorch/blob/9477211e7d609ce382c0e22d7721c14c36d083de/aten/src/ATen/native/TensorShape.cpp#L120): If out= is not the correct shape then the memory format is used from the first tensor.
* [`cat_out_cuda`](https://github.com/pytorch/pytorch/blob/9477211e7d609ce382c0e22d7721c14c36d083de/aten/src/ATen/native/cuda/Shape.cu#L454):  If out= is not the correct shape then the memory format is computed using [`compute_output_memory_format `](https://github.com/pytorch/pytorch/blob/9477211e7d609ce382c0e22d7721c14c36d083de/aten/src/ATen/native/cuda/Shape.cu#L435), calling it [here](https://github.com/pytorch/pytorch/blob/9477211e7d609ce382c0e22d7721c14c36d083de/aten/src/ATen/native/cuda/Shape.cu#L512) (**point of difference**) and then that is finally used for reszing. If you look into the implementation it is not directly utilizing the first tensors' memory format unlike the `cpu` variant. To be precise according to [L446 in Shape.cu](https://github.com/pytorch/pytorch/blob/9477211e7d609ce382c0e22d7721c14c36d083de/aten/src/ATen/native/cuda/Shape.cu#L446) `bool contiguous=True` will be assigned even if the first tensor has a different format (eg. `channels_last`) and some other tensor has a `contiguous_format`.

## To Reproduce

<details>
<summary>Test to verify the inconsistent behavior (being added in #62560)</summary>

```python
  @onlyCUDA
  def test_cat_out_memory_format(self, device):
      inp_size = (4, 4, 4, 4)
      expected_size = (8, 4, 4, 4)
      a_cuda = torch.randn(inp_size, device=device).contiguous(memory_format=torch.channels_last)
      a_cpu = torch.randn(inp_size, device='cpu').contiguous(memory_format=torch.channels_last)
      b_cuda = torch.randn(inp_size, device=device).contiguous(memory_format=torch.contiguous_format)
      b_cpu = torch.randn(inp_size, device='cpu').contiguous(memory_format=torch.contiguous_format)
      c_cuda = torch.randn(inp_size, device=device).contiguous(memory_format=torch.channels_last)

      # Case 1: if out= is the correct shape then the memory format of out= is respected

      out_cuda = torch.empty(expected_size, device=device).contiguous(memory_format=torch.contiguous_format)
      res1_cuda = torch.cat((a_cuda, b_cuda), out=out_cuda)

      out_cpu = torch.empty(expected_size, device='cpu').contiguous(memory_format=torch.contiguous_format)
      res1_cpu = torch.cat((a_cpu, b_cpu), out=out_cpu)

      self.assertTrue(res1_cuda.is_contiguous(memory_format=torch.contiguous_format))
      self.assertTrue(res1_cpu.is_contiguous(memory_format=torch.contiguous_format))

      # Case 2: if out= is not the correct shape then the output it is resized internally
      # - For the CPU variant the memory format is that of the first tensor
      # - For the CUDA variant it only propagates memory format if all the tensors have
      #   the same memory format, otherwise it just uses contiguous_format as a default

      out_cuda = torch.empty((0), device=device).contiguous(memory_format=torch.contiguous_format)
      # a_cuda and b_cuda have different memory_format
      res2_cuda = torch.cat((a_cuda, b_cuda), out=out_cuda)

      out_cpu = torch.empty((0), device='cpu').contiguous(memory_format=torch.contiguous_format)
      res2_cpu = torch.cat((a_cpu, b_cpu), out=out_cpu)

      self.assertTrue(res2_cuda.is_contiguous(memory_format=torch.contiguous_format))
      self.assertTrue(res2_cpu.is_contiguous(memory_format=torch.channels_last))

      out_cuda = torch.empty((0), device=device).contiguous(memory_format=torch.contiguous_format)
      # a_cuda and c_cuda have same memory_format
      res3_cuda = torch.cat((a_cuda, c_cuda), out=out_cuda)

      self.assertTrue(res3_cuda.is_contiguous(memory_format=torch.channels_last))
```
</details>

## Expected behavior

Both the `cpu` and `cuda` variants of `cat_out` should follow the same behaviour and logic for `memory_format`.

## Pitch

A suggested fix is to make the behavior consistently like CUDA, only propagating when all the inputs have the same `memory_format`. In short we utilize [`compute_output_memory_format `](https://github.com/pytorch/pytorch/blob/9477211e7d609ce382c0e22d7721c14c36d083de/aten/src/ATen/native/cuda/Shape.cu#L435) for the CPU variant as well.

**Note**: Though this is a BC breaking change, it is probably not encountered by a lot of users so we can get this fixed easily.

After the 1.10 branch cut, I can send a fix according to the final discussion on this and validate it doesn't break anything.

## Additional context
Refer [#62560 comment](https://github.com/pytorch/pytorch/pull/62560#issuecomment-901597040) for a more elaborate and initial discussion.

cc @ngimel @VitalyFedyunin @jamesr66a @mruberry @kimishpatel