Repository: pytorch
Issue: Changing behavior of module.to() to better support mixed real- and complex-valued parameters
Labels: module: nn,triaged,module: complex,needs design

### 🚀 The feature, motivation and pitch

Motivation
--------------

It is incredibly common for modules which use complex-valued parameters to also contain real-valued parameters. This causes problems wherever module.to() is called.

Consider the example below:

```python
real = torch.ones([10,10], dtype=torch.float32)

# returns torch.complex64
print(real.to(dtype=torch.complex64).dtype)

complex = torch.ones([10,10], dtype=torch.complex64)

# returns torch.float32 and raises UserWarning for casting complex to real
print(complex.to(dtype=torch.float32).dtype)
```

This behavior seems obviously correct, but a problem appears at the module level:

```python
class MixedModule(torch.nn.Module):
     def __init__(self):
         super().__init__()
         self.complex_param = torch.nn.Parameter(torch.ones([10,10], dtype=torch.complex64))
         self.real_param = torch.nn.Parameter(torch.ones([10,10], dtype=torch.float32))

mm = MixedModule()

mm.to(dtype=torch.float64) 

 # returns torch.float64
print(mm.real_param.dtype)

# returns torch.float64
print(mm.complex_param.dtype)
```

This is makes it a dangerous game to use `module.to()` on any module which has mixed real and complex-valued parameters, for example to ensure that all parameters loaded from some external data source are stored at the proper precision for the module to be run. 

This issue also exists when modules have mixed integer and floating-point variables, which is why [module.to](https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.to) only accepts floating-point dtypes, and will not update the dtype of any integer variables.

Proposed Feature
-----------------------

I propose add a new kwarg, `precision`, to both `Tensor.to` and `Module.to`, which will update the precision but not affect the real-ness or complex-ness of the tensor(s) to be updated. Here is a brief proposal for it's behavior:

1) It accepts as input a `torch.dtype`, whose precision it will attempt to match, e.g.:
```python
real = torch.ones([10,10], dtype=torch.float32)

# returns torch.float32
print(real.to(precision=torch.complex64).dtype)

# returns torch.float64
print(real.to(precision=torch.float64).dtype)

# returns torch.float64
print(real.to(precision=torch.complex128).dtype)
```

2) If both a dtype and precision are input, it chooses dtype over precision and raises a UserWarning

3) It raises a TypeError when asked to convert a complex tensor to a non-supported real dtype, e.g.:
```python
complex = torch.ones([10,10], dtype=torch.complex64)

# raises TypeError
complex.to(precision=torch.float16)
```

4) In all other respects, it acts just like the dtype argument. For example, in `Tensor.to`, it would also accept integer dtypes and happily operate on integer dtypes. In `Module.to`, it would treat integers just like they are treated by the `dtype` argument in `Module.to`. The value of this choice is that it enables other code which wants the "precision"-style behavior to make one change everywhere.

Notes
--------

It is possible that other code within pytorch which uses `Module.to` to update dtypes would want to have it's default behavior changed to use the `precision` kwarg, in which case this feature addition would require making backwards-incompatible changes to those functions.

### Alternatives

Alternative 1
----------------
Instead of introducing a `precision` kwarg to `Tensor.to` and `Module.to`, we could introduce a boolean `preserve_realness` kwarg which would switch the `dtype` kwarg to have the behaviour outlined above. Is there a better name for this kwarg? In this case, ideally the default setting for `Tensor.to` would be `False` but the default setting for `Module.to` would be `True`.

The advantage is that other functions which use `Module.to` do not need to be explicitly updated, but the disadvantage is that `Module.to` itself becomes explicitly backwards-incompatible.

Alternative 2
----------------
Introduce two new functions, `Tensor.change_precision` and `Module.change_precision`. I think this is much less clean, because pytorch already chooses to combine device changes and dtype changes into one function - why introduce a new function just for this one weird thing?

Alternative 3
----------------
The exact behavior of the `precision` kwarg could also be different in a few ways:

1) It could accept integer (e.g. 32, 64) or string (e.g. 'float', 'double') input, or multiple different input styles
2) It could only accept `t.float32`, `t.float64`, `t.complex64`, and `t.complex128`, rather than replicating the full functionality of the `dtype` arg.
3) It could raise an error instead of a warning if both a dtype and precision argument are input



### Additional context

Let's all remember that complex numbers and real numbers are friends!!!

cc @albanD @mruberry @jbschlosser @walterddr @saketh-are @ezyang @anjali411 @dylanbespalko @Lezcano @nikitaved