Repository: pytorch
Issue: Activation functions for complex tensors
Labels: module: nn,triaged,module: complex,function request

## 🚀 Feature
<!-- A clear and concise description of the feature proposal -->
Activation functions in `torch.nn` should support complex activation functions.

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request related to a problem? e.g., I'm always frustrated when [...]. If this is related to another GitHub issue, please link here too -->

Complex numbers are different from just a simple pair of numbers with their inherent periodicity.
Complex neural networks are a growing research field.
Allowing for these would allow [physical simulation](https://arxiv.org/abs/2009.11810) of complex neural networks with optics.
Complex neural networks need complex activation functions

Similar to https://github.com/pytorch/pytorch/issues/46642.

## Pitch

These activation functions should be implemented for both forward and backward operations on the CPU and CUDA.
The documentation should be clear on whether it calculates the activation function separately for the real and complex parts, or uses the argument and the magnitude.

<!-- A clear and concise description of what you want to happen. -->
- [ ] [nn.ELU](https://pytorch.org/docs/stable/generated/torch.nn.ELU)
- [ ] [nn.Hardshrink](https://pytorch.org/docs/stable/generated/torch.nn.Hardshrink)
- [ ] [nn.Hardsigmoid](https://pytorch.org/docs/stable/generated/torch.nn.Hardsigmoid)
- [ ] [nn.Hardtanh](https://pytorch.org/docs/stable/generated/torch.nn.Hardtanh)
- [ ] [nn.Hardswish](https://pytorch.org/docs/stable/generated/torch.nn.Hardswish)
- [ ] [nn.LeakyReLU](https://pytorch.org/docs/stable/generated/torch.nn.LeakyReLU)
- [ ] [nn.LogSigmoid](https://pytorch.org/docs/stable/generated/torch.nn.LogSigmoid)
- [ ] [nn.MultiheadAttention](https://pytorch.org/docs/stable/generated/torch.nn.MultiheadAttention)
- [ ] [nn.PReLU](https://pytorch.org/docs/stable/generated/torch.nn.PReLU)
- [ ] [nn.ReLU](https://pytorch.org/docs/stable/generated/torch.nn.ReLU)
- [ ] [nn.ReLU6](https://pytorch.org/docs/stable/generated/torch.nn.ReLU6)
- [ ] [nn.RReLU](https://pytorch.org/docs/stable/generated/torch.nn.RReLU)
- [ ] [nn.SELU](https://pytorch.org/docs/stable/generated/torch.nn.SELU)
- [ ] [nn.CELU](https://pytorch.org/docs/stable/generated/torch.nn.CELU)
- [ ] [nn.GELU](https://pytorch.org/docs/stable/generated/torch.nn.GELU)
- [ ] [nn.Sigmoid](https://pytorch.org/docs/stable/generated/torch.nn.Sigmoid)
- [ ] [nn.SiLU](https://pytorch.org/docs/stable/generated/torch.nn.SiLU)
- [ ] [nn.Softplus](https://pytorch.org/docs/stable/generated/torch.nn.Softplus)
- [ ] [nn.Softshrink](https://pytorch.org/docs/stable/generated/torch.nn.Softshrink)
- [ ] [nn.Softsign](https://pytorch.org/docs/stable/generated/torch.nn.Softsign)
- [ ] [nn.Tanh](https://pytorch.org/docs/stable/generated/torch.nn.Tanh)
- [ ] [nn.Tanhshrink](https://pytorch.org/docs/stable/generated/torch.nn.Tanhshrink)
- [ ] [nn.Threshold](https://pytorch.org/docs/stable/generated/torch.nn.Threshold)

With priority given to frequently used activation functions e.g. ReLU

## Additional context

<!-- Add any other context or screenshots about the feature request here. -->
See <https://giggleliu.github.io/2018/02/01/complex_bp.html> for an explanation on how complex numbers are deeper than mere pairs of real numbers.

cc @ezyang @anjali411 @dylanbespalko @mruberry @albanD