Repository: pytorch
Issue: Exponentiating floating number with cuda tensor is slow
Labels: module: cuda,triaged,topic: performance

### 🐛 Describe the bug

Exponentiating a floating number with a cuda tensor is slower than doing the same operation on cpu (even with numpy). 

The dtype of the tensor does not seem to impact significantly the results. 
Perhaps it's expected since it isn't an operation that is expected to benefit from being executed on cuda, but still the difference in performance is quite significant.
Strangely, changing to `a ** b` to `(a.log() * b).exp()` seems to bring quite a speedup.

```python
import torch
import timeit
import math

if __name__ == "__main__":
    steps = torch.randint(5, (512,))
    gamma = 0.99

    print("numpy")
    steps_numpy = steps.numpy()
    print(timeit.timeit("gamma ** steps_numpy", globals={"steps_numpy": steps_numpy, "gamma": gamma}))

    print("torch, cpu")
    print(timeit.timeit("gamma ** steps", globals={"steps": steps, "gamma": gamma}))

    print("torch, cpu, float")
    steps_float = steps.float()
    print(timeit.timeit("gamma ** steps_float", globals={"steps_float": steps_float, "gamma": gamma}))

    print("torch, cpu, int8")
    steps_int8 = steps.to(torch.int8)
    print(timeit.timeit("gamma ** steps_int8", globals={"steps_int8": steps_int8, "gamma": gamma}))

    print("torch, cuda")
    steps_cuda = steps.cuda()
    print(timeit.timeit("gamma ** steps_cuda", globals={"steps_cuda": steps_cuda, "gamma": gamma}))

    print("torch, cuda, float")
    steps_cuda_float = steps.cuda().float()
    print(timeit.timeit("gamma ** steps_cuda_float", globals={"steps_cuda_float": steps_cuda_float, "gamma": gamma}))

    print("torch, cuda, int8")
    steps_cuda_int8 = steps.cuda().to(torch.int8)
    print(timeit.timeit("gamma ** steps_cuda_int8", globals={"steps_cuda_int8": steps_cuda_int8, "gamma": gamma}))

    print("torch, cuda, log")
    print(timeit.timeit("(math.log(gamma) * steps_cuda).exp()", globals={"steps_cuda": steps_cuda, "gamma": gamma, "torch": torch, "math": math}))

    print("torch, cuda, log, int8")
    print(timeit.timeit("(math.log(gamma) * steps_cuda_int8).exp()", globals={"steps_cuda_int8": steps_cuda_int8, "gamma": gamma, "torch": torch, "math": math}))
```

Heres' the log on a cluster with recent nvidia gpus:
```
numpy
15.628631395753473
torch, cpu
12.830785953439772
torch, cpu, float
9.56215474428609
torch, cpu, int8
12.628445492126048
torch, cuda
31.448247560299933
torch, cuda, float
30.8037648932077
torch, cuda, int8
31.459348308388144
torch, cuda, log
13.928261326625943
torch, cuda, log, int8
13.960974802728742
```

### Versions

```
PyTorch version: 1.13.0.dev20220526
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.6 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
CMake version: version 3.22.3
Libc version: glibc-2.27

Python version: 3.9.13 | packaged by conda-forge | (main, May 27 2022, 16:58:50)  [GCC 10.3.0] (64-bit runtime)
Python platform: Linux-5.4.0-1069-aws-x86_64-with-glibc2.27
Is CUDA available: True
CUDA runtime version: 11.3.109
GPU models and configuration:
GPU 0: NVIDIA A100-SXM4-40GB
GPU 1: NVIDIA A100-SXM4-40GB
GPU 2: NVIDIA A100-SXM4-40GB
GPU 3: NVIDIA A100-SXM4-40GB
GPU 4: NVIDIA A100-SXM4-40GB
GPU 5: NVIDIA A100-SXM4-40GB
GPU 6: NVIDIA A100-SXM4-40GB
GPU 7: NVIDIA A100-SXM4-40GB

Nvidia driver version: 510.47.03
cuDNN version: Probably one of the following: 
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.0.5
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.1.1

HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] functorch==0.3.0a0+9fa0265
[pip3] numpy==1.22.4
[pip3] torch==1.13.0.dev20220526
[pip3] torchaudio==0.12.0.dev20220529
[pip3] torchrl==0.1
[pip3] torchvision==0.14.0.dev20220529
[conda] _tflow_select             2.3.0                       mkl
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               11.3.1               h2bc3f7f_2
[conda] functorch                 0.3.0a0+9fa0265          pypi_0    pypi
[conda] mkl                       2021.4.0           h06a4308_640
[conda] mkl-service               2.4.0            py39h7e14d7c_0    conda-forge
[conda] mkl_fft                   1.3.1            py39h0c7bc48_1    conda-forge
[conda] mkl_random                1.2.2            py39hde0f152_0    conda-forge
[conda] numpy                     1.22.4                   pypi_0    pypi
[conda] numpy-base                1.22.3           py39hf524024_0
[conda] pytorch                   1.13.0.dev20220526 py3.9_cuda11.3_cudnn8.3.2_0    pytorch-nightly
[conda] pytorch-mutex             1.0                        cuda    pytorch-nightly
[conda] tensorflow                2.4.1           mkl_py39h4683426_0
[conda] tensorflow-base           2.4.1           mkl_py39h43e0292_0
[conda] torchaudio                0.12.0.dev20220529      py39_cu113    pytorch-nightly
[conda] torchrl                   0.1                       dev_0    <develop>
[conda] torchvision               0.14.0.dev20220529      py39_cu113    pytorch-nightly
```

cc @ngimel