Repository: pytorch
Issue: Make BetterTransformer implementation non-blocking
Labels: module: nn,triaged

### 🚀 The feature, motivation and pitch

I am using `optimum` integration for `BetterTransformer` with AMP.

Here is what I get without `BetterTransformer`:
![Screenshot from 2023-04-04 12-46-46](https://user-images.githubusercontent.com/56936206/229759565-cf3162dd-13d1-4b11-bbc9-e7d6782ec315.png)

![Screenshot from 2023-04-04 12-44-00](https://user-images.githubusercontent.com/56936206/229759632-9a423006-9bff-446b-ba4c-e95c657e7f5b.png)

The key point here is `_process_doc_contents` is a CPU-intensive preprocessing function and `repad` is GPU-blocking. Notice how we seemingly spend no time in `transformers` code.

Now after I introduce `BetterTransformer` I get the following picture:
![Screenshot from 2023-04-04 12-46-50](https://user-images.githubusercontent.com/56936206/229761459-04c8d070-229d-4162-9d4f-dc1331531370.png)
![Screenshot from 2023-04-04 12-51-58](https://user-images.githubusercontent.com/56936206/229761625-a8a1ff00-05b3-493e-8efa-f0a1fdf95fc3.png)

Suddenly, we start spending a lot of time in `transformers` code, which tells me some operation is GPU-blocking there. Furthermore, my guess is confirmed by the drop in GPU saturation and increase in total time by ~30s (the time needed to preprocess the inputs).

Why is this issue not in `transformers` repo? As far as I can tell from the profiler's output, the execution was blocked by this function:

![image](https://user-images.githubusercontent.com/56936206/229764685-1f02c589-eaa8-402a-852a-b0bfd7b7bec3.png)

**Software:**
Ubuntu 18.04.6 LTS
```
torch==2.0.0
transformers==4.27.4
optimum==1.7.3
```

**Hardware:**
NVIDIA GeForce RTX 2060
```
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.108.03   Driver Version: 510.108.03   CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0 Off |                  N/A |
| 32%   38C    P8     9W / 160W |      5MiB /  6144MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      2541      G   /usr/lib/xorg/Xorg                  4MiB |
+-----------------------------------------------------------------------------+

```

Sorry for not providing a reproducible example. I will work on it later since for now I am not sure how to implement it.

### Alternatives

_No response_

### Additional context

_No response_

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki @bhosmer @cpuhrsch @erichan1 @drisspg