Repository: pytorch
Issue: Performance regression on interpolation in Kornia
Labels: triaged,bug

### 🐛 Describe the bug

When compiling the rescale function of kornia.geometry.transform.rescale (which conv2d  + interpolation) with dynamo/inductor we see a performance regression of up to 4x in CUDA and smaller but constant regression on CPU.

These are timing results for eager vs dynamo:
```
[---------------------------------------- Rescale ----------------------------------------]
                     |  eager_cpu  |  eager_cuda  |  dynamo_cpu  |  dynamo_cuda  |   opencv
1 threads: --------------------------------------------------------------------------------
      [1, 32, 0.5]   |       23.8  |     14.2     |       45.3   |      53.0     |     18.5
      [1, 32, 1.5]   |       65.1  |     14.2     |       86.4   |      52.6     |     18.7
      [1, 64, 0.5]   |       48.9  |     13.9     |       71.4   |      52.8     |     39.5
      [1, 64, 1.5]   |      210.2  |     14.3     |      239.6   |      53.3     |     39.6
      [1, 128, 0.5]  |      147.7  |     14.0     |      182.4   |      53.0     |    120.3
      [1, 128, 1.5]  |      808.8  |     14.2     |      867.7   |      53.2     |    120.2
      [1, 256, 0.5]  |      557.6  |     13.9     |      612.4   |      53.5     |    442.3
      [1, 256, 1.5]  |     3214.4  |     14.2     |     3269.5   |      54.2     |    440.6
      [1, 512, 0.5]  |     2155.5  |     14.3     |     2210.9   |      53.0     |   1736.9
      [1, 512, 1.5]  |    12733.5  |     18.1     |    12769.2   |      61.7     |   1736.2
      [2, 32, 0.5]   |       32.3  |     14.3     |       55.0   |      53.4     |     36.1
      [2, 32, 1.5]   |      114.5  |     14.2     |      136.9   |      53.7     |     35.9
      [2, 64, 0.5]   |       81.3  |     14.0     |      111.0   |      53.6     |     77.8
      [2, 64, 1.5]   |      409.1  |     14.2     |      455.3   |      53.3     |     77.8
      [2, 128, 0.5]  |      287.9  |     14.2     |      332.5   |      53.1     |    238.7
      [2, 128, 1.5]  |     1597.3  |     14.3     |     1652.7   |      53.2     |    238.9
      [2, 256, 0.5]  |     1085.8  |     14.0     |     1134.3   |      53.1     |    881.6
      [2, 256, 1.5]  |     6348.1  |     14.1     |     6415.3   |      55.8     |    881.0
      [2, 512, 0.5]  |     4236.6  |     14.1     |     4303.3   |      53.2     |   3464.7
      [2, 512, 1.5]  |    25832.0  |     26.2     |    26018.7   |      70.9     |   3469.2
      [5, 32, 0.5]   |       57.4  |     14.2     |       81.3   |      53.4     |     88.1
      [5, 32, 1.5]   |      261.4  |     14.1     |      290.9   |      55.8     |     88.3
      [5, 64, 0.5]   |      183.2  |     13.9     |      220.0   |      56.3     |    192.6
      [5, 64, 1.5]   |     1009.4  |     14.3     |     1058.6   |      55.7     |    192.8
      [5, 128, 0.5]  |      688.6  |     14.3     |      734.5   |      56.1     |    594.0
      [5, 128, 1.5]  |     3953.8  |     13.9     |     4003.5   |      55.9     |    595.4
      [5, 256, 0.5]  |     2658.2  |     14.2     |     2720.4   |      55.5     |   2210.5
      [5, 256, 1.5]  |    15882.5  |     17.8     |    20457.2   |      61.2     |   2206.8
      [5, 512, 0.5]  |    10680.2  |     14.0     |    10915.5   |      66.2     |   8679.6
      [5, 512, 1.5]  |   111027.6  |     87.6     |   111251.9   |     138.8     |   8665.6
      [9, 32, 0.5]   |       91.0  |     14.1     |      118.0   |      57.2     |    158.5
      [9, 32, 1.5]   |      463.1  |     15.2     |      505.1   |      61.3     |    158.7
      [9, 64, 0.5]   |      323.9  |     15.0     |      368.8   |      66.7     |    344.5
      [9, 64, 1.5]   |     1799.3  |     15.2     |     1857.8   |      65.9     |    345.1
      [9, 128, 0.5]  |     1222.4  |     15.4     |     1282.7   |      66.1     |   1072.3
      [9, 128, 1.5]  |     7070.1  |     15.3     |     7134.2   |      65.8     |   1073.4
      [9, 256, 0.5]  |     4778.1  |     15.5     |     4862.1   |      65.5     |   3974.5
      [9, 128, 1.5]  |     7070.1  |     15.3     |     7134.2   |      65.8     |   1073.4 
      [9, 256, 0.5]  |     4778.1  |     15.5     |     4862.1   |      65.5     |   3974.5
      [9, 256, 1.5]  |    29078.0  |     30.2     |    32856.6   |      79.8     |   3972.6
      [9, 512, 0.5]  |    19593.3  |     33.4     |    19761.1   |     117.8     |  15543.2
      [9, 512, 1.5]  |   202600.8  |    147.0     |   202055.1   |     227.9     |  15562.6
4 threads: --------------------------------------------------------------------------------
      [1, 32, 0.5]   |       23.7  |     14.3     |       31.3   |      34.7     |     18.7
      [1, 32, 1.5]   |       65.0  |     14.4     |       72.6   |      34.1     |     18.8
      [1, 64, 0.5]   |       48.9  |     14.1     |       57.1   |      34.3     |     39.6
      [1, 64, 1.5]   |      156.6  |     14.2     |      164.9   |      34.7     |     40.0
      [1, 128, 0.5]  |      166.4  |     14.0     |      182.7   |      34.2     |    120.3
      [1, 128, 1.5]  |      634.1  |     14.2     |      651.7   |      34.0     |    120.3
      [1, 256, 0.5]  |      318.9  |     14.0     |      337.9   |      34.1     |    441.1
      [1, 256, 1.5]  |     2394.8  |     14.3     |     2397.3   |      34.2     |    442.4
      [1, 512, 0.5]  |     1121.7  |     14.3     |     1146.3   |      34.2     |   1738.7
      [1, 512, 1.5]  |     9302.0  |     18.1     |     9266.6   |      34.3     |   1737.7
      [2, 32, 0.5]   |       32.3  |     14.3     |       39.9   |      34.4     |     35.9
      [2, 32, 1.5]   |      137.7  |     14.2     |      146.8   |      34.3     |     36.0
      [2, 64, 0.5]   |       81.4  |     14.0     |       91.0   |      34.3     |     78.0
      [2, 64, 1.5]   |      335.8  |     14.1     |      345.3   |      34.0     |     77.9
      [2, 128, 0.5]  |      316.6  |     14.2     |      346.8   |      34.0     |    238.9
      [2, 128, 1.5]  |     1219.4  |     14.4     |     1240.1   |      34.1     |    238.9
      [2, 256, 0.5]  |      590.4  |     14.0     |      611.9   |      34.3     |    880.0
      [2, 256, 1.5]  |     4657.5  |     14.1     |     4697.3   |      34.0     |    881.5
      [2, 512, 0.5]  |     2165.1  |     14.1     |     2186.8   |      34.1     |   3466.0
      [2, 512, 1.5]  |    18391.2  |     26.1     |    18398.1   |      41.8     |   3476.7
      [5, 32, 0.5]   |       57.6  |     14.4     |       65.4   |      33.8     |     88.2
      [5, 32, 1.5]   |      326.9  |     14.2     |      339.6   |      34.1     |     88.2
      [5, 64, 0.5]   |      223.6  |     14.2     |      240.0   |      34.3     |    192.8
      [5, 64, 1.5]   |      789.4  |     14.2     |      810.0   |      34.0     |    192.7
      [5, 128, 0.5]  |      760.8  |     14.2     |      784.4   |      33.8     |    595.3
      [5, 128, 1.5]  |     2983.4  |     14.2     |     2994.6   |      34.3     |    605.7
      [5, 256, 0.5]  |     1405.6  |     14.3     |     1424.7   |      33.7     |   2210.3
      [5, 256, 1.5]  |    11539.5  |     17.8     |    11553.5   |      34.2     |   2205.0
      [5, 512, 0.5]  |     5361.9  |     14.0     |     5373.1   |      41.2     |   8668.3
      [5, 512, 1.5]  |    61519.4  |     87.3     |    61506.9   |     113.1     |   8679.6
      [9, 32, 0.5]   |       91.1  |     14.2     |       98.5   |      34.3     |    158.1
      [9, 32, 1.5]   |      538.4  |     15.2     |      553.1   |      34.5     |    157.9
      [9, 64, 0.5]   |      365.3  |     15.0     |      387.8   |      37.7     |    345.4
      [9, 64, 1.5]   |     1388.3  |     15.2     |     1416.0   |      37.6     |    345.8
      [9, 128, 0.5]  |     1284.0  |     15.4     |     1306.2   |      37.4     |   1073.4
      [9, 128, 1.5]  |     5332.5  |     15.3     |     5407.2   |      37.5     |   1072.3
      [9, 256, 0.5]  |     2490.2  |     15.5     |     2503.7   |      37.7     |   3969.1
      [9, 256, 1.5]  |    20731.2  |     29.9     |    20729.8   |      48.7     |   3976.7
      [9, 512, 0.5]  |     9657.5  |     33.4     |     9725.7   |      87.8     |  15561.7
      [9, 512, 1.5]  |   113171.0  |    147.0     |   113178.5   |     197.6     |  15578.7

Times are in microseconds (us).
```

To reproduce:
```
pip install kornia opencv-python
git clone https://github.com/kornia/kornia-benchmark
cd kornia-benchmark/dynamo
python test_rescale.py
```


### Error logs

_No response_

### Minified repro

_No response_