Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.12.0.dev20220322+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220320+cu113

Base PyTorch commit: 38512d90c1380aa7eb66bdfbc52c744bc06b17c6

Affected PyTorch version: 1.12.0.dev20220322+cu113

Affected PyTorch commit: 7c2103ad5ffdc1ef91231c966988f7f2a61b4166

Affected Tests:
- test_eval[densenet121-cpu-eager]: -7.87439%
- test_train[drq-cuda-eager]: +11.52762%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2025016670

Empty result. Will revisit later.

Update local reproduction result.
20220320: 
```
Running train method from drq on cuda in eager mode.
GPU Time:            311.469 milliseconds
CPU Dispatch Time:   275.483 milliseconds
CPU Total Wall Time: 311.497 milliseconds
Running train method from drq on cuda in eager mode.
GPU Time:            317.457 milliseconds
CPU Dispatch Time:   282.534 milliseconds
CPU Total Wall Time: 317.487 milliseconds
Running train method from drq on cuda in eager mode.
GPU Time:            340.330 milliseconds
CPU Dispatch Time:   307.145 milliseconds
CPU Total Wall Time: 340.361 milliseconds
Running train method from drq on cuda in eager mode.
GPU Time:            314.320 milliseconds
CPU Dispatch Time:   277.322 milliseconds
CPU Total Wall Time: 314.353 milliseconds
Running train method from drq on cuda in eager mode.
GPU Time:            322.174 milliseconds
CPU Dispatch Time:   286.564 milliseconds
CPU Total Wall Time: 322.210 milliseconds
Running train method from drq on cuda in eager mode.
GPU Time:            315.084 milliseconds
CPU Dispatch Time:   278.212 milliseconds
CPU Total Wall Time: 315.118 milliseconds
Running train method from drq on cuda in eager mode.
GPU Time:            314.162 milliseconds
CPU Dispatch Time:   277.816 milliseconds
CPU Total Wall Time: 314.195 milliseconds
Running train method from drq on cuda in eager mode.
GPU Time:            314.624 milliseconds
CPU Dispatch Time:   277.483 milliseconds
CPU Total Wall Time: 314.658 milliseconds
Running train method from drq on cuda in eager mode.
GPU Time:            335.913 milliseconds
CPU Dispatch Time:   299.789 milliseconds
CPU Total Wall Time: 335.954 milliseconds
Running train method from drq on cuda in eager mode.
GPU Time:            317.884 milliseconds
CPU Dispatch Time:   280.970 milliseconds
CPU Total Wall Time: 317.928 milliseconds
```

20220322: 
```
Running train method from drq on cuda in eager mode.
GPU Time:            313.069 milliseconds
CPU Dispatch Time:   277.470 milliseconds
CPU Total Wall Time: 313.213 milliseconds
Running train method from drq on cuda in eager mode.
GPU Time:            316.980 milliseconds
CPU Dispatch Time:   282.127 milliseconds
CPU Total Wall Time: 317.015 milliseconds
Running train method from drq on cuda in eager mode.
GPU Time:            312.834 milliseconds
CPU Dispatch Time:   277.320 milliseconds
CPU Total Wall Time: 312.870 milliseconds
Running train method from drq on cuda in eager mode.
GPU Time:            327.293 milliseconds
CPU Dispatch Time:   292.023 milliseconds
CPU Total Wall Time: 327.330 milliseconds
Running train method from drq on cuda in eager mode.
GPU Time:            327.811 milliseconds
CPU Dispatch Time:   292.385 milliseconds
CPU Total Wall Time: 327.922 milliseconds
Running train method from drq on cuda in eager mode.
GPU Time:            315.365 milliseconds
CPU Dispatch Time:   278.924 milliseconds
CPU Total Wall Time: 315.401 milliseconds
Running train method from drq on cuda in eager mode.
GPU Time:            315.187 milliseconds
CPU Dispatch Time:   278.923 milliseconds
CPU Total Wall Time: 315.224 milliseconds
Running train method from drq on cuda in eager mode.
GPU Time:            316.011 milliseconds
CPU Dispatch Time:   279.901 milliseconds
CPU Total Wall Time: 316.040 milliseconds
Running train method from drq on cuda in eager mode.
GPU Time:            317.862 milliseconds
CPU Dispatch Time:   281.137 milliseconds
CPU Total Wall Time: 317.899 milliseconds
Running train method from drq on cuda in eager mode.
GPU Time:            320.306 milliseconds
CPU Dispatch Time:   284.071 milliseconds
CPU Total Wall Time: 320.341 milliseconds
```

There is no noticeable difference in latency. Closed.

V1 performance:
```
# pytorch 20220320
Ran in 0.3175199031829834 seconds, gpu time 366.21795654296875.
Ran in 0.3111572265625 seconds, gpu time 354.811767578125.
Ran in 0.3147120475769043 seconds, gpu time 359.29705810546875.
Ran in 0.31801557540893555 seconds, gpu time 365.5249938964844.
Ran in 0.3069474697113037 seconds, gpu time 351.0711975097656.
Ran in 0.3273293972015381 seconds, gpu time 374.24822998046875.
Ran in 0.30251431465148926 seconds, gpu time 348.1863098144531.
Ran in 0.3320591449737549 seconds, gpu time 373.36883544921875.
Ran in 0.32295918464660645 seconds, gpu time 367.07904052734375.
Ran in 0.3360574245452881 seconds, gpu time 380.9996337890625.
```
```
# pytorch 20220322
Ran in 0.29651474952697754 seconds, gpu time 341.79205322265625.
Ran in 0.29657530784606934 seconds, gpu time 341.3385009765625.
Ran in 0.30380702018737793 seconds, gpu time 349.1574401855469.
Ran in 0.6331512928009033 seconds, gpu time 762.4970092773438.
Ran in 0.6546080112457275 seconds, gpu time 785.723388671875.
Ran in 0.4512946605682373 seconds, gpu time 494.329833984375.
Ran in 0.6517105102539062 seconds, gpu time 795.0052490234375.
Ran in 0.6477818489074707 seconds, gpu time 786.6122436523438.
Ran in 0.6416068077087402 seconds, gpu time 777.970458984375.
Ran in 0.6522409915924072 seconds, gpu time 789.6398315429688.
```

The test seems really unstable and should be disabled for v1.