Repository: pytorch/benchmark
Issue: Look into the broken bisection workflow backlogs
Root cause: Validation Gap

20230209 BISECT_ISSUE="bisection-gh4139273032" ([link](https://github.com/pytorch/benchmark/actions/runs/4195864401/jobs/7276018136))
20230214 BISECT_ISSUE="bisection-gh4174579288" ([link](https://github.com/pytorch/benchmark/actions/runs/4176323526/jobs/7274556118))
20230209 BISECT_ISSUE="bisection-gh4144603051" ([link](https://github.com/pytorch/benchmark/actions/runs/4146164644/jobs/7171476840))
20230211 BISECT_ISSUE="bisection-gh4151690668" ([link](https://github.com/pytorch/benchmark/actions/runs/4152459544/jobs/7183427822))
20230212 BISECT_ISSUE="bisection-gh4156683692" ([link](https://github.com/pytorch/benchmark/actions/runs/4157406412/jobs/7214160276))

20230209 Speedup details:
  test_eval[Super_SloMo-cuda-eager]: -16.130904297590032
  test_eval[Super_SloMo-cuda-jit]: -16.158961661914102
  test_eval[alexnet-cuda-eager]: -9.51970309891636
  test_eval[alexnet-cuda-jit]: -11.900160063287066
  test_eval[mnasnet1_0-cuda-jit]: -11.400669645704458
  test_eval[mobilenet_v3_large-cuda-jit]: -14.857222260420228
  test_eval[pyhpc_equation_of_state-cpu-jit]: 10.812980692740258
  test_eval[shufflenet_v2_x1_0-cuda-eager]: 16.67776900148924
  test_eval[shufflenet_v2_x1_0-cuda-jit]: -19.102175528241897
  test_eval[squeezenet1_1-cuda-eager]: -11.167168342628822
  test_eval[squeezenet1_1-cuda-jit]: -8.741464479628002
  test_train[Background_Matting-cuda-jit]: -8.833282419798975
  test_train[dcgan-cuda-eager]: -22.842457973298274
  test_train[dcgan-cuda-jit]: -22.746188469514394
  test_train[timm_regnet-cpu-eager]: 10.392593774447857
  test_train[timm_regnet-cuda-eager]: 8.120880684558957
  test_train[timm_resnest-cuda-eager]: -7.383987378061144
  test_train[timm_resnest-cuda-jit]: -7.171501096163819
  test_train[timm_vision_transformer-cpu-eager]: -11.371849657755236

Test workflow: https://github.com/pytorch/benchmark/actions/runs/4458597621

bisection-gh4174579288: https://github.com/pytorch/benchmark/actions/runs/4460556968
```
{
  "start": "6fadd5e94a273d62ac3f245a7615f818b6a33500",
  "end": "d567df9f36094e1efa89d11599067773f45032c8",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": []
}
```

bisection-gh4144603051: https://github.com/pytorch/benchmark/actions/runs/4484360871
Empty result.

bisection-gh4151690668: https://github.com/pytorch/benchmark/actions/runs/4462177989

Result:
```
{
  "start": "cb715c26e270759e3955968fbd3d763d51b0b83d",
  "end": "948cd61afc90e1b9067b35d4aec4ec74deeb73f6",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": []
}
```

bisection-gh4156683692: https://github.com/pytorch/benchmark/actions/runs/4461925556
Result:
```
{
  "start": "948cd61afc90e1b9067b35d4aec4ec74deeb73f6",
  "end": "67d979098567fd61dfdb918d837426535eb9883b",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": []
}
```

0131 to 0209: https://github.com/pytorch/benchmark/actions/runs/4472378678/jobs/7858467171

```
{
  "start": "2a6e08570442c8fcbad6659e074c226809b1ed22",
  "end": "cb715c26e270759e3955968fbd3d763d51b0b83d",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "f3bf46e801d",
      "commit1_time": "2023-02-06 07:11:40 +0000",
      "commit1_digest": {
        "test_eval[demucs-cpu-jit]": 2.3416957347653806,
        "test_eval[demucs-cpu-eager]": 2.3733991852030156,
        "test_train[pytorch_stargan-cpu-jit]": 1.1215154441306368,
        "test_eval[pytorch_stargan-cpu-jit]": 0.5444436671910807,
        "test_train[pytorch_stargan-cpu-eager]": 1.1224970313953235,
        "test_eval[pytorch_stargan-cpu-eager]": 0.5089264643378556
      },
      "commit2": "26cba842ada",
      "commit2_time": "2023-02-06 10:11:25 +0000",
      "commit2_digest": {
        "test_eval[demucs-cpu-jit]": 1.8380495666991918,
        "test_eval[demucs-cpu-eager]": 1.8787339514587074,
        "test_train[pytorch_stargan-cpu-jit]": 0.8897267851978541,
        "test_eval[pytorch_stargan-cpu-jit]": 0.3102563458727673,
        "test_train[pytorch_stargan-cpu-eager]": 0.8812472485704348,
        "test_eval[pytorch_stargan-cpu-eager]": 0.26982887524645777
      }
    },
    {
      "commit1": "16387bee4ac",
      "commit1_time": "2023-02-06 13:56:07 +0000",
      "commit1_digest": {
        "test_eval[demucs-cpu-jit]": 1.796384311770089,
        "test_eval[demucs-cpu-eager]": 1.8645849246997386,
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.06318514882586897,
        "test_train[pytorch_stargan-cpu-jit]": 0.876325546996668,
        "test_eval[pytorch_stargan-cpu-jit]": 0.31762100730556997,
        "test_train[pytorch_stargan-cpu-eager]": 0.9107461318839342,
        "test_eval[pytorch_stargan-cpu-eager]": 0.27297083106823267
      },
      "commit2": "db011e11eab",
      "commit2_time": "2023-02-06 14:58:54 +0000",
      "commit2_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.06796045338269323
      }
    }
  ]
}
```


bisection-gh4479951054: https://github.com/pytorch/benchmark/actions/runs/4485353249


All resolved, fixed.