zhiminy/EvalEng huggingface dataset:
Repository: pytorch/benchmark
Issue: fambench_dlrm is very slow
Root cause: Algorithmic Error

The following takes more than an hour:
```
./benchmarks/torchbench.py -dcuda -k fambench_dlrm --nothing -n100
```

This hurts time to signal a lot.

Yeah I have also noticed that, the model is too large (~13G after `torch.save()`). However, this model represents internal use case and is still useful. I am thinking to add a variable `VERY_LARGE_MODEL=True` to the class, and users skip the large models when they want to get quick results.

What do you think?

IMO this model is slow enough that it is not useful.  We could put it behind a flag like you suggest, but I think then people just won't use it.

Yesterday I updated torchbench, did an overnight run (which usually takes a couple of hours) and by morning it was still stuck on `fambench_dlrm` for one of the baselines and I got zero result for the actual experiment.  That ruined my experiment and wasted time.  I killed the running experiment, deleted the model in my working tree, restarted the run, then got results in the afternoon.

Is there a way we could have a faster time-to-signal model that still represents the important patterns we want to test?  Is the pattern already well captured in the other dlrm model in torchbench?













I see, that makes sense. We are working on replacing the current oss dlrm (which is old) with torchrec version to make it more representative and useful. But as a quick mitigation I will remove this `fambench_dlrm` model in the `models/` directory and put it in a separate directory for archive (https://github.com/pytorch/benchmark/pull/1107). So if later people want to dive deep into the performance of this specfic model, they can still do that by explicitly running it with, say, `run.py` (which is not part of the PR and will be implemented later).
Can you help stamp?
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220814+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220813+cu113

Base PyTorch commit: c2808571bf823a26e97231d83f5f3f2c5e3d3fe3

Affected PyTorch version: 1.13.0.dev20220814+cu113

Affected PyTorch commit: 408fa38f33d10e02fe762ef9d444ec9fc4f41ffc

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: -38.78630%
- test_train[timm_vision_transformer-cpu-eager]: +10.75725%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2856543660
Empty result, closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220815+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220814+cu113

Base PyTorch commit: 408fa38f33d10e02fe762ef9d444ec9fc4f41ffc

Affected PyTorch version: 1.13.0.dev20220815+cu113

Affected PyTorch commit: fbe8c77427ea95eb76876f20239e9493f2a1d6ec

Affected Tests:
- test_eval[mobilenet_v3_large-cpu-jit]: +11.76872%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2863669590
Empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220816+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220815+cu113

Base PyTorch commit: fbe8c77427ea95eb76876f20239e9493f2a1d6ec

Affected PyTorch version: 1.13.0.dev20220816+cu113

Affected PyTorch commit: 641d75d0ba0053816a73a6c977ac4a2d6e00e896

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: +19.87168%
- test_eval[pytorch_stargan-cpu-eager]: -8.96696%
- test_train[timm_vision_transformer-cpu-eager]: -9.45065%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2869810474
Empty result, closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220816+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220815+cu113

Base PyTorch commit: fbe8c77427ea95eb76876f20239e9493f2a1d6ec

Affected PyTorch version: 1.13.0.dev20220816+cu113

Affected PyTorch commit: 641d75d0ba0053816a73a6c977ac4a2d6e00e896

Affected Tests:
- test_eval[mobilenet_v3_large-cpu-jit]: -10.00587%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2871273956
Empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220817+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220816+cu113

Base PyTorch commit: 641d75d0ba0053816a73a6c977ac4a2d6e00e896

Affected PyTorch version: 1.13.0.dev20220817+cu113

Affected PyTorch commit: 343b5f86512f75f8f3bd4b90749c0459743b9e72

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: -33.30442%
- test_eval[pytorch_stargan-cpu-eager]: +8.85922%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2936651655

Empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220825+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220824+cu113

Base PyTorch commit: 754d7f05b6841e555cea5a4b2c505dd9e0baec1d

Affected PyTorch version: 1.13.0.dev20220825+cu113

Affected PyTorch commit: b21a6ff6397b74c148c12e4fc41ef12b382443e2

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: +26.82674%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2930184520
Empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220826+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220825+cu113

Base PyTorch commit: b21a6ff6397b74c148c12e4fc41ef12b382443e2

Affected PyTorch version: 1.13.0.dev20220826+cu113

Affected PyTorch commit: 15b560a5c4d638c82e738f3496e2faf95fc328a5

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: +7.72762%
- test_eval[resnext50_32x4d-cpu-jit]: +9.49977%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2935371010
Empty result, closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220826+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220824+cu113

Base PyTorch commit: 754d7f05b6841e555cea5a4b2c505dd9e0baec1d

Affected PyTorch version: 1.13.0.dev20220826+cu113

Affected PyTorch commit: 15b560a5c4d638c82e738f3496e2faf95fc328a5

Affected Tests:
- test_eval[nvidia_deeprecommender-cpu-eager]: +15.58932%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2936696200
Empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220827+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220826+cu113

Base PyTorch commit: 15b560a5c4d638c82e738f3496e2faf95fc328a5

Affected PyTorch version: 1.13.0.dev20220827+cu113

Affected PyTorch commit: e33897cb9999f124bce126c7e43f96c0755413ef

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: -7.48774%
- test_eval[resnext50_32x4d-cpu-jit]: -8.45477%
- test_train[timm_vision_transformer-cpu-eager]: +12.47377%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2940015223
Empty result, closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220827+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220826+cu113

Base PyTorch commit: 15b560a5c4d638c82e738f3496e2faf95fc328a5

Affected PyTorch version: 1.13.0.dev20220827+cu113

Affected PyTorch commit: e33897cb9999f124bce126c7e43f96c0755413ef

Affected Tests:
- test_eval[nvidia_deeprecommender-cpu-eager]: -13.18814%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2940730861
Empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220828+cu113'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220827+cu113

Base PyTorch commit: e33897cb9999f124bce126c7e43f96c0755413ef

Affected PyTorch version: 1.13.0.dev20220828+cu113

Affected PyTorch commit: c9b144ff47ff3b6f358752976d29ac61f2b9b070

Affected Tests:
- test_train[timm_resnest-cpu-eager]: +8.10243%
- test_train[timm_vision_transformer-cpu-eager]: -9.18065%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2943851164

result:
```
{
  "start": "e33897cb9999f124bce126c7e43f96c0755413ef",
  "end": "c9b144ff47ff3b6f358752976d29ac61f2b9b070",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "6446da17305",
      "commit1_time": "2022-08-27 04:40:41 +0000",
      "commit1_digest": {
        "test_train[timm_vision_transformer-cpu-eager]": 0.21746781094843753
      },
      "commit2": "7e7694b6615",
      "commit2_time": "2022-08-27 09:05:20 +0000",
      "commit2_digest": {
        "test_train[timm_vision_transformer-cpu-eager]": 0.19957570464757737
      }
    }
  ]
}
```

The root cause pr is not cpu related. Noise. Closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220829+cu113'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220828+cu113

Base PyTorch commit: c9b144ff47ff3b6f358752976d29ac61f2b9b070

Affected PyTorch version: 1.13.0.dev20220829+cu113

Affected PyTorch commit: 533203f5aaa9f8987f25d828e1c37e755a2ba4ea

Affected Tests:
- test_eval[resnet18-cpu-jit]: +26.66543%
- test_train[timm_resnest-cpu-eager]: -8.56157%
- test_train[timm_vision_transformer-cpu-eager]: +7.42556%
- test_eval[tts_angular-cuda-eager]: +10.69778%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2950361403
Result:
```
{
  "start": "c9b144ff47ff3b6f358752976d29ac61f2b9b070",
  "end": "533203f5aaa9f8987f25d828e1c37e755a2ba4ea",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "c9b144ff47f",
      "commit1_time": "2022-08-28 01:25:07 +0000",
      "commit1_digest": {
        "test_eval[resnet18-cpu-jit]": 0.014202548859519621,
        "test_train[timm_resnest-cpu-eager]": 1.099269716248091,
        "test_train[timm_vision_transformer-cpu-eager]": 0.21436139394936618,
        "test_eval[tts_angular-cuda-eager]": 0.08058981244685129
      },
      "commit2": "b078d242c48",
      "commit2_time": "2022-08-28 04:26:36 +0000",
      "commit2_digest": {
        "test_train[timm_resnest-cpu-eager]": 0.9874842420016648
      }
    }
  ]
}
```

Looks like noise, closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220829+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220828+cu113

Base PyTorch commit: c9b144ff47ff3b6f358752976d29ac61f2b9b070

Affected PyTorch version: 1.13.0.dev20220829+cu113

Affected PyTorch commit: 533203f5aaa9f8987f25d828e1c37e755a2ba4ea

Affected Tests:
- test_eval[mobilenet_v3_large-cpu-jit]: +10.14276%
- test_eval[squeezenet1_1-cpu-jit]: +7.03461%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2951899282

Empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220831+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220830+cu113

Base PyTorch commit: df98c529480b2ece3809b19fc850f57d2054605a

Affected PyTorch version: 1.13.0.dev20220831+cu113

Affected PyTorch commit: 71d99662a0d7f8a9ad68999c9a014b71591cbb68

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: +9.48090%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2965718876
Empty result, closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220831+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220830+cu113

Base PyTorch commit: df98c529480b2ece3809b19fc850f57d2054605a

Affected PyTorch version: 1.13.0.dev20220831+cu113

Affected PyTorch commit: 71d99662a0d7f8a9ad68999c9a014b71591cbb68

Affected Tests:
- test_eval[mobilenet_v3_large-cpu-jit]: -8.79072%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2967201841
Empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220831+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220830+cu113

Base PyTorch commit: df98c529480b2ece3809b19fc850f57d2054605a

Affected PyTorch version: 1.13.0.dev20220831+cu113

Affected PyTorch commit: 71d99662a0d7f8a9ad68999c9a014b71591cbb68

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: +9.06650%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2973561826
Empty result, closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220831+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220830+cu113

Base PyTorch commit: df98c529480b2ece3809b19fc850f57d2054605a

Affected PyTorch version: 1.13.0.dev20220831+cu113

Affected PyTorch commit: 71d99662a0d7f8a9ad68999c9a014b71591cbb68

Affected Tests:
- test_eval[mobilenet_v3_large-cpu-jit]: -7.07057%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2967201841
Empty result, closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220902+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220831+cu113

Base PyTorch commit: 71d99662a0d7f8a9ad68999c9a014b71591cbb68

Affected PyTorch version: 1.13.0.dev20220902+cu113

Affected PyTorch commit: 97b2dff60081e1092cfd6d1b3a80c995ff3d6148

Affected Tests:
- test_train[hf_Bert-cpu-eager]: +9.53001%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2981969161
Empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220903+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220902+cu113

Base PyTorch commit: 97b2dff60081e1092cfd6d1b3a80c995ff3d6148

Affected PyTorch version: 1.13.0.dev20220903+cu113

Affected PyTorch commit: 2a332afbf41b68080a9436e910b93af7cd336fbc

Affected Tests:
- test_eval[resnet18-cpu-jit]: +27.26439%
- test_eval[resnext50_32x4d-cpu-jit]: +10.96092%
- test_train[timm_vision_transformer-cpu-eager]: -11.04257%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2985159359
Result

```
{
  "start": "97b2dff60081e1092cfd6d1b3a80c995ff3d6148",
  "end": "2a332afbf41b68080a9436e910b93af7cd336fbc",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "f0e5b733641",
      "commit1_time": "2022-09-02 17:42:58 +0000",
      "commit1_digest": {
        "test_train[timm_vision_transformer-cpu-eager]": 0.19480935494648294
      },
      "commit2": "a5a01e443ce",
      "commit2_time": "2022-09-02 17:42:58 +0000",
      "commit2_digest": {
        "test_train[timm_vision_transformer-cpu-eager]": 0.22091167060425504
      }
    }
  ]
}
```

Improvement, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220904+cu113'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220903+cu113

Base PyTorch commit: 2a332afbf41b68080a9436e910b93af7cd336fbc

Affected PyTorch version: 1.13.0.dev20220904+cu113

Affected PyTorch commit: e46c1c7931da2d723a6cad4ec307ff4ed4e9cb7f

Affected Tests:
- test_eval[resnet18-cpu-jit]: -20.86129%
- test_eval[resnext50_32x4d-cpu-jit]: -9.18207%
- test_train[timm_vision_transformer-cpu-eager]: +11.19265%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2988894812
Empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220905+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220904+cu113

Base PyTorch commit: e46c1c7931da2d723a6cad4ec307ff4ed4e9cb7f

Affected PyTorch version: 1.13.0.dev20220905+cu113

Affected PyTorch commit: 9a6aa9053f79127721875e371addd9c3baeaaac0

Affected Tests:
- test_train[timm_regnet-cpu-eager]: -7.86523%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2995027309
Empty result, closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220905+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220904+cu113

Base PyTorch commit: e46c1c7931da2d723a6cad4ec307ff4ed4e9cb7f

Affected PyTorch version: 1.13.0.dev20220905+cu113

Affected PyTorch commit: 9a6aa9053f79127721875e371addd9c3baeaaac0

Affected Tests:
- test_train[hf_Bert-cpu-eager]: +10.43813%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2996128915
Empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220905+cu113'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220904+cu113

Base PyTorch commit: e46c1c7931da2d723a6cad4ec307ff4ed4e9cb7f

Affected PyTorch version: 1.13.0.dev20220905+cu113

Affected PyTorch commit: 9a6aa9053f79127721875e371addd9c3baeaaac0

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: +8.32191%
- test_train[timm_vision_transformer-cpu-eager]: -12.70812%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/3002073443
Empty result, closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220905+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220904+cu113

Base PyTorch commit: e46c1c7931da2d723a6cad4ec307ff4ed4e9cb7f

Affected PyTorch version: 1.13.0.dev20220905+cu113

Affected PyTorch commit: 9a6aa9053f79127721875e371addd9c3baeaaac0

Affected Tests:
- test_train[hf_Bert-cpu-eager]: +7.51248%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/3003514578
Empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220907+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220905+cu113

Base PyTorch commit: 9a6aa9053f79127721875e371addd9c3baeaaac0

Affected PyTorch version: 1.13.0.dev20220907+cu113

Affected PyTorch commit: 06ebe2d5bc1055f226f56ed2fe26a29038a466e5

Affected Tests:
- test_eval[resnet50_quantized_qat-cpu-eager]: -7.39373%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/3009492175
Empty result, closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220907+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220905+cu113

Base PyTorch commit: 9a6aa9053f79127721875e371addd9c3baeaaac0

Affected PyTorch version: 1.13.0.dev20220907+cu113

Affected PyTorch commit: 06ebe2d5bc1055f226f56ed2fe26a29038a466e5

Affected Tests:
- test_eval[pytorch_struct-cpu-jit]: -7.13831%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/3010862769
Empty result, closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220910+cu116'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220907+cu113

Base PyTorch commit: 06ebe2d5bc1055f226f56ed2fe26a29038a466e5

Affected PyTorch version: 1.13.0.dev20220910+cu116

Affected PyTorch commit: 96e4bd950027a2f472fafa98616c92403a890bd2

Affected Tests:
- test_train[hf_Bert-cpu-eager]: +7.91619%
- test_eval[nvidia_deeprecommender-cuda-eager]: -8.04810%
- test_train[speech_transformer-cuda-eager]: -13.30991%


cc @xuzhao9
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220914+cu116'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220913+cu116

Base PyTorch commit: 2fbc0fab20d4af520f69f158f8777e99ad761e1d

Affected PyTorch version: 1.13.0.dev20220914+cu116

Affected PyTorch commit: 55ca297d4e048c641d149a76f2fda7c9ce630ff6

Affected Tests:
- test_eval[nvidia_deeprecommender-cpu-eager]: +21.48532%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/3056832589
Closed. Empty result.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220915+cu116'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220914+cu116

Base PyTorch commit: 55ca297d4e048c641d149a76f2fda7c9ce630ff6

Affected PyTorch version: 1.13.0.dev20220915+cu116

Affected PyTorch commit: 625e44c1df211d6753609a9b391cb10f2f94367f

Affected Tests:
- test_eval[nvidia_deeprecommender-cpu-eager]: -17.88537%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/3063848614
Empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220916+cu116'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220915+cu116

Base PyTorch commit: 625e44c1df211d6753609a9b391cb10f2f94367f

Affected PyTorch version: 1.13.0.dev20220916+cu116

Affected PyTorch commit: 0ec19db7ac88e307135100ddcfc418ae3925844f

Affected Tests:
- test_eval[demucs-cpu-jit]: -7.33554%
- test_eval[demucs-cpu-eager]: -8.06421%
- test_eval[mnasnet1_0-cpu-jit]: -23.19689%
- test_eval[pytorch_stargan-cpu-jit]: -11.37964%
- test_train[pytorch_stargan-cpu-eager]: -7.00331%
- test_eval[pytorch_stargan-cpu-eager]: -14.45801%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/3069546784
Empty result, closed.
Repository: pytorch/benchmark
Issue: Add FSDP to core model set
Root cause: Unimplemented Feature Gap

The following command should work:

```python run_benchmark.py distributed --ngpus 8 --nodes 1 --model torchbenchmark.models.hf_T5.Model --trainer torchbenchmark.util.distributed.core_model.trainer.Trainer --distributed fsdp --job_dir $PWD/.userbenchmark/distributed```

Fixed by https://github.com/pytorch/benchmark/pull/1190