zhiminy/EvalEng huggingface dataset:
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.12.0.dev20220309+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20220203+cu113

Base PyTorch commit: 58dabebcd746aad95a37bdfc7e60e5d22f0f5641

Affected PyTorch version: 1.12.0.dev20220309+cu113

Affected PyTorch commit: beda4e8b2f644434202c3d27eac21b5bf9b82321

Affected Tests:
- test_eval[dlrm-cuda-eager]: +14.90477%
- test_eval[soft_actor_critic-cuda-eager]: -7.81608%
- test_eval[timm_vovnet-cpu-jit]: -7.46800%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/runs/5560253747?check_suite_focus=true

New workflow: https://github.com/pytorch/benchmark/runs/5576515589?check_suite_focus=true

- test_eval[dlrm-cuda-eager] slowdown: commit [aa5dab02b2](https://github.com/pytorch/pytorch/commit/aa5dab02b2)
   Verify PR: https://github.com/pytorch/pytorch/pull/74722 (Signal)
- test_eval[timm_vovnet-cpu-jit] speedup: commit [727debb18e](https://github.com/pytorch/pytorch/commit/727debb18e)
  Verify PR: https://github.com/pytorch/pytorch/pull/74723 (Closed as noise signal)

Escalated to PyTorch issue: https://github.com/pytorch/pytorch/issues/74751
Proposed fix: https://github.com/pytorch/pytorch/pull/74767

Fix landed. Closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.12.0.dev20220311+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220310+cu113

Base PyTorch commit: d8dbf9d8b2d73c4799555af04d8fdf9af9d027a3

Affected PyTorch version: 1.12.0.dev20220311+cu113

Affected PyTorch commit: 794f813522deb8afb5e576cec264ce9224cfe067

Affected Tests:
- test_eval[shufflenet_v2_x1_0-cpu-eager]: -7.03285%


cc @xuzhao9


bisection workflow id: bisection-gh1970512558

Workflow: https://github.com/pytorch/benchmark/runs/5587530503?check_suite_focus=true

Empty result. Closed.
Repository: pytorch/benchmark
Issue: CPU implementation for pytorch_CycleGAN_and_pix2pix
Root cause: Unimplemented Feature Gap

Add CPU implementation for `pytorch_CycleGAN_and_pix2pix` to help with accuracy testing. 

https://github.com/pytorch/benchmark/blob/main/torchbenchmark/models/pytorch_CycleGAN_and_pix2pix/metadata.yaml
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.12.0.dev20220311+cu113'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220310+cu113

Base PyTorch commit: d8dbf9d8b2d73c4799555af04d8fdf9af9d027a3

Affected PyTorch version: 1.12.0.dev20220311+cu113

Affected PyTorch commit: 794f813522deb8afb5e576cec264ce9224cfe067

Affected Tests:
- test_eval[nvidia_deeprecommender-cuda-eager]: +8.48147%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/runs/5517933976?check_suite_focus=true

New workflow: https://github.com/pytorch/benchmark/actions/runs/1988358876

Affecting commit: https://github.com/pytorch/pytorch/commit/00b01ec056. This is a known problem caused by cublasLt.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.12.0.dev20220314+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220308+cu113

Base PyTorch commit: 07410207c487369d87a03e4602f72ec870abe4b7

Affected PyTorch version: 1.12.0.dev20220314+cu113

Affected PyTorch commit: ef9023e93dabdcfe328bb55869ded08096e2d993

Affected Tests:
- test_train[timm_nfnet-cpu-eager]: -8.68816%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/1984095209
Empty result. Close.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.12.0.dev20220315+cu113'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20220202+cu113

Base PyTorch commit: f20fa66f704a61b4570ce372add59c7469f44a96

Affected PyTorch version: 1.12.0.dev20220315+cu113

Affected PyTorch commit: 060f1b822a6b71e1db07b94f68fdfb53fe47a7d0

Affected Tests:
- test_eval[nvidia_deeprecommender-cpu-eager]: +8.07126%
- test_eval[squeezenet1_1-cpu-jit]: -10.41098%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/1989391432

test_eval[squeezenet1_1-cpu-jit] speedup commit: [452c26bbeb](https://github.com/pytorch/pytorch/commit/452c26bbeb)
test_eval[nvidia_deeprecommender-cpu-eager]: noise

The commit only removes warnings and this should be a noise signal, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.12.0.dev20220318+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220314+cu113

Base PyTorch commit: ef9023e93dabdcfe328bb55869ded08096e2d993

Affected PyTorch version: 1.12.0.dev20220318+cu113

Affected PyTorch commit: 6ffe16662f1f4792bf4dc2c2ec35f21b6ff1cc7e

Affected Tests:
- test_train[timm_nfnet-cpu-eager]: +9.54442%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/runs/5604247168?check_suite_focus=true

Result is empty. Close and mark as a noise.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.12.0.dev20220321+cu113'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220320+cu113

Base PyTorch commit: 38512d90c1380aa7eb66bdfbc52c744bc06b17c6

Affected PyTorch version: 1.12.0.dev20220321+cu113

Affected PyTorch commit: 7c2103ad5ffdc1ef91231c966988f7f2a61b4166

Affected Tests:
- test_eval[shufflenet_v2_x1_0-cpu-eager]: -10.77144%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2017645335. Closed because of empty result (noise).
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.12.0.dev20220321+cu113'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220315+cu113

Base PyTorch commit: 060f1b822a6b71e1db07b94f68fdfb53fe47a7d0

Affected PyTorch version: 1.12.0.dev20220321+cu113

Affected PyTorch commit: 7c2103ad5ffdc1ef91231c966988f7f2a61b4166

Affected Tests:
- test_train[drq-cuda-eager]: +11.68477%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/runs/5636149055?check_suite_focus=true

Empty result. Closed as noise.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.12.0.dev20220322+cu113'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220320+cu113

Base PyTorch commit: 38512d90c1380aa7eb66bdfbc52c744bc06b17c6

Affected PyTorch version: 1.12.0.dev20220322+cu113

Affected PyTorch commit: 7c2103ad5ffdc1ef91231c966988f7f2a61b4166

Affected Tests:
- test_train[drq-cuda-eager]: +11.67764%
- test_eval[mnasnet1_0-cpu-jit]: +32.01403%
- test_train[timm_nfnet-cpu-eager]: -10.85303%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2023830626

Empty result. Noise.

Reopen, the drq regression is worth a closer look

Can't reproduce the problem locally. Closing this issue.
Repository: pytorch/benchmark
Issue: HF errors from network calls
Root cause: Architectural Constraint

I got this error running TorchBench:
```
500 Server Error: Internal Server Error for url: https://huggingface.co/facebook/bart-base/resolve/main/config.json
Traceback (most recent call last):
  File "/home/jansel/conda/envs/torchdynamo/lib/python3.8/site-packages/transformers/configuration_utils.py", line 457, in get_config_dict
    resolved_config_file = cached_path(
  File "/home/jansel/conda/envs/torchdynamo/lib/python3.8/site-packages/transformers/file_utils.py", line 1165, in cached_path
    output_path = get_from_cache(
  File "/home/jansel/conda/envs/torchdynamo/lib/python3.8/site-packages/transformers/file_utils.py", line 1336, in get_from_cache
    r.raise_for_status()
  File "/home/jansel/conda/envs/torchdynamo/lib/python3.8/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://huggingface.co/facebook/bart-base/resolve/main/config.json

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "torchbench.py", line 1011, in <module>
    main()
  File "torchbench.py", line 902, in main
    for device, name, model, example_inputs in iter_models(args):
  File "torchbench.py", line 112, in iter_models
    yield load_model(device, model_name, args.training, args.check_accuracy)
  File "torchbench.py", line 140, in load_model
    benchmark = benchmark_cls(test="eval", device=device, jit=False)
  File "/home/jansel/torchbenchmark/torchbenchmark/util/model.py", line 16, in __call__
    obj = type.__call__(cls, *args, **kwargs)
  File "/home/jansel/torchbenchmark/torchbenchmark/models/hf_Bart/__init__.py", line 25, in __init__
    config = AutoConfig.from_pretrained("facebook/bart-base")
  File "/home/jansel/conda/envs/torchdynamo/lib/python3.8/site-packages/transformers/models/auto/configuration_auto.py", line 398, in from_pretrained
    config_dict, _ = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/home/jansel/conda/envs/torchdynamo/lib/python3.8/site-packages/transformers/configuration_utils.py", line 477, in get_config_dict
    raise EnvironmentError(msg)
OSError: Can't load config for 'facebook/bart-base'. Make sure that:

- 'facebook/bart-base' is a correct model identifier listed on 'https://huggingface.co/models'

- or 'facebook/bart-base' is the correct path to a directory containing a config.json file
```

This run was from an existing warmed-up checkout, so I would expect the model to already be downloaded.

Is there a way to make the network calls happen in install.py rather than every time I run the model?
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.12.0.dev20220324+cu113'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220322+cu113

Base PyTorch commit: 7c2103ad5ffdc1ef91231c966988f7f2a61b4166

Affected PyTorch version: 1.12.0.dev20220324+cu113

Affected PyTorch commit: d583f9c9d2bd0cc5fb5d369cd498bd62b62c43fe

Affected Tests:
- test_train[Super_SloMo-cuda-jit]: -16.93950%
- test_eval[Super_SloMo-cuda-jit]: +12.79589%
- test_train[Super_SloMo-cuda-eager]: -16.86279%
- test_eval[Super_SloMo-cuda-eager]: +12.64786%
- test_eval[alexnet-cuda-jit]: +9.62829%
- test_eval[alexnet-cuda-eager]: +7.18450%
- test_train[dcgan-cuda-jit]: -15.63837%
- test_train[dcgan-cuda-eager]: -15.70387%
- test_train[maml-cuda-eager]: -12.93909%
- test_eval[maml-cuda-eager]: -14.68961%
- test_eval[mnasnet1_0-cpu-jit]: -24.29900%
- test_eval[mnasnet1_0-cuda-jit]: -92.04965%
- test_eval[mobilenet_v2-cuda-jit]: -7.18633%
- test_eval[mobilenet_v3_large-cuda-jit]: -83.60196%
- test_eval[opacus_cifar10-cuda-eager]: -27.23624%
- test_train[pytorch_unet-cuda-jit]: -14.07633%
- test_train[pytorch_unet-cuda-eager]: -14.08312%
- test_train[resnet18-cuda-jit]: -9.28816%
- test_train[resnet18-cuda-eager]: -9.27021%
- test_eval[resnet18-cuda-eager]: -15.39782%
- test_train[resnet50-cuda-jit]: -16.94554%
- test_train[resnet50-cuda-eager]: -16.91977%
- test_train[resnet50_quantized_qat-cuda-eager]: -12.06298%
- test_eval[resnext50_32x4d-cuda-jit]: -22.95051%
- test_eval[resnext50_32x4d-cuda-eager]: -15.09114%
- test_eval[shufflenet_v2_x1_0-cuda-jit]: +41.48391%
- test_eval[shufflenet_v2_x1_0-cuda-eager]: -8.30074%
- test_eval[timm_efficientnet-cuda-jit]: -61.07355%
- test_train[timm_nfnet-cpu-eager]: +8.05564%
- test_train[timm_regnet-cuda-jit]: +41.94406%
- test_eval[timm_regnet-cuda-jit]: +185.41225%
- test_train[timm_regnet-cuda-eager]: +39.05237%
- test_eval[timm_regnet-cuda-eager]: +86.40320%
- test_train[timm_resnest-cuda-jit]: -16.07832%
- test_train[timm_resnest-cuda-eager]: -16.01351%
- test_train[tts_angular-cuda-eager]: -7.27285%
- test_eval[tts_angular-cuda-eager]: -8.88685%
- test_train[vision_maskrcnn-cuda-eager]: -14.04072%
- test_train[yolov3-cuda-eager]: -19.14568%
- test_eval[yolov3-cuda-eager]: -7.32508%


cc @xuzhao9


Root cause: CUDNN version upgrade: https://github.com/pytorch/builder/pull/1000 . Closing this issue.

@xuzhao9 are you using conda or pip wheels? Also, does negative percentage means tests are running slower or faster?

@malfet I am using pip wheels (https://github.com/pytorch/benchmark/blob/main/.github/workflows/v2-nightly.yml#L50). Negative means the tests are running faster, and positive means slower.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.12.0.dev20220325+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220324+cu113

Base PyTorch commit: d583f9c9d2bd0cc5fb5d369cd498bd62b62c43fe

Affected PyTorch version: 1.12.0.dev20220325+cu113

Affected PyTorch commit: 4d82e5bf449929c2732250f8620976e5be96a7f3

Affected Tests:
- test_eval[pyhpc_turbulent_kinetic_energy-cpu-jit]: +9.82353%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2041261628

Empty result. Close as noise.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.12.0.dev20220326+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220325+cu113

Base PyTorch commit: 4d82e5bf449929c2732250f8620976e5be96a7f3

Affected PyTorch version: 1.12.0.dev20220326+cu113

Affected PyTorch commit: 90459ba9dc6a812ef74c814d8fd1fa1edaaa27ee

Affected Tests:
- test_eval[demucs-cpu-jit]: +13.11172%


cc @xuzhao9


Empty result. Closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.12.0.dev20220327+cu113'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220326+cu113

Base PyTorch commit: 90459ba9dc6a812ef74c814d8fd1fa1edaaa27ee

Affected PyTorch version: 1.12.0.dev20220327+cu113

Affected PyTorch commit: 1c5a8125798392f8d7c57e88735f43a14ae0beca

Affected Tests:
- test_eval[demucs-cpu-jit]: -8.81259%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2048434344

Bisection returns empty result. Closed as noise.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.12.0.dev20220328+cu113'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220323+cu113

Base PyTorch commit: 65329f4fac8fb22318b7a3eb115e9da207d8d41a

Affected PyTorch version: 1.12.0.dev20220328+cu113

Affected PyTorch commit: 1c5a8125798392f8d7c57e88735f43a14ae0beca

Affected Tests:
- test_train[Super_SloMo-cuda-jit]: -16.92249%
- test_eval[Super_SloMo-cuda-jit]: +12.22803%
- test_train[Super_SloMo-cuda-eager]: -16.85775%
- test_eval[Super_SloMo-cuda-eager]: +12.09915%
- test_eval[alexnet-cuda-jit]: -14.82252%
- test_eval[alexnet-cuda-eager]: -14.13340%
- test_train[densenet121-cuda-jit]: -7.82857%
- test_train[densenet121-cuda-eager]: -7.64626%
- test_train[maml-cuda-eager]: -12.12154%
- test_eval[maml-cuda-eager]: -13.42175%
- test_eval[maml_omniglot-cuda-eager]: -7.97607%
- test_eval[mnasnet1_0-cuda-jit]: -95.44156%
- test_train[mobilenet_v2-cuda-jit]: -10.52894%
- test_eval[mobilenet_v2-cuda-jit]: -14.17461%
- test_train[mobilenet_v2-cuda-eager]: -10.54444%
- test_eval[mobilenet_v2-cuda-eager]: -10.08228%
- test_eval[mobilenet_v3_large-cuda-jit]: -23.09441%
- test_train[moco-cuda-jit]: -9.83372%
- test_train[moco-cuda-eager]: -9.84092%
- test_eval[opacus_cifar10-cuda-eager]: -27.95975%
- test_eval[pytorch_stargan-cuda-jit]: -30.42362%
- test_eval[pytorch_stargan-cuda-eager]: -30.45168%
- test_train[resnet18-cuda-jit]: -17.28054%
- test_train[resnet18-cuda-eager]: -17.31519%
- test_train[resnet50-cuda-jit]: -16.86644%
- test_eval[resnet50-cuda-jit]: -20.43296%
- test_train[resnet50-cuda-eager]: -16.85397%
- test_eval[resnet50-cuda-eager]: -8.91445%
- test_train[resnet50_quantized_qat-cuda-eager]: -11.99613%
- test_train[resnext50_32x4d-cuda-jit]: -18.83313%
- test_train[resnext50_32x4d-cuda-eager]: -18.96089%
- test_eval[resnext50_32x4d-cuda-eager]: -12.01397%
- test_eval[shufflenet_v2_x1_0-cuda-jit]: +8.22455%
- test_train[speech_transformer-cuda-eager]: -8.09863%
- test_train[squeezenet1_1-cuda-jit]: -10.53504%
- test_eval[squeezenet1_1-cuda-jit]: -9.60593%
- test_train[squeezenet1_1-cuda-eager]: -10.52895%
- test_eval[squeezenet1_1-cuda-eager]: -10.20297%
- test_train[timm_resnest-cuda-jit]: -35.27120%
- test_eval[timm_resnest-cuda-jit]: -8.42915%
- test_train[timm_resnest-cuda-eager]: -35.60262%
- test_eval[timm_resnest-cuda-eager]: -10.86361%
- test_train[vgg16-cuda-jit]: -9.51113%
- test_train[vgg16-cuda-eager]: -9.53890%


cc @xuzhao9


Speedup caused by cudnn version upgrade.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.12.0.dev20220329+cu113'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220327+cu113

Base PyTorch commit: 1c5a8125798392f8d7c57e88735f43a14ae0beca

Affected PyTorch version: 1.12.0.dev20220329+cu113

Affected PyTorch commit: 51e50a2ddb9844e0abaf6028fe5e4246448dffff

Affected Tests:
- test_eval[pytorch_stargan-cpu-jit]: +7.31959%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2060033275

Result: facdbe6d72 shows speedup, not regression, looks like this is noise. Closed.
Repository: pytorch/benchmark
Issue: timm_efficientdet prints out spam while loading
Root cause: External Dependency Breakage

```
./torchbench.py -k timm_efficientdet --nothing
loading annotations into memory...
Done (t=0.01s)
creating index...
index created!
loading annotations into memory...
Done (t=0.01s)
creating index...
index created!
```

Loading a module should not print.

These are printed by a dependency of the model, pycocotools: https://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocotools/coco.py#L82, we can disable these printing statements by patching the package.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.12.0.dev20220331+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220330+cu113

Base PyTorch commit: 3269729c683ac01d7e91b109187a69686167bb08

Affected PyTorch version: 1.12.0.dev20220331+cu113

Affected PyTorch commit: 3036a0309dba11d0601d19b5f2d2f4a7e6afa456

Affected Tests:
- test_eval[dlrm-cuda-eager]: -10.86805%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/runs/5777664241?check_suite_focus=true

New workflow: https://github.com/pytorch/benchmark/actions/runs/2074448277

Root cause commit: [273c2f01241](https://github.com/pytorch/pytorch/commit/273c2f01241)
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.12.0.dev20220401+cu113'
Root cause: Environment Incompatibility


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220329+cu113

Base PyTorch commit: 51e50a2ddb9844e0abaf6028fe5e4246448dffff

Affected PyTorch version: 1.12.0.dev20220401+cu113

Affected PyTorch commit: c0a6add7eef8ab3715c0f4281b3b7a2a0f1e24b5

Affected Tests:
- test_train[Super_SloMo-cuda-jit]: +20.27914%
- test_eval[Super_SloMo-cuda-jit]: -10.91610%
- test_train[Super_SloMo-cuda-eager]: +20.16960%
- test_eval[Super_SloMo-cuda-eager]: -10.81445%
- test_eval[alexnet-cuda-jit]: +16.35704%
- test_eval[alexnet-cuda-eager]: +15.10636%
- test_train[densenet121-cuda-jit]: +8.53451%
- test_train[densenet121-cuda-eager]: +8.48672%
- test_train[dlrm-cpu-eager]: +10.67012%
- test_train[dlrm-cuda-eager]: +7.36037%
- test_train[maml-cuda-eager]: +15.64464%
- test_eval[maml-cuda-eager]: +16.14958%
- test_eval[mnasnet1_0-cuda-jit]: +1744.01834%
- test_train[mobilenet_v2-cuda-jit]: +11.83316%
- test_eval[mobilenet_v2-cuda-jit]: +13.51792%
- test_train[mobilenet_v2-cuda-eager]: +11.87469%
- test_eval[mobilenet_v2-cuda-eager]: +9.73801%
- test_eval[mobilenet_v3_large-cuda-jit]: +27.55735%
- test_train[moco-cuda-jit]: +10.91887%
- test_train[moco-cuda-eager]: +10.92378%
- test_eval[opacus_cifar10-cuda-eager]: +38.42694%
- test_eval[pytorch_stargan-cuda-jit]: +43.60693%
- test_eval[pytorch_stargan-cuda-eager]: +43.67630%
- test_train[resnet18-cuda-jit]: +20.78688%
- test_train[resnet18-cuda-eager]: +20.89248%
- test_train[resnet50-cuda-jit]: +20.28754%
- test_eval[resnet50-cuda-jit]: +24.83848%
- test_train[resnet50-cuda-eager]: +20.17807%
- test_eval[resnet50-cuda-eager]: +11.55894%
- test_train[resnet50_quantized_qat-cuda-eager]: +13.64049%
- test_train[resnext50_32x4d-cuda-jit]: +23.20692%
- test_train[resnext50_32x4d-cuda-eager]: +22.97922%
- test_eval[resnext50_32x4d-cuda-eager]: +9.84660%
- test_eval[shufflenet_v2_x1_0-cuda-jit]: -7.15736%
- test_train[squeezenet1_1-cuda-jit]: +11.90014%
- test_eval[squeezenet1_1-cuda-jit]: +7.94940%
- test_train[squeezenet1_1-cuda-eager]: +12.07959%
- test_eval[squeezenet1_1-cuda-eager]: +10.23720%
- test_train[timm_resnest-cuda-jit]: +54.80388%
- test_eval[timm_resnest-cuda-jit]: +8.23392%
- test_train[timm_resnest-cuda-eager]: +52.33217%
- test_eval[timm_resnest-cuda-eager]: +9.97490%
- test_train[vgg16-cuda-jit]: +10.51193%
- test_train[vgg16-cuda-eager]: +10.59504%
- test_train[yolov3-cuda-eager]: +22.84987%


cc @xuzhao9


This is due to new instance was using an old AMI image (hence old cudnn verison), closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.12.0.dev20220405+cu113'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220403+cu113

Base PyTorch commit: bd032cd8d65e36b9d16af42520f5d63e861cb9d4

Affected PyTorch version: 1.12.0.dev20220405+cu113

Affected PyTorch commit: b0e047b59da4d6434bf347dd6ae501169218bb1a

Affected Tests:
- test_train[Super_SloMo-cuda-jit]: -16.88734%
- test_eval[Super_SloMo-cuda-jit]: +12.23077%
- test_train[Super_SloMo-cuda-eager]: -16.81555%
- test_eval[Super_SloMo-cuda-eager]: +12.08973%
- test_eval[alexnet-cuda-jit]: -13.94041%
- test_eval[alexnet-cuda-eager]: -13.07031%
- test_train[densenet121-cuda-jit]: -8.04015%
- test_train[densenet121-cuda-eager]: -7.95190%
- test_eval[densenet121-cuda-eager]: -7.07512%
- test_train[maml-cuda-eager]: -12.91613%
- test_eval[maml-cuda-eager]: -13.65270%
- test_eval[mnasnet1_0-cuda-jit]: -94.54541%
- test_train[mobilenet_v2-cuda-jit]: -10.56092%
- test_eval[mobilenet_v2-cuda-jit]: -12.50417%
- test_train[mobilenet_v2-cuda-eager]: -10.55601%
- test_eval[mobilenet_v2-cuda-eager]: -7.39146%
- test_eval[mobilenet_v3_large-cuda-jit]: -21.50287%
- test_train[moco-cuda-jit]: -9.83134%
- test_train[moco-cuda-eager]: -9.80068%
- test_eval[opacus_cifar10-cuda-eager]: -27.63870%
- test_eval[pytorch_stargan-cuda-jit]: -30.44466%
- test_eval[pytorch_stargan-cuda-eager]: -30.47270%
- test_train[resnet18-cuda-jit]: -17.19763%
- test_train[resnet18-cuda-eager]: -17.29127%
- test_train[resnet50-cuda-jit]: -16.88958%
- test_eval[resnet50-cuda-jit]: -20.29944%
- test_train[resnet50-cuda-eager]: -16.84506%
- test_eval[resnet50-cuda-eager]: -9.26120%
- test_train[resnet50_quantized_qat-cuda-eager]: -12.04411%
- test_train[resnext50_32x4d-cuda-jit]: -18.92614%
- test_train[resnext50_32x4d-cuda-eager]: -18.85035%
- test_eval[resnext50_32x4d-cuda-eager]: -9.40264%
- test_eval[shufflenet_v2_x1_0-cuda-jit]: +8.64341%
- test_train[squeezenet1_1-cuda-jit]: -10.69087%
- test_eval[squeezenet1_1-cuda-jit]: -8.07538%
- test_train[squeezenet1_1-cuda-eager]: -10.74510%
- test_eval[squeezenet1_1-cuda-eager]: -8.71605%
- test_train[timm_resnest-cuda-jit]: -34.31089%
- test_eval[timm_resnest-cuda-jit]: -7.79768%
- test_train[timm_resnest-cuda-eager]: -34.35969%
- test_eval[timm_resnest-cuda-eager]: -7.83851%
- test_train[vgg16-cuda-jit]: -9.49614%
- test_train[vgg16-cuda-eager]: -9.56612%


cc @xuzhao9


Again this is because of infra issue <cudnn upgrade churn>. Closed.