zhiminy/EvalEng huggingface dataset:
Repository: pytorch/benchmark
Issue: Add DeOldify Model
Root cause: Unimplemented Feature Gap

Follow instructions at https://github.com/pytorch/benchmark/blob/master/torchbenchmark/models/ADDING_MODELS.md

Model source:
https://github.com/bwasti/DeOldify

This model has already been forked and modified as part of an earlier hackathon, so while it needs work to get into the current benchmark format,  the install/data/train steps should already be clear.  Script/tracing has already been attempted without success, so that can be skipped.
Repository: pytorch/benchmark
Issue: Improve the dind dockerfile
Root cause: Unimplemented Feature Gap

We would like to see the following improvements to the dind docker file:

- [x] Use `ENV` to update PATH and LD_LIBRARY_PATH environment variables (https://www.baeldung.com/ops/dockerfile-path-environment-variable)
- [x] Install DCGM package
- [ ] Assign the runner user with SYS_ADMIN capability and is able to run dcgm profiling
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.14.0.dev20221020+cu116'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.14.0.dev20221018+cu116

Base PyTorch commit: 4814270708cb6141c1fb6202f883c084c71290b4

Affected PyTorch version: 1.14.0.dev20221020+cu116

Affected PyTorch commit: 472bdb3aa84678b2faa4afe1cb5757f55e14ed9a

Affected Tests:
- test_train[LearningToPaint-cpu-eager]: +7.56226%
- test_eval[demucs-cpu-eager]: +9.47538%
- test_eval[hf_Longformer-cpu-eager]: -11.60031%
- test_eval[mnasnet1_0-cpu-jit]: +29.56326%
- test_train[opacus_cifar10-cpu-eager]: -22.58740%
- test_train[opacus_cifar10-cuda-eager]: -16.32688%
- test_train[pytorch_stargan-cpu-jit]: +7.58695%
- test_eval[pytorch_stargan-cpu-jit]: +16.37559%
- test_train[pytorch_stargan-cpu-eager]: +8.24575%
- test_eval[pytorch_stargan-cpu-eager]: +20.01417%
- test_train[timm_vision_transformer-cpu-eager]: -18.04775%


cc @xuzhao9


Result:
```
{
  "start": "4814270708cb6141c1fb6202f883c084c71290b4",
  "end": "472bdb3aa84678b2faa4afe1cb5757f55e14ed9a",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "c4a03e4da19",
      "commit1_time": "2022-10-18 22:58:44 +0000",
      "commit1_digest": {
        "test_eval[hf_Longformer-cpu-eager]": 2.356933786101581,
        "test_train[opacus_cifar10-cpu-eager]": 1.2877291915545357,
        "test_train[opacus_cifar10-cuda-eager]": 0.1358206262026215
      },
      "commit2": "80790ecee4f",
      "commit2_time": "2022-10-18 23:01:28 +0000",
      "commit2_digest": {
        "test_eval[hf_Longformer-cpu-eager]": 2.0866590807563625,
        "test_train[opacus_cifar10-cpu-eager]": 0.9853979319435894,
        "test_train[opacus_cifar10-cuda-eager]": 0.11364018479653168
      }
    },
    {
      "commit1": "888e15408e8",
      "commit1_time": "2022-10-20 01:04:42 +0000",
      "commit1_digest": {
        "test_train[timm_vision_transformer-cpu-eager]": 0.22920883405022324
      },
      "commit2": "0d2c2110f17",
      "commit2_time": "2022-10-20 01:17:00 +0000",
      "commit2_digest": {
        "test_train[timm_vision_transformer-cpu-eager]": 0.2078797268026392
      }
    }
  ]
}
```

Noise, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.14.0.dev20221022+cu116'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.14.0.dev20221020+cu116

Base PyTorch commit: 472bdb3aa84678b2faa4afe1cb5757f55e14ed9a

Affected PyTorch version: 1.14.0.dev20221022+cu116

Affected PyTorch commit: 8461460d55c2474b236a5d7198067ed299631b76

Affected Tests:
- test_eval[demucs-cpu-jit]: -7.72292%
- test_eval[demucs-cpu-eager]: -8.06288%
- test_eval[pytorch_stargan-cpu-jit]: -14.83687%
- test_train[pytorch_stargan-cpu-eager]: -8.05975%
- test_eval[pytorch_stargan-cpu-eager]: -14.44575%
- test_train[timm_vision_transformer-cpu-eager]: +22.39146%


cc @xuzhao9


Result:
```
{
  "start": "472bdb3aa84678b2faa4afe1cb5757f55e14ed9a",
  "end": "8461460d55c2474b236a5d7198067ed299631b76",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "472bdb3aa84",
      "commit1_time": "2022-10-20 03:45:18 +0000",
      "commit1_digest": {
        "test_eval[demucs-cpu-jit]": 1.9188905177084963,
        "test_eval[demucs-cpu-eager]": 2.0636679343588185,
        "test_eval[pytorch_stargan-cpu-jit]": 0.48076599889609495,
        "test_train[pytorch_stargan-cpu-eager]": 1.0074269479489886,
        "test_eval[pytorch_stargan-cpu-eager]": 0.4370430368027883,
        "test_train[timm_vision_transformer-cpu-eager]": 0.23410833064408507
      },
      "commit2": "d94e33f041f",
      "commit2_time": "2022-10-20 03:46:50 +0000",
      "commit2_digest": {
        "test_eval[demucs-cpu-jit]": 2.262464324652683
      }
    },
    {
      "commit1": "1e4a2742489",
      "commit1_time": "2022-10-20 17:28:27 +0000",
      "commit1_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.48501308874983806,
        "test_eval[pytorch_stargan-cpu-eager]": 0.4443765416974202
      },
      "commit2": "c16b7b41f76",
      "commit2_time": "2022-10-20 17:34:16 +0000",
      "commit2_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.5319523978425422,
        "test_eval[pytorch_stargan-cpu-eager]": 0.5121736923465505
      }
    }
  ]
}
```

https://github.com/pytorch/pytorch/commit/c16b7b41f76 should be noise - profiler only changes
https://github.com/pytorch/pytorch/commit/d94e33f041f should also be noise - nested tensor only changes

Workflow: 
https://github.com/pytorch/benchmark/actions/runs/3307926864
Repository: pytorch/benchmark
Issue: Identify Graph NNs to add and write up issues for them 
Root cause: Unimplemented Feature Gap

Would any of these JIT and be useful for dynamic shapes work? Still good to cover these use cases in our benchmark suite.

https://github.com/facebookresearch/PyTorch-BigGraph (Partitioning)
  - CPU-only, GPU-experimental, suggests only useful with huge data/distributed system, haven't checked JIT compatibility
https://github.com/facebookresearch/poincare-embeddings (Embedding)
https://github.com/benedekrozemberczki/CapsGNN (Classification)
https://github.com/Diego999/pyGAT (Graph Regression)
Repository: pytorch/benchmark
Issue: Fix broken bisection workflows
Root cause: Configuration Error

v2:
bisection-gh3329753297
bisection-gh3338141919
bisection-gh3346070799

bisection-gh3346070799 workflow: https://github.com/pytorch/benchmark/actions/runs/3348939147
Done

bisection-gh3338141919 Done (https://github.com/pytorch/benchmark/actions/runs/3349864259)

bisection-gh3329753297 (https://github.com/pytorch/benchmark/actions/runs/3349864874)

All resolved.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.14.0.dev20221027+cu116'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.14.0.dev20221026+cu116

Base PyTorch commit: 0d13ffbbae0ae12e72ed8856ccdd822bf840344c

Affected PyTorch version: 1.14.0.dev20221027+cu116

Affected PyTorch commit: 21bef8e944c90cdf98c2ead4369410db252944e1

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: -15.62159%
- test_eval[pytorch_stargan-cpu-eager]: -13.80175%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/3349464817
Result:
```
{
  "start": "0d13ffbbae0ae12e72ed8856ccdd822bf840344c",
  "end": "21bef8e944c90cdf98c2ead4369410db252944e1",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "926827b89cc",
      "commit1_time": "2022-10-26 21:01:09 +0000",
      "commit1_digest": {
        "test_eval[pytorch_stargan-cpu-eager]": 0.4369667133491021
      },
      "commit2": "adb76ef510e",
      "commit2_time": "2022-10-26 21:28:45 +0000",
      "commit2_digest": {
        "test_eval[pytorch_stargan-cpu-eager]": 0.5065934277314227
      }
    },
    {
      "commit1": "30ea8f5c207",
      "commit1_time": "2022-10-27 01:24:03 +0000",
      "commit1_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.035671123424046006
      },
      "commit2": "195a13f48ce",
      "commit2_time": "2022-10-27 01:50:00 +0000",
      "commit2_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.0421335926609269
      }
    }
  ]
}
```

Noise, closed.
Repository: pytorch/benchmark
Issue: Add Speech Reco ESPNet
Root cause: Unimplemented Feature Gap

This hasn't been looked at by the benchmark team, there could be issues at any point in the process (torchbenchmark/models/ADDING_MODELS.md)
https://github.com/pytorch/benchmark/blob/master/torchbenchmark/models/ADDING_MODELS.md
 - JIT optional.
 
https://github.com/espnet/espnet
Repository: pytorch/benchmark
Issue: soft_actor_critic does not use the new gym API and crash on env initialization
Root cause: External Dependency Breakage

Gym deprecated the method `env.seed` and needs to use `env.reset(seed=seed)` instead

The method `end.seed` was removed in https://github.com/openai/gym/commit/3a8daafce113c6161190c0252c6a3a340cc38fd4#diff-8f3ae46be618d936b191b5f32bad0dcc4674ea5385bafb31a6c02840a43fe76aL196

Fixed in https://github.com/pytorch/benchmark/pull/1272
Repository: pytorch/benchmark
Issue: Remove bloat from git repo

These account for a large % of the repo size but aren't needed.  However, some work is needed before they can all be removed, and then the process of rewriting git history to clean the files up should be done all at once and carefully.  This issue can accumulate a wish list of files to delete, and at some point we can purge them all at once. 

demucs/results/* -- can be removed, not used by benchmark
Background_matting/sample_data -- apparently not used, another `ak` folder contains used data
yolov3/data --contains mini coco, which is also checked in under torchbenchmark/data, so that should be used instead

This has been resolved as part of the Facebook internal integration process.
Repository: pytorch/benchmark
Issue: Installation error
Root cause: External Dependency Breakage

### PyTorch Version:
nightly torch-1.14.0.dev20221104%2Bcu116-cp38-cp38-linux_x86_64.whl
### Error 
```
from torch.testing import assert_allclose as _assert_close
ImportError: cannot import name 'assert_allclose' from 'torch.testing' 
(miniconda3/envs/pt_nov4/lib/python3.8/site-packages/torch/testing/__init__.py)
```
Error occurs when install DALLE2_pytorch. It is related to this issue https://github.com/pytorch/pytorch/issues/61844

It looks like the nightly version deprecated some functions. 


Full error stack:
```
Traceback (most recent call last):
  File "install.py", line 20, in <module>
    patch_dalle2()
  File "install.py", line 7, in patch_dalle2
    import dalle2_pytorch
  File "/home/ubuntu/miniconda3/envs/pt_nov4/lib/python3.8/site-packages/dalle2_pytorch/__init__.py", line 2, in <module>
    from dalle2_pytorch.dalle2_pytorch import DALLE2, DiffusionPriorNetwork, DiffusionPrior, Unet, Decoder
  File "/home/ubuntu/miniconda3/envs/pt_nov4/lib/python3.8/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 20, in <module>
    from kornia.filters import gaussian_blur2d
  File "/home/ubuntu/miniconda3/envs/pt_nov4/lib/python3.8/site-packages/kornia/__init__.py", line 3, in <module>
    from . import filters
  File "/home/ubuntu/miniconda3/envs/pt_nov4/lib/python3.8/site-packages/kornia/filters/__init__.py", line 2, in <module>
    from .blur_pool import BlurPool2D, MaxBlurPool2D, blur_pool2d, edge_aware_blur_pool2d, max_blur_pool2d
  File "/home/ubuntu/miniconda3/envs/pt_nov4/lib/python3.8/site-packages/kornia/filters/blur_pool.py", line 7, in <module>
    from kornia.testing import KORNIA_CHECK, KORNIA_CHECK_SHAPE
  File "/home/ubuntu/miniconda3/envs/pt_nov4/lib/python3.8/site-packages/kornia/testing/__init__.py", line 254, in <module>
    from torch.testing import assert_allclose as _assert_close
ImportError: cannot import name 'assert_allclose' from 'torch.testing' (/home/ubuntu/miniconda3/envs/pt_nov4/lib/python3.8/site-packages/torch/testing/__init__.py)
```

This is because upstream issue: https://github.com/kornia/kornia/issues/1981, we will update after upstream fixes

Fixed
Repository: pytorch/benchmark
Issue: bug for DCGM fp32 metric collection
Root cause: Algorithmic Error

When there are multiple metrics, the DCGM API may return `-1` values for some metrics. In general, most metrics use `max`  as the default aggregator function while our FP32 metric use `mean`. It will introduce wrong values when DCGM returns `-1`. Update the FP32 aggregator function to fix this.
Repository: pytorch/benchmark
Issue: E2E model running bug 
Root cause: Validation Gap

Run:
```
python run_e2e.py hf_bert -t eval
```
Error:
```
Traceback (most recent call last):
  File "/scratch/ybliang/work/repos/benchmark/run_e2e.py", line 43, in <module>
    m = Model(test=args.test, batch_size=[args.bs](http://args.bs/), extra_args=extra_args)
  File "/scratch/ybliang/work/repos/benchmark/torchbenchmark/util/e2emodel.py", line 8, in _call_
    obj = type.__call__(cls, *args, **kwargs)
  File "/scratch/ybliang/work/repos/benchmark/torchbenchmark/e2e_models/hf_bert/__init__.py", line 83, in _init_
    self.prep(hf_args)
  File "/scratch/ybliang/work/repos/benchmark/torchbenchmark/e2e_models/hf_bert/__init__.py", line 132, in prep
    model = DDP(
  File "/scratch/ybliang/work/repos/pytorch/torch/nn/parallel/distributed.py", line 616, in _init_
    self.process_group = _get_default_group()
  File "/scratch/ybliang/work/repos/pytorch/torch/distributed/distributed_c10d.py", line 588, in _get_default_group
    raise RuntimeError(
RuntimeError: Default process group has not been initialized, please make sure to call init_process_group.
```

Fixed in https://github.com/pytorch/benchmark/pull/1291
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.14.0.dev20221112+cu116'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.14.0.dev20221111+cu116

Base PyTorch commit: a6d72f44a4e8b6e9d2e878f30fd8b1d3e1197f0e

Affected PyTorch version: 1.14.0.dev20221112+cu116

Affected PyTorch commit: 7c3adddd6c3fe1bda4a9e5bfb9f992a802329551

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: -21.80489%
- test_eval[pyhpc_turbulent_kinetic_energy-cuda-eager]: -9.73592%


cc @xuzhao9

Result json: 
```
{
  "start": "a6d72f44a4e8b6e9d2e878f30fd8b1d3e1197f0e",
  "end": "7c3adddd6c3fe1bda4a9e5bfb9f992a802329551",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "6fe47b682fe",
      "commit1_time": "2022-11-11 22:31:32 +0000",
      "commit1_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.03596256369853328,
        "test_eval[pyhpc_turbulent_kinetic_energy-cuda-eager]": 0.033363515449066954
      },
      "commit2": "a7fa423f48a",
      "commit2_time": "2022-11-11 23:31:15 +0000",
      "commit2_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.035997570572613644,
        "test_eval[pyhpc_turbulent_kinetic_energy-cuda-eager]": 0.03018386963316623
      }
    },
    {
      "commit1": "a7fa423f48a",
      "commit1_time": "2022-11-11 23:31:15 +0000",
      "commit1_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.035997570572613644,
        "test_eval[pyhpc_turbulent_kinetic_energy-cuda-eager]": 0.03018386963316623
      },
      "commit2": "7c3adddd6c3",
      "commit2_time": "2022-11-11 23:58:51 +0000",
      "commit2_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.0416570108756423,
        "test_eval[pyhpc_turbulent_kinetic_energy-cuda-eager]": 0.030176670879454297
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/3451836909


Noise, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.14.0.dev20221113+cu116'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.14.0.dev20221112+cu116

Base PyTorch commit: 7c3adddd6c3fe1bda4a9e5bfb9f992a802329551

Affected PyTorch version: 1.14.0.dev20221113+cu116

Affected PyTorch commit: 46796fe5e9b74602d45927304773fdcda1c3215a

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: +16.02686%


cc @xuzhao9

Result json: 
```
{
  "start": "7c3adddd6c3fe1bda4a9e5bfb9f992a802329551",
  "end": "46796fe5e9b74602d45927304773fdcda1c3215a",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "6e5f736d86b",
      "commit1_time": "2022-11-12 14:23:45 +0000",
      "commit1_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.03593457474148479
      },
      "commit2": "4270bb37dac",
      "commit2_time": "2022-11-12 15:03:50 +0000",
      "commit2_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.04201923768656949
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/3456208271


Noise, closed.
Repository: pytorch/benchmark
Issue: E2E hf_t5 error
Root cause: Configuration Error

Run ```python run_e2e.py hf_t5 -t eval```, you'll get the DDP error:
```
RuntimeError: Default process group has not been initialized, please make sure to call init_process_group.
```
I followed https://github.com/pytorch/benchmark/pull/1291 and made ```--distributed``` by default none. Then the following error occurs:
```
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument index in method wrapper__index_select)
```
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.14.0.dev20221117+cu116'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.14.0.dev20221115+cu116

Base PyTorch commit: 73d71ae3d62607f2e480af37c470375ea405eb1c

Affected PyTorch version: 1.14.0.dev20221117+cu116

Affected PyTorch commit: c3acb9c8859fb5cfa1959ee49849f07942c40ccc

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: +16.86073%
- test_eval[pytorch_stargan-cpu-eager]: +7.20564%
- test_eval[resnext50_32x4d-cpu-jit]: +10.52181%


cc @xuzhao9

Result json: 
```
{
  "start": "73d71ae3d62607f2e480af37c470375ea405eb1c",
  "end": "c3acb9c8859fb5cfa1959ee49849f07942c40ccc",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "73d71ae3d62",
      "commit1_time": "2022-11-15 04:40:13 +0000",
      "commit1_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.042947927257046103,
        "test_eval[pytorch_stargan-cpu-eager]": 0.5162501436891034,
        "test_eval[resnext50_32x4d-cpu-jit]": 0.04543359462252777
      },
      "commit2": "21dd311077d",
      "commit2_time": "2022-11-15 05:08:26 +0000",
      "commit2_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.03537731137576288
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/3490500826


Unrelated noise, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.14.0.dev20221121+cu116'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.14.0.dev20221119+cu116

Base PyTorch commit: 419ef2cdcfe84442de5232739284c6a51a18632f

Affected PyTorch version: 1.14.0.dev20221121+cu116

Affected PyTorch commit: 1856fa5df7fda9950da26eff2ef885e845bf6b6c

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: -29.79170%
- test_eval[pytorch_stargan-cpu-jit]: -7.13346%


cc @xuzhao9

Result json: 
```
{
  "start": "419ef2cdcfe84442de5232739284c6a51a18632f",
  "end": "1856fa5df7fda9950da26eff2ef885e845bf6b6c",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "6b8c1b19b51",
      "commit1_time": "2022-11-19 17:49:39 +0000",
      "commit1_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.03598900142543275
      },
      "commit2": "6afe341276f",
      "commit2_time": "2022-11-19 18:01:25 +0000",
      "commit2_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.04869316048210576
      }
    },
    {
      "commit1": "6afe341276f",
      "commit1_time": "2022-11-19 18:01:25 +0000",
      "commit1_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.04869316048210576
      },
      "commit2": "5582001bd5e",
      "commit2_time": "2022-11-19 21:14:31 +0000",
      "commit2_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.03569615320784265
      }
    },
    {
      "commit1": "5582001bd5e",
      "commit1_time": "2022-11-19 21:14:31 +0000",
      "commit1_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.03569615320784265
      },
      "commit2": "8ac58bc2e34",
      "commit2_time": "2022-11-19 21:40:07 +0000",
      "commit2_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.04271561873611063
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/3516629929


Noisy test, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.14.0.dev20221122+cu116'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.14.0.dev20221121+cu116

Base PyTorch commit: 1856fa5df7fda9950da26eff2ef885e845bf6b6c

Affected PyTorch version: 1.14.0.dev20221122+cu116

Affected PyTorch commit: 40cf214f2d18b3b8af5354ddc5dad8156ea32520

Affected Tests:
- test_eval[demucs-cpu-jit]: +9.62922%
- test_train[pytorch_stargan-cpu-jit]: +11.74031%
- test_eval[pytorch_stargan-cpu-jit]: +7.81116%
- test_train[pytorch_stargan-cpu-eager]: +9.96892%
- test_eval[pytorch_stargan-cpu-eager]: +9.64781%
- test_train[timm_vision_transformer-cpu-eager]: +7.87736%


cc @xuzhao9

Result json: 
```
{
  "start": "1856fa5df7fda9950da26eff2ef885e845bf6b6c",
  "end": "40cf214f2d18b3b8af5354ddc5dad8156ea32520",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "e545caa50f3",
      "commit1_time": "2022-11-22 03:57:04 +0000",
      "commit1_digest": {
        "test_train[pytorch_stargan-cpu-eager]": 1.1681347470264882
      },
      "commit2": "40cf214f2d1",
      "commit2_time": "2022-11-22 04:12:43 +0000",
      "commit2_digest": {
        "test_eval[demucs-cpu-jit]": 2.2809481856646014,
        "test_train[pytorch_stargan-cpu-jit]": 1.0932917105965316,
        "test_eval[pytorch_stargan-cpu-jit]": 0.5589741758536547,
        "test_train[pytorch_stargan-cpu-eager]": 1.0663489607162773,
        "test_eval[pytorch_stargan-cpu-eager]": 0.5160945279989392,
        "test_train[timm_vision_transformer-cpu-eager]": 0.24042469570413233
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/3525626376


Noise, closed.
Repository: pytorch/benchmark
Issue: errors for configured CUDA_VISIBLE_DEVICES on multiple gpus
Root cause: Validation Gap

If we specify the visibility of a subset of installed GPUs via CUDA_VISIBLE_DEVICES, there will be an error 
```
components.model_analyzer.tb_dcgm_types.da_exceptions.TorchBenchAnalyzerException:  GPU with GPU-XXXX uuid is not present!
```