zhiminy/EvalEng huggingface dataset:
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.0.0.dev20230114+cu116'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.0.0.dev20230113+cu116

Base PyTorch commit: fbbb19599a1d162e5927542ed251fd2ba63d5163

Affected PyTorch version: 2.0.0.dev20230114+cu116

Affected PyTorch commit: b8057aa16d376eefe081d852335541e2a7609c40

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: +36.26565%
- test_eval[pytorch_stargan-cpu-eager]: -7.42422%
- test_train[timm_regnet-cpu-eager]: +7.55228%
- test_train[timm_vision_transformer-cpu-eager]: +11.64020%


cc @xuzhao9

Result json: 
```
{
  "start": "fbbb19599a1d162e5927542ed251fd2ba63d5163",
  "end": "b8057aa16d376eefe081d852335541e2a7609c40",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "7568484d54d",
      "commit1_time": "2023-01-14 02:30:54 +0000",
      "commit1_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.03593359055832542,
        "test_train[timm_regnet-cpu-eager]": 0.7164395724423229
      },
      "commit2": "3a0053abd62",
      "commit2_time": "2023-01-14 02:55:32 +0000",
      "commit2_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.04268081490105639,
        "test_train[timm_regnet-cpu-eager]": 0.7319657129235566
      }
    },
    {
      "commit1": "3a0053abd62",
      "commit1_time": "2023-01-14 02:55:32 +0000",
      "commit1_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.04268081490105639,
        "test_train[timm_regnet-cpu-eager]": 0.7319657129235566
      },
      "commit2": "b8057aa16d3",
      "commit2_time": "2023-01-14 03:38:03 +0000",
      "commit2_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.04878867881118574,
        "test_eval[pytorch_stargan-cpu-eager]": 0.5130617402959615,
        "test_train[timm_regnet-cpu-eager]": 0.6529049536213278,
        "test_train[timm_vision_transformer-cpu-eager]": 0.22451452531386168
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/3919531755


Noise, closed.
Repository: pytorch/benchmark
Issue: Add stable diffusion to the benchmark
Root cause: Unimplemented Feature Gap

Example inference code

```import torch
from diffusers import EulerDiscreteScheduler, StableDiffusionPipeline

model_id = "stabilityai/stable-diffusion-2"
scheduler = EulerDiscreteScheduler.from_pretrained(model_id, subfolder="scheduler")
pipe = StableDiffusionPipeline.from_pretrained(
    model_id,
    scheduler=scheduler,
    torch_dtype=torch.float16,
)
pipe = pipe.to("cuda")

@torch.compile(backend="inductor")
def inference_func(promt):
    image = pipe(prompt, num_inference_steps=1).images[0]
    return image

prompt = "a photo of an astronaut riding a horse on mars"
image = inference_func(prompt)

```
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.0.0.dev20230119+cu116'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.0.0.dev20230118+cu116

Base PyTorch commit: 00fe63d1d8c5c6411e093fa4388dc98c3683d236

Affected PyTorch version: 2.0.0.dev20230119+cu116

Affected PyTorch commit: de459bdfaa44a25565d4c4eedda49906feb6d993

Affected Tests:
- test_eval[demucs-cpu-jit]: +7.38607%
- test_eval[pytorch_stargan-cpu-eager]: -7.11284%
- test_train[timm_vision_transformer-cpu-eager]: +8.40390%
- test_train[tts_angular-cpu-eager]: -63.65962%
- test_eval[tts_angular-cpu-eager]: -38.14227%


cc @xuzhao9

Result json: 
```
{
  "start": "00fe63d1d8c5c6411e093fa4388dc98c3683d236",
  "end": "de459bdfaa44a25565d4c4eedda49906feb6d993",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "00fe63d1d8c",
      "commit1_time": "2023-01-18 02:49:14 +0000",
      "commit1_digest": {
        "test_eval[demucs-cpu-jit]": 2.08232683215756,
        "test_eval[pytorch_stargan-cpu-eager]": 0.49604378377553077,
        "test_train[timm_vision_transformer-cpu-eager]": 0.21524148634634913,
        "test_train[tts_angular-cpu-eager]": 0.5573350619524717,
        "test_eval[tts_angular-cpu-eager]": 0.35106956111267207
      },
      "commit2": "98b78aa11c5",
      "commit2_time": "2023-01-18 02:55:42 +0000",
      "commit2_digest": {
        "test_train[timm_vision_transformer-cpu-eager]": 0.2305441338568926,
        "test_train[tts_angular-cpu-eager]": 0.5476256367517636,
        "test_eval[tts_angular-cpu-eager]": 0.3459392771823332
      }
    },
    {
      "commit1": "a41f00ed708",
      "commit1_time": "2023-01-18 04:02:41 +0000",
      "commit1_digest": {
        "test_train[tts_angular-cpu-eager]": 0.5542491457192227,
        "test_eval[tts_angular-cpu-eager]": 0.3472385982517153
      },
      "commit2": "94a7c011590",
      "commit2_time": "2023-01-18 04:41:18 +0000",
      "commit2_digest": {
        "test_eval[demucs-cpu-jit]": 2.1444463058374823,
        "test_train[timm_vision_transformer-cpu-eager]": 0.2379927942994982,
        "test_train[tts_angular-cpu-eager]": 0.20029154166113586,
        "test_eval[tts_angular-cpu-eager]": 0.2180802782997489
      }
    },
    {
      "commit1": "07800c52af3",
      "commit1_time": "2023-01-18 22:05:42 +0000",
      "commit1_digest": {
        "test_eval[pytorch_stargan-cpu-eager]": 0.5097044834867119
      },
      "commit2": "de459bdfaa4",
      "commit2_time": "2023-01-18 22:28:52 +0000",
      "commit2_digest": {
        "test_eval[demucs-cpu-jit]": 2.2861412693280725,
        "test_eval[pytorch_stargan-cpu-eager]": 0.4512327048927546,
        "test_train[timm_vision_transformer-cpu-eager]": 0.23950233801733703,
        "test_train[tts_angular-cpu-eager]": 0.20482875173911452,
        "test_eval[tts_angular-cpu-eager]": 0.21431025105994195
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/3960834267


tts_angular cpu speedup: https://github.com/pytorch/pytorch/commit/94a7c011590
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.0.0.dev20230121+cu116'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.0.0.dev20230120+cu116

Base PyTorch commit: 620846c8b4f00f58dbd8b62746ec8e4328552b58

Affected PyTorch version: 2.0.0.dev20230121+cu116

Affected PyTorch commit: 1af40d5108ee5ddd408eed8115ce3abf6512149c

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: +28.80307%
- test_eval[vision_maskrcnn-cpu-eager]: +8.63309%


cc @xuzhao9

Result json: 
```
{
  "start": "620846c8b4f00f58dbd8b62746ec8e4328552b58",
  "end": "1af40d5108ee5ddd408eed8115ce3abf6512149c",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "1464db08b40",
      "commit1_time": "2023-01-20 03:18:21 +0000",
      "commit1_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.04818593615412035
      },
      "commit2": "663bf4ba152",
      "commit2_time": "2023-01-20 04:08:45 +0000",
      "commit2_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.03554418482335991
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/3975832035


Noise, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.0.0.dev20230123+cu116'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.0.0.dev20230122+cu116

Base PyTorch commit: ed07070a11802dd561995c20e89d33cc986de2fa

Affected PyTorch version: 2.0.0.dev20230123+cu116

Affected PyTorch commit: 20bf77f9bd3cd7cb4da758afb58042e2980ee6ba

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: -24.96495%
- test_eval[pytorch_stargan-cpu-jit]: +7.41999%
- test_eval[pytorch_stargan-cpu-eager]: +9.06000%
- test_eval[resnext50_32x4d-cpu-jit]: -9.40575%


cc @xuzhao9

Result json: 
```
{
  "start": "ed07070a11802dd561995c20e89d33cc986de2fa",
  "end": "20bf77f9bd3cd7cb4da758afb58042e2980ee6ba",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "ed07070a118",
      "commit1_time": "2023-01-22 04:03:04 +0000",
      "commit1_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.046871273185719146,
        "test_eval[pytorch_stargan-cpu-jit]": 0.5473578092176468,
        "test_eval[pytorch_stargan-cpu-eager]": 0.5048869968624785,
        "test_eval[resnext50_32x4d-cpu-jit]": 0.04497129318263868
      },
      "commit2": "6016e4c707a",
      "commit2_time": "2023-01-22 04:53:03 +0000",
      "commit2_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.03647711675148457
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/3988888938


Noise, closed.
Repository: pytorch/benchmark
Issue: enable running a model script directly 
Root cause: Documentation Deficiency

right now, you can't `python torchbenchmark/models/dlrm/__init__.py` becuase "ImportError: attempted relative import with no known parent package".  

how can the __main__ block in __init__.py be executed?  If it's possible we should write up the workaround in README, or else it would be nice to enable.

I think this ` python -m torchbenchmark.models.BERT_pytorch.__init__` works pretty well now, but we should add it to `README.md`

> I think this ` python -m torchbenchmark.models.BERT_pytorch.__init__` works pretty well now, but we should add it to `README.md`

The command works for the BERT_pytorch model, but when I am running it on other models, say "Background_Matting", the command returns nothing because there is no main function in the module: https://github.com/pytorch/benchmark/blob/master/torchbenchmark/models/Background_Matting/__init__.py
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.0.0.dev20230124+cu116'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.0.0.dev20230123+cu116

Base PyTorch commit: 20bf77f9bd3cd7cb4da758afb58042e2980ee6ba

Affected PyTorch version: 2.0.0.dev20230124+cu116

Affected PyTorch commit: 2a8669c54cb29bf42692185340fd68f341ac65ef

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: +38.26343%
- test_train[timm_regnet-cpu-eager]: -7.89869%


cc @xuzhao9

Result json: 
```
{
  "start": "20bf77f9bd3cd7cb4da758afb58042e2980ee6ba",
  "end": "2a8669c54cb29bf42692185340fd68f341ac65ef",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "20bf77f9bd3",
      "commit1_time": "2023-01-22 22:19:40 +0000",
      "commit1_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.04958496998906845,
        "test_train[timm_regnet-cpu-eager]": 0.6716958157485351
      },
      "commit2": "b985c2ef4a5",
      "commit2_time": "2023-01-23 04:38:11 +0000",
      "commit2_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.03575181448832154
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/3998771421


Noise, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.0.0.dev20230126+cu116'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.0.0.dev20230125+cu116

Base PyTorch commit: 4bc0491752b6d891c5bb33a3d18c5da7334fc6df

Affected PyTorch version: 2.0.0.dev20230126+cu116

Affected PyTorch commit: 0a57a20c02132577a48f5283ffbdf04257af1dbf

Affected Tests:
- test_train[BERT_pytorch-cpu-jit]: -7.56120%
- test_train[dlrm-cuda-eager]: -14.11087%
- test_eval[mnasnet1_0-cpu-jit]: +25.70337%
- test_train[nvidia_deeprecommender-cpu-eager]: -8.70715%
- test_train[nvidia_deeprecommender-cuda-eager]: -10.57407%
- test_train[resnext50_32x4d-cpu-jit]: -8.06190%
- test_train[resnext50_32x4d-cpu-eager]: -8.57346%
- test_train[timm_nfnet-cpu-jit]: -8.83225%
- test_train[timm_nfnet-cpu-eager]: -25.50586%


cc @xuzhao9

Result json: 
```
{
  "start": "4bc0491752b6d891c5bb33a3d18c5da7334fc6df",
  "end": "0a57a20c02132577a48f5283ffbdf04257af1dbf",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "5441f2c067e",
      "commit1_time": "2023-01-26 00:37:54 +0000",
      "commit1_digest": {
        "test_train[BERT_pytorch-cpu-jit]": 1.1679606192745269,
        "test_train[dlrm-cuda-eager]": 0.027036831383348295,
        "test_train[nvidia_deeprecommender-cpu-eager]": 0.5046387678012252,
        "test_train[nvidia_deeprecommender-cuda-eager]": 0.12282024077139794,
        "test_train[timm_nfnet-cpu-jit]": 13.804846601490862,
        "test_train[timm_nfnet-cpu-eager]": 13.893649057205767
      },
      "commit2": "b90496eef56",
      "commit2_time": "2023-01-26 01:04:28 +0000",
      "commit2_digest": {
        "test_train[BERT_pytorch-cpu-jit]": 1.065794260217808,
        "test_train[dlrm-cuda-eager]": 0.02321390499657189,
        "test_train[nvidia_deeprecommender-cpu-eager]": 0.4591219003079459,
        "test_train[nvidia_deeprecommender-cuda-eager]": 0.1096938799135387,
        "test_train[timm_nfnet-cpu-jit]": 11.290817176364362,
        "test_train[timm_nfnet-cpu-eager]": 10.649523812555708
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/4017547691


Root cause commit: https://github.com/pytorch/pytorch/commit/b90496eef56

@janeyx99 Does this performance speedup make sense to you?

Yes! The setting the grads to None (thus avoiding instantiating tensors) should save memory and time. :D 

Thanks! Closing this issue.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.0.0.dev20230130+cu116'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.0.0.dev20230129+cu116

Base PyTorch commit: 7cc91f4002cb5dd3290318a967fb1e830d812666

Affected PyTorch version: 2.0.0.dev20230130+cu116

Affected PyTorch commit: e790281a85fe3693fc1d38bf0e2c6e874d5e10b0

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: +39.80514%


cc @xuzhao9

Result json: 
```
{
  "start": "7cc91f4002cb5dd3290318a967fb1e830d812666",
  "end": "e790281a85fe3693fc1d38bf0e2c6e874d5e10b0",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "7cc91f4002c",
      "commit1_time": "2023-01-29 03:33:34 +0000",
      "commit1_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.04774356392127546
      },
      "commit2": "68a1065bd76",
      "commit2_time": "2023-01-29 04:35:46 +0000",
      "commit2_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.03561251404984244
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/4046424250


Noise, closed.
Repository: pytorch/benchmark
Issue: Print memory metrics by default
Root cause: Unimplemented Feature Gap

Let `run.py` print CPU and GPU peak memory information by default.

some notes:
1. I didn't add the auto downgradation form `dcgm` to `nvml`. It means if you set the `metrics_backend` to `dcgm` but the system doesn't install dcgm or has permission errors for dcgm, there will be errors thrown rather than downgrading to `nvml`.
2. the results of metrics backend `nvml` are slightly larger than `dcgm` because of the overhead of `nvml`.
3. the `nvml` has a dependent python package `nvidia-ml-py`
4. `nvml` is more compatible with GPUs. it works on most gpus if these gpus could run `nvidia-smi`. But `dcgm` only works on Tesla GPUs for profile metrics.

1. Sounds good to me.
2. Also sounds fine, doesn't really matter.
3. Since we have added to `requirements.txt`, it should be good
4. The availability of dcgm is a problem because it requires setting up special permissions to performance counters, which might not be available everywhere (e.g. sudo not available or can't set the special option to nvidia kernel module)

Closed by https://github.com/pytorch/benchmark/pull/1392
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.0.0.dev20230131+cu116'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.0.0.dev20230130+cu116

Base PyTorch commit: e790281a85fe3693fc1d38bf0e2c6e874d5e10b0

Affected PyTorch version: 2.0.0.dev20230131+cu116

Affected PyTorch commit: 2a6e08570442c8fcbad6659e074c226809b1ed22

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: -28.42596%
- test_train[timm_regnet-cpu-eager]: +7.34110%
- test_eval[tts_angular-cpu-eager]: +7.94197%


cc @xuzhao9

Result json: 
```
{
  "start": "e790281a85fe3693fc1d38bf0e2c6e874d5e10b0",
  "end": "2a6e08570442c8fcbad6659e074c226809b1ed22",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "e790281a85f",
      "commit1_time": "2023-01-30 01:56:50 +0000",
      "commit1_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.035996599058652747,
        "test_train[timm_regnet-cpu-eager]": 0.6833136462606489,
        "test_eval[tts_angular-cpu-eager]": 0.24577847882173956
      },
      "commit2": "4d51c8532c4",
      "commit2_time": "2023-01-30 05:14:03 +0000",
      "commit2_digest": {
        "test_eval[tts_angular-cpu-eager]": 0.21461955411359668
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/4057310851


Noise, closed.
Repository: pytorch/benchmark
Issue: Https proxy comment in test.py
Root cause: Documentation Deficiency

I am trying to run the benchmarks, but the benchmarks hangs when running `python test.py`. The script does not run and gets stuck somewhere. I discovered the comment in `test.py`

> Make sure to enable an https proxy if necessary, or the setup steps may hang.

What does that mean? What script does need the proxy for what? 


The test script needs to be able to access the internet (on first run, at least) to download test data and the like.  So if you're inside a firewall you'll need a proxy server to get out.  This is a common setup for facebook engineers (hence the comment) but may not really apply to the rest of the world.

@milutter were you able to get past the stuck issue? 

Thanks for the help. I got it to partially work but it was just surprising to see that it is downloading something without an progress bar. I stopped using it because the test did not work easily and my need for benchmarking was not that great.
Repository: pytorch/benchmark
Issue: GPU Test stableness tracker
Root cause: Resource Mishandling

# List of unstable GPU tests

- [x] pytorch_CycleGAN_and_pix2pix train: https://github.com/pytorch/benchmark/actions/runs/4147131894
- [x] DALLE2_pytorch eval (33% variation): https://github.com/pytorch/benchmark/actions/runs/4147979273
       Reason: eval is dominated by memory copy
![image](https://user-images.githubusercontent.com/502017/223216857-2ee376e7-c0c6-4257-91cc-315b8c14f8bd.png)

- [ ] LearningToPaint train (115% variation): https://github.com/pytorch/benchmark/actions/runs/4287347886
        Try run 1: https://github.com/pytorch/benchmark/actions/runs/4295256456 (remove CPU tensors)
        Try run 2: https://github.com/pytorch/benchmark/actions/runs/4347585042
- [ ] dense121 eval (11.2% variation):
- [ ] detectron2_fasterrcnn_r_50_c4 eval (16% variation):
- [ ] dlrm eval (15% variation):
- [ ] mnasnet1_0 train (17% variation):
- [ ] phlippe_densenet train (10% variation):
- [ ] pytorch_CycleGAN_and_pix2pix eval (15% variation):
- [ ] resnext50_32x4d train (10.7% variation):
- [ ] shufflenet_v2_x1_0 eval (14.8% variation):
- [x] timm_vision_transformer train (16% variation): https://github.com/pytorch/benchmark/actions/runs/4147202172
        Fix: https://github.com/pytorch/benchmark/pull/1419
- [x] Yolov3 train: https://github.com/pytorch/benchmark/actions/runs/4145521493
- [x] Yolov3 eval: https://github.com/pytorch/benchmark/actions/runs/4146281655


# CPU
CPU eval test: https://github.com/pytorch/benchmark/actions/runs/4145537778


Run test workflow: https://github.com/pytorch/benchmark/actions/runs/4585968352
Repository: pytorch/benchmark
Issue: Simple PytorchBenchmark script gives CUDA forked subprocess error
Root cause: Interface Contract Mismatch

When I try to run a simple Pytorch Benchmark test I get this error:

```
Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
Traceback (most recent call last):
  File "/home/cbarkhof/code-thesis/Experimentation/Benchmarking/benchmarking-models-claartje.py", line 31, in <module>
    benchmark.run()
  File "/home/cbarkhof/.local/lib/python3.6/site-packages/transformers/benchmark/benchmark_utils.py", line 674, in run
    memory, inference_summary = self.inference_memory(model_name, batch_size, sequence_length)
ValueError: too many values to unpack (expected 2)
```

The script is as simple as this:
```
from transformers import PyTorchBenchmark, PyTorchBenchmarkArguments

benchmark_args = PyTorchBenchmarkArguments(models=["bert-base-uncased"],
                                           batch_sizes=[8],
                                           sequence_lengths=[8, 32, 128, 512],
                                           save_to_csv=True,
                                           log_filename='log',
                                           env_info_csv_file='env_info')

benchmark = PyTorchBenchmark(benchmark_args)
benchmark.run()
```

This happens on my machine, but also in this notebook running on Colab: https://github.com/huggingface/transformers/blob/master/notebooks/05-benchmark.ipynb

Closed, this is not relevant anymore.
Repository: pytorch/benchmark
Issue: profile report CUDA failed in CPU only machine.
Root cause: Validation Gap

Windows:
---------------------------------------------------------------------------------
PS D:\xu_github\benchmark> python.exe ./run.py -d cpu -m jit --bs=12 alexnet --stress 200 --profile
Running eval method from alexnet on cpu in jit mode with input batch size 12.
STAGE:2023-02-15 13:24:24 11468:11628 ..\third_party\kineto\libkineto\src\ActivityProfilerController.cpp:294] Completed Stage: Warm Up
STAGE:2023-02-15 13:24:24 11468:11628 ..\third_party\kineto\libkineto\src\ActivityProfilerController.cpp:300] Completed Stage: Collection
Traceback (most recent call last):
  File "D:\xu_github\benchmark\run.py", line 333, in <module>
    profile_one_step(test)
  File "D:\xu_github\benchmark\run.py", line 229, in profile_one_step
    start_event = torch.cuda.Event(enable_timing=True)
  File "C:\Users\1\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\cuda\streams.py", line 164, in __new__
    return super(Event, cls).__new__(
TypeError: object.__new__() takes exactly one argument (the type to instantiate)

Linux:
---------------------------------------------------------------------------------
xu@xu-G3-3590:~/torchbench/benchmark$ python3 run.py -d cpu -m jit --bs=12 resnet50 --stress 200 --profile --profile-devices cpu
Running eval method from resnet50 on cpu in jit mode with input batch size 12.
STAGE:2023-02-15 21:20:54 2315:2315 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
STAGE:2023-02-15 21:20:54 2315:2315 ActivityProfilerController.cpp:300] Completed Stage: Collection
Traceback (most recent call last):
  File "run.py", line 333, in <module>
    profile_one_step(test)
  File "run.py", line 233, in profile_one_step
    start_event.record()
  File "/home/xu/.local/lib/python3.8/site-packages/torch/cuda/streams.py", line 179, in record
    stream = torch.cuda.current_stream()
  File "/home/xu/.local/lib/python3.8/site-packages/torch/cuda/__init__.py", line 591, in current_stream
    _lazy_init()
  File "/home/xu/.local/lib/python3.8/site-packages/torch/cuda/__init__.py", line 229, in _lazy_init
    torch._C._cuda_init()
RuntimeError: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx

Can you please try https://github.com/pytorch/benchmark/pull/1415 fixes your problem?

Verified as solved, thanks.