zhiminy/EvalEng huggingface dataset:
Repository: pytorch/benchmark
Issue: RuntimeError When Enabling Accuracy Checks in yolov3 Training on GPU.
Root cause: Interface Contract Mismatch

Issue Description
I encounter a RuntimeError related to gradient computation when enabling accuracy checks during the training of yolov3 in a GPU docker environment. The training runs without issues when the --accuracy flag is not used.

Steps to Reproduce
python install.py yolov3
python run.py yolov3 -d cuda -t train --accuracy

Expected Behavior
The training process should run without errors and perform accuracy checks without causing runtime errors.

Actual Behavior
The script executes successfully without the --accuracy flag.
However, when the accuracy check is enabled, it fails with the following error message:
 
 TypeError: Darknet.forward() takes from 2 to 4 positional arguments but 6 were given
Running train method from yolov3 on cuda in eager mode with input batch size 4 and precision fp32.

env：pytorch-cuda=12.1   python=3.11

The following command will pass the test:

```
$ python run_benchmark.py dynamo --only yolov3 --accuracy --training --amp --backend=inductor
loading model: 0it [00:02, ?it/s]
cuda train yolov3
W0619 23:26:48.274000 139707225552704 torch/_inductor/utils.py:1219] [9/0] DeviceCopy in input program
W0619 23:26:48.275000 139707225552704 torch/_inductor/utils.py:1219] [9/0] DeviceCopy in input program
skipping cudagraphs due to skipping cudagraphs due to cpu device (primals_2). Found from :
   File "/home/xz/git/benchmark/torchbenchmark/models/yolov3/yolo_models.py", line 188, in forward
    self.create_grids((nx, ny), p.device)
  File "/home/xz/git/benchmark/torchbenchmark/models/yolov3/yolo_models.py", line 159, in create_grids
    self.anchor_vec = self.anchor_vec.to(device)

W0619 23:26:52.032000 139707225552704 torch/_inductor/utils.py:1219] [9/1] DeviceCopy in input program
W0619 23:26:52.032000 139707225552704 torch/_inductor/utils.py:1219] [9/1] DeviceCopy in input program
skipping cudagraphs due to skipping cudagraphs due to cpu device (primals_4). Found from :
   File "/home/xz/git/benchmark/torchbenchmark/models/yolov3/yolo_models.py", line 188, in forward
    self.create_grids((nx, ny), p.device)
  File "/home/xz/git/benchmark/torchbenchmark/models/yolov3/yolo_models.py", line 159, in create_grids
    self.anchor_vec = self.anchor_vec.to(device)

W0619 23:26:54.131000 139707225552704 torch/_logging/_internal.py:1034] [13/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
pass
```

Looking into why the result is different in `run.py` runner.

Fix: https://github.com/pytorch/benchmark/pull/2321
Can you help verify it fixes the issue?

> Fix: #2321 Can you help verify it fixes the issue?

Thank you. I tested with your fix commit, and the accuracy test is passing.
Repository: pytorch/benchmark
Issue: torchbench: torch.no_grad() not working for dynamo inductor backend
Root cause: Interface Contract Mismatch

torch dynamo inductor backend is not seeing the` torch.no_grad()` setting from torchbench framework.

This is the reproducer
`python run_benchmark.py cpu --model hf_Bert --test eval --torchdynamo inductor`

Here is the code snippet from torchbench that sets the no_grad()
https://github.com/pytorch/benchmark/blob/main/torchbenchmark/util/framework/huggingface/model_factory.py#L120
```
with torch.no_grad():
            with self.amp_context():
                out = self.model(**self.example_inputs)
```

but the torch dynamo inductor backend isn't seeing the grad disabled, and hence is not triggering the graph level optimizations.
https://github.com/pytorch/pytorch/blob/v2.3.0/torch/_inductor/compile_fx.py#L1270


I found this stackover discussion, but not sure if it's related.
https://stackoverflow.com/questions/75022490/pytorch-torch-no-grad-doesnt-affect-modules

I can reproduce it using a toy module, reported to inductor upstream

great, thanks for reproducing it!
I had raised the [issue](https://github.com/pytorch/pytorch/issues/125236) on pytorch repo too.





The bug is confirmed by upstream Inductor developers but there is no quick way to fix it. I believe the best shot is to move the `no_grad()` context from model level to the framework level. I will work on that.
Repository: pytorch/benchmark
Issue: hf_Bert_large failed
Root cause: Validation Gap

Hi 

I can't run the hf_Bert_large model on my local site, I was running it from pytorch repo and my command is:
./benchmarks/dynamo/torchbench.py --performance --inference --bfloat16 --backend=inductor --only=hf_Bert_large --output=torchbench_inference.csv

the error message is:
![image](https://github.com/pytorch/benchmark/assets/146043398/35ec85ff-a378-454e-9694-7dff4e1eb3cf)

looks something wrong with the input. 

I found that this model is good to run on dashboard, 
https://hud.pytorch.org/benchmark/torchbench/inductor_max_autotune?startTime=Mon,%2006%20May%202024%2010:12:38%20GMT&stopTime=Mon,%2013%20May%202024%2010:12:38%20GMT&granularity=hour&mode=training&model=hf_Bert_large&dtype=amp&lBranch=main&lCommit=02093b6c6ae1046368e2500881d0bb5880873386&rBranch=main&rCommit=68a1f787c8ce0783f39c35a91fa1e6614b974050

anybody can help to resolve this issue?  many thanks.

Can you try this version of transformers? `transformers==4.38.1`

@FindHao I checked my environment and make sure that the version of transformers is 4.38.1,  and the issue is still exists.

@FindHao BTW, I tried to build a docker with the dockerfile from the repo and ran the same test, the model hf_Bert_large still failed with the same issue, looks the environment to run the dashboard is different?


The torchbench in dynamo is slightly different from this repo. You can try the above PR I submitted to solve this issue. Or, you can run with this repo's `run.py` in your tests. 

@FindHao Thanks for the fixing, it works for inference, there is another similar bug in pytorch benchmark repo: https://github.com/pytorch/pytorch/blob/main/benchmarks/dynamo/torchbench.py#L435
![image](https://github.com/pytorch/benchmark/assets/146043398/7aea9fa7-694b-42de-8ae5-29cef1d22d99)

I think it should be "if isinstance(cloned_inputs, dict):" , which looks like a typo,  and the hf model training will fail because of this. 

thanks


Thanks for your reply! It is a typo. Fixed in https://github.com/pytorch/pytorch/pull/126981
Repository: pytorch/benchmark
Issue: Exported HF models contain SoftmaxCrossEntropyLoss node
Root cause: Validation Gap

I notice that using the torchscript exporter, ie using the `--torchscript-onnx` flag, all the exported HF models contain a `SoftmaxCrossEntropyLoss` node at the end that compares the model output with the true labels. Why is training-related ops showing up in the exported model and is there a way to disable such ops?

<img width="353" alt="image" src="https://github.com/pytorch/benchmark/assets/99693700/85ecef39-4a6f-4472-9472-86397a299885">

The export command I'm using is:
```
python pytorch/benchmarks/dynamo/huggingface.py --performance --amp -dcuda --output=/workspace/output/dynamo-onnx_huggingface_amp_inference_cuda_performance.csv --inference --use-eval-mode -n1 --torchscript-onnx --no-skip --dashboard -k <model_name> 
```

I think this is why https://github.com/pytorch/pytorch/blob/8184cd85fcfe663019edb3c1e502e03dcbaba4f0/benchmarks/dynamo/common.py#L3785-L3787C14

Maybe you would want to hack around here to make it eval mode: https://github.com/pytorch/pytorch/blob/01694eaa56adb343f5d3d15b53d2962615dafe17/benchmarks/dynamo/huggingface.py#L513-L520

> Maybe you would want to hack around here to make it eval mode: https://github.com/pytorch/pytorch/blob/01694eaa56adb343f5d3d15b53d2962615dafe17/benchmarks/dynamo/huggingface.py#L513-L520

I think it is the eval path that's being run, since `is_training` is still False. One can also replace `--performance` with `--accuracy` so that `args.use_eval_mode` doesn't get overwritten to False. But the exported model still contains a SoftmaxCrossEntropyLoss node.

We found it's this: https://github.com/pytorch/pytorch/blob/a1ca4dfe0ba65de3a8293aabd9e25504f4771e94/benchmarks/dynamo/huggingface.py#L435
Repository: pytorch/benchmark
Issue: tts_angular failed at training
Root cause: External Dependency Breakage

Hi, the tts_angular failed at training with the following error message:
![image](https://github.com/pytorch/benchmark/assets/146043398/e2f6b0cb-d4ce-47b7-a444-621a7c34f21d)

Looks it's also failed in the dashboard, what's the root cause of this issue? any plan to make it work? 

Thanks

I can reproduce this with the command:

```
$ python run_benchmark.py dynamo --only tts_angular --performance --training --amp --backend=inductor
```

This should be an issue of PT2 (TorchInductor) rather than the benchmark itself. Please report this issue to `pytorch/pytorch` and add inductor tag.
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '2.4.0.dev20240601+cu121'
Root cause: Others


TorchBench CI has detected a performance signal or runtime regression, and bisected its result.

Control PyTorch commit: 1485621ccbb5367936ba6da51ba896387416c353
Control PyTorch version: 2.4.0.dev20240515+cu121

Treatment PyTorch commit: 121c55d8d12a878b12eab00a7cebae2e2fa47ee7
Treatment PyTorch version: 2.4.0.dev20240601+cu121

Bisection result:

```
[
    {
        "commit1": "699db7988d8",
        "commit1_time": "2024-05-29 11:58:03 +0000",
        "commit1_digest": {
            "name": "torch-nightly",
            "environ": {
                "pytorch_git_version": "699db7988d84d163ebb6919f78885e4630182a7a",
                "pytorch_version": "2.4.0a0+git699db79",
                "device": "NVIDIA A100-SXM4-40GB",
                "git_commit_hash": "699db7988d84d163ebb6919f78885e4630182a7a"
            },
            "metrics": {
                "test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_latency": 23.396359,
                "test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_cmem": 1.2490234375,
                "test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_gmem": 2.95623779296875
            }
        },
        "commit2": "749a132fb0a",
        "commit2_time": "2024-05-29 12:09:27 +0000",
        "commit2_digest": {
            "name": "torch-nightly",
            "environ": {
                "pytorch_git_version": "749a132fb0a8325cbad4734a563aa459ca611991",
                "pytorch_version": "2.4.0a0+git749a132",
                "device": "NVIDIA A100-SXM4-40GB",
                "git_commit_hash": "749a132fb0a8325cbad4734a563aa459ca611991"
            },
            "metrics": {
                "test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_latency": 26.862159,
                "test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_cmem": 1.2587890625,
                "test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_gmem": 2.95623779296875
            }
        }
    }
]
```

cc @xuzhao9


Noisy result, closed
Repository: pytorch/benchmark
Issue: CI Failure bisection
Root cause: External Dependency Breakage

Success: 20240515 (e5cd7bea0283f82d32097626f86475fd9c127354)
Fail: 20240606 (529bedad3e6cbd98387b9e46ce0c0e290ed12ab5)

Userbenchmark name: test_bench

Userbenchmark args: -m detectron2_maskrcnn_r_50_fpn --accuracy

python run.py detectron2_maskrcnn_r_50_fpn -d cuda -t eval --accuracy


Test workflow: https://github.com/pytorch/benchmark/actions/runs/9454579616
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '2.4.0.dev20240610+cu121'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression, and bisected its result.

Control PyTorch commit: 65aa16f968af2cd18ff8c25cc657e7abda594bfc
Control PyTorch version: 2.4.0.dev20240607+cu121

Treatment PyTorch commit: df43d5843edd9abd95d2c039670bce51375d9c06
Treatment PyTorch version: 2.4.0.dev20240610+cu121

Bisection result:

```
[
    {
        "commit1": "917387f66d5",
        "commit1_time": "2024-06-08 13:23:49 +0000",
        "commit1_digest": {
            "name": "torch-nightly",
            "environ": {
                "pytorch_git_version": "917387f66d59a198b0f8313ddded4913d94d22d6",
                "pytorch_version": "2.4.0a0+git917387f",
                "device": "NVIDIA A100-SXM4-40GB",
                "git_commit_hash": "917387f66d59a198b0f8313ddded4913d94d22d6"
            },
            "metrics": {
                "test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_latency": 23.178642,
                "test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_cmem": 1.255859375,
                "test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_gmem": 2.95623779296875
            }
        },
        "commit2": "348b181a97a",
        "commit2_time": "2024-06-08 15:25:03 +0000",
        "commit2_digest": {
            "name": "torch-nightly",
            "environ": {
                "pytorch_git_version": "348b181a97abc2e636a6c18e5880a78e5d1dab94",
                "pytorch_version": "2.4.0a0+git348b181",
                "device": "NVIDIA A100-SXM4-40GB",
                "git_commit_hash": "348b181a97abc2e636a6c18e5880a78e5d1dab94"
            },
            "metrics": {
                "test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_latency": 26.773922,
                "test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_cmem": 1.263671875,
                "test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_gmem": 2.95623779296875
            }
        }
    }
]
```

cc @xuzhao9


The PR is reverted, closing
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '2.5.0.dev20240613+cu124'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal or runtime regression, and bisected its result.

Control PyTorch commit: df43d5843edd9abd95d2c039670bce51375d9c06
Control PyTorch version: 2.4.0.dev20240610+cu124

Treatment PyTorch commit: d630e1e838e7416b6e27572a91407c4447a00734
Treatment PyTorch version: 2.5.0.dev20240613+cu124

Bisection result:

```
[
    {
        "commit1": "46a35a1ed4f",
        "commit1_time": "2024-06-12 10:40:23 +0000",
        "commit1_digest": {
            "name": "torch-nightly",
            "environ": {
                "pytorch_git_version": "46a35a1ed4fe0b13a2730043318ce77f1aa02542",
                "pytorch_version": "2.4.0a0+git46a35a1",
                "device": "NVIDIA A100-SXM4-40GB",
                "git_commit_hash": "46a35a1ed4fe0b13a2730043318ce77f1aa02542"
            },
            "metrics": {
                "test_eval[doctr_det_predictor-cuda-eager]_latency": 52.614842,
                "test_eval[doctr_det_predictor-cuda-eager]_cmem": 0.99609375,
                "test_eval[doctr_det_predictor-cuda-eager]_gmem": 3.49334716796875
            }
        },
        "commit2": "2e065f2486e",
        "commit2_time": "2024-06-12 10:49:16 +0000",
        "commit2_digest": {
            "name": "torch-nightly",
            "environ": {
                "pytorch_git_version": "2e065f2486e92d8b89b058f1d561bb801f226179",
                "pytorch_version": "2.4.0a0+git2e065f2",
                "device": "NVIDIA A100-SXM4-40GB",
                "git_commit_hash": "2e065f2486e92d8b89b058f1d561bb801f226179"
            },
            "metrics": {
                "test_eval[doctr_det_predictor-cuda-eager]_latency": 58.452785,
                "test_eval[doctr_det_predictor-cuda-eager]_cmem": 0.9951171875,
                "test_eval[doctr_det_predictor-cuda-eager]_gmem": 3.49334716796875
            }
        }
    }
]
```

cc @xuzhao9
Repository: pytorch/benchmark
Issue: fixture 'compiler' not found while executing "pytest test_bench.py"
Root cause: Validation Gap

I'm profiling the execution of BERT_pytorch with `test_bench.py` using PyTorch 2.0.1 with CUDA 11.7, in which the benchmark keeps updated to the [latest commit](https://github.com/pytorch/benchmark/commit/5f6633d3fbd7b8341f1314a3d87a0137eb8d9632). Scripts listed in README worked perfectly until I tried `test_bench.py`.

![image](https://github.com/pytorch/benchmark/assets/68550432/f7ae7dbe-fb2a-42ff-86ab-7e0982f70a08)

Python 3.10.12, and here are the packages I installed: [requirements.txt](https://github.com/user-attachments/files/15902499/requirements.txt)

Perhaps I missed something. Thanks!

Hi @yuxuan-z19, can you please test on https://github.com/pytorch/benchmark/pull/2320 ?

Thanks @xuzhao9 , it's working!
Repository: pytorch/benchmark
Issue: `DALLE2_pytorch` moved to `canary_models` due to NumPy 2.0 upgrade.
Root cause: External Dependency Breakage

In #2311, `DALLE2_pytorch` benchmark was moved into `canary_models` due to the NumPy 2.0 upgrade. I'm opening this issue for tracking the progress towards moving it back to the `models` folder.

We remove `DALLE2_pytorch` because the upstream model depends on `embedding-reader`, which further depends on numpy<2: https://github.com/rom1504/embedding-reader/blob/a4fd55830a502685600ed8ef07947cd1cb92b083/requirements.txt#L3

We don't have plan to add numpy>=2 support to embedding-reader. Therefore, we can't really add back this model  unless upstream repo fixes this issue.
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '2.5.0.dev20240627+cu124'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression, and bisected its result.

Control PyTorch commit: 00f675bb4c2ec02bb5ffecfc75571026e220701c
Control PyTorch version: 2.5.0.dev20240626+cu124

Treatment PyTorch commit: cdbd6542d015bae9b850602834c4f22e3001084b
Treatment PyTorch version: 2.5.0.dev20240627+cu124

Bisection result:

```
[
    {
        "commit1": "c4dd752d97",
        "commit1_time": "2024-06-25 01:31:26 +0000",
        "commit1_digest": {
            "name": "torch-nightly",
            "environ": {
                "pytorch_git_version": "c4dd752d9763d9c306ffee5c7ada749294c9ba31",
                "pytorch_version": "2.5.0a0+gitc4dd752",
                "device": "NVIDIA A100-SXM4-40GB",
                "git_commit_hash": "c4dd752d9763d9c306ffee5c7ada749294c9ba31"
            },
            "metrics": {
                "test_eval[maml_omniglot-cuda-eager]_latency": 1.210188,
                "test_eval[maml_omniglot-cuda-eager]_cmem": 0.5537109375,
                "test_eval[maml_omniglot-cuda-eager]_gmem": 1.53826904296875
            }
        },
        "commit2": "533395e204",
        "commit2_time": "2024-06-25 02:39:13 +0000",
        "commit2_digest": {
            "name": "torch-nightly",
            "environ": {
                "pytorch_git_version": "533395e2043e260e5a0124a45b83049c98919e60",
                "pytorch_version": "2.5.0a0+git533395e",
                "device": "NVIDIA A100-SXM4-40GB",
                "git_commit_hash": "533395e2043e260e5a0124a45b83049c98919e60"
            },
            "metrics": {
                "test_eval[maml_omniglot-cuda-eager]_latency": 1.024718,
                "test_eval[maml_omniglot-cuda-eager]_cmem": 0.5537109375,
                "test_eval[maml_omniglot-cuda-eager]_gmem": 1.53826904296875
            }
        }
    }
]
```

cc @xuzhao9
Repository: pytorch/benchmark
Issue: Use ARC Kubernetes mode to run benchmark
Root cause: Unimplemented Feature Gap

ARC Kubernetes mode has the highest security level among all runner modes.
We can keep using CPU-only instance for docker building, but all GPU instances should use ARC Kubernetes mode.

Reference: https://www.stepsecurity.io/blog/how-to-use-docker-in-actions-runner-controller-runners-securelly


On PyTorch, we decided to stop ARC migration.
Is this still relevant for TorchBench?

Yes, it is still relevant for TorchBench, TB uses ARC runners on GCP K8s cluster for PR and nightly testing: https://github.com/pytorch/benchmark/tree/main/docker/infra

If we migrate all OSS workflows to Internal, or if pytorch CI provides capacity for GCP A100 runners, we can also deprecate the ARC runner setup.
Repository: pytorch/benchmark
Issue: Flop counter crashes with multiple compiled benchmarks
Root cause: Interface Contract Mismatch

When running several benchmarks with `run.py` in compile mode and attempting to collect flops metrics, the flop counter crashes complaining about multiple values for keyword argument 'out'.

Examples:

command:
`python benchmark/run.py hf_T5 -t train --metrics model_flops,cpu_peak_mem,ttfb --torchdynamo inductor`

output:
```
flop_count = flop_count_func(*args, **kwargs, out=out)  # type: ignore[operator]
TypeError: torch.utils.flop_counter.mm_flop() got multiple values for keyword argument 'out'
```

command:
`python benchmark/run.py torchrec_dlrm -t eval --metrics model_flops,cpu_peak_mem,ttfb --torchdynamo inductor`

output:
```
flop_count = flop_count_func(*args, **kwargs, out=out)  # type: ignore[operator]
TypeError: torch.utils.flop_counter.addmm_flop() got multiple values for keyword argument 'out'
```

Tests impacted include:
hf_GPT2_large, hf_T5, hf_T5_base, hf_T5_large, timm_vision_transformer_large, hf_GPT2_generate, hf_Yi, orca_2

It should be noted that metrics are collected fine if `model_flops` is not included.


Tested on:
Ubuntu 22.04.4 LTS
Python 3.10.12
torch 2.3.0+cpu
torchaudio 2.3.0+cpu
torchvision 0.18.0+cpu



We are using the `__torch_dispatch__` based flops counter: https://dev-discuss.pytorch.org/t/the-ideal-pytorch-flop-counter-with-torch-dispatch/505.

I thought it works only in the eager mode and does not work with inductor, can @Chillee help confirm? 

Yeah, we need to disable `torch.compile` when running under the flop counter I think.

@JasonMts Can you help try https://github.com/pytorch/benchmark/pull/2390 to see if it fixes the problem?
Repository: pytorch/benchmark
Issue: sam_fast model memory leak
Root cause: External Dependency Breakage

https://github.com/pytorch/benchmark/actions/runs/10168992530/job/28133362873

good: 2.5.0.dev20240729+cu124 (500aea8d5033fd3540c6ed325dd80e7e1420b0f3)
bad: torch: 2.5.0.dev20240730+cu124 (05a8540041cea936a63355c2e38b7b3beb5ce168)
bisect userbenchmark: test_bench
arguments: -m sam_fast -t eval --memleak

Bisection workflow: https://github.com/pytorch/benchmark/actions/runs/10173197195

```
{
  "target_repo": "pytorch",
  "start": "500aea8d5033fd3540c6ed325dd80e7e1420b0f3",
  "start_version": "2.5.0.dev20240730+cu124",
  "end": "05a8540041cea936a63355c2e38b7b3beb5ce168",
  "end_version": "2.5.0.dev20240730+cu124",
  "result": [
    {
      "commit1": "4c2bcf92cb",
      "commit1_time": "2024-07-29 19:19:54 +0000",
      "commit1_digest": {
        "name": "test_bench",
        "environ": {
          "pytorch_git_version": "4c2bcf92cbecd36b7881904bceb8dc50c9b9741d",
          "pytorch_version": "2.5.0a0+git4c2bcf9",
          "device": "NVIDIA A100-SXM4-40GB",
          "git_commit_hash": "4c2bcf92cbecd36b7881904bceb8dc50c9b9741d"
        },
        "metrics": {
          "model=sam_fast, test=eval, device=cuda, bs=None, extra_args=['--memleak'], metric=memleak": "False"
        }
      },
      "commit2": "f44446e851",
      "commit2_time": "2024-07-29 20:01:51 +0000",
      "commit2_digest": {
        "name": "test_bench",
        "environ": {
          "pytorch_git_version": "f44446e851294569fabb1b7f354618a33e06a75e",
          "pytorch_version": "2.5.0a0+gitf44446e",
          "device": "NVIDIA A100-SXM4-40GB",
          "git_commit_hash": "f44446e851294569fabb1b7f354618a33e06a75e"
        },
        "metrics": {
          "model=sam_fast, test=eval, device=cuda, bs=None, extra_args=['--memleak'], metric=memleak": "True"
        }
      }
    }
  ]
}
```

sam_fast uses torch.compile: https://github.com/pytorch-labs/segment-anything-fast/blob/main/segment_anything_fast/build_sam.py#L70

Root cause: https://github.com/pytorch/pytorch/commit/f44446e851

Confirmed by PR author, but will take ~1 week to fix

cc @mlazos @bdhirsh if this could be related to `tensor_dict` holding on to references.