zhiminy/EvalEng huggingface dataset:
Repository: pytorch/benchmark
Issue: Failed TorchAO nightly workflow
Root cause: Configuration Error

https://github.com/pytorch/benchmark/actions/runs/10203374784/job/28229575495 failed because of missing the newly added config `huggingface.yaml`

How do commits get from pytorch to benchmark?

I see that the commit was copied https://github.com/pytorch/benchmark/commit/0dc03849bd93b4584a715239d5f55789e9ed214c , but it misses YAML from the original commit https://github.com/pytorch/pytorch/commit/8458980bbf78714a0fbe703785c100cad523fade

I will probably just add the YAML separately now.

Internally we have ShipIt config to sync between pytorch/benchmarks/dynamo and benchmark/userbenchmark/dynamo/dynamobench, we need to change the config to sync the new files or sync the whole directory.
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '2.5.0.dev20240805+cu124'
Root cause: Others


TorchBench CI has detected a performance signal or runtime regression, and bisected its result.

Control PyTorch commit: f0da167ce55ddbbe4f0138ed0f0804008069ddad
Control PyTorch version: 2.5.0.dev20240802+cu124

Treatment PyTorch commit: c7cfa5172139737bf75afbd4a7920b1a02b1dcb2
Treatment PyTorch version: 2.5.0.dev20240805+cu124

Bisection result:

```
[
    {
        "commit1": "159d508f03",
        "commit1_time": "2024-08-03 08:07:21 +0000",
        "commit1_digest": {
            "name": "torch-nightly",
            "environ": {
                "pytorch_git_version": "159d508f03d40a2df3b0a2f3855afdc61e22b2cb",
                "pytorch_version": "2.5.0a0+git159d508",
                "device": "NVIDIA A100-SXM4-40GB",
                "git_commit_hash": "159d508f03d40a2df3b0a2f3855afdc61e22b2cb"
            },
            "metrics": {
                "test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_latency": 24.23883,
                "test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_cmem": 1.28125,
                "test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_gmem": 2.82342529296875
            }
        },
        "commit2": "0e7e61f7ce",
        "commit2_time": "2024-08-03 09:43:38 +0000",
        "commit2_digest": {
            "name": "torch-nightly",
            "environ": {
                "pytorch_git_version": "0e7e61f7cec82a43f2de52b83eff152d703be7a3",
                "pytorch_version": "2.5.0a0+git0e7e61f",
                "device": "NVIDIA A100-SXM4-40GB",
                "git_commit_hash": "0e7e61f7cec82a43f2de52b83eff152d703be7a3"
            },
            "metrics": {
                "test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_latency": 27.1415,
                "test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_cmem": 1.2880859375,
                "test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_gmem": 2.82342529296875
            }
        }
    }
]
```

cc @xuzhao9


Instable test, closing

<img width="1437" alt="image" src="https://github.com/user-attachments/assets/d58e4b48-7751-46b2-a8cb-007a904a79e2">
Repository: pytorch/benchmark
Issue: [tritonbench] A better way to encode input shapes with parameters
Root cause: Unimplemented Feature Gap

Right now developers need to manually specify the input shape metadata to generate the input tensors.
I am thinking if we could use decorators to make this process easier and more user-friendly.

Old style:


```
def get_input_iter(self) -> Generator:
    shape_a = range(x, y)
    shape_b = range(m, n)
    for a in shape_a:
        for b in shape_b:
            yield generate_input(a, b)
```

New style:

```
@given(
        a=st.integers(min_value=x, max_value=y),
        b=st.integers(min_value=m, max_value=n),
)
def get_input(self, a, b):
      return generate_input(a, b)
```
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '2.5.0.dev20240816+cu124'
Root cause: Others


TorchBench CI has detected a performance signal or runtime regression, and bisected its result.

Control PyTorch commit: 3965f118376a01382b0d1d3eea612188c4847ec8
Control PyTorch version: 2.5.0.dev20240815+cu124

Treatment PyTorch commit: 29c4b4ea5a23ec41b98def0f50a0294c0bdc5c18
Treatment PyTorch version: 2.5.0.dev20240816+cu124

Bisection result:

```
[
    {
        "commit1": "7470ae85e4",
        "commit1_time": "2024-08-15 16:38:26 +0000",
        "commit1_digest": {
            "name": "torch-nightly",
            "environ": {
                "pytorch_git_version": "7470ae85e4b090c6b331242a978ec025f03e537c",
                "pytorch_version": "2.5.0a0+git7470ae8",
                "device": "NVIDIA A100-SXM4-40GB",
                "git_commit_hash": "7470ae85e4b090c6b331242a978ec025f03e537c"
            },
            "metrics": {
                "test_eval[doctr_det_predictor-cuda-eager]_latency": 57.147894,
                "test_eval[doctr_det_predictor-cuda-eager]_cmem": 1.029296875,
                "test_eval[doctr_det_predictor-cuda-eager]_gmem": 3.49334716796875
            }
        },
        "commit2": "3434a54fba",
        "commit2_time": "2024-08-15 16:41:51 +0000",
        "commit2_digest": {
            "name": "torch-nightly",
            "environ": {
                "pytorch_git_version": "3434a54fba537f67d152b50142b211d4aa059e67",
                "pytorch_version": "2.5.0a0+git3434a54",
                "device": "NVIDIA A100-SXM4-40GB",
                "git_commit_hash": "3434a54fba537f67d152b50142b211d4aa059e67"
            },
            "metrics": {
                "test_eval[doctr_det_predictor-cuda-eager]_latency": 48.885768,
                "test_eval[doctr_det_predictor-cuda-eager]_cmem": 0.9404296875,
                "test_eval[doctr_det_predictor-cuda-eager]_gmem": 3.49334716796875
            }
        }
    }
]
```

cc @xuzhao9


This is an unstable test:

<img width="1437" alt="image" src="https://github.com/user-attachments/assets/7ecb1c51-6aba-4da3-84a8-dba7edc06eda">
Repository: pytorch/benchmark
Issue: Fix build and installation error due to maskrcnn benchmark
Root cause: Validation Gap

- `python test.py` fails to run due to missing `model` attr from `maskrcnn_benchmark` 
- `python install` also fails to install due to `maskrcnn_benchmark`

Instead of aborting the entire script, this fix wll printout a WARNING message and skip the benchmark, so that the rest of the benchmarks may run. 
```
Warning: <module 'torchbenchmark.models.maskrcnn_benchmark' (namespace)> does not define attribute Model, skip it
```

In addition, improved the error message for `python install --continue_on_fail`. After the fix, the error message is:
```
Warning: some benchmarks were not installed due to failure
```

Fixed by #244
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.6.0.dev20240914+cu124'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.4.0.dev20240606+cu121

Base PyTorch commit: ffaea656b5d8ff6518669494cc8f664b94f8e8b1

Affected PyTorch version: 2.6.0.dev20240914+cu124

Affected PyTorch commit: 081c4a966de77dd5187c1cb97d287cae6f3fbc02

Affected Tests:
- test_train[Super_SloMo-cuda-eager]: +14.62073%
- test_eval[Super_SloMo-cuda-eager]: +11.06524%
- test_train[Super_SloMo-cuda-jit]: +14.75104%
- test_eval[Super_SloMo-cuda-jit]: +11.22865%
- test_eval[alexnet-cuda-eager]: +10.52647%
- test_eval[alexnet-cuda-jit]: +13.86864%
- test_train[dcgan-cuda-eager]: +63.60124%
- test_train[dcgan-cuda-jit]: +63.60849%
- test_eval[pytorch_stargan-cpu-eager]: -13.51456%
- test_eval[shufflenet_v2_x1_0-cpu-eager]: +14.21934%
- test_eval[shufflenet_v2_x1_0-cuda-jit]: -27.54311%
- test_train[timm_nfnet-cpu-jit]: -13.11850%
- test_eval[timm_regnet-cuda-eager]: +170.74196%
- test_eval[timm_regnet-cuda-jit]: +355.22499%
- test_train[tts_angular-cpu-eager]: +12.80220%
- test_train[yolov3-cuda-eager]: -9.78467%


cc @xuzhao9

Result json: 
```
{
  "start": "ffaea656b5d8ff6518669494cc8f664b94f8e8b1",
  "end": "081c4a966de77dd5187c1cb97d287cae6f3fbc02",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "fedae41c57b",
      "commit1_time": "2024-07-16 06:55:46 +0000",
      "commit1_digest": {
        "test_train[tts_angular-cpu-eager]": 0.23825253788381814
      },
      "commit2": "096dc444cee",
      "commit2_time": "2024-07-16 08:39:00 +0000",
      "commit2_digest": {
        "test_train[tts_angular-cpu-eager]": 0.21400512969121338
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/10864154281
Repository: pytorch/benchmark
Issue: Bug in torch_multimodal_clip: IndexError: list index out of range
Root cause: Algorithmic Error

Hi, I was testing some of torchbench models and I found a simple bug that can be resolved.

**Model**: torch_multimodal_clip

**Reproducing steps**:
1. python3 install.py model torch_multimodal_clip
2. python3 test.py -v -k "test_torch_multimodal_clip_eval_cuda"

**Error**:

Running tests...
----------------------------------------------------------------------
  test_torch_multimodal_clip_eval_cuda (__main__.TestBenchmark) ... ERROR (9.481s)

======================================================================
ERROR [9.481s]: test_torch_multimodal_clip_eval_cuda (__main__.TestBenchmark)
----------------------------------------------------------------------
torchbenchmark._components._impl.workers.subprocess_rpc.ChildTraceException: Traceback (most recent call last):
  File "/root/torchbench/torchbenchmark/_components/_impl/workers/subprocess_rpc.py", line 482, in _run_block
    exec(  # noqa: P204
  File "<subprocess-worker>", line 2, in <module>
  File "/root/torchbench/torchbenchmark/util/model.py", line 317, in invoke
    out = self.eval()
  File "/root/torchbench/torchbenchmark/models/torch_multimodal_clip/__init__.py", line 91, in eval
    return self.text[torch.argmax(score)]
IndexError: list index out of range

**Solution**:
The problem is inside __init__.py script. Default batch size is equal to 32, so the product of matrix multiplication returned as score is size [32,32]. In my opinion the goal is to find the highest score in each batch, but the method torch.argmax(score) returns global index  (960). I think there's a missing argument of dim=1 to focus on each batch.
Moreover, after getting index or rather indices, in next step we want to get values from self.text field. I assume that we want to refer to self.texts according to proper batch_size.

I propose to change some lines in eval() method:

```
def eval(self):
        self.model.eval()
        image_embedding, text_embedding = self.model(
            self.image_tensor, self.text_tensor
        )
        score = image_embedding @ text_embedding.t()

        return self.text[torch.argmax(score)]
```
to
```
def eval(self):
        self.model.eval()
        image_embedding, text_embedding = self.model(
            self.image_tensor, self.text_tensor
        )
        score = image_embedding @ text_embedding.t()

        indices = torch.argmax(score, dim=1)
        return [self.texts[i][indices[i].item()] for i in range(self.batch_size)]
```

This works for any batch_size:

Running tests...
----------------------------------------------------------------------
  test_torch_multimodal_clip_eval_cuda (__main__.TestBenchmark) ... ok (9.999s)

----------------------------------------------------------------------
Ran 1 test in 9.999s

OK
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.6.0.dev20240917+cu124'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.4.0.dev20240606+cu121

Base PyTorch commit: ffaea656b5d8ff6518669494cc8f664b94f8e8b1

Affected PyTorch version: 2.6.0.dev20240917+cu124

Affected PyTorch commit: 37a08b33bb86ad0fce3db6fc98f90c9e02d77cba

Affected Tests:
- test_train[Super_SloMo-cuda-eager]: +14.62766%
- test_eval[Super_SloMo-cuda-eager]: +11.07496%
- test_train[Super_SloMo-cuda-jit]: +14.74094%
- test_eval[Super_SloMo-cuda-jit]: +11.20760%
- test_eval[alexnet-cuda-eager]: +10.33393%
- test_eval[alexnet-cuda-jit]: +13.68206%
- test_train[dcgan-cuda-eager]: +63.62549%
- test_train[dcgan-cuda-jit]: +63.68281%
- test_eval[pyhpc_equation_of_state-cpu-jit]: +10.55933%
- test_eval[pytorch_stargan-cpu-eager]: -14.15109%
- test_eval[shufflenet_v2_x1_0-cpu-eager]: +12.14505%
- test_eval[shufflenet_v2_x1_0-cuda-jit]: -27.39666%
- test_eval[timm_regnet-cuda-eager]: +169.47718%
- test_eval[timm_regnet-cuda-jit]: +356.91370%
- test_train[yolov3-cuda-eager]: -9.73554%


cc @xuzhao9

Result json: 
```
{
  "start": "ffaea656b5d8ff6518669494cc8f664b94f8e8b1",
  "end": "37a08b33bb86ad0fce3db6fc98f90c9e02d77cba",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "8759ed2ac50",
      "commit1_time": "2024-09-04 17:18:21 +0000",
      "commit1_digest": {
        "test_eval[shufflenet_v2_x1_0-cpu-eager]": 0.16593277025967837
      },
      "commit2": "7858045491f",
      "commit2_time": "2024-09-04 17:21:32 +0000",
      "commit2_digest": {
        "test_eval[shufflenet_v2_x1_0-cpu-eager]": 0.15055387476459145
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/10919529125
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.6.0.dev20240918+cu124'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.4.0.dev20240606+cu121

Base PyTorch commit: ffaea656b5d8ff6518669494cc8f664b94f8e8b1

Affected PyTorch version: 2.6.0.dev20240918+cu124

Affected PyTorch commit: 48d18fbd4cf785e1f69a6555d97a39023a5d199e

Affected Tests:
- test_train[Super_SloMo-cuda-eager]: +14.61260%
- test_eval[Super_SloMo-cuda-eager]: +11.06704%
- test_train[Super_SloMo-cuda-jit]: +14.74181%
- test_eval[Super_SloMo-cuda-jit]: +11.21563%
- test_eval[alexnet-cuda-eager]: +10.33117%
- test_eval[alexnet-cuda-jit]: +13.69378%
- test_train[dcgan-cuda-eager]: +63.62368%
- test_train[dcgan-cuda-jit]: +63.61366%
- test_eval[pyhpc_equation_of_state-cpu-jit]: +15.99175%
- test_eval[pytorch_stargan-cpu-eager]: -11.83703%
- test_eval[shufflenet_v2_x1_0-cpu-eager]: +10.94334%
- test_eval[shufflenet_v2_x1_0-cuda-jit]: -27.58940%
- test_eval[timm_regnet-cuda-eager]: +170.00818%
- test_eval[timm_regnet-cuda-jit]: +355.25202%
- test_train[yolov3-cuda-eager]: -9.95197%


cc @xuzhao9

Result json: 
```
{
  "start": "ffaea656b5d8ff6518669494cc8f664b94f8e8b1",
  "end": "48d18fbd4cf785e1f69a6555d97a39023a5d199e",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "5f7d9563620",
      "commit1_time": "2024-09-09 18:24:21 +0000",
      "commit1_digest": {
        "test_eval[pytorch_stargan-cpu-eager]": 0.2655519244261086
      },
      "commit2": "2032f107d7e",
      "commit2_time": "2024-09-09 19:07:48 +0000",
      "commit2_digest": {
        "test_eval[pytorch_stargan-cpu-eager]": 0.2415499879978597
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/10928523124
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.6.0.dev20240919+cu124'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.4.0.dev20240606+cu121

Base PyTorch commit: ffaea656b5d8ff6518669494cc8f664b94f8e8b1

Affected PyTorch version: 2.6.0.dev20240919+cu124

Affected PyTorch commit: e037bb326fdafca243bdb08023bbef93b29a4513

Affected Tests:
- test_train[Super_SloMo-cuda-eager]: +14.62943%
- test_eval[Super_SloMo-cuda-eager]: +11.07308%
- test_train[Super_SloMo-cuda-jit]: +14.75310%
- test_eval[Super_SloMo-cuda-jit]: +11.21929%
- test_eval[alexnet-cuda-eager]: +10.56490%
- test_eval[alexnet-cuda-jit]: +13.79772%
- test_train[dcgan-cuda-eager]: +63.66190%
- test_train[dcgan-cuda-jit]: +63.58292%
- test_eval[pyhpc_equation_of_state-cpu-jit]: +17.08329%
- test_eval[pytorch_stargan-cpu-eager]: -14.51072%
- test_eval[pytorch_stargan-cpu-jit]: -9.68358%
- test_eval[resnext50_32x4d-cpu-jit]: +26.96671%
- test_eval[shufflenet_v2_x1_0-cpu-eager]: +14.71172%
- test_eval[shufflenet_v2_x1_0-cuda-jit]: -27.58208%
- test_train[timm_nfnet-cpu-jit]: -13.35092%
- test_eval[timm_regnet-cuda-eager]: +169.96523%
- test_eval[timm_regnet-cuda-jit]: +355.28978%
- test_train[yolov3-cuda-eager]: -9.77666%


cc @xuzhao9

Result json: 
```
{
  "start": "ffaea656b5d8ff6518669494cc8f664b94f8e8b1",
  "end": "e037bb326fdafca243bdb08023bbef93b29a4513",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "187d55018ae",
      "commit1_time": "2024-08-20 16:12:04 +0000",
      "commit1_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.24741940470412374
      },
      "commit2": "8d93fe510e5",
      "commit2_time": "2024-08-20 16:16:30 +0000",
      "commit2_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.2774291805922985
      }
    },
    {
      "commit1": "8d93fe510e5",
      "commit1_time": "2024-08-20 16:16:30 +0000",
      "commit1_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.2774291805922985
      },
      "commit2": "33f1ee036ef",
      "commit2_time": "2024-08-20 16:27:44 +0000",
      "commit2_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.24583374997600912
      }
    },
    {
      "commit1": "33f1ee036ef",
      "commit1_time": "2024-08-20 16:27:44 +0000",
      "commit1_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.24583374997600912
      },
      "commit2": "c3d02fa3906",
      "commit2_time": "2024-08-20 16:33:26 +0000",
      "commit2_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.30004308177158234
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/10945734159
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.6.0.dev20240920+cu124'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.4.0.dev20240606+cu121

Base PyTorch commit: ffaea656b5d8ff6518669494cc8f664b94f8e8b1

Affected PyTorch version: 2.6.0.dev20240920+cu124

Affected PyTorch commit: fe0e9fb385d6e75a01c9feaa459252db09b71839

Affected Tests:
- test_train[Super_SloMo-cuda-eager]: +14.61819%
- test_eval[Super_SloMo-cuda-eager]: +11.07743%
- test_train[Super_SloMo-cuda-jit]: +14.76292%
- test_eval[Super_SloMo-cuda-jit]: +11.20991%
- test_eval[alexnet-cuda-eager]: +10.64854%
- test_eval[alexnet-cuda-jit]: +13.80642%
- test_train[dcgan-cuda-eager]: +63.63194%
- test_train[dcgan-cuda-jit]: +63.55866%
- test_eval[pyhpc_equation_of_state-cpu-jit]: +20.65562%
- test_eval[pytorch_stargan-cpu-eager]: -9.60231%
- test_eval[pytorch_stargan-cpu-jit]: -8.86550%
- test_eval[resnext50_32x4d-cpu-jit]: +22.49616%
- test_eval[shufflenet_v2_x1_0-cpu-eager]: +13.39576%
- test_eval[shufflenet_v2_x1_0-cuda-jit]: -27.56676%
- test_eval[timm_regnet-cuda-eager]: +169.96574%
- test_eval[timm_regnet-cuda-jit]: +355.17728%
- test_train[vision_maskrcnn-cpu-eager]: -7.12621%
- test_train[yolov3-cuda-eager]: -9.80941%


cc @xuzhao9

Result json: 
```
{
  "start": "ffaea656b5d8ff6518669494cc8f664b94f8e8b1",
  "end": "fe0e9fb385d6e75a01c9feaa459252db09b71839",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "3cb5d251224",
      "commit1_time": "2024-09-02 00:28:34 +0000",
      "commit1_digest": {
        "test_eval[pytorch_stargan-cpu-eager]": 0.26277237487956884
      },
      "commit2": "de3a641476c",
      "commit2_time": "2024-09-02 03:52:40 +0000",
      "commit2_digest": {
        "test_eval[pytorch_stargan-cpu-eager]": 0.23902534460648894
      }
    },
    {
      "commit1": "08184aa85cf",
      "commit1_time": "2024-08-30 16:52:28 +0000",
      "commit1_digest": {
        "test_eval[pytorch_stargan-cpu-eager]": 0.239402234274894
      },
      "commit2": "577a93514f1",
      "commit2_time": "2024-08-30 17:06:57 +0000",
      "commit2_digest": {
        "test_eval[pytorch_stargan-cpu-eager]": 0.25945915998890995
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/10963487492
Repository: pytorch/benchmark
Issue: PyTorch release benchmark suite Improvements
Root cause: Unimplemented Feature Gap

For PyTorch Releases we execute following benchmarks:
https://github.com/pytorch/benchmark/tree/main/userbenchmark/release-test 

These are the tests that we run:
```
# run mnist
mkdir -p "${RESULT_DIR}/mnist"
pushd "${EXAMPLES_DIR}/mnist"
export LOG_FILE=${RESULT_DIR}/mnist/result.log
export MEM_FILE=${RESULT_DIR}/mnist/result_mem.log
${PREFIX} bash "${CURRENT_DIR}/monitor_proc.sh" python main.py --epochs 10
# run mnist-hogwild
mkdir -p ${RESULT_DIR}/mnist_hogwild
pushd "${EXAMPLES_DIR}/mnist_hogwild"
export LOG_FILE=${RESULT_DIR}/mnist_hogwild/result.log
export MEM_FILE=${RESULT_DIR}/mnist_hogwild/result_mem.log
${PREFIX} bash "${CURRENT_DIR}/monitor_proc.sh" python main.py --epochs 10
# run CPU WLM LSTM
mkdir -p ${RESULT_DIR}/wlm_cpu_lstm
pushd "${EXAMPLES_DIR}/word_language_model"
export LOG_FILE=${RESULT_DIR}/wlm_cpu_lstm/result.log
export MEM_FILE=${RESULT_DIR}/wlm_cpu_lstm/result_mem.log
${PREFIX} bash "${CURRENT_DIR}/monitor_proc.sh" python main.py --epochs 10 --model LSTM
# run GPU WLM LSTM
mkdir -p ${RESULT_DIR}/wlm_gpu_lstm
pushd "${EXAMPLES_DIR}/word_language_model"
export LOG_FILE=${RESULT_DIR}/wlm_gpu_lstm/result.log
export MEM_FILE=${RESULT_DIR}/wlm_gpu_lstm/result_mem.log
${PREFIX} bash "${CURRENT_DIR}/monitor_proc.sh" python main.py --epochs 10 --model LSTM --cuda
# run CPU WLM Transformer
mkdir -p ${RESULT_DIR}/wlm_cpu_trans
pushd "${EXAMPLES_DIR}/word_language_model"
export LOG_FILE=${RESULT_DIR}/wlm_cpu_trans/result.log
export MEM_FILE=${RESULT_DIR}/wlm_cpu_trans/result_mem.log
${PREFIX} bash "${CURRENT_DIR}/monitor_proc.sh" python main.py --epochs 10 --model Transformer
# run GPU WLM Transformer
mkdir -p ${RESULT_DIR}/wlm_gpu_trans
pushd "${EXAMPLES_DIR}/word_language_model"
export LOG_FILE=${RESULT_DIR}/wlm_gpu_trans/result.log
export MEM_FILE=${RESULT_DIR}/wlm_gpu_trans/result_mem.log
${PREFIX} bash "${CURRENT_DIR}/monitor_proc.sh" python main.py --epochs 10 --model Transformer --cuda
```

Models are taken from here: https://github.com/pytorch/examples


Improvement suggestions:

- Lower the epoch to 2-3 to make it run faster. Run the results and compare.
- Introduce Additional models.  Something to look into: https://github.com/mlcommons/algorithmic-efficiency/tree/main/algorithmic_efficiency/workloads
- Add an option test test torch only without the domains by running models that does not require torchvision/torchaudio
Repository: pytorch/benchmark
Issue: Hardcoded Shape of `x_keep` in `low_mem_dropout` Benchmark
Root cause: Architectural Constraint

I have noticed that in the low_mem_dropout benchmark, the shape of `x_keep` is incorrectly hardcoded as 10. [Code Link.](https://github.com/pytorch/benchmark/blob/0f05015140467c80395308f98748c8cb85aab4b6/torchbenchmark/operators/low_mem_dropout/operator.py#L41). This could potentially cause memory access violations, especially when the actual size of the input differs from this hardcoded value.

# Steps to Reproduce:

Run the low_mem_dropout benchmark with input sizes different from the hardcoded value of 10.
Then you can observe potential memory access violations or unexpected behavior.

# Expected Behavior: 
The shape of x_keep should be dynamically assigned based on the input size to prevent any out-of-bounds memory access.

# Suggested Fix: 

See Pull Request #2475 for details.



cc @Jokeren