Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '2.4.0.dev20240425+cu121'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression, and bisected its result.

Control PyTorch commit: b91f83f18139a6ed2626c30979a38e533d4c7d7c
Control PyTorch version: 2.4.0.dev20240424+cu121

Treatment PyTorch commit: 13ab24f1923359cdfe795a32cdd3d753f940efbe
Treatment PyTorch version: 2.4.0.dev20240425+cu121

Bisection result:

```
[
    {
        "commit1": "02ed2992d94",
        "commit1_time": "2024-04-24 23:12:19 +0000",
        "commit1_digest": {
            "name": "torch-nightly",
            "environ": {
                "pytorch_git_version": "02ed2992d94e6bb09d95fb1409883fc61cf19e13",
                "pytorch_version": "2.4.0a0+git02ed299",
                "device": "NVIDIA A100-SXM4-40GB",
                "git_commit_hash": "02ed2992d94e6bb09d95fb1409883fc61cf19e13"
            },
            "metrics": {
                "test_eval[pytorch_unet-cuda-eager]_latency": 45.703445,
                "test_eval[pytorch_unet-cuda-eager]_cmem": 0.76953125,
                "test_eval[pytorch_unet-cuda-eager]_gmem": 4.95037841796875
            }
        },
        "commit2": "cb94845b148",
        "commit2_time": "2024-04-24 23:14:41 +0000",
        "commit2_digest": {
            "name": "torch-nightly",
            "environ": {
                "pytorch_git_version": "cb94845b148fd47090a9f26f09d5e714bb91265d",
                "pytorch_version": "2.4.0a0+gitcb94845",
                "device": "NVIDIA A100-SXM4-40GB",
                "git_commit_hash": "cb94845b148fd47090a9f26f09d5e714bb91265d"
            },
            "metrics": {
                "test_eval[pytorch_unet-cuda-eager]_latency": 51.428472,
                "test_eval[pytorch_unet-cuda-eager]_cmem": 0.7646484375,
                "test_eval[pytorch_unet-cuda-eager]_gmem": 7.00115966796875
            }
        }
    }
]
```

cc @xuzhao9


@bhack https://github.com/pytorch/pytorch/pull/121324 slows down pytorch_unet (45ms -> 51ms) with higher gpu memory usage. This model is using the amp precision.
Is this expected?

/cc @albanD 
It is expected as the upsampling is now `float32` with amp. Is there a way for you to test the `torch.comile` perf with amp?

Not sure, @xuzhao9 are we running any compile-d perf measurement with amp?



This was the big topic on the PR thread as we didn't see the same gradient limit with the compiled path.
So I suppose compiled doesn't need to be forced `float32` with amp. At least with the inputs we have tested at:
https://github.com/pytorch/pytorch/pull/121324#issuecomment-2021411957

> Not sure, @xuzhao9 are we running any compile-d perf measurement with amp?

In this CI workflow we do not run compile, the PR affects the eager mode path (non-compiled).
For the compile-d workflow result we need to check the HUD: https://hud.pytorch.org/benchmark/compilers

>  For the compile-d workflow result we need to check the HUD: https://hud.pytorch.org/benchmark/compilers

Is there a way to isolate `pytorch_unet` on that HUD page?

@bhack Check: https://hud.pytorch.org/benchmark/torchbench/inductor_with_cudagraphs?startTime=Fri,%2019%20Apr%202024%2019:18:11%20GMT&stopTime=Fri,%2026%20Apr%202024%2019:18:11%20GMT&granularity=hour&mode=training&model=pytorch_unet&dtype=amp&lBranch=main&lCommit=59a1f1f308545e3ac1d81940a51f8dc0db3d82d4&rBranch=main&rCommit=b2f6cfd9c061a212cde8c8768fda41cc75a3110c

It seems we cannot isolate before and after the PR on that page but on a coarse daily timescale I don't see the perf drop on the compiled version so I think it could be ok. 
Do we care about the final effect on trained network in eager mode of:
https://github.com/pytorch/pytorch/issues/121072

How is it going to impact accuracy this increased precision in eager mode? 
Cause the problem is that now we skipped the accuracy check it as it is not deterministic in eager mode.

@bhack One thing I noticed: in eager mode, the regression happens only on amp+inference. However, Inductor CI does not test this combination: https://hud.pytorch.org/benchmark/torchbench/inductor_with_cudagraphs?startTime=Fri%2C+19+Apr+2024+19%3A18%3A11+GMT&stopTime=Fri%2C+26+Apr+2024+19%3A18%3A11+GMT&granularity=hour&mode=training&model=pytorch_unet&dtype=amp&lBranch=main&lCommit=59a1f1f308545e3ac1d81940a51f8dc0db3d82d4&rBranch=main&rCommit=b2f6cfd9c061a212cde8c8768fda41cc75a3110c.
.
It does not regress on train+amp or inference+bf16, but I am not sure about inference+amp since there is no data.

> eager mode, the regression happens only on amp+inference.

It is strange. Are you sure this is  tested also eager + amp training?

Is the ops list in the PR complete to cover the backward or we need to add something else?

@bhack Yes, we test both train and eval: 

Left: 20240424
Right: 20240426

<img width="1839" alt="image" src="https://github.com/pytorch/benchmark/assets/502017/67e02412-5f30-48b2-afd1-012de31420eb">

<img width="1839" alt="image" src="https://github.com/pytorch/benchmark/assets/502017/a5832208-e5a7-4dc3-b9b0-9653b4000e5a">

Train has a much smaller regression than eval.

Is this because we have different `torch.backends.cudnn.deterministic`  setup for train and eval tests?  https://github.com/pytorch/benchmark/blob/main/torchbenchmark/models/pytorch_unet/__init__.py#L91C9-L91C43

I think the regression happens only when `torch.backends.cudnn.deterministic = True`.

I think it is expected.  See @lezcano comment at https://github.com/pytorch/pytorch/pull/121769#issuecomment-2030738866.

But I don't know how it could be connected to my PR. If you git blame for the sort fallback can you  check when it is introduced?

The deterministic fallback was merged on 29 march 2023:
https://github.com/pytorch/pytorch/pull/96898

So I suppose that in pure eager we have a similar `sort` fallback when we ask for determinism (I have not checked it in the source code but I suppose is there).

Is it that the sort fallback is going to be more impacted by working at `float32` then the non fallback `atomic_add` version?

This is the only explanation I have in mind for your regression only in deterministic mode.

@bhack The regression happened on 20240425, so it can't be https://github.com/pytorch/pytorch/pull/96898

Yes It Is what I meant that PR is more than 1 year old but the fallback code with sort when deterministic is required it is still there.

So if you see the perf regression only in deterministic mode  it could be that forcing upsample to be float32 is going to significantly impact only the `sort` fallback.


cc @malfet I am wondering what should we do about this regression - looks like it is expected since we are upsampling the float32 tensors in amp and it only affects the deterministic mode?

Closing this case since it is expected result of upsampling.

Marked the PR bc-breaking to make sure we properly warn in the release notes.