Repository: pytorch
Issue: Stop calling torch.cuda.synchronize() in triton/testing.py do_bench
Labels: triaged,oncall: pt2

### 🐛 Describe the bug

When a job hangs on a torch.cuda.synchronize(), it is rarely due to that particular torch.cuda.synchronize() call. Instead, something else in the CUDA queue that was already asynchronously queued that is causing the hang (typically a NCCL call or something). When this has happened, we will unpredictably hang at some later point in time, either when the next kernel put on the queue gets stuck because the queue is full, OR we block waiting for the entire queue to flush, which is what happens with torch.cuda.synchronize(). Because the synchronize() call is so likely to trigger the hang, it is the most likely thing to show up when you see hung jobs and then everyone concludes it's a PT2 problem when it's not.

Unfortunately, PT2 generates a lot of torch.cuda.synchronize() calls when it hasn't quiesced yet. Specifically, when we do autotuning, we end up doing a synchronize:

```
  File "/dev/shm/uid-99/dea1e454-seed-nspid4026560790_cgpid9965291-ns-4026560787/torch/_inductor/runtime/triton_heuristics.py", line 818, in run
    self.autotune_to_one_config(*args, grid=grid, **kwargs)
  File "/dev/shm/uid-99/dea1e454-seed-nspid4026560790_cgpid9965291-ns-4026560787/torch/_inductor/runtime/triton_heuristics.py", line 720, in autotune_to_one_config
    timings = self.benchmark_all_configs(*args, **kwargs)
  File "/dev/shm/uid-99/dea1e454-seed-nspid4026560790_cgpid9965291-ns-4026560787/torch/_dynamo/utils.py", line 272, in time_wrapper
    r = func(*args, **kwargs)
  File "/dev/shm/uid-99/dea1e454-seed-nspid4026560790_cgpid9965291-ns-4026560787/torch/_inductor/runtime/triton_heuristics.py", line 695, in benchmark_all_configs
    timings = {
  File "/dev/shm/uid-99/dea1e454-seed-nspid4026560790_cgpid9965291-ns-4026560787/torch/_inductor/runtime/triton_heuristics.py", line 696, in <dictcomp>
    launcher: self.bench(launcher, *args, **kwargs)
  File "/dev/shm/uid-99/dea1e454-seed-nspid4026560790_cgpid9965291-ns-4026560787/torch/_inductor/runtime/triton_heuristics.py", line 667, in bench
    return do_bench_gpu(kernel_call, rep=40, fast_flush=True)
  File "/dev/shm/uid-99/dea1e454-seed-nspid4026560790_cgpid9965291-ns-4026560787/torch/_inductor/runtime/runtime_utils.py", line 118, in do_bench_gpu
    return triton_do_bench(*args, **kwargs)[0]
  File "/dev/shm/uid-99/dea1e454-seed-nspid4026560790_cgpid9965291-ns-4026560787/triton/testing.py", line 104, in do_bench
    torch.cuda.synchronize()
  File "/dev/shm/uid-99/dea1e454-seed-nspid4026560790_cgpid9965291-ns-4026560787/torch/cuda/__init__.py", line 949, in synchronize
    return torch._C._cuda_synchronize()
```

NB: this code was recently changed in https://github.com/pytorch/pytorch/pull/132827 but I think synchronize calls are still made.

It would be really great if we could figure out a way to not have to do a full CUDA synchronization. Remote autotuning service would be a sledgehammer way to solve the problem, but maybe there is something simpler we can do short term.

cc @chauhang @penguinwu @eellison @nmacchioni @aorenste 

### Versions

main