Repository: pytorch
Issue: Better document that non-blocking GPU -> CPU memory requires device sync
Labels: module: docs,module: cuda,triaged,module: correctness (silent)

### 🐛 Describe the bug

Here is a minimal working example of the issue.

```
import torch
torch.manual_seed(0)

acc_blocking = None
acc_non_blocking = None

for j in range(3):
    print(f"\n{j=}")
    t_blocking = torch.randn(10_000, 1_000, dtype=torch.float16).cuda()
    t_non_blocking = t_blocking + 5
    print(f"{t_blocking[0, 0].item() = }")
    print(f"minima of original tensors:\n\t{t_blocking.min().item()=}, {t_non_blocking.min().item()=}")
    if j == 0:
        acc_blocking = t_blocking.to(device="cpu", non_blocking=False)
        acc_non_blocking = t_non_blocking.to(device="cpu", non_blocking=True)
    else:
        acc_blocking = torch.cat([acc_blocking, t_blocking.to(device="cpu", non_blocking=False)], dim=0)
        acc_non_blocking = torch.cat([acc_non_blocking, t_non_blocking.to(device="cpu", non_blocking=True)], dim=0)
    print(f"minima of accumulators:\n\t{acc_blocking.min().item()=}, {acc_non_blocking.min().item()=}")
```

Here is the terminal output.

```
j=0
t_blocking[0, 0].item() = -0.9013671875
minima of original tensors:
        t_blocking.min().item()=-3.904296875, t_non_blocking.min().item()=1.095703125
minima of accumulators:
        acc_blocking.min().item()=-3.904296875, acc_non_blocking.min().item()=1.095703125

j=1
t_blocking[0, 0].item() = 0.07061767578125
minima of original tensors:
        t_blocking.min().item()=-3.904296875, t_non_blocking.min().item()=1.095703125
minima of accumulators:
        acc_blocking.min().item()=-3.904296875, acc_non_blocking.min().item()=0.0

j=2
t_blocking[0, 0].item() = 0.662109375
minima of original tensors:
        t_blocking.min().item()=-3.904296875, t_non_blocking.min().item()=1.095703125
minima of accumulators:
        acc_blocking.min().item()=-3.904296875, acc_non_blocking.min().item()=0.0
```

As you can see, starting at j=1, `acc_non_blocking` clearly has the wrong minimum value. (It should always be the minimum of the minima of all previous instances of `t_non_blocking`.)

Here are a few further comments.
* This also occurs with bfloat16, but seemingly not with float32.
* This occurs with `10 ** 7` entries in the tensors (as above), but seemingly not with `10 ** 6` or fewer entries.
* This seems not to occur if the summand 5 is replaced by a smaller number that keeps `t_non_blocking.min().item()` negative. (For instance, it occurs when the summand is 3.91, but not when it's 3.90.)
* The fact that in each iteration the tensors' minima are all the same seems likely due to the imprecision of (b)float16. The top-left entry is printed just to confirm that the tensor is indeed changing at each iteration.

P.S. This is my first bug report, please let me know if anything is missing or could be improved.

CC: @dhruvbpai

### Versions

Collecting environment information...
PyTorch version: 2.1.2+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Amazon Linux 2 (x86_64)
GCC version: (GCC) 7.3.1 20180712 (Red Hat 7.3.1-17)
Clang version: Could not collect
CMake version: version 2.8.12.2
Libc version: glibc-2.26

Python version: 3.10.14 | packaged by conda-forge | (main, Mar 20 2024, 12:45:18) [GCC 12.3.0] (64-bit runtime)
Python platform: Linux-5.10.223-212.873.amzn2.x86_64-x86_64-with-glibc2.26
Is CUDA available: True
CUDA runtime version: 12.1.105
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA A100-SXM4-40GB
GPU 1: NVIDIA A100-SXM4-40GB
GPU 2: NVIDIA A100-SXM4-40GB
GPU 3: NVIDIA A100-SXM4-40GB
GPU 4: NVIDIA A100-SXM4-40GB
GPU 5: NVIDIA A100-SXM4-40GB
GPU 6: NVIDIA A100-SXM4-40GB
GPU 7: NVIDIA A100-SXM4-40GB

Nvidia driver version: 535.183.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              96
On-line CPU(s) list: 0-95
Thread(s) per core:  2
Core(s) per socket:  24
Socket(s):           2
NUMA node(s):        2
Vendor ID:           GenuineIntel
CPU family:          6
Model:               85
Model name:          Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz
Stepping:            7
CPU MHz:             3599.918
BogoMIPS:            5999.99
Hypervisor vendor:   KVM
Virtualization type: full
L1d cache:           32K
L1i cache:           32K
L2 cache:            1024K
L3 cache:            36608K
NUMA node0 CPU(s):   0-23,48-71
NUMA node1 CPU(s):   24-47,72-95
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke

Versions of relevant libraries:
[pip3] galore-torch==1.0
[pip3] numpy==1.26.4
[pip3] torch==2.1.2
[pip3] torchtyping==0.1.5
[pip3] torchvision==0.16.2
[pip3] triton==2.1.0
[conda] No relevant packages

cc @svekars @brycebortree @sekyondaMeta @ptrblck @msaroufim