Repository: pytorch
Issue: Perf-Drop (factor=2) Ubuntu-vs-Windows on same PC (dual-boot)
Labels: module: windows,module: cuda,triaged

### 🐛 Describe the bug

On my Dual-Boot-PC (Ubuntu + Windows) same Torch-Code (same folder on NTFS-drive) shows factor 2 runtime slowdown on Windows.
I disabled write Parquet-Files and shuffle turned off, to exclude Filesystem or other issues. I do not use the Compile-feature, afaik.
Both: torch.device("cuda"), torch.get_default_dtype=torch.float32, torch.cuda.is_available()=True
python -m cProfile -o test.cprof mnistBenchmark.py # (.cprof attached for both systems)
Overall Runtume: Windows=30s, Ubtuntu=15s, Ubuntu+cProfile=23s (cProfile does some Ubuntu-only slowdown, but this is not the issue here).

It would need some hours to cleanup the code form confidential details, so I dont want to upload it to public at the moment, but could send python files on a non-public-channel. I think the issue is of general nature as it happens with textbook-MNIST-examples and affects all solvers (at least Adam and my own new solver in the same way).

### Versions

Hardware: PC, 32GB-DDR5, Intel i5-12400, RTX-4060-GPU, fast M2-PCIe4-SSD.
Common-Software:  Python 3.11.5, CUDA-12.2-Update2, all System-Updates.
Linux: Ubuntu 22.04, NVIDIA-Linux-x86_64-535.113.01
Windows 10, 537.32 (but last 2 verions had same issue).
Simple Net-config: MNIST, 7960 parameter, Float32, Relu, Shuffle=Off, WriteToParquet=Off, 60000/256=235 steps x 10 epochs, many optimizers (e.g. Adam).

pip list # Windows 10 + Python 3.11.5 + Cuda-12.2
numpy   1.25.2
torch   2.2.0.dev20230929+cu121 # use dev for Cuda on Windows
// no nvidia- modules

pip list # Ubuntu 22.04 + Python 3.11.5 + Cuda-12.2
numpy                    1.26.0
nvidia-cublas-cu11       11.10.3.66
nvidia-cuda-runtime-cu11 11.7.99
nvidia-cudnn-cu11        8.5.0.96
torch                    2.0.1

##################################
Collecting environment information...
PyTorch version: 2.2.0.dev20230929+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 10 Pro
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect
Libc version: N/A

Python version: 3.11.5 (tags/v3.11.5:cce6ba9, Aug 24 2023, 14:38:34) [MSC v.1936 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.19045-SP0
Is CUDA available: True
CUDA runtime version: 12.2.140
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 4060
Nvidia driver version: 537.42
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture=9
CurrentClockSpeed=2500
DeviceID=CPU0
Family=205
L2CacheSize=7680
L2CacheSpeed=
Manufacturer=GenuineIntel
MaxClockSpeed=2500
Name=12th Gen Intel(R) Core(TM) i5-12400
ProcessorType=3
Revision=

Versions of relevant libraries:
[pip3] numpy==1.25.2
[pip3] torch==2.2.0.dev20230929+cu121
[pip3] torchaudio==2.2.0.dev20230929+cu121
[pip3] torchvideo==0.0.0
[pip3] torchvision==0.17.0.dev20230929+cu121
[conda] Could not collect

cc @peterjc123 @mszhanyi @skyline75489 @nbcsm @vladimir-aubrecht @iremyux @Blackhex @cristianPanaite @ptrblck