Repository: pytorch
Issue: Discrepancy in einsum when done in batch vs non-batch
Labels: triaged,module: linear algebra,module: tf32

### 🐛 Describe the bug

Hello,

As shown in the notebook cell paste image below, I don't understand why einsten summation varies when using `float` precision between batch and non-batch. It makes sense from my previous issue : https://github.com/pytorch/pytorch/issues/75249 that this can happen between CPU and CUDA tensors, but between two same CUDA tensors, I don't understand why there should be discrepancy. I can simply use double precision as a workaround but it just becomes very obscure as there are many places in my codebase that requires such change. Also, I cannot find any documentation that explicitly mentions where such conversion has to be done. A fix for this to support consistency among float32 tensors would be appreciated as ubiquitous double precision is an unviable option due to memory constraints.




### Versions




### Versions


```
Collecting environment information...
PyTorch version: 1.10.0+cu111
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: 10.0.0-4ubuntu1 
CMake version: version 3.22.1
Libc version: glibc-2.31

Python version: 3.8.10 (default, Jun  4 2021, 15:09:15)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.4.0-99-generic-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: 
GPU 0: NVIDIA A100-PCIE-40GB
GPU 1: NVIDIA A100-PCIE-40GB
GPU 2: NVIDIA A100-PCIE-40GB

Nvidia driver version: 470.57.02
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] mypy==0.931
[pip3] mypy-extensions==0.4.3
[pip3] neuralnet-pytorch==0.0.3
[pip3] numpy==1.20.3
[pip3] numpy-stl==2.16.0
[pip3] pytorch-lightning==1.5.10
[pip3] pytorch-lightning-bolts==0.3.2.post1
[pip3] pytorch-metric-learning==0.9.99
[pip3] pytorch3d==0.6.1
[pip3] siren-pytorch==0.1.5
[pip3] torch==1.10.0+cu111
[pip3] torch-cluster==1.5.9
[pip3] torch-geometric==1.7.2
[pip3] torch-points-kernels==0.6.10
[pip3] torch-points3d==1.3.0
[pip3] torch-scatter==2.0.9
[pip3] torch-sparse==0.6.10
[pip3] torch-tb-profiler==0.3.1
[pip3] torchfile==0.1.0
[pip3] torchmeta==1.4.0
[pip3] torchmetrics==0.7.0
[pip3] torchnet==0.0.4
[pip3] torchsummary==1.5.1
[pip3] torchviz==0.0.2
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.1.74              h6bb024c_0    nvidia
[conda] libblas                   3.9.0            11_linux64_mkl    conda-forge
[conda] libcblas                  3.9.0            11_linux64_mkl    conda-forge
[conda] liblapack                 3.9.0            11_linux64_mkl    conda-forge
[conda] mkl                       2021.3.0           h06a4308_520  
[conda] mkl-service               2.4.0            py38h7f8727e_0  
[conda] mkl_fft                   1.3.0            py38h42c9631_2  
[conda] mkl_random                1.2.2            py38h51133e4_0  
[conda] numpy                     1.19.0                   pypi_0    pypi
[conda] numpy-base                1.20.3           py38h74d4b33_0  
[conda] pytorch-lightning         1.5.10                   pypi_0    pypi
[conda] pytorch-metric-learning   0.9.99                   pypi_0    pypi
[conda] pytorch3d                 0.6.1                    pypi_0    pypi
[conda] siren-pytorch             0.1.5                    pypi_0    pypi
[conda] torch                     1.10.0+cu111             pypi_0    pypi
[conda] torch-cluster             1.5.9                    pypi_0    pypi
[conda] torch-geometric           1.7.2                    pypi_0    pypi
[conda] torch-points-kernels      0.6.10                   pypi_0    pypi
[conda] torch-points3d            1.3.0                    pypi_0    pypi
[conda] torch-scatter             2.0.9                    pypi_0    pypi
[conda] torch-sparse              0.6.10                   pypi_0    pypi
[conda] torch-tb-profiler         0.3.1                    pypi_0    pypi
[conda] torchfile                 0.1.0                    pypi_0    pypi
[conda] torchmeta                 1.4.0                    pypi_0    pypi
[conda] torchnet                  0.0.4                    pypi_0    pypi
[conda] torchviz                  0.0.2                    pypi_0    pypi
```
<img width="935" alt="image" src="https://user-images.githubusercontent.com/6195312/165736487-bfb9fb68-7e0f-4f0d-ad66-d8cde57f5bb6.png">


Edit: To motivate, when there are multiple einsums, these errors which initially are in magnitudes of 1e-4 accumulate to 1e-1 which isn't insignificant anymore. Without sharing my codebase, I hope it's not too hard to imagine. 

cc @jianyuh @nikitaved @pearu @mruberry @walterddr @IvanYashchuk @xwang233 @Lezcano @zasdfgbnm @ptrblck