Repository: pytorch
Issue: [torch.arange] Output mismatch with Numpy / TensorFlow
Labels: module: numerical-stability,triaged

### 🐛 Describe the bug

Output of torch.arange:

- Does not match with **Numpy / TensorFlow arange** output for **FP32** for specific inputs.
- Does not match with **TensorFlow arange** output for **BF16** for specific inputs

Mismatch occur for specific elements of the output (not entire output).

## Case 1:

For BF16, start=2.2999999999999998, end=165.5 and step=3, output of PyTorch arange:

```
 tensor([  2.2969,   5.3125,   8.3125,  11.3125,  14.3125,  17.2500,  20.2500,
         23.2500,  26.2500,  29.2500,  32.2500,  35.2500,  38.2500,  41.2500,
         44.2500,  47.2500,  50.2500,  53.2500,  56.2500,  59.2500,  62.2500,
         65.0000,  68.0000,  71.0000,  74.0000,  77.0000,  80.0000,  83.0000,
         86.0000,  89.0000,  92.0000,  95.0000,  98.5000, 101.5000, 104.5000,
        107.5000, 110.5000, 113.5000, 116.5000, 119.5000, 122.5000, 125.5000,
        128.0000, 131.0000, 134.0000, 137.0000, 140.0000, 143.0000, 146.0000,
        149.0000, 152.0000, 155.0000, 158.0000, 161.0000, 164.0000],
       dtype=torch.bfloat16)
```

Output of TensorFlow:

```
 tf.Tensor(
[2.29688 5.3125 8.3125 11.3125 14.3125 17.25 20.25 23.25 26.25 29.25 32.25
 35.25 38.25 41.25 44.25 47.25 50.25 53.25 56.25 59.25 62.25 65.5 68.5
 71.5 74.5 77.5 80.5 83.5 86.5 89.5 92.5 95.5 98.5 101.5 104.5 107.5 110.5
 113.5 116.5 119.5 122.5 125.5 128 131 134 137 140 143 146 149 152 155 158
 161 164], shape=(55,), dtype=bfloat16)
```

Note the mismatch for element 65.50 vs 65.00, 68.50 vs 68.00 and so on (error of 0.5).

## Case 2:

For FP32, if start=2.555599, end=50 and step=1, output of torch.arange is:

```
 tensor([ 2.5556,  3.5556,  4.5556,  5.5556,  6.5556,  7.5556,  8.5556,  9.5556,
        10.5556, 11.5556, 12.5556, 13.5556, 14.5556, 15.5556, 16.5556, 17.5556,
        18.5556, 19.5556, 20.5556, 21.5556, 22.5556, 23.5556, 24.5556, 25.5556,
        26.5556, 27.5556, 28.5556, 29.5556, 30.5556, 31.5556, 32.5556, 33.5556,
        34.5556, 35.5556, 36.5556, 37.5556, 38.5556, 39.5556, 40.5556, 41.5556,
        42.5556, 43.5556, 44.5556, 45.5556, 46.5556, 47.5556, 48.5556, 49.5556])
``` 

Output of Numpy arange:

```
 [ 2.555599  3.555599  4.555599  5.555599  6.555599  7.555599  8.555599
  9.555599 10.555599 11.555599 12.555599 13.555599 14.555599 15.555599
 16.5556   17.5556   18.5556   19.5556   20.5556   21.5556   22.5556
 23.5556   24.5556   25.5556   26.5556   27.5556   28.5556   29.5556
 30.5556   31.5556   32.5556   33.5556   34.5556   35.5556   36.5556
 37.5556   38.5556   39.5556   40.5556   41.5556   42.5556   43.5556
 44.5556   45.5556   46.5556   47.5556   48.5556   49.5556  ]
```

Note the output of Numpy and TensorFlow match for this case. Note the mismatch for the initial elements.

Code for replicating the error:

```python
import torch
import numpy as np
import tensorflow as tf


def test_arange_accuracy():

    input_shapes = [
       (2.555599, 50, 1, torch.strided, torch.float),
       (2.2999999999999998, 165.5, 3, torch.strided, torch.bfloat16),
    ]
    def fn(start, layout, step, end, device, dtype):
        return torch.arange(start=start, step=step, end=end, device=device, dtype=dtype, layout=layout)

    count = 0
    for input in input_shapes:
        count = count + 1
        start = input[0]
        end = input[1]
        step = input[2]
        layout = input[3]
        dtype = input[4]
        pytorch_arange_output = fn(start, layout, step, end, "cpu", dtype)
        numpy_arange_output = np.arange(start, end, step, np.float32)
        tf_dtype = tf.bfloat16
        if dtype == torch.float:
            tf_dtype = tf.float32
        tf_arange_output = tf.experimental.numpy.arange(start, end, step, tf_dtype)
        print("Test ", count, ":")
        print("PyTorch output:\n", pytorch_arange_output)
        print("TensorFlow output:\n", tf_arange_output)
        if dtype == torch.float:
            print("Numpy output:\n", numpy_arange_output)
        print("===================")
```

### Versions

```
PyTorch version: 2.2.0a0+git377f0cb
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: 14.0.5
CMake version: version 3.28.3
Libc version: glibc-2.35

Python version: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-5.15.0-102-generic-x86_64-with-glibc2.35
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
```