Repository: pytorch
Issue: test_learnable_forward_per_channel fails due to integer overflow
Labels: oncall: quantization,triaged

### 🐛 Describe the bug

`test_quantization.py` fails in `test_learnable_forward_per_channel` due to an integer overflow in the kernel code.

```
AssertionError: False is not true : Expected kernel forward function to have results match the reference forward function
Falsifying example: test_learnable_forward_per_channel_cpu(
    X=(array([9.223372e+14], dtype=float32),
     (array([1.]), array([0]), 0, torch.quint8)),
    self=<quantization.core.test_workflow_ops.TestFakeQuantizeOps testMethod=test_learnable_forward_per_channel_cpu>,
)
```

I traced the issue to an integer overflow leading to different results in the kernel and reference function.

https://github.com/pytorch/pytorch/blob/a4391f085bff409dca93a8b3eff8e379f0ef8f68/test/quantization/core/test_workflow_ops.py#L812
returns a scale of `1.e-4` the zero-point is (in my case) actually zero.

The reason is obvious when comparing the reference function and the kernel. The reference is called at https://github.com/pytorch/pytorch/blob/a4391f085bff409dca93a8b3eff8e379f0ef8f68/test/quantization/core/test_workflow_ops.py#L796

The [calculation](https://github.com/pytorch/pytorch/blob/a4391f085bff409dca93a8b3eff8e379f0ef8f68/torch/testing/_internal/common_quantized.py#L200) is `(torch.clamp(torch.round(X[i] * (1.0 / per_channel_scale[i]) + per_channel_zero_point[i]), quant_min, quant_max) - per_channel_zero_point[i]) * per_channel_scale[i]`

When putting the values for the parameter to `clamp` we get `9.223372e+14 * (1.0 / 1e-4)`

Similar in the actual kernel it also rounds, clamps and converts the zero_point to int: https://github.com/pytorch/pytorch/blob/a4391f085bff409dca93a8b3eff8e379f0ef8f68/aten/src/ATen/native/quantized/FakeQuantPerChannelAffine.cpp#L147

So we get to [this kernel code:](https://github.com/pytorch/pytorch/blob/a4391f085bff409dca93a8b3eff8e379f0ef8f68/aten/src/ATen/native/quantized/cpu/kernels/QuantizedOpKernels.cpp#L2702-L2708)  `std::fmin(std::fmax(static_cast<int64_t>(zero_point + std::nearbyint(self * inv_scale)), quant_min), quant_max) - zero_point) * scale;`

We can see a similar "clamp" with a strange cast to `int64` of the value (in this case)  `9.223372e+14 * 1e4` and sure enough `INT64_MAX` is ~ `9.223372e+18` hence we get **undefined behavior** here.

For the reference function I get a result of `0.0015` so it (correctly) clamps to `quant_max=15` (and multiplies by `scale=1.e-4`)

As I get a result of zero from the kernel I assume the UB manifests in a negative integer which then gets clamped to `quant_min=0`

Note that a similar issue exists in the [mask-calculation](https://github.com/pytorch/pytorch/blob/a4391f085bff409dca93a8b3eff8e379f0ef8f68/aten/src/ATen/native/quantized/cpu/kernels/QuantizedOpKernels.cpp#L2694) and in the path for floating-point zero-points where `std::lrint` is used which has:

> If the result of the rounding is outside the range of the return type, [FE_INVALID](https://en.cppreference.com/w/cpp/numeric/fenv/FE_exceptions) is raised and an implementation-defined value is returned.

Given that `fmin/fmax` is used, the cast to `int64_t` can likely just be removed.

### Versions

PyTorch 2.0.1 with examples above from current main (2.1.0)

cc @jerryzh168 @jianyuh @raghuramank100 @jamesr66a @vkuzo @jgong5 @Xia-Weiwen @leslie-fang-intel