Repository: pytorch
Issue: Instability in `test_input_weight_equalization_activation_values` test for random test values.
Labels: oncall: quantization,low priority,module: tests,triaged

## 🐛 Bug

Initialization of additional, unused, random tensors in global or local scope causes the `test_input_weight_equalization_activation_values` test in `test/quantization/fx/test_equalize_fx.py` to fail.

## To Reproduce

Steps to reproduce the behavior:

1. add initialization of random pytorch tensor in `test/quantization/fx/test_equalize_fx.py`
```
git diff --patch test/quantization/fx/test_equalize_fx.py
diff --git a/test/quantization/fx/test_equalize_fx.py b/test/quantization/fx/test_equalize_fx.py
index a552fc0a0f..ec964afc86 100644
--- a/test/quantization/fx/test_equalize_fx.py
+++ b/test/quantization/fx/test_equalize_fx.py
@@ -505,6 +505,8 @@ class TestEqualizeFx(QuantizationTestCase):
         observer's min/max values are as expected
         """
 
+        foo = [[torch.randint(0, 10, (12, 1))]]
+
         tests = [SingleLayerLinearModel, TwoLayerLinearModel, SingleLayerFunctionalLinearModel]
 
         x = torch.rand((5, 5))
```
2. run `test_input_weight_equalization_activation_values` test

`pytest test/quantization/fx/test_equalize_fx.py -k "test_input_weight_equalization_activation_values" `

3. Observe that activation comparison is well outside tolerance

```
pytest test/quantization/fx/test_equalize_fx.py -k "test_input_weight_equalization_activation_values" 
======================================================================================================================== test session starts ========================================================================================================================
platform linux -- Python 3.9.6, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /home/bkoopman/dev/pytorch, configfile: pytest.ini
plugins: hypothesis-6.14.1
collected 10 items / 9 deselected / 1 selected                                                                                                                                                                                                                      

test/quantization/fx/test_equalize_fx.py F                                                                                                                                                                                                                    [100%]

============================================================================================================================= FAILURES ==============================================================================================================================
__________________________________________________________________________________________________ TestEqualizeFx.test_input_weight_equalization_activation_values __________________________________________________________________________________________________
Traceback (most recent call last):
  File "/home/bkoopman/dev/pytorch/test/quantization/fx/test_equalize_fx.py", line 541, in test_input_weight_equalization_activation_values
    self.assertEqual(modules[str(node.target)].min_val, exp_min_val)
  File "/home/bkoopman/local/miniconda3/envs/pytorch/lib/python3.9/site-packages/torch/testing/_internal/common_utils.py", line 1903, in assertEqual
    super().assertTrue(result, msg=self._get_assert_msg(msg, debug_msg=debug_msg))
  File "/home/bkoopman/local/miniconda3/envs/pytorch/lib/python3.9/unittest/case.py", line 682, in assertTrue
    raise self.failureException(msg)
AssertionError: False is not true : Tensors failed to compare as equal!With rtol=1.3e-06 and atol=1e-05, found 1 element(s) (out of 1) whose difference(s) exceeded the margin of error (including 0 nan comparisons). The greatest difference was 0.126822829246521 (-0.8828725814819336 vs. -0.7560497522354126), which occurred at index 0.
========================================================================================================================= warnings summary ==========================================================================================================================
test/quantization/fx/test_equalize_fx.py::TestEqualizeFx::test_input_weight_equalization_activation_values
  /home/bkoopman/local/miniconda3/envs/pytorch/lib/python3.9/site-packages/torch/ao/quantization/observer.py:177: UserWarning: Please use quant_min and quant_max to specify the range for observers.                     reduce_range will be deprecated in a future release of PyTorch.
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/warnings.html
====================================================================================================================== short test summary info ======================================================================================================================
FAILED test/quantization/fx/test_equalize_fx.py::TestEqualizeFx::test_input_weight_equalization_activation_values - AssertionError: False is not true : Tensors failed to compare as equal!With rtol=1.3e-06 and atol=1e-05, found 1 element(s) (out of 1) whose d...
============================================================================================================ 1 failed, 9 deselected, 1 warning in 1.80s =============================================================================================================
```

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

## Expected behavior

I would expect this test to succeed regardless of the state of the RNG, as this prevents additions of new random test-data to test classes, such as `torch.testing._internal.common_quantization.QuantizationTestCase`.

## Environment

PyTorch version: 1.11.0a0+gitbc1dec9
Is debug build: True
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: CentOS Stream 8 (x86_64)
GCC version: (GCC) 8.5.0 20210514 (Red Hat 8.5.0-3)
Clang version: Could not collect
CMake version: version 3.20.2
Libc version: glibc-2.28

Python version: 3.9.6 (default, Jul 30 2021, 16:35:19)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.6.13-0_fbk17_hardened_5815_gc01d8dbd2635-x86_64-with-glibc2.28
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] mypy==0.812
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.21.2
[pip3] torch==1.11.0a0+gitfc4836f
[conda] mypy                      0.812                    pypi_0    pypi
[conda] mypy_extensions           0.4.3            py39h06a4308_0  
[conda] numpy                     1.21.2                   pypi_0    pypi
[conda] torch                     1.11.0a0+gitbc1dec9          pypi_0    pypi


## Additional context

This was uncovered when adding additional test-data to `torch.testing._internal.common_quantization.QuantizationTestCase` as part of https://github.com/pytorch/pytorch/pull/66429  for use in testing embeddings.

```
diff --git a/torch/testing/_internal/common_quantization.py b/torch/testing/_internal/common_quantization.py
index b49260b980..f26fd79853 100644
--- a/torch/testing/_internal/common_quantization.py
+++ b/torch/testing/_internal/common_quantization.py
@@ -340,6 +340,11 @@ class QuantizationTestCase(TestCase):
                               2 : self.img_data_2d,
                               3 : self.img_data_3d}
 
+        self.embed_linear_data_train = [[torch.randint(0, 10, (12, 12), dtype=torch.long),
+                                         torch.randn((12, 1), dtype=torch.float)]
+                                        for _ in range(2)]
+        self.embed_data = [[torch.randint(0, 10, (12, 1))]]
+
         # Quant types that produce statically quantized ops
         self.static_quant_types = [QuantType.STATIC, QuantType.QAT]
         # All quant types for (fx based) graph mode quantization
```

cc @jerryzh168 @jianyuh @raghuramank100 @jamesr66a @vkuzo @jgong5 @Xia-Weiwen @leslie-fang-intel @mruberry @ZainRizvi @supriyar