Repository: pytorch
Issue: After XNNPack update `TestXNNPACKSerDes.test_linear` started to fail
Labels: high priority,module: rocm,triaged,module: regression,module: xnnpack

### 🐛 Describe the bug

TestXNNPACKSerDes.test_linear started to fail (but only on ROCm testers) after updating https://github.com/google/XNNPACK/commit/ae108ef49aa5623b896fc93d4298c49d1750d9b
with the following signature:
```
======================================================================
FAIL [0.446s]: test_linear (__main__.TestXNNPACKOps)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_xnnpack_integration.py", line 25, in test_linear
    data_shape=hu.array_shapes(1, 3, 2, 64),
  File "/opt/conda/lib/python3.7/site-packages/hypothesis/core.py", line 1116, in wrapped_test
    raise the_error_hypothesis_found
  File "test_xnnpack_integration.py", line 39, in test_linear
    torch.testing.assert_close(ref_result, output_linearprepacked, rtol=1e-2, atol=1e-3)
  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_comparison.py", line 1321, in assert_close
    msg=msg,
  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_comparison.py", line 1074, in assert_equal
    raise error_metas[0].to_error()
AssertionError: Tensor-likes are not close!

Mismatched elements: 446 / 1250 (35.7%)
Greatest absolute difference: 1.7944009453058243 at index (1, 17, 22) (up to 0.001 allowed)
Greatest relative difference: 1.1037873159796914e+18 at index (1, 24, 16) (up to 0.01 allowed)


======================================================================
FAIL [3.459s]: test_combined_model (__main__.TestXNNPACKSerDes)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_xnnpack_integration.py", line 440, in test_combined_model
    input_channels_per_group=st.integers(1, 32),
  File "/opt/conda/lib/python3.7/site-packages/hypothesis/core.py", line 1116, in wrapped_test
    raise the_error_hypothesis_found
  File "test_xnnpack_integration.py", line 553, in test_combined_model
    torch.testing.assert_close(ref_result, xnnpack_result, rtol=1e-2, atol=1e-3)
  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_comparison.py", line 1321, in assert_close
    msg=msg,
  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_comparison.py", line 1074, in assert_equal
    raise error_metas[0].to_error()
AssertionError: Tensor-likes are not close!

Mismatched elements: 195 / 1235 (15.8%)
Greatest absolute difference: 0.2513161301612854 at index (0, 4, 12, 17) (up to 0.001 allowed)
Greatest relative difference: inf at index (0, 4, 10, 16) (up to 0.01 allowed)


======================================================================
FAIL [1.274s]: test_linear (__main__.TestXNNPACKSerDes)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_xnnpack_integration.py", line 186, in test_linear
    data_shape=hu.array_shapes(1, 3, 2, 64),
  File "/opt/conda/lib/python3.7/site-packages/hypothesis/core.py", line 1116, in wrapped_test
    raise the_error_hypothesis_found
  File "test_xnnpack_integration.py", line 218, in test_linear
    torch.testing.assert_close(ref_result, output_linearprepacked, rtol=1e-2, atol=1e-3)
  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_comparison.py", line 1321, in assert_close
    msg=msg,
  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_comparison.py", line 1074, in assert_equal
    raise error_metas[0].to_error()
AssertionError: Tensor-likes are not close!

Mismatched elements: 480 / 1188 (40.4%)
Greatest absolute difference: 7.112247561849998e+22 at index (1, 10, 50) (up to 0.001 allowed)
Greatest relative difference: 42.11850658475976 at index (1, 6, 39) (up to 0.01 allowed)


----------------------------------------------------------------------
```
For more details, see 
https://github.com/pytorch/pytorch/runs/5342822407?check_suite_focus=true

### Versions

Nightly

cc @ezyang @gchanan @zou3519 @jeffdaily @sunway513 @jithunnair-amd @ROCmSupport @KyleCZH