Repository: pytorch
Issue: test_stream_event_nogil: Is the test making a wrong assumption?
Labels: in progress,module: rocm,module: tests,triaged,actionable

## 🐛 Bug
It seems there is a problem with the test itself.

It assumes that if each thread takes t amount of time, then in Pytorch, which does not have Python's GIL,
and threads can actually run asynchronously, the total time taken for running 2 threads should be less than 2t.
But the test assumes that the total time taken will be at least 1.4t. This assumption seems to be arbitrary.
In case of ROCM, the test is failing because the total execution time for 2 threads is less than 1.4t.
Which seems to be better.

Here are the timings for failure cases. In the following examples, the P+C times are the parent and child thread total
time, Total time is the total test time. Ideally, the lesser the P+C the better.

============================================================

    self.assertGreater(parent_time + child_time, total_time * 1.4)
AssertionError: 179.880952835083 not greater than 209.75072326660154

Ran 1 test in 4.450s
FAILED (failures=1)
P+C time:179.880952835083, Total Time:149.8219451904297, 1*4:209.75072326660154

===========================================================

    self.assertGreater(parent_time + child_time, total_time * 1.4)
AssertionError: 9.652470231056213 not greater than 15.782816505432129

----------------------------------------------------------------------
Ran 1 test in 4.643s

FAILED (failures=1)
P+C time:9.652470231056213, Total Time:11.27344036102295, 1*4:15.782816505432129

=====================================================================

    self.assertGreater(parent_time + child_time, total_time * 1.4)
AssertionError: 9.332950949668884 not greater than 16.89542407989502

----------------------------------------------------------------------
Ran 1 test in 4.612s

FAILED (failures=1)
P+C time:9.332950949668884, Total Time:12.068160057067871, 1*4:16.89542407989502

======================================================================

<!-- A clear and concise description of what the bug is. -->

## To Reproduce

Steps to reproduce the behavior:
export PYTORCH_TEST_WITH_ROCM=1
 while [ 1 ]; do python3.6 test/test_cuda.py TestCuda.test_stream_event_nogil -v; done

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

## Expected behavior

The test should check whether the time taken to running two threads asynchronously does not exceed
the twice the time taken to run a single thread.

<!-- A clear and concise description of what you expected to happen. -->

## Environment

Please copy and paste the output from our
[environment collection script](https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py)
(or fill out the checklist below manually).

You can get the script and run it with:
```
wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
```
PyTorch version: 1.9.0a0+git3bb1f59
Is debug build: False
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: 4.1.21114-e9025c25

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: 12.0.0 (/src/external/llvm-project/clang 8bdba68022bb7238e77d52748e8b4415846b861c)
CMake version: version 3.19.6

Python version: 3.6 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: Device 66a1
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: 3.24.1
MIOpen runtime version: 2.10.0

Versions of relevant libraries:
[pip3] numpy==1.18.5
[pip3] torch==1.9.0a0+git3bb1f59
[pip3] torchvision==0.8.0a0+e70c91a
[conda] blas                      1.0                         mkl  
[conda] mkl                       2020.2                      256  
[conda] mkl-include               2020.2                      256  
[conda] mkl-service               2.3.0            py36he8ac12f_0  
[conda] mkl_fft                   1.3.0            py36h54f3939_0  
[conda] mkl_random                1.1.1            py36h0573a6f_0  
[conda] numpy                     1.18.5           py36ha1c710e_0  
[conda] numpy-base                1.18.5           py36hde5b4d6_0  
[conda] torch                     1.9.0a0+git3bb1f59          pypi_0    pypi

## Additional context
The test is currently skipped for ROCM. We are triaging to enable the test.

<!-- Add any other context about the problem here. -->


cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang @mruberry @VitalyFedyunin @walterddr