Repository: pytorch
Issue: DISABLED test_profiler_rpc_key_names (__main__.TensorPipeRpcTest)
Labels: oncall: distributed,triaged,module: flaky-tests,skipped

Platforms: linux

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_profiler_rpc_key_names&suite=TensorPipeRpcTest&limit=100) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/24478184208).

Over the past 3 hours, it has been determined flaky in 3 workflow(s) with 3 failures and 3 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_profiler_rpc_key_names`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.



<details><summary>Sample error message</summary>

```
Traceback (most recent call last):
  File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 535, in wrapper
    self._join_processes(fn)
  File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 754, in _join_processes
    self._check_return_codes(elapsed_time)
  File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
    raise RuntimeError(error)
RuntimeError: Process 1 exited with error code 10 and exception:
Traceback (most recent call last):
  File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 651, in run_test
    getattr(self, test_name)()
  File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 537, in wrapper
    fn()
  File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 2757, in wrapper
    method(*args, **kwargs)
  File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/dist_utils.py", line 81, in new_test_method
    return_value = old_test_method(self, *arg, **kwargs)
  File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 1910, in test_profiler_rpc_key_names
    fut.result()
  File "/opt/conda/envs/py_3.8/lib/python3.8/concurrent/futures/_base.py", line 444, in result
    return self.__get_result()
  File "/opt/conda/envs/py_3.8/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/opt/conda/envs/py_3.8/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 1893, in rpc_with_profiling
    self.assertTrue(
  File "/opt/conda/envs/py_3.8/lib/python3.8/unittest/case.py", line 765, in assertTrue
    raise self.failureException(msg)
AssertionError: False is not true : Expected {'aten::ones', 'aten::clamp_min', 'aten::add', 'aten::relu', 'aten::mul', 'aten::sigmoid'} to be included in remote profiler output.

To execute this test, run the following from the base repo dir:
     python test/distributed/rpc/test_tensorpipe_agent.py -k test_profiler_rpc_key_names

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
```

</details>


Test file path: `distributed/rpc/test_tensorpipe_agent.py`

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu @fegin @XilunWu @wanchaol @fduwjj @wz337 @tianyu-l @wconstab @yf225 @chauhang @d4l3k @clee2000