Repository: pytorch
Issue: Add distributed examples into PyTorch CI tests
Labels: oncall: distributed,module: bootcamp,module: tests,triaged

### Targeted Examples

This will help us to capture tutorial/examples failures caused by BC-breaking changes. So the test should ideally be the same as the examples. However, as some examples depends on non-PyTorch libraries, we will need to replace those with some dummy implementation. Below is a list of existing examples.


- [ ] [Distributed Reinforcement Learning using RPC and RRef](https://pytorch.org/tutorials/intermediate/rpc_tutorial.html#distributed-reinforcement-learning-using-rpc-and-rref). Example source code is [here](https://github.com/pytorch/examples/tree/master/distributed/rpc/rl). One caveat is that, since we cannot let PyTorch depend on `gym`, we will need to create a dummy environment. This is fine, as we are only testing API correctness. It should be sufficient to return random states and let the training loop to run a small number of iterations.
- [x] [#51016] [Distributed RNN using Distributed Autograd and Distributed Optimizer](https://pytorch.org/tutorials/intermediate/rpc_tutorial.html#distributed-rnn-using-distributed-autograd-and-distributed-optimizer). Example source code is [here](https://github.com/pytorch/examples/tree/master/distributed/rpc/rnn).
- [x] [#50930] [Implementing a Parameter Server using Distributed RPC Framework](https://pytorch.org/tutorials/intermediate/rpc_param_server_tutorial.html). Example source code is [here](https://github.com/pytorch/examples/tree/master/distributed/rpc/parameter_server).
- [x] [#46510 cc @H-Huang] [Batch-Updating Parameter Server](https://pytorch.org/tutorials/intermediate/rpc_async_execution.html#batch-updating-parameter-server). Source code is [here](https://github.com/pytorch/examples/blob/master/distributed/rpc/batch/parameter_server.py). Similar to the first item, we cannot make PyTorch depend on torchvision, so please choose some dummy models instead.
- [ ] [Batch-Processing CartPole Solver](https://pytorch.org/tutorials/intermediate/rpc_async_execution.html#batch-processing-cartpole-solver). Source code is [here](https://github.com/pytorch/examples/blob/master/distributed/rpc/batch/reinforce.py). The test cannot depend on `gym`, so please use some dummy environments.
- [x] [#51037] [Distributed Pipeline Parallelism Using RPC](https://pytorch.org/tutorials/intermediate/dist_pipeline_parallel_tutorial.html). Source code is [here](https://github.com/pytorch/examples/tree/master/distributed/rpc/pipeline). Again, cannot depend on `torchvision`.
- [ ] [Combining DDP with RPC](https://pytorch.org/tutorials/advanced/rpc_ddp_tutorial.html). Source code is [here](https://github.com/pytorch/examples/tree/master/distributed/rpc/ddp_rpc).
- [ ] [#54678][CUDA RPC](https://pytorch.org/tutorials/recipes/cuda_rpc.html) [reserved for @gcramer23]

### Implementation

1. Create an `example_test.py` file in [this folder](https://github.com/pytorch/pytorch/tree/e7dbaa252e85779660c1c13262340ae2b891c7b4/torch/testing/_internal/distributed/rpc), and add an `ExampleTest` class in that file. Please refer to [RpcTest](https://github.com/pytorch/pytorch/blob/e7dbaa252e85779660c1c13262340ae2b891c7b4/torch/testing/_internal/distributed/rpc/rpc_test.py#L506) as an example. Then, add this new test class into [GENERIC_TESTS](https://github.com/pytorch/pytorch/blob/e7dbaa252e85779660c1c13262340ae2b891c7b4/torch/testing/_internal/distributed/rpc_utils.py#L84), which will allow RPC agent tests to pick it up in the CI. This only needs to be done once, so adding the first example will be a little complicated than the following ones. 
2. Add a test function in `ExampleTest` for the specific example you are working on.
3. Run the following command to verify that your test works:
   ```
   pytest test/distributed/rpc/test_tensorpipe_agent.py -k test_func_name -vs
   pytest test/distributed/rpc/test_process_group_agent.py -k test_func_name -vs
   ```

cc @mruberry @VitalyFedyunin @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @xush6528 @osalpekar @jiayisuse @agolynski