Repository: pytorch
Issue: torchrun: Hostname/endpoint mismatch not handled
Labels: triaged,module: c10d

### 🐛 Describe the bug

TL;DR: I need to monkey patch `torchrun` to be able to handle a `--rendezvous_endpoint` with a format different from `socket.gehostname()`.

# Description

I have a very peculiar setup for a Slurm cluster: if I want to communicate across InfiniBand cells, the addresses must have an "i" appended. Let's assume node addresses follow the regular expression format `node[0-9]` (or `node[0-9]i`). Optionally, the name of the cluster may follow these addresses, so we get `node[0-9].cluster` or `node[0-9]i.cluster`.

## Correct Case
I'm starting jobs on nodes 0 to 3, with `node0` being the master node; so I pass `--rendezvous_endpoint node0` to `torchrun`. Everything works fine because all nodes are on the same InfiniBand cell and don't require the "i" appended.

## Failure Case
I now start a job on nodes 0 to 7. Assume 0 to 3 are on one InfiniBand cell, 4 to 7 on another; I thus have to append the "i" so the nodes can communicate between one another. I pass `--rendezvous_endpoint node0i` to `torchrun`.

### First Error
With the `"c10d"` backend, PyTorch [tries to check whether we are host](https://github.com/pytorch/pytorch/blob/v1.10.2/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py#L144) using `sockets.gethostname()` and some other variants.
In the [`_matches_machine_hostname` function](https://github.com/pytorch/pytorch/blob/v1.10.2/torch/distributed/elastic/rendezvous/utils.py#L121-L139), these variables get the following values:
- `addr` becomes `None` due to an error (can't parse an IP address from our endpoint hostname),
- `this_host` becomes `"node0"` or similar,
- the single `addr_info` in `addr_list` contains a hostname similar to `"node0.cluster"` at index 3.

Now, the host argument to that function is `"node0i"` (note the suffixed "i") as I want to communicate over InfiniBand cells.
This means all the comparisons fail for all nodes, meaning PyTorch doesn't find a host! PyTorch expects a host to be found, so this is where we run into a deadlock and consequent timeout.

#### First Solution
However, luckily there is an out [via some extra config values for rendezvous backends](https://github.com/pytorch/pytorch/blob/v1.10.2/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py#L136-L144). The section I linked is the original `if`-query from which the (in my case erroneous) `_matches_machine_hostname` is called.
If I specify an `"is_host"` config, I avoid that call; `torchrun` has the `--rdzv_conf` argument for these config values, so it's just a matter of passing it in correctly depending on if `torchrun` is started from the master node or any other.

### Second Error
Now the script starts but I run into another issue when `torch.distributed.init_process_group` is called. That's because [the `MASTER_ADDR` envvar is set](https://github.com/pytorch/pytorch/blob/v1.10.2/torch/distributed/elastic/agent/server/api.py#L514-L515) based [on `socket.gethostname()` again](https://github.com/pytorch/pytorch/blob/v1.10.2/torch/distributed/elastic/agent/server/api.py#L384-L385) (the end result looks like `addr_info[3]` from before, so `_get_fq_hostname() == "node0.cluster"`)! Note the missing "i" suffix – the nodes can't communicate across InfiniBand cells again, so I run into another deadlock.

Now you could argue that I can avoid this erroneous `_get_fq_hostname` result by passing a `--master_addr` to `torchrun` so that it's not `None` in the `if`-query. However, [that argument is ignored unless the backend is `"static"`](https://github.com/pytorch/pytorch/blob/v1.10.2/torch/distributed/launcher/api.py#L169-L173); the `MASTER_ADDR` will be obtained dynamically in any other case. So as a non-root user I have two options:

#### Second Solution
1. Monkey patch `_get_fq_hostname` to append an "i" in the `torchrun` script. Advantage is that I can use `torchrun` for any other scripts as well without any further changes.
2. Modify my script and insert the "i" in the `MASTER_ADDR` envvar at the correct location. Advantage is that I don't have to patch `torchrun` for each different Python environment.

## Possible Fixes
Having an "out" for the second error case like another `--rdzv_conf` flag or not ignoring `--master_addr` would be nice. Currently, solutions are involved and very manual. Another option would be to allow methods other than `socket.gethostname` for grabbing the host addresses (in both `_matches_machine_hostname` and `_get_fq_hostname`).

### Versions

1.10.2