Repository: pytorch
Issue: Possible bug of tools::flight_recorder
Labels: oncall: distributed

### 🐛 Describe the bug

Hello,
I'm a new user of PyTorch and recently tried to run the Flight Recorder code provided in the tools. But I cannot get the code to execute as expected.

I use ngc 24.10 container, and all pytorch code of commit e000cf0ad980e5d140dc895a646174e9b945cf26.
I run Megatron Pretrain_gpt.py in single node with 8 GPUs.
I set NCCL_SHM_DISABLE=1 and NCCL_P2P_DISABLE=1 to force communicating by IBNetwork.

I try to solve the problem, but still have new errors, so I would like to ask if this is a problem with my usage or if there may be some bugs in Flight Recorder.

Thank you very very much for reply.

---

1. find_coalesced_group() cannot append the last event in coalesced_group.
  In my test, the last event in coalesced_group's::'is_group' is False, then it cannot be append to list in find_coalesced_group(). And then error...

```{'frames': [{'name': 'isend', 'filename': '/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py', 'line': 2064}, {'name': 'batch_isend_irecv', 'filename': '/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py', 'line': 2374}, {'name': '_batched_p2p_ops', 'filename': '/workspace/deep_learning_examples/thirdparty/Megatron-LM/megatron/core/pipeline_parallel/p2p_communication.py', 'line': 153}, {'name': '_communicate', 'filename': '/workspace/deep_learning_examples/thirdparty/Megatron-LM/megatron/core/pipeline_parallel/p2p_communication.py', 'line': 370}, {'name': 'send_forward_recv_backward', 'filename': '/workspace/deep_learning_examples/thirdparty/Megatron-LM/megatron/core/pipeline_parallel/p2p_communication.py', 'line': 510}, {'name': 'send_forward_recv_backward', 'filename': '/workspace/deep_learning_examples/thirdparty/Megatron-LM/megatron/core/pipeline_parallel/schedules.py', 'line': 1252}, {'name': 'forward_backward_pipelining_without_interleaving', 'filename': '/workspace/deep_learning_examples/thirdparty/Megatron-LM/megatron/core/pipeline_parallel/schedules.py', 'line': 1467}, {'name': 'train_step', 'filename': '/workspace/deep_learning_examples/thirdparty/Megatron-LM/megatron/training/training.py', 'line': 731}, {'name': 'train', 'filename': '/workspace/deep_learning_examples/thirdparty/Megatron-LM/megatron/training/training.py', 'line': 1247}, {'name': 'pretrain', 'filename': '/workspace/deep_learning_examples/thirdparty/Megatron-LM/megatron/training/training.py', 'line': 361}, {'name': '<module>', 'filename': '/workspace/deep_learning_examples/thirdparty/Megatron-LM/pretrain_gpt.py', 'line': 265}], 
'record_id': 5054, 'pg_id': 7, 'process_group': ('33', 'undefined'), 'collective_seq_id': 58, 'p2p_seq_id': 0, 'op_id': 92, 'profiling_name': 'nccl:send 0->1', 'time_created_ns': 1730266770405425544, 'input_sizes': [[1024, 1, 8192]], 'input_dtypes': ['BFloat16'], 'output_sizes': [[1024, 1, 8192]], 'output_dtypes': ['BFloat16'], 'state': 'scheduled', 'time_discovered_started_ns': None, 'time_discovered_completed_ns': None, 'retired': False, 'timeout_ms': 600000, 'is_p2p': True}, 

{'frames': [{'name': 'irecv', 'filename': '/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py', 'line': 2112}, {'name': 'batch_isend_irecv', 'filename': '/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py', 'line': 2374}, {'name': '_batched_p2p_ops', 'filename': '/workspace/deep_learning_examples/thirdparty/Megatron-LM/megatron/core/pipeline_parallel/p2p_communication.py', 'line': 153}, {'name': '_communicate', 'filename': '/workspace/deep_learning_examples/thirdparty/Megatron-LM/megatron/core/pipeline_parallel/p2p_communication.py', 'line': 370}, {'name': 'send_forward_recv_backward', 'filename': '/workspace/deep_learning_examples/thirdparty/Megatron-LM/megatron/core/pipeline_parallel/p2p_communication.py', 'line': 510}, {'name': 'send_forward_recv_backward', 'filename': '/workspace/deep_learning_examples/thirdparty/Megatron-LM/megatron/core/pipeline_parallel/schedules.py', 'line': 1252}, {'name': 'forward_backward_pipelining_without_interleaving', 'filename': '/workspace/deep_learning_examples/thirdparty/Megatron-LM/megatron/core/pipeline_parallel/schedules.py', 'line': 1467}, {'name': 'train_step', 'filename': '/workspace/deep_learning_examples/thirdparty/Megatron-LM/megatron/training/training.py', 'line': 731}, {'name': 'train', 'filename': '/workspace/deep_learning_examples/thirdparty/Megatron-LM/megatron/training/training.py', 'line': 1247}, {'name': 'pretrain', 'filename': '/workspace/deep_learning_examples/thirdparty/Megatron-LM/megatron/training/training.py', 'line': 361}, {'name': '<module>', 'filename': '/workspace/deep_learning_examples/thirdparty/Megatron-LM/pretrain_gpt.py', 'line': 265}], 
'record_id': 5055, 'pg_id': 7, 'process_group': ('33', 'undefined'), 'collective_seq_id': 58, 'p2p_seq_id': 0, 'op_id': 93, 'profiling_name': 'nccl:recv 0<-1', 'time_created_ns': 1730266770405447264, 'input_sizes': [[1024, 1, 8192]], 'input_dtypes': ['BFloat16'], 'output_sizes': [[1024, 1, 8192]], 'output_dtypes': ['BFloat16'], 'state': 'scheduled', 'time_discovered_started_ns': None, 'time_discovered_completed_ns': None, 'retired': False, 'timeout_ms': 600000, 'is_p2p': True}, 

{'frames': [{'name': '_coalescing_manager', 'filename': '/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py', 'line': 2319}, {'name': '__exit__', 'filename': '/usr/lib/python3.10/contextlib.py', 'line': 142}, {'name': 'batch_isend_irecv', 'filename': '/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py', 'line': 2372}, {'name': '_batched_p2p_ops', 'filename': '/workspace/deep_learning_examples/thirdparty/Megatron-LM/megatron/core/pipeline_parallel/p2p_communication.py', 'line': 153}, {'name': '_communicate', 'filename': '/workspace/deep_learning_examples/thirdparty/Megatron-LM/megatron/core/pipeline_parallel/p2p_communication.py', 'line': 370}, {'name': 'send_forward_recv_backward', 'filename': '/workspace/deep_learning_examples/thirdparty/Megatron-LM/megatron/core/pipeline_parallel/p2p_communication.py', 'line': 510}, {'name': 'send_forward_recv_backward', 'filename': '/workspace/deep_learning_examples/thirdparty/Megatron-LM/megatron/core/pipeline_parallel/schedules.py', 'line': 1252}, {'name': 'forward_backward_pipelining_without_interleaving', 'filename': '/workspace/deep_learning_examples/thirdparty/Megatron-LM/megatron/core/pipeline_parallel/schedules.py', 'line': 1467}, {'name': 'train_step', 'filename': '/workspace/deep_learning_examples/thirdparty/Megatron-LM/megatron/training/training.py', 'line': 731}, {'name': 'train', 'filename': '/workspace/deep_learning_examples/thirdparty/Megatron-LM/megatron/training/training.py', 'line': 1247}, {'name': 'pretrain', 'filename': '/workspace/deep_learning_examples/thirdparty/Megatron-LM/megatron/training/training.py', 'line': 361}, {'name': '<module>', 'filename': '/workspace/deep_learning_examples/thirdparty/Megatron-LM/pretrain_gpt.py', 'line': 265}], 
'record_id': 5056, 'pg_id': 7, 'process_group': ('33', 'undefined'), 'collective_seq_id': 58, 'p2p_seq_id': 0, 'op_id': 93, 'profiling_name': 'nccl:coalesced', 'time_created_ns': 1730266770405457818, 'duration_ms': 4.395232200622559, 'input_sizes': [], 'input_dtypes': [], 'output_sizes': [], 'output_dtypes': [], 'state': 'completed', 'time_discovered_started_ns': 1730266770561774281, 'time_discovered_completed_ns': 1730266770561776108, 'retired': True, 'timeout_ms': 600000, 'is_p2p': False}, 
```

2. dst_global_rank in match_coalesced_groups() is wrong in "receive" op.
In match_coalesced_groups(), `dst_global_rank = sorted(memberships[op.pg_name])[op.dst]`.
But when op.type=="receive", op.dst is current rank, match() will fail after it.

3. `rank, event = all_rank_events[r][i]` in visualize_ops(). 
I see the first element of all_rank_events[r][i] is index of prev entries, why it has assigned to rank?

4. ....

I see other problems, but I donot know whether I havenot understood the code or it has bugs.
So I eagerly need your help and discussion! Thank u again~

### Versions

**ngc 24.10 container**
all pytorch code of commit **e000cf0ad980e5d140dc895a646174e9b945cf26**

Megatron Pretrain_gpt.py in single node with 8 GPUs.

NCCL_SHM_DISABLE=1.
NCCL_P2P_DISABLE=1.

cc @H-Huang @awgu @kwen2501 @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @c-p-i-o