Repository: pytorch
Issue: [FSDP]coding to multi-node save optimizer error
Labels: triaged,module: fsdp

### 🐛 Describe the bug

I'm updating DPO into multi-nodes, 
trans batch into 
```
train_sampler = DistributedSampler(train_dataset,
                                           rank=self.rank,
                                           num_replicas=self.world_size,
                                           seed=self.seed,
                                           shuffle=True)
        self.train_dataloader = DataLoader(train_dataset, sampler=train_sampler, collate_fn=collate_fn,
                                           batch_size=config.batch_size, drop_last=True)
```
```
    def save(self, output_dir=None, metrics=None):
        """Save policy, optimizer, and scheduler state to disk, gathering from all processes and saving only on the rank 0 process."""
        save_policy = FullStateDictConfig(offload_to_cpu=True, rank0_only=True)
        with FSDP.state_dict_type(self.policy, StateDictType.FULL_STATE_DICT, state_dict_config=save_policy):
            policy_state_dict = self.policy.state_dict()

        if self.rank == 0:
            self.write_state_dict(self.example_counter, policy_state_dict, metrics, 'policy.pt', output_dir)
        del policy_state_dict
        dist.barrier()

        save_policy = FullOptimStateDictConfig(offload_to_cpu=True, rank0_only=True)
        with FSDP.state_dict_type(self.policy, StateDictType.FULL_STATE_DICT, optim_state_dict_config=save_policy):
            optimizer_state_dict = FSDP.optim_state_dict(self.policy, self.optimizer)

        if self.rank == 0:
            self.write_state_dict(self.example_counter, optimizer_state_dict, metrics, 'optimizer.pt', output_dir)
        del optimizer_state_dict
        dist.barrier()

        if self.rank == 0:
            scheduler_state_dict = self.scheduler.state_dict()
            self.write_state_dict(self.example_counter, scheduler_state_dict, metrics, 'scheduler.pt', output_dir)
        dist.barrier()
```
but when I saving optimizer, the error came:

```

Traceback (most recent call last):
  File "/altenli/direct-preference-optimization/train.py", line 143, in main
    worker_main(config, policy, reference_model)
  File "/altenli/direct-preference-optimization/train.py", line 59, in worker_main
    trainer.train()
  File "/altenli/direct-preference-optimization/trainers.py", line 457, in train
    self.save(output_dir, mean_eval_metrics)
  File "/altenli/direct-preference-optimization/trainers.py", line 631, in save
    optimizer_state_dict = FSDP.full_optim_state_dict(self.policy, self.optimizer)
  File "/altenli/dpoenv/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 1330, in full_optim_state_dict
    return FullyShardedDataParallel._optim_state_dict_impl(
  File "/altenli/dpoenv/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 1154, in _optim_state_dict_impl
    return _optim_state_dict(
  File "/altenli/dpoenv/lib/python3.9/site-packages/torch/distributed/fsdp/_optim_utils.py", line 1508, in _optim_state_dict
    fsdp_osd["param_groups"] = _unflatten_param_groups(
  File "/altenli/dpoenv/lib/python3.9/site-packages/torch/distributed/fsdp/_optim_utils.py", line 1311, in _unflatten_param_groups
    nested_unflat_param_names = [
  File "/altenli/dpoenv/lib/python3.9/site-packages/torch/distributed/fsdp/_optim_utils.py", line 1312, in <listcomp>
    param_to_fqns[param] for param in param_group_params
KeyError: Parameter containing:
tensor([[ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000],
        [ 0.0001,  0.0012,  0.0003,  ...,  0.0006,  0.0011,  0.0003],
        [ 0.0009,  0.0074, -0.0137,  ...,  0.0053,  0.0146,  0.0013],
        ...,
        [-0.0043, -0.0063,  0.0071,  ..., -0.0029, -0.0024, -0.0065],
        [-0.0015, -0.0015, -0.0015,  ..., -0.0037, -0.0009, -0.0010],
        [ 0.0060, -0.0078,  0.0006,  ..., -0.0086,  0.0082, -0.0028]],
       device='cuda:0', dtype=torch.bfloat16, requires_grad=True)
```
 I've been searched, but no answer found. How could I do


### Versions

PyTorch version: 2.0.1+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: CentOS Linux release 7.9.2009 (Core) (x86_64)
GCC version: (GCC) 9.3.1 20200408 (Red Hat 9.3.1-2)
Clang version: Could not collect
CMake version: version 3.26.4
Libc version: glibc-2.17

Python version: 3.9.13 (main, Jul  4 2023, 19:04:55)  [GCC 9.3.1 20200408 (Red Hat 9.3.1-2)] (64-bit runtime)
Python platform: Linux-4.18.0-240.el8.x86_64-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: 11.7.64
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: A100-SXM4-40GB
GPU 1: A100-SXM4-40GB
GPU 2: A100-SXM4-40GB
GPU 3: A100-SXM4-40GB
GPU 4: A100-SXM4-40GB
GPU 5: A100-SXM4-40GB
GPU 6: A100-SXM4-40GB
GPU 7: A100-SXM4-40GB

Nvidia driver version: 460.73.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                96
On-line CPU(s) list:   0-95
Thread(s) per core:    2
Core(s) per socket:    24
Socket(s):             2
NUMA node(s):          8
Vendor ID:             AuthenticAMD
CPU family:            23
Model:                 49
Model name:            AMD EPYC 7402 24-Core Processor
Stepping:              0
CPU MHz:               3342.578
CPU max MHz:           2800.0000
CPU min MHz:           1500.0000
BogoMIPS:              5589.29
Virtualization:        AMD-V
L1d cache:             32K
L1i cache:             32K
L2 cache:              512K
L3 cache:              16384K
NUMA node0 CPU(s):     0-5,48-53
NUMA node1 CPU(s):     6-11,54-59
NUMA node2 CPU(s):     12-17,60-65
NUMA node3 CPU(s):     18-23,66-71
NUMA node4 CPU(s):     24-29,72-77
NUMA node5 CPU(s):     30-35,78-83
NUMA node6 CPU(s):     36-41,84-89
NUMA node7 CPU(s):     42-47,90-95
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate sme ssbd mba sev ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif umip rdpid overflow_recov succor smca

Versions of relevant libraries:
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.24.3
[pip3] torch==2.0.1
[pip3] triton==2.0.0
[conda] Could not collect

cc @zhaojuanmao @mrshenli @rohan-varma @awgu @fegin @penguinwu