Repository: pytorch
Issue: FSDP.forward() fails "_is_root should not have been set" error after saving a distributed checkpoint
Labels: oncall: distributed,triaged,module: fsdp

### 🐛 Describe the bug

In the example below, we're wrapping a simple model with FSDP, then saving a checkpoint using `torch.distributed.checkpoint` API. The checkpoint saves successfully, no errors. But in forward, an error occurs:

```
-- Process 1 terminated with the following error:
Traceback (most recent call last):
  File "/home/adrian/.conda/envs/lightning/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 74, in _wrap
    fn(i, *args)
  File "/home/adrian/repositories/lightning/examples/pytorch/bug_report/bug_report_model.py", line 47, in main
    _ = model(x1, x2)
  File "/home/adrian/.conda/envs/lightning/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1519, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/adrian/.conda/envs/lightning/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1528, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/adrian/.conda/envs/lightning/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 844, in forward
    args, kwargs = _root_pre_forward(self, self, args, kwargs)
  File "/home/adrian/.conda/envs/lightning/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 540, in _root_pre_forward
    _lazy_init(state, module)
  File "/home/adrian/.conda/envs/lightning/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 171, in _lazy_init
    _share_state_and_init_handle_attrs(state, root_module)
  File "/home/adrian/.conda/envs/lightning/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 239, in _share_state_and_init_handle_attrs
    _p_assert(
  File "/home/adrian/.conda/envs/lightning/lib/python3.10/site-packages/torch/distributed/utils.py", line 147, in _p_assert
    raise AssertionError(s)
AssertionError: Non-root FSDP instance's `_is_root` should not have been set yet or should have been set to `False`

```

**Reproducing code:**
```py
import os
import torch
import torch.multiprocessing as mp
import torch.distributed as dist
from torch.distributed.checkpoint import FileSystemWriter, save_state_dict
from torch.distributed.fsdp import FullyShardedDataParallel as FSDP
from torch.distributed.fsdp.api import StateDictType
from torch.distributed.fsdp.wrap import ModuleWrapPolicy


class BoringModel(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.model = torch.nn.Sequential(torch.nn.Linear(5, 5), torch.nn.ReLU(), torch.nn.Linear(5, 1))
        self.pretrained_model = torch.nn.Sequential(
            torch.nn.Linear(5, 5),
            torch.nn.ReLU(),
        )

    def forward(self, x1, x2):
        x2_hidden = self.pretrained_model(x2)
        hiddens = torch.cat([x1, x2_hidden], dim=1)
        return self.model(hiddens)


def main(rank):
    os.environ["MASTER_ADDR"] = "localhost"
    os.environ["MASTER_PORT"] = "12355 "
    dist.init_process_group("nccl", rank=rank, world_size=2)
    torch.cuda.set_device(rank)
    
    model = BoringModel()
    model = FSDP(model, auto_wrap_policy=ModuleWrapPolicy({torch.nn.Linear}), device_id=rank)

    state_dict_type_context = FSDP.state_dict_type(
        module=model, state_dict_type=StateDictType.SHARDED_STATE_DICT,
    )

    with state_dict_type_context:
        state_dict = model.pretrained_model.state_dict()
    
    writer = FileSystemWriter(path="model.ckpt")
    save_state_dict(state_dict, writer)

    x1 = torch.ones((2, 6, 5))
    x2 = torch.ones((2, 6, 5)) * 2
    _ = model(x1, x2)


if __name__ == "__main__":
    mp.spawn(main, nprocs=2)

```

PyTorch nightly 2.2.0.dev20230920+cu121
Reported originally in Lightning: https://github.com/Lightning-AI/lightning/issues/18944

### Versions

Collecting environment information...
PyTorch version: 2.2.0.dev20230920+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.26.0
Libc version: glibc-2.35

Python version: 3.10.9 (main, Jan 11 2023, 15:21:40) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-75-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA A100-SXM4-40GB
GPU 1: NVIDIA A100-SXM4-40GB
GPU 2: NVIDIA A100-SXM4-40GB
GPU 3: NVIDIA A100-SXM4-40GB
GPU 4: NVIDIA A100-SXM4-40GB
GPU 5: NVIDIA A100-SXM4-40GB
GPU 6: NVIDIA A100-SXM4-40GB
GPU 7: NVIDIA A100-SXM4-40GB

Nvidia driver version: 525.125.06
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.5
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Address sizes:                   43 bits physical, 48 bits virtual
Byte Order:                      Little Endian
CPU(s):                          256
On-line CPU(s) list:             0-254
Off-line CPU(s) list:            255
Vendor ID:                       AuthenticAMD
Model name:                      AMD EPYC 7742 64-Core Processor
CPU family:                      23
Model:                           49
Thread(s) per core:              2
Core(s) per socket:              64
Socket(s):                       2
Stepping:                        0
Frequency boost:                 enabled
CPU max MHz:                     2250.0000
CPU min MHz:                     0.0000
BogoMIPS:                        4499.83
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sme sev sev_es
Virtualization:                  AMD-V
L1d cache:                       4 MiB (128 instances)
L1i cache:                       4 MiB (128 instances)
L2 cache:                        64 MiB (128 instances)
L3 cache:                        512 MiB (32 instances)
NUMA node(s):                    2
NUMA node0 CPU(s):               0-63,128-191
NUMA node1 CPU(s):               64-127,192-254
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Not affected
Vulnerability Retbleed:          Mitigation; untrained return thunk; SMT enabled with STIBP protection
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Retpolines, IBPB conditional, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected

Versions of relevant libraries:
[pip3] mypy==1.4.1
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.24.2
[pip3] onnx==1.12.0
[pip3] onnxruntime==1.14.1
[pip3] pytorch-lightning==2.0.6
[pip3] pytorch-triton==2.1.0+6e4932cda8
[pip3] torch==2.2.0.dev20230920+cu121
[pip3] torchmetrics==1.0.3
[pip3] torchvision==0.17.0.dev20230920+cu121
[pip3] triton==2.0.0
[conda] numpy                     1.24.2                   pypi_0    pypi
[conda] pytorch-lightning         2.0.6                    pypi_0    pypi
[conda] pytorch-triton            2.1.0+6e4932cda8          pypi_0    pypi
[conda] torch                     2.2.0.dev20230920+cu121          pypi_0    pypi
[conda] torchmetrics              1.0.3                    pypi_0    pypi
[conda] torchvision               0.17.0.dev20230920+cu121          pypi_0    pypi
[conda] triton                    2.0.0                    pypi_0    pypi

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu @fegin @XilunWu @wanchaol @fduwjj @wz337 @tianyu-l @wconstab @yf225 @kiukchung @lucasllc @d4l3k