Repository: pytorch
Issue: consume_prefix_in_state_dict_if_present can not remove prefix of _metadata.
Labels: oncall: distributed,module: ddp

### 🐛 Describe the bug

https://github.com/pytorch/pytorch/blob/0b1f3bd158bec2cf4ab6bfde573d691c65b0742d/torch/nn/modules/utils.py#L64

```
    if "_metadata" in state_dict:
```
This line always returns `False`, because `_metadta` is a Protected Attributes of state_dict, not a key.

![image](https://user-images.githubusercontent.com/34199488/158140580-a369288f-c429-4d26-9a0e-afa8b72b763e.png)

![image](https://user-images.githubusercontent.com/34199488/158141181-32fe8c14-710b-403f-937d-4308817e0279.png)


### Versions

```
Collecting environment information...
PyTorch version: 1.10.0
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.17

Python version: 3.7.11 (default, Jul 27 2021, 14:32:16)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-4.15.0-151-generic-x86_64-with-debian-buster-sid
Is CUDA available: True
CUDA runtime version: 11.3.109
GPU models and configuration: 
GPU 0: Tesla K80
GPU 1: Tesla K80

Nvidia driver version: 470.63.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.2.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.2
[pip3] torch==1.10.0
[pip3] torchdomain==0.0.1
[pip3] torchelastic==0.2.0
[pip3] torchtext==0.11.0
[pip3] torchvision==0.11.0
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.3.1               ha36c431_9    nvidia
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] libblas                   3.9.0            11_linux64_mkl    conda-forge
[conda] libcblas                  3.9.0            11_linux64_mkl    conda-forge
[conda] liblapack                 3.9.0            11_linux64_mkl    conda-forge
[conda] mkl                       2021.3.0           h06a4308_520  
[conda] mkl-service               2.4.0            py37h7f8727e_0  
[conda] mkl_fft                   1.3.1            py37hd3c417c_0  
[conda] mkl_random                1.2.2            py37h51133e4_0  
[conda] numpy                     1.21.2           py37h20f2e39_0  
[conda] numpy-base                1.21.2           py37h79a1101_0  
[conda] pytorch                   1.10.0          py3.7_cuda11.3_cudnn8.2.0_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchdomain               0.0.1                     dev_0    <develop>
[conda] torchelastic              0.2.0                    pypi_0    pypi
[conda] torchtext                 0.11.0                     py37    pytorch
[conda] torchvision               0.11.0               py37_cu113    pytorch
```

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang