Repository: pytorch
Issue: Unable to programmatically update models using references from model.named_modules()...requires additional parsing
Labels: module: nn,triaged,enhancement

### 🐛 Describe the bug

Users may need to programatically update models - for example, changing out specific layers or dynamically adding activation checkpoint_wrappers. 

While attempting to do this, I used a simple enumeration of a DeepViT model with 
~~~
for index, (name, layer) in enumerate(model.named_modules()):
        if isinstance(layer, type):
~~~
This correctly isolated the layers of interest.  However, when you go to update these layers with the name returned (i.e. transformer.layers.0.0.fn.fn.to_qkv) using setattr, you will find that rather than replacing the layer, you have simply appended a new layer to the model.   Setattr doesn't return any response, and so it's even harder programmatically to determine if the updates worked, or you just mangled your model with a bunch of new layers appended.

To fix this and actually update the model using the returned name from named_modules, requires some additional parsing:
~~~
pieces = name.strip().split(".")

            layer = model
            for item in pieces[:-1]:
                if not item.isnumeric():
                    layer = getattr(layer, item)
                else:
                    layer = layer[int(item)]
~~~
This is b/c when items are in a list, we would need to reference them via [0] instead of .0 which is what is returned by named_modules.
However, this is highly confusing and shouldn't require a user to figure this out and write their own parsing routine to offer programmatic updates. 

Details on this are here:
https://discuss.pytorch.org/t/how-to-replace-a-layer-with-own-custom-variant/43586/11




### Versions

PyTorch version: 1.13.0.dev20220523+cu113
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.6 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.22.4
Libc version: glibc-2.27

Python version: 3.8.12 | packaged by conda-forge | (default, Oct 12 2021, 21:59:51)  [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.4.0-1072-aws-x86_64-with-glibc2.10
Is CUDA available: True
CUDA runtime version: 11.1.105
GPU models and configuration: 
GPU 0: NVIDIA A100-SXM4-40GB
GPU 1: NVIDIA A100-SXM4-40GB
GPU 2: NVIDIA A100-SXM4-40GB
GPU 3: NVIDIA A100-SXM4-40GB
GPU 4: NVIDIA A100-SXM4-40GB
GPU 5: NVIDIA A100-SXM4-40GB
GPU 6: NVIDIA A100-SXM4-40GB
GPU 7: NVIDIA A100-SXM4-40GB

Nvidia driver version: 510.47.03
cuDNN version: Probably one of the following:
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.0.5
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.1.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] functorch==0.2.0a0+ae70048
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.21.2
[pip3] numpydoc==1.1.0
[pip3] torch==1.13.0.dev20220523+cu113
[pip3] torch-model-archiver==0.5.0b20211117
[pip3] torch-workflow-archiver==0.2.0b20211118
[pip3] torchdynamo==0.2.0
[pip3] torchserve==0.5.0b20211117
[pip3] torchtext==0.11.0
[pip3] torchvision==0.13.0.dev20220523+cu113
[pip3] vit-pytorch==0.33.2
[conda] blas                      1.0                         mkl  
[conda] captum                    0.4.1                         0    pytorch
[conda] cudatoolkit               11.1.1               h6406543_9    conda-forge
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] libblas                   3.9.0            12_linux64_mkl    conda-forge
[conda] libcblas                  3.9.0            12_linux64_mkl    conda-forge
[conda] liblapack                 3.9.0            12_linux64_mkl    conda-forge
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py38h95df7f1_0    conda-forge
[conda] mkl_fft                   1.3.1            py38h8666266_1    conda-forge
[conda] mkl_random                1.2.2            py38h1abd341_0    conda-forge
[conda] numpy                     1.21.2           py38h20f2e39_0  
[conda] numpy-base                1.21.2           py38h79a1101_0  
[conda] numpydoc                  1.1.0                      py_1    conda-forge
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torch                     1.13.0.dev20220523+cu113          pypi_0    pypi
[conda] torch-model-archiver      0.5.0                    py38_0    pytorch
[conda] torch-workflow-archiver   0.2.0                    py38_0    pytorch
[conda] torchdynamo               0.2.0                     dev_0    <develop>
[conda] torchserve                0.5.0                    py38_0    pytorch
[conda] torchtext                 0.11.0                     py38    pytorch
[conda] torchvision               0.13.0.dev20220523+cu113          pypi_0    pypi
[conda] vit-pytorch               0.33.2                   pypi_0    pypi

cc @albanD @mruberry @jbschlosser @walterddr @kshitij12345