Repository: pytorch
Issue: Converting to Torch Script: cpp_module does not match nn_module
Labels: high priority,triage review,oncall: jit,triaged

## 🐛 Bug

When trying to convert a yolov3 model to Torch Script I get the following error:
`  File "D:\python\lib\site-packages\torch\nn\modules\module.py", line 576, in __getattr__
    type(self).__name__, name))
AttributeError: 'Sequential' object has no attribute 'shortcut_4'` (full stacktrace below)

The model itself works fine during training and inferencing. 

While debugging the conversion of the script, I can clearly see that the network's layers are correct: [https://i.imgur.com/ckx1AaN.png](https://i.imgur.com/ckx1AaN.png)
During conversion to Torch Script, the issue is caused by 9th Sequential module , which contains the EmptyLayer with id `shortcut_4`

Debugging further, I got to the following function in '**torch.jit._recursive.py**':
```
def create_script_module(nn_module, stubs):
    check_module_initialized(nn_module)
    concrete_type = concrete_type_store.get_or_create_concrete_type(nn_module)
    cpp_module = torch._C._create_module_with_type(concrete_type.jit_type)

    return create_script_module_impl(nn_module, concrete_type, cpp_module, stubs)
```
I don't understand what exactly is happening but the resulting `cpp_module` variable is created with `shortcut_4` as a property, while it should have `shortcut_8` as shown in the image above, since it is the 9th Module. It seems the `get_or_create_concrete_type` function gets the module created before with property `shortcut_4`. For some reason the cpp_module created from the nn_module has a different key/name, see here: [https://i.imgur.com/eMol4oX.png](https://i.imgur.com/eMol4oX.png)

## To Reproduce

Steps to reproduce the behavior:

1. Run the code below:
```
import cv2
import numpy as np
import torch
import torch.nn as nn


def parse_cfg(cfgfile):
    """
    Takes a configuration file

    Returns a list of blocks. Each blocks describes a block in the neural
    network to be built. Block is represented as a dictionary in the list
    """
    file = open(cfgfile, 'r')
    lines = file.read().split('\n')  # store the lines in a list
    lines = [x for x in lines if len(x) > 0]  # get read of the empty lines
    lines = [x for x in lines if x[0] != '#']
    lines = [x.rstrip().lstrip() for x in lines]

    block = {}
    blocks = []

    for line in lines:
        if line[0] == "[":  # This marks the start of a new block
            if len(block) != 0:
                blocks.append(block)
                block = {}
            block["type"] = line[1:-1].rstrip()
        else:
            key, value = line.split("=")
            block[key.rstrip()] = value.lstrip()
    blocks.append(block)

    return blocks


def create_modules(blocks):
    net_info = blocks[0]  # Captures the information about the input and pre-processing
    module_list = nn.ModuleList()
    index = 0  # indexing blocks helps with implementing route  layers (skip connections)
    prev_filters = 3
    output_filters = []

    for x in blocks:
        module = nn.Sequential()

        if x["type"] == "net":
            continue

        # If it's a convolutional layer
        if x["type"] == "convolutional":
            # Get the info about the layer
            activation = x["activation"]
            if "batch_normalize" in x:
                batch_normalize = int(x["batch_normalize"])
                bias = False
            else:
                batch_normalize = 0
                bias = True

            filters = int(x["filters"])
            padding = int(x["pad"])
            kernel_size = int(x["size"])
            stride = int(x["stride"])

            if padding:
                pad = (kernel_size - 1) // 2
            else:
                pad = 0

            # Add the convolutional layer
            conv = nn.Conv2d(prev_filters, filters, kernel_size, stride, pad, bias=bias)
            module.add_module("conv_{0}".format(index), conv)

            # Add the Batch Norm Layer
            if batch_normalize:
                bn = nn.BatchNorm2d(filters)
                module.add_module("batch_norm_{0}".format(index), bn)

            # Check the activation.
            # It is either Linear or a Leaky ReLU for YOLO
            if activation == "leaky":
                activn = nn.LeakyReLU(0.1, inplace=True)
                module.add_module("leaky_{0}".format(index), activn)

        # If it's an upsampling layer
        # We use Bilinear2dUpsampling
        elif x["type"] == "upsample":
            stride = int(x["stride"])
            upsample = nn.Upsample(scale_factor=2, mode="nearest")
            module.add_module("upsample_{}".format(index), upsample)

        # If it is a route layer
        elif x["type"] == "route":
            x["layers"] = x["layers"].split(',')

            # Start  of a route
            start = int(x["layers"][0])
            if len(x["layers"]) <= 2:
                # end, if there exists one.
                if len(x["layers"]) > 1:
                    end = int(x["layers"][1])
                else:
                    end = 0

                # Positive anotation
                if start > 0:
                    start = start - index

                if end > 0:
                    end = end - index

                route = EmptyLayer("route_{0}".format(index))
                module.add_module("route_{0}".format(index), route)

                if end < 0:
                    filters = output_filters[index + start] + output_filters[index + end]
                else:
                    filters = output_filters[index + start]
            else:  # SPP-route
                route = EmptyLayer("route_{0}".format(index))
                module.add_module("route_{0}".format(index), route)
                filters = output_filters[index + start] + output_filters[index + int(x["layers"][1])] \
                          + output_filters[index + int(x["layers"][2])] + output_filters[index + int(x["layers"][3])]

        # shortcut corresponds to skip connection
        elif x["type"] == "shortcut":
            shortcut = EmptyLayer("shortcut_{}".format(index))
            module.add_module("shortcut_{}".format(index), shortcut)
        elif x["type"] == "maxpool":
            stride = int(x["stride"])
            size = int(x["size"])
            if stride != 1:
                maxpool = nn.MaxPool2d(size, stride)
            else:
                maxpool = MaxPoolStride1(size)
            module.add_module("maxpool_{}".format(index), maxpool)

        # Yolo is the detection layer
        elif x["type"] == "yolo":
            mask = x["mask"].split(",")
            mask = [int(x) for x in mask]

            anchors = x["anchors"].split(",")
            anchors = [int(a) for a in anchors]
            anchors = [(anchors[i], anchors[i + 1]) for i in range(0, len(anchors), 2)]
            anchors = [anchors[i] for i in mask]

            detection = DetectionLayer(anchors)
            module.add_module("Detection_{}".format(index), detection)

        else:
            pass

        module_list.append(module)
        prev_filters = filters
        output_filters.append(filters)
        index += 1

    return net_info, module_list


def convert2cpu(matrix):
    if matrix.is_cuda:
        return torch.FloatTensor(matrix.size()).copy_(matrix)
    else:
        return matrix


def predict_transform(prediction, inp_dim, anchors, num_classes):
    batch_size = prediction.size(0)
    stride = inp_dim // prediction.size(2)
    grid_size = inp_dim // stride
    bbox_attrs = 5 + num_classes
    num_anchors = len(anchors)

    anchors = [(a[0] / stride, a[1] / stride) for a in anchors]

    prediction = prediction.view(batch_size, bbox_attrs * num_anchors, grid_size * grid_size)
    prediction = prediction.transpose(1, 2).contiguous()
    prediction = prediction.view(batch_size, grid_size * grid_size * num_anchors, bbox_attrs)

    # Sigmoid the  centre_X, centre_Y. and object confidencce
    prediction[:, :, 0] = torch.sigmoid(prediction[:, :, 0])
    prediction[:, :, 1] = torch.sigmoid(prediction[:, :, 1])
    prediction[:, :, 4] = torch.sigmoid(prediction[:, :, 4])

    # Add the center offsets
    grid_len = np.arange(grid_size)
    a, b = np.meshgrid(grid_len, grid_len)

    x_offset = torch.FloatTensor(a).view(-1, 1)
    y_offset = torch.FloatTensor(b).view(-1, 1)

    x_y_offset = torch.cat((x_offset, y_offset), 1).repeat(1, num_anchors).view(-1, 2).unsqueeze(0)

    prediction[:, :, :2] += x_y_offset

    # log space transform height and the width
    anchors = torch.FloatTensor(anchors)

    anchors = anchors.repeat(grid_size * grid_size, 1).unsqueeze(0)
    prediction[:, :, 2:4] = torch.exp(prediction[:, :, 2:4]) * anchors

    # Softmax the class scores
    prediction[:, :, 5: 5 + num_classes] = torch.sigmoid((prediction[:, :, 5: 5 + num_classes]))

    prediction[:, :, :4] *= stride

    return prediction


class Darknet(nn.Module):

    def __init__(self, cfgfile, weightfile, input_dim):
        super(Darknet, self).__init__()
        self.blocks = parse_cfg(cfgfile)
        self.net_info, self.module_list = create_modules(self.blocks)
        self.net_info['height'] = input_dim
        self.header = torch.IntTensor([0, 0, 0, 0])
        self.seen = 0
        self.load_weights(weightfile)

    def forward(self, x):
        detections = []
        modules = self.blocks[1:]
        outputs = {}  # We cache the outputs for the route layer

        write = 0
        for i in range(len(modules)):
            module_type = (modules[i]["type"])
            if module_type == "convolutional" or module_type == "upsample" or module_type == "maxpool":

                x = self.module_list[i](x)
                outputs[i] = x
            elif module_type == "route":
                layers = modules[i]["layers"]
                layers = [int(a) for a in layers]

                if (layers[0]) > 0:
                    layers[0] = layers[0] - i

                if len(layers) == 1:
                    x = outputs[i + (layers[0])]

                elif len(layers) == 2:
                    if (layers[1]) > 0:
                        layers[1] = layers[1] - i

                    map1 = outputs[i + layers[0]]
                    map2 = outputs[i + layers[1]]

                    x = torch.cat((map1, map2), 1)
                elif len(layers) == 4:  # SPP
                    map1 = outputs[i + layers[0]]
                    map2 = outputs[i + layers[1]]
                    map3 = outputs[i + layers[2]]
                    map4 = outputs[i + layers[3]]

                    x = torch.cat((map1, map2, map3, map4), 1)
                outputs[i] = x

            elif module_type == "shortcut":
                from_ = int(modules[i]["from"])
                x = outputs[i - 1] + outputs[i + from_]
                outputs[i] = x
            elif module_type == 'yolo':
                anchors = self.module_list[i][0].anchors
                # Get the input dimensions
                inp_dim = int(self.net_info["height"])

                # Get the number of classes
                num_classes = int(modules[i]["classes"])

                # Output the result
                x = x.data
                x = predict_transform(x, inp_dim, anchors, num_classes)

                if type(x) == int:
                    continue

                if not write:
                    detections = x
                    write = 1

                else:
                    detections = torch.cat((detections, x), 1)

                outputs[i] = outputs[i - 1]

        return detections

    def load_weights(self, weightfile):
        # Open the weights file
        fp = open(weightfile, "rb")

        # The first 4 values are header information
        # 1. Major version number
        # 2. Minor Version Number
        # 3. Subversion number
        # 4. IMages seen
        header = np.fromfile(fp, dtype=np.int32, count=5)
        self.header = torch.from_numpy(header)
        self.seen = self.header[3]

        # The rest of the values are the weights
        # Let's load them up
        weights = np.fromfile(fp, dtype=np.float32)

        ptr = 0
        for i in range(len(self.module_list)):
            module_type = self.blocks[i + 1]["type"]

            if module_type == "convolutional":
                model = self.module_list[i]

                if "batch_normalize" in self.blocks[i + 1]:
                    batch_normalize = int(self.blocks[i + 1]["batch_normalize"])
                else:
                    batch_normalize = 0

                conv = model[0]

                if (batch_normalize):
                    bn = model[1]

                    # Get the number of weights of Batch Norm Layer
                    num_bn_biases = bn.bias.numel()

                    # Load the weights
                    bn_biases = torch.from_numpy(weights[ptr:ptr + num_bn_biases])
                    ptr += num_bn_biases

                    bn_weights = torch.from_numpy(weights[ptr: ptr + num_bn_biases])
                    ptr += num_bn_biases

                    bn_running_mean = torch.from_numpy(weights[ptr: ptr + num_bn_biases])
                    ptr += num_bn_biases

                    bn_running_var = torch.from_numpy(weights[ptr: ptr + num_bn_biases])
                    ptr += num_bn_biases

                    # Cast the loaded weights into dims of model weights.
                    bn_biases = bn_biases.view_as(bn.bias.data)
                    bn_weights = bn_weights.view_as(bn.weight.data)
                    bn_running_mean = bn_running_mean.view_as(bn.running_mean)
                    bn_running_var = bn_running_var.view_as(bn.running_var)

                    # Copy the data to model
                    bn.bias.data.copy_(bn_biases)
                    bn.weight.data.copy_(bn_weights)
                    bn.running_mean.copy_(bn_running_mean)
                    bn.running_var.copy_(bn_running_var)

                else:
                    # Number of biases
                    num_biases = conv.bias.numel()

                    # Load the weights
                    conv_biases = torch.from_numpy(weights[ptr: ptr + num_biases])
                    ptr = ptr + num_biases

                    # reshape the loaded weights according to the dims of the model weights
                    conv_biases = conv_biases.view_as(conv.bias.data)

                    # Finally copy the data
                    conv.bias.data.copy_(conv_biases)

                # Let us load the weights for the Convolutional layers
                num_weights = conv.weight.numel()

                # Do the same as above for weights
                conv_weights = torch.from_numpy(weights[ptr:ptr + num_weights])
                ptr = ptr + num_weights

                conv_weights = conv_weights.view_as(conv.weight.data)
                conv.weight.data.copy_(conv_weights)


class EmptyLayer(nn.Module):
    def __init__(self, name):
        super(EmptyLayer, self).__init__()
        self.name = name

    def forward(self, x):
        return x


class DetectionLayer(nn.Module):
    def __init__(self, anchors):
        super(DetectionLayer, self).__init__()
        self.anchors = anchors

    def forward(self, x, inp_dim, num_classes, confidence):
        x = x.data
        prediction = x
        prediction = predict_transform(prediction, inp_dim, self.anchors, num_classes, confidence)
        return prediction


class MaxPoolStride1(nn.Module):
    def __init__(self, kernel_size):
        super(MaxPoolStride1, self).__init__()
        self.kernel_size = kernel_size
        self.pad = kernel_size - 1

    def forward(self, x):
        padding = int(self.pad / 2)
        padded_x = nn.functional.pad(x, (padding, padding, padding, padding), mode="constant", value=0)
        pooled_x = nn.MaxPool2d(self.kernel_size, 1)(padded_x)
        return pooled_x


def letterbox_image(img, inp_dim):
    img_w, img_h = img.shape[1], img.shape[0]
    w, h = inp_dim
    new_w = int(img_w * min(w / img_w, h / img_h))
    new_h = int(img_h * min(w / img_w, h / img_h))
    resized_image = cv2.resize(img, (new_w, new_h), interpolation=cv2.INTER_CUBIC)

    canvas = np.full((inp_dim[1], inp_dim[0], 3), 128)

    canvas[(h - new_h) // 2:(h - new_h) // 2 + new_h, (w - new_w) // 2:(w - new_w) // 2 + new_w, :] = resized_image

    return canvas


def prep_frame(frame, inp_dim):
    dim = frame.shape[1], frame.shape[0]
    dim = torch.FloatTensor(dim).repeat(1, 2)
    img = (letterbox_image(frame, (inp_dim, inp_dim)))
    img_ = img[:, :, ::-1].transpose((2, 0, 1)).copy()
    img_ = torch.from_numpy(img_).float().div(255.0).unsqueeze(0)
    return img_, dim


if __name__ == "__main__":
    input_dim = 608

    img = cv2.imread('test.jpg')
    image, dimensions = prep_frame(img, input_dim)

    model = Darknet('/models/yolo/yolov3-spp.cfg',
                    '/models/yolo/yolov3-spp.weights', input_dim)
    model.eval()

    script_module = torch.jit.script(model)
    print(script_module.code)

```
#### Full stacktrace
```
Traceback (most recent call last):
  File "D:/dev/yolo/script.py", line 228, in <module>
    script_module = torch.jit.script(model)
  File "D:\python\lib\site-packages\torch\jit\__init__.py", line 1255, in script
    return torch.jit._recursive.recursive_script(obj)
  File "D:\python\lib\site-packages\torch\jit\_recursive.py", line 534, in recursive_script
    return create_script_module(nn_module, infer_methods_to_compile(nn_module))
  File "D:\python\lib\site-packages\torch\jit\_recursive.py", line 296, in create_script_module
    return create_script_module_impl(nn_module, concrete_type, cpp_module, stubs)
  File "D:\python\lib\site-packages\torch\jit\_recursive.py", line 336, in create_script_module_impl
    script_module = torch.jit.RecursiveScriptModule._construct(cpp_module, init_fn)
  File "D:\python\lib\site-packages\torch\jit\__init__.py", line 1593, in _construct
    init_fn(script_module)
  File "D:\python\lib\site-packages\torch\jit\_recursive.py", line 328, in init_fn
    scripted = recursive_script(orig_value)
  File "D:\python\lib\site-packages\torch\jit\_recursive.py", line 534, in recursive_script
    return create_script_module(nn_module, infer_methods_to_compile(nn_module))
  File "D:\python\lib\site-packages\torch\jit\_recursive.py", line 296, in create_script_module
    return create_script_module_impl(nn_module, concrete_type, cpp_module, stubs)
  File "D:\python\lib\site-packages\torch\jit\_recursive.py", line 336, in create_script_module_impl
    script_module = torch.jit.RecursiveScriptModule._construct(cpp_module, init_fn)
  File "D:\python\lib\site-packages\torch\jit\__init__.py", line 1593, in _construct
    init_fn(script_module)
  File "D:\python\lib\site-packages\torch\jit\_recursive.py", line 328, in init_fn
    scripted = recursive_script(orig_value)
  File "D:\python\lib\site-packages\torch\jit\_recursive.py", line 534, in recursive_script
    return create_script_module(nn_module, infer_methods_to_compile(nn_module))
  File "D:\python\lib\site-packages\torch\jit\_recursive.py", line 296, in create_script_module
    return create_script_module_impl(nn_module, concrete_type, cpp_module, stubs)
  File "D:\python\lib\site-packages\torch\jit\_recursive.py", line 336, in create_script_module_impl
    script_module = torch.jit.RecursiveScriptModule._construct(cpp_module, init_fn)
  File "D:\python\lib\site-packages\torch\jit\__init__.py", line 1593, in _construct
    init_fn(script_module)
  File "D:\python\lib\site-packages\torch\jit\_recursive.py", line 321, in init_fn
    orig_value = getattr(nn_module, name)
  File "D:\python\lib\site-packages\torch\nn\modules\module.py", line 576, in __getattr__
    type(self).__name__, name))
AttributeError: 'Sequential' object has no attribute 'shortcut_4'
```
## Expected behavior

Script should convert model to Torch Script succesfully.

## Environment

PyTorch version: 1.4.0
Is debug build: No
CUDA used to build PyTorch: 10.1

OS: Microsoft Windows 10 Pro
GCC version: Could not collect
CMake version: version 3.17.1

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 10.2.89
GPU models and configuration: GPU 0: GeForce GTX 980
Nvidia driver version: 441.22
cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\bin\cudnn64_7.dll

Versions of relevant libraries:
[pip3] numpy==1.18.1
[pip3] torch==1.4.0
[pip3] torchfile==0.1.0
[pip3] torchtext==0.4.0
[pip3] torchvision==0.5.0
[conda] Could not collect

cc @ezyang @gchanan @zou3519 @suo