Repository: pytorch
Issue: Model with Caffe2 runs much slower than it with pytorch in GPU mode !!!!
Labels: caffe2

Firstly, I run my model(CRNN model) on Pytorch GPU. With code:
```
directory = os.path.dirname(os.path.realpath(__file__))
model_file = os.path.join(directory, './pth/netCRNN_epoch17_idx78105_step2733675.pth')
keys = new_key.KEY_4498
model = Resnet18(False, num_classes=len(keys) + 2)

state_dict = torch.load(model_file)
model.load_state_dict(state_dict)
model.cuda()
model.train(False)

begin = time.time()
tol_time = 0
for i in range(5000):
    x = torch.randn(1, 3, 32, 512, requires_grad=False).cuda()
    begin = time.time()
    out = model(x)
    end = time.time()
    tol_time += (end - begin)

print('cost time:', tol_time / 5000)
```
**Result:  cost time: 0.002339872884750366**

Secondly, I convert my pytorch model to onnx and then convert onnx to pb in caffe2. With code:

```
directory = os.path.dirname(os.path.realpath(__file__))
model_file = os.path.join(directory, './pth/netCRNN_epoch17_idx78105_step2733675.pth')
keys = new_key.KEY_4498
model = Resnet18(False, num_classes=len(keys) + 2)

state_dict = torch.load(model_file)
model.load_state_dict(state_dict)
model.cuda()
model.train(False)

x = torch.randn(1, 3, 32, 512, requires_grad=False).cuda()

# Export the model
torch_out = torch.onnx._export(model,  # model being run
                               x,  # model input (or a tuple for multiple inputs)
                               "ocr_api.onnx",  # where to save the model (can be a file or file-like object)
                               export_params=True)
```
and 
```
convert-onnx-to-caffe2 ocr_api.onnx --output predict_net.pb --init-net-output init_net.pb
```

Thirdly,I run model in Caffe2 on GPU.  With code:
```
workspace.ResetWorkspace()
device_opts = core.DeviceOption(caffe2_pb2.CUDA, 0)

input_data = np.random.rand(1, 3, 32, 128).astype(np.float32)  # NCHW

init_def = caffe2_pb2.NetDef()
with open('init_net.pb', 'rb') as f:
    init_def.ParseFromString(f.read())
    init_def.device_option.CopyFrom(device_opts)
    workspace.RunNetOnce(init_def)
    # workspace.RunNetOnce(init_def.SerializeToString())

net_def = caffe2_pb2.NetDef()
with open('predict_net.pb', 'rb') as f:
    net_def.ParseFromString(f.read())
    net_def.device_option.CopyFrom(device_opts)

    input_name = net_def.external_input[0]
    workspace.FeedBlob(input_name, input_data,device_opts)
    workspace.CreateNet(net_def)
    # workspace.CreateNet(net_def.SerializeToString())

name = net_def.name
output_name = net_def.external_output[-1]
input_name = net_def.external_input[0]
print(name, input_name, output_name)

num_iters = 5000

tol_time = 0
for i in range(num_iters):
    input_data = np.random.rand(1, 3, 32, 512).astype(np.float32)  # NCHW
    start = time.time()
    workspace.FeedBlob(input_name, input_data,device_opts)
    workspace.RunNet(name, 1)
    results = workspace.FetchBlob(output_name)
    end = time.time()
    tol_time += (end - start)

print('Run time per RunNet: {}'.format(tol_time / num_iters))
```
**Run time per RunNet: 0.005980247449874878**

**But, the speed is slower than that in Pytorch !  How weird!** 

Any one konws what happens!