Repository: pytorch
Issue: TensorRTOpTest.test_vgg19 is flaky
Labels: triaged,module: flaky-tests

Sample failure: https://circleci.com/gh/pytorch/pytorch/632436?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link

```
Jan 25 01:19:46 =================================== FAILURES ===================================
Jan 25 01:19:46 __________________________ TensorRTOpTest.test_vgg19 ___________________________
Jan 25 01:19:46 
Jan 25 01:19:46 self = <caffe2.python.trt.test_trt.TensorRTOpTest testMethod=test_vgg19>
Jan 25 01:19:46 
Jan 25 01:19:46     @unittest.skipIf(not workspace.C.use_trt, "No TensortRT support")
Jan 25 01:19:46     def test_vgg19(self):
Jan 25 01:19:46 >       self._test_onnx_importer('vgg19', -2, 9)
Jan 25 01:19:46 
Jan 25 01:19:46 ../.local/lib/python2.7/site-packages/caffe2/python/trt/test_trt.py:176: 
Jan 25 01:19:46 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
Jan 25 01:19:46 ../.local/lib/python2.7/site-packages/caffe2/python/trt/test_trt.py:127: in _test_onnx_importer
Jan 25 01:19:46     op = convert_onnx_model_to_trt_op(model_def, verbosity=3)
Jan 25 01:19:46 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
Jan 25 01:19:46 
Jan 25 01:19:46 onnx_model = ir_version: 3
Jan 25 01:19:46 producer_name: "onnx-caffe2"
Jan 25 01:19:46 graph {
Jan 25 01:19:46   node {
Jan 25 01:19:46     input: "data_0...    }
Jan 25 01:19:46         }
Jan 25 01:19:46       }
Jan 25 01:19:46     }
Jan 25 01:19:46   }
Jan 25 01:19:46 }
Jan 25 01:19:46 opset_import {
Jan 25 01:19:46   domain: ""
Jan 25 01:19:46   version: 9
Jan 25 01:19:46 }
Jan 25 01:19:46 
Jan 25 01:19:46 max_batch_size = 64, max_workspace_size = 2097152, verbosity = 3
Jan 25 01:19:46 debug_builder = False
Jan 25 01:19:46 
Jan 25 01:19:46     def convert_onnx_model_to_trt_op(onnx_model,
Jan 25 01:19:46             max_batch_size=64,
Jan 25 01:19:46             max_workspace_size=2*1024*1024,
Jan 25 01:19:46             verbosity=1,
Jan 25 01:19:46             debug_builder=False):
Jan 25 01:19:46         """
Jan 25 01:19:46         Convert the whole ONNX model to a TensorRT C2 op
Jan 25 01:19:46         """
Jan 25 01:19:46         check_gpu_()
Jan 25 01:19:46         trt_str = C.onnx_to_trt_op(onnx_model.SerializeToString(),
Jan 25 01:19:46                                    _get_output_shapes(onnx_model.graph.output),
Jan 25 01:19:46                                    max_batch_size,
Jan 25 01:19:46                                    max_workspace_size,
Jan 25 01:19:46                                    verbosity,
Jan 25 01:19:46 >                                  debug_builder)
Jan 25 01:19:46 E       RuntimeError: [enforce fail at trt_utils.h:43] obj. Failed to create TensorRt object
Jan 25 01:19:46 E       frame #0: c10::ThrowEnforceNotMet(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void const*) + 0x68 (0x7febb5761be8 in /var/lib/jenkins/.local/lib/python2.7/site-packages/caffe2/python/../../torch/lib/libc10.so)
Jan 25 01:19:46 E       frame #1: caffe2::tensorrt::BuildTrtEngine(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, caffe2::tensorrt::TrtLogger*, unsigned long, unsigned long, bool) + 0xbe2 (0x7febb6df95e2 in /var/lib/jenkins/.local/lib/python2.7/site-packages/caffe2/python/../../torch/lib/libcaffe2_gpu.so)
Jan 25 01:19:46 E       frame #2: caffe2::TensorRTTransformer::BuildTrtOp(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<int, std::allocator<int> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::vector<int, std::allocator<int> > > > > const&) + 0xbd (0x7febb6dfdb4d in /var/lib/jenkins/.local/lib/python2.7/site-packages/caffe2/python/../../torch/lib/libcaffe2_gpu.so)
Jan 25 01:19:46 E       frame #3: <unknown function> + 0xea487 (0x7febd7734487 in /var/lib/jenkins/.local/lib/python2.7/site-packages/caffe2/python/caffe2_pybind11_state_gpu.so)
Jan 25 01:19:46 E       frame #4: <unknown function> + 0x90c00 (0x7febd76dac00 in /var/lib/jenkins/.local/lib/python2.7/site-packages/caffe2/python/caffe2_pybind11_state_gpu.so)
Jan 25 01:19:46 E       frame #5: PyEval_EvalFrameEx + 0x5ca (0x4bc4aa in /usr/bin/python2)
Jan 25 01:19:46 E       frame #6: PyEval_EvalCodeEx + 0x306 (0x4b9b66 in /usr/bin/python2)
Jan 25 01:19:46 E       frame #7: PyEval_EvalFrameEx + 0x58e6 (0x4c17c6 in /usr/bin/python2)
Jan 25 01:19:46 E       frame #8: PyEval_EvalCodeEx + 0x306 (0x4b9b66 in /usr/bin/python2)
Jan 25 01:19:46 E       frame #9: PyEval_EvalFrameEx + 0x58e6 (0x4c17c6 in /usr/bin/python2)
Jan 25 01:19:46 E       frame #10: PyEval_EvalCodeEx + 0x306 (0x4b9b66 in /usr/bin/python2)
Jan 25 01:19:46 E       frame #11: PyEval_EvalFrameEx + 0x6076 (0x4c1f56 in /usr/bin/python2)
Jan 25 01:19:46 E       frame #12: PyEval_EvalCodeEx + 0x306 (0x4b9b66 in /usr/bin/python2)
Jan 25 01:19:46 E       frame #13: /usr/bin/python2() [0x4d57a3]
Jan 25 01:19:46 E       frame #14: PyObject_Call + 0x3e (0x4a587e in /usr/bin/python2)
Jan 25 01:19:46 E       frame #15: PyEval_EvalFrameEx + 0x263e (0x4be51e in /usr/bin/python2)
Jan 25 01:19:46 E       frame #16: PyEval_EvalCodeEx + 0x306 (0x4b9b66 in /usr/bin/python2)
Jan 25 01:19:46 E       frame #17: /usr/bin/python2() [0x4d57a3]
Jan 25 01:19:46 E       frame #18: /usr/bin/python2() [0x4eef5e]
Jan 25 01:19:46 E       frame #19: PyObject_Call + 0x3e (0x4a587e in /usr/bin/python2)
Jan 25 01:19:46 E       frame #20: /usr/bin/python2() [0x548fc3]
Jan 25 01:19:46 E       frame #21: PyEval_EvalFrameEx + 0x578d (0x4c166d in /usr/bin/python2)
Jan 25 01:19:46 E       frame #22: PyEval_EvalCodeEx + 0x306 (0x4b9b66 in /usr/bin/python2)
Jan 25 01:19:46 E       frame #23: PyEval_EvalFrameEx + 0x6076 (0x4c1f56 in /usr/bin/python2)
Jan 25 01:19:46 E       frame #24: PyEval_EvalCodeEx + 0x306 (0x4b9b66 in /usr/bin/python2)
Jan 25 01:19:46 E       frame #25: /usr/bin/python2() [0x4d5669]
Jan 25 01:19:46 E       frame #26: PyObject_Call + 0x3e (0x4a587e in /usr/bin/python2)
Jan 25 01:19:46 E       frame #27: PyEval_EvalFrameEx + 0x263e (0x4be51e in /usr/bin/python2)
Jan 25 01:19:46 E       frame #28: PyEval_EvalCodeEx + 0x306 (0x4b9b66 in /usr/bin/python2)
Jan 25 01:19:46 E       frame #29: PyEval_EvalFrameEx + 0x58e6 (0x4c17c6 in /usr/bin/python2)
Jan 25 01:19:46 E       frame #30: PyEval_EvalCodeEx + 0x306 (0x4b9b66 in /usr/bin/python2)
Jan 25 01:19:46 E       frame #31: PyEval_EvalFrameEx + 0x6076 (0x4c1f56 in /usr/bin/python2)
Jan 25 01:19:46 E       frame #32: PyEval_EvalFrameEx + 0x553f (0x4c141f in /usr/bin/python2)
Jan 25 01:19:46 E       frame #33: PyEval_EvalCodeEx + 0x306 (0x4b9b66 in /usr/bin/python2)
Jan 25 01:19:46 E       frame #34: /usr/bin/python2() [0x4d57a3]
Jan 25 01:19:46 E       frame #35: /usr/bin/python2() [0x4eef5e]
Jan 25 01:19:46 E       frame #36: PyObject_Call + 0x3e (0x4a587e in /usr/bin/python2)
Jan 25 01:19:46 E       frame #37: /usr/bin/python2() [0x548fc3]
Jan 25 01:19:46 E       frame #38: PyObject_Call + 0x3e (0x4a587e in /usr/bin/python2)
Jan 25 01:19:46 E       frame #39: PyEval_EvalFrameEx + 0x263e (0x4be51e in /usr/bin/python2)
Jan 25 01:19:46 E       frame #40: PyEval_EvalCodeEx + 0x306 (0x4b9b66 in /usr/bin/python2)
Jan 25 01:19:46 E       frame #41: PyEval_EvalFrameEx + 0x6076 (0x4c1f56 in /usr/bin/python2)
Jan 25 01:19:46 E       frame #42: PyEval_EvalCodeEx + 0x306 (0x4b9b66 in /usr/bin/python2)
Jan 25 01:19:46 E       frame #43: /usr/bin/python2() [0x4d57a3]
Jan 25 01:19:46 E       frame #44: /usr/bin/python2() [0x4eef5e]
Jan 25 01:19:46 E       frame #45: /usr/bin/python2() [0x4eeb66]
Jan 25 01:19:46 E       frame #46: /usr/bin/python2() [0x4aaafb]
Jan 25 01:19:46 E       frame #47: PyEval_EvalFrameEx + 0x578d (0x4c166d in /usr/bin/python2)
Jan 25 01:19:46 E       frame #48: PyEval_EvalCodeEx + 0x306 (0x4b9b66 in /usr/bin/python2)
Jan 25 01:19:46 E       frame #49: /usr/bin/python2() [0x4d57a3]
Jan 25 01:19:46 E       frame #50: PyObject_Call + 0x3e (0x4a587e in /usr/bin/python2)
Jan 25 01:19:46 E       frame #51: PyEval_EvalFrameEx + 0x263e (0x4be51e in /usr/bin/python2)
Jan 25 01:19:46 E       frame #52: PyEval_EvalCodeEx + 0x306 (0x4b9b66 in /usr/bin/python2)
Jan 25 01:19:46 E       frame #53: PyEval_EvalFrameEx + 0x58e6 (0x4c17c6 in /usr/bin/python2)
Jan 25 01:19:46 E       frame #54: PyEval_EvalCodeEx + 0x306 (0x4b9b66 in /usr/bin/python2)
Jan 25 01:19:46 E       frame #55: PyEval_EvalFrameEx + 0x58e6 (0x4c17c6 in /usr/bin/python2)
Jan 25 01:19:46 E       frame #56: PyEval_EvalCodeEx + 0x306 (0x4b9b66 in /usr/bin/python2)
Jan 25 01:19:46 E       frame #57: /usr/bin/python2() [0x4d5669]
Jan 25 01:19:46 E       frame #58: PyObject_Call + 0x3e (0x4a587e in /usr/bin/python2)
Jan 25 01:19:46 E       frame #59: PyEval_EvalFrameEx + 0x263e (0x4be51e in /usr/bin/python2)
Jan 25 01:19:46 E       frame #60: PyEval_EvalCodeEx + 0x306 (0x4b9b66 in /usr/bin/python2)
Jan 25 01:19:46 E       frame #61: PyEval_EvalFrameEx + 0x58e6 (0x4c17c6 in /usr/bin/python2)
Jan 25 01:19:46 E       frame #62: PyEval_EvalCodeEx + 0x306 (0x4b9b66 in /usr/bin/python2)
Jan 25 01:19:46 E       frame #63: PyEval_EvalFrameEx + 0x6076 (0x4c1f56 in /usr/bin/python2)
Jan 25 01:19:46 
Jan 25 01:19:46 ../.local/lib/python2.7/site-packages/caffe2/python/trt/transform.py:51: RuntimeError
Jan 25 01:19:46 ----------------------------- Captured stdout call -----------------------------
```