Repository: pytorch/glow

Issue #2247: [model zoo] Different results running ResNet50 models
Created: 2019-01-09T10:33:14Z
In provided examples there are 2 ResNet50 available. First one (let's call it RN1) is Caffe2 model from fb-glow-assets.s3.amazonaws.com/models/ and second one (RN2) is ONNX model from s3.amazonaws.com/download.onnx/models/opset_6/
What I noticed is that when running batch size bigger than 1 RN2 is returning only first result correctly and rest has random values. The problem can be spotted on the low-level IR level.
For RN1 in declare:
%gpu_0_data = WeightVar float<16 x 3 x 224 x 224> mutable // size: 9633792 // Users: @in 1
%save_gpu_0_softmax = WeightVar float<16 x 1000> mutable // size: 64000 // Users: @out 237 

For RN2 in declare:
%gpu_0_data_0 = WeightVar float<16 x 3 x 224 x 224> mutable // size: 9633792 // Users: @in 1
%save_gpu_0_softmax_1 = WeightVar float<1 x 1000> mutable // size: 4000 // Users: @out 243

And the results I have are following:
RN1:
./bin/image-classifier tests/images/imagenet/16/*.png -use-imagenet-normalization -image-mode=0to1 -m=resnet50 -model-input-name=gpu_0/data -cpu -dump-ir
Model: resnet50
 File: tests/images/imagenet/16/cat_285_1.png   Label-K1: 281 (probability: 0.7190)
 File: tests/images/imagenet/16/cat_285_2.png   Label-K1: 281 (probability: 0.7190)
 File: tests/images/imagenet/16/cat_285_4.png   Label-K1: 281 (probability: 0.7190)
 File: tests/images/imagenet/16/cat_285_5.png   Label-K1: 281 (probability: 0.7190)
 File: tests/images/imagenet/16/cat_285.png     Label-K1: 281 (probability: 0.7190)
 File: tests/images/imagenet/16/dog_207_1.png   Label-K1: 207 (probability: 0.9446)
 File: tests/images/imagenet/16/dog_207_2.png   Label-K1: 207 (probability: 0.9446)
 File: tests/images/imagenet/16/dog_207_4.png   Label-K1: 207 (probability: 0.9446)
 File: tests/images/imagenet/16/dog_207_5.png   Label-K1: 207 (probability: 0.9446)
 File: tests/images/imagenet/16/dog_207.png     Label-K1: 207 (probability: 0.9446)
 File: tests/images/imagenet/16/zebra_340_1.png Label-K1: 340 (probability: 0.9984)
 File: tests/images/imagenet/16/zebra_340_2.png Label-K1: 340 (probability: 0.9984)
 File: tests/images/imagenet/16/zebra_340_3.png Label-K1: 340 (probability: 0.9984)
 File: tests/images/imagenet/16/zebra_340_4.png Label-K1: 340 (probability: 0.9984)
 File: tests/images/imagenet/16/zebra_340_5.png Label-K1: 340 (probability: 0.9984)
 File: tests/images/imagenet/16/zebra_340.png   Label-K1: 340 (probability: 0.9984)

RN2:
./bin/image-classifier tests/images/imagenet/16/*.png -use-imagenet-normalization -image-mode=0to1 -m=onnx_models/resnet50/model.onnx -model-input-name=gpu_0/data_0 -cpu -dump-ir
Model: onnx_models/resnet50/model.onnx
 File: tests/images/imagenet/16/cat_285_1.png   Label-K1: 281 (probability: 0.7190)
 File: tests/images/imagenet/16/cat_285_2.png   Label-K1: 803 (probability: 109501926753866426517948838075957248.0000)
 File: tests/images/imagenet/16/cat_285_4.png   Label-K1: 8 (probability: 1163137744525318537797146186350592.0000)
 File: tests/images/imagenet/16/cat_285_5.png   Label-K1: 163 (probability: 200735535189990277129236968606233788416.0000)
 File: tests/images/imagenet/16/cat_285.png     Label-K1: 140 (probability: 73989688674039732580357437915136.0000)
 File: tests/images/imagenet/16/dog_207_1.png   Label-K1: 492 (probability: 1198804886080946637596203847516160.0000)
 File: tests/images/imagenet/16/dog_207_2.png   Label-K1: 200 (probability: 76717526031150600707147634076286976.0000)
 File: tests/images/imagenet/16/dog_207_4.png   Label-K1: 444 (probability: 1163137744525318537797146186350592.0000)
 File: tests/images/imagenet/16/dog_207_5.png   Label-K1: 60 (probability: 76717526031150600707147634076286976.0000)
 File: tests/images/imagenet/16/dog_207.png     Label-K1: 308 (probability: 76717526031150600707147634076286976.0000)
 File: tests/images/imagenet/16/zebra_340_1.png Label-K1: 674 (probability: 1209009612681038303883533234470912.0000)
 File: tests/images/imagenet/16/zebra_340_2.png Label-K1: 212 (probability: 1163137744525318537797146186350592.0000)
 File: tests/images/imagenet/16/zebra_340_3.png Label-K1: 760 (probability: 1163137744525318537797146186350592.0000)
 File: tests/images/imagenet/16/zebra_340_4.png Label-K1: 679 (probability: 77457197409704310398483475464192.0000)
 File: tests/images/imagenet/16/zebra_340_5.png Label-K1: 890 (probability: 5948536641811781986122581567044845568.0000)
 File: tests/images/imagenet/16/zebra_340.png   Label-K1: 392 (probability: 1163137744525318537797146186350592.0000)

Looking at the dumped IR I can see that for some reason in RN2 Glow adds at the very end extra tensorview instruction which is effectively reducing batch size from 16 to 1.
232 %gpu_0_pool5_12_res = allocactivation  { Ty: float<16 x 2048 x 1 x 1>} // size: 131072 // Users: @out 238, @in 235, @out 233
233 %gpu_0_pool5_12 = transpose @out %gpu_0_pool5_12_res, @in %gpu_0_pool5_11_res { Shuffle: [0, 3, 1, 2]}
234 %dealloc121 = deallocactivation @out %gpu_0_pool5_11_res // size: 131072
235 %tensorview_reshape = tensorview @in %gpu_0_pool5_12_res { Ty: float<1 x 2048>, Offsets: [0, 0, 0, 0]} // Users: @in 237
236 %copy_reshape_res = allocactivation  { Ty: float<1 x 2048>} // size: 8192 // Users: @out 241, @in 240, @out 237
237 %copy_reshape = copy @out %copy_reshape_res, @in %tensorview_reshape
238 %dealloc122 = deallocactivation @out %gpu_0_pool5_12_res // size: 131072
239 %gpu_0_pred_11_res = allocactivation  { Ty: float<1 x 1000>} // size: 4000 // Users: @in 243, @out 242, @out 244, @in 242, @out 240
240 %gpu_0_pred_11 = matmul @out %gpu_0_pred_11_res, @in %copy_reshape_res, @in %gpu_0_pred_w_01
241 %dealloc123 = deallocactivation @out %copy_reshape_res // size: 8192
242 %gpu_0_pred_12 = elementadd @out %gpu_0_pred_11_res, @in %gpu_0_pred_11_res, @in %gpu_0_pred_b_01
243 %gpu_0_softmax_1 = softmax @out %save_gpu_0_softmax_1, @in %gpu_0_pred_11_res
244 %dealloc124 = deallocactivation @out %gpu_0_pred_11_res // size: 4000

Duplicate issue #1517: [RFC] Error reporting, malformed graph, and general resilience
Created: 2018-08-23T19:30:24Z
Context
Glow checks are for the most part bound to whether or not asserts are enabled in the build.
The rationale is we expect the inputs (network, weights) to be correctly shaped and sized when we invoke the compiler/loaders and thus this design eliminates unnecessary overhead.
Problems
Given most checks are removed in release mode, we end up in a situation where we cannot use this variant when bringing up new works.
For instance, silly mistakes like shaping incorrectly a transpose will produce random compiler errors or just invalid/inaccurate outputs with no way to tell what is going wrong.
Then, running the same network with a debug variant would trigger a verifier failure from the beginning, hinting directly at what the problem was. This process is far from being intuitive and is cumbersome.
In essence, we have the following problems:

The debug mode gives us diagnostics but is slow to run.
The release mode is fast to run but does not gives any information on what is going on.
The error reporting through asserts has too little context to be actionable on its own.

To make a comparison, Glow behaves like a compiler that wouldn't issue a syntax error diagnostic if compiled with assert disabled. On top of that, when the assertions are enabled, errors like type mismatch would be reported bluntly: no location in the source file, no naming of the type that was causing the problem and so on.
In a nutshell, when working on a new model, the process may look like this:

Run the network with the release variant
=> compiler crashes
Run the network with the debug variant
=> assertion failure "invalid..."
Run the network with the debug variant in a debugger
=> assertion failure "invalid...", then print variables, examine the memory, etc. to see what is wrong.

Finally, for large networks, it is possible that the debug mode is too slow to be practical, making debugging very hard.
Ultimate Goal
Ideally, we would like to be sure that only valid IR hit the compiler. Then, in the compiler itself, it seems reasonable to rely on asserts for things related to the IR, like the invariants after such and such optimizations and so on. Indeed, at this point, any problem occurring in the compiler is the result of a wrong doing of the compiler itself and thus not actionable by the user.
Of course, we would still need to have some kind of non-assert based error reporting for anything that is not in the control of the compiler. E.g., failing to write to the filesystem, out-of-memory allocation, network too big for the targeted backend, etc.
How Do We Get There?
We basically have two approaches to make sure only valid IR hit the compiler:

Have every "frontend" of Glow performs the checks (e.g., like the protobuf loaders).
Have Glow performs the checks before doing anything.

The second approach would be the preferred way as it focused the burden of validating the IR in only one place. However, this potentially makes error reporting more difficult for the backends. Therefore, we likely would want both.
Regarding the general error reporting, I believe we would want to have the front-end to register a diagnostic handler in Glow, so that the reporting is properly plumbed there. This is less critical than ensuring that the IR is valid as most users's errors should be caught by the front-end itself.
What About Getting Rid of the Unnecessary Checks Then?
With the proposed approach, we don't really have a way to get rid of the checks that valid the IR, so how do we avoid to waste resources when we are running for the same network over and over?
We see two possible directions here:

Something higher level of Glow, could cache the result of the compilation and such, then it wouldn't invoke Glow or its front-end/loader again and again and instead would directly run what it already had.
We could feature a "reckless" mode that effectively disable every error reporting in both the front-end and Glow. This mode wouldn't be the default like it is right now.

For #2, the idea would be that only trusted networks would be run. This could be checked for instance with a whitelist embedded in the compiler/loader.
What do people think?
cc: @nadavrot, @artemrakhov, @opti-mix, @stoklund, @jfix71, @rdzhabarov, @hegemanjwh2, @beicy, @bertmaher, @tlepley.

Closing comment (2019-01-09T18:57:06Z): It seems we aren't calling verify anywhere in Release mode. At this point is there any reason we can't/shouldn't add verify checks in the ONNX/Caffe2 model importers? E.g. before returning success here, add RETURN_ERR_IF_NOT(F.verify(), "Function verification failed.");:

  
    
      glow/lib/Importer/ONNXModelLoader.cpp
    
    
        Lines 963 to 979
      in
      884d40b
    
  
  
    

        
          
           auto setup = [&]() -> llvm::Error { 
        

        
          
             // The ONNX model that we are deserializing. 
        

        
          
             ONNX_NAMESPACE::ModelProto modelDef; 
        

        
          
             ASSIGN_VALUE_OR_RETURN_ERR(modelDef, loadProto(modelDescFilename)); 
        

        
          
            
        

        
          
             RETURN_IF_ERR(setVersion(modelDef)); 
        

        
          
            
        

        
          
             ONNX_NAMESPACE::GraphProto graphDef = modelDef.graph(); 
        

        
          
             RETURN_IF_ERR(checkInputs(graphDef, tensorNames, types)); 
        

        
          
            
        

        
          
             RETURN_IF_ERR(loadInitializers(graphDef)); 
        

        
          
             RETURN_IF_ERR(loadNetwork(graphDef)); 
        

        
          
            
        

        
          
             RETURN_IF_ERR(setOutputNodes(graphDef)); 
        

        
          
            
        

        
          
             return llvm::Error::success(); 
        

        
          
           }; 
        
    
  


Related to #2247