Repository: pytorch
Issue: Forward method slows down at some point if it is executed repeatedly on Android with NNAPI
Labels: module: android,oncall: mobile,mobile_perf

## 🐛 Bug

We built pytorch for Android with `LITE_INTERPRETER` and converted our original fully quantized model (with static quantization) that contains only conv and relu to NNAPI supported format.
Test inference code worked successfully on Android. But forward method slows down at some point if it is executed repeatedly.  
We'd like to apply our model to video frame sequence. For real time processing, the latency is important. So, this behavior is critical issue for us.  

## To Reproduce

Steps to reproduce the behavior:

1. Follow this [link](https://github.com/pytorch/pytorch/tree/master/android#building-pytorch-android-from-source) to build pytorch from source with `BUILD_LITE_INTERPRETER=1`.
1. Convert our original model to NNAPI format by referring to [the conversion of MobileNetV2](https://pytorch.org/tutorials/prototype/nnapi_mobilenetv2.html).
1. Prebuilt library and our original converted model is attached here. [aar_and_model.zip](https://github.com/pytorch/pytorch/files/7336585/aar_and_model.zip)
1. Link to prebuilt libtorch library from gradle dependency.
1. Run inference repeatedly on native code via jni.

App `build.gradle`  

```build.gradle
plugins {
    id 'com.android.application'
    id 'kotlin-android'
}

android {
    viewBinding {
        enabled = true
    }
    configurations {
        extractForNativeBuild
    }
    compileSdkVersion 30
    buildToolsVersion "30.0.3"

    defaultConfig {
        applicationId "jp.co.navier.pytorchmobilesample"
        minSdkVersion 28
        targetSdkVersion 30
        versionCode 1
        versionName "1.0"

        testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
        externalNativeBuild {
            cmake {
                cppFlags ""
                arguments "-DANDROID_STL=c++_shared"
            }
        }
        ndk {
            abiFilters 'arm64-v8a'
        }
    }

    buildTypes {
        release {
            minifyEnabled false
            proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro'
        }
    }
    externalNativeBuild {
        cmake {
            path "src/main/cpp/CMakeLists.txt"
            version "3.10.2"
        }
    }
    compileOptions {
        sourceCompatibility JavaVersion.VERSION_1_8
        targetCompatibility JavaVersion.VERSION_1_8
    }
    kotlinOptions {
        jvmTarget = '1.8'
    }
    aaptOptions {
        noCompress "pt"
    }

}

dependencies {

    implementation "org.jetbrains.kotlin:kotlin-stdlib:$kotlin_version"
    implementation 'androidx.core:core-ktx:1.6.0'
    implementation 'androidx.appcompat:appcompat:1.3.1'
    implementation 'com.google.android.material:material:1.4.0'
    implementation 'androidx.constraintlayout:constraintlayout:2.1.0'
    testImplementation 'junit:junit:4.+'
    androidTestImplementation 'androidx.test.ext:junit:1.1.3'
    androidTestImplementation 'androidx.test.espresso:espresso-core:3.4.0'

    implementation project(path: ':pytorch_android-release')
    extractForNativeBuild project(path: ':pytorch_android-release')
}

task extractAARForNativeBuild {
    doLast {
        configurations.extractForNativeBuild.files.each {
            def file = it.absoluteFile
            copy {
                from zipTree(file)
                into "$buildDir/$file.name"
                include "headers/**"
                include "jni/**"
            }
        }
    }
}

tasks.whenTaskAdded { task ->
    if (task.name.contains('externalNativeBuild')) {
        task.dependsOn(extractAARForNativeBuild)
    }
}
```

Inference code sample:

```c++
void testLoadAndForward(
        JNIEnv *env,
        jstring jModelPath)
{
    c10::InferenceMode guard;
    const char* modelPath = env->GetStringUTFChars(jModelPath, 0);

    torch::jit::mobile::Module module = torch::jit::_load_for_mobile(modelPath);

    auto get_method = module.find_method("get_all_bundled_inputs");
    auto all_inputs = (*get_method)({}).toList();
    auto inputs = all_inputs.get(0).toTuple()->elements();

    c10::IValue t_out;
    int iter = 100;
    for (int i = 0; i < iter; ++i) {
        std::chrono::system_clock::time_point start = std::chrono::system_clock::now();
        t_out = module.forward(inputs);
        std::chrono::system_clock::time_point end = std::chrono::system_clock::now();
        auto duration = std::chrono::duration_cast<std::chrono::nanoseconds>(end - start).count() / 1000000.0;
        LOGD("%d duration: %lf msec", i, duration);
    }
}
```

Log (From 38 iteration, inference time is obviously slower than previous one. 0th iteration is for warmup.):

```
D/NATIVE_CODE: 0 duration: 9284.693000 msec
D/NATIVE_CODE: 1 duration: 79.760000 msec
D/NATIVE_CODE: 2 duration: 78.123000 msec
D/NATIVE_CODE: 3 duration: 78.468000 msec
D/NATIVE_CODE: 4 duration: 77.882000 msec
D/NATIVE_CODE: 5 duration: 78.987000 msec
D/NATIVE_CODE: 6 duration: 77.897000 msec
D/NATIVE_CODE: 7 duration: 78.845000 msec
D/NATIVE_CODE: 8 duration: 78.278000 msec
D/NATIVE_CODE: 9 duration: 77.879000 msec
D/NATIVE_CODE: 10 duration: 78.108000 msec
D/NATIVE_CODE: 11 duration: 78.109000 msec
D/NATIVE_CODE: 12 duration: 78.286000 msec
D/NATIVE_CODE: 13 duration: 78.376000 msec
D/NATIVE_CODE: 14 duration: 78.368000 msec
D/NATIVE_CODE: 15 duration: 78.295000 msec
D/NATIVE_CODE: 16 duration: 78.213000 msec
D/NATIVE_CODE: 17 duration: 78.345000 msec
D/NATIVE_CODE: 18 duration: 77.938000 msec
D/NATIVE_CODE: 19 duration: 78.280000 msec
D/NATIVE_CODE: 20 duration: 78.160000 msec
D/NATIVE_CODE: 21 duration: 78.574000 msec
D/NATIVE_CODE: 22 duration: 78.711000 msec
D/NATIVE_CODE: 23 duration: 78.490000 msec
D/NATIVE_CODE: 24 duration: 78.499000 msec
D/NATIVE_CODE: 25 duration: 78.521000 msec
D/NATIVE_CODE: 26 duration: 78.173000 msec
D/NATIVE_CODE: 27 duration: 78.130000 msec
D/NATIVE_CODE: 28 duration: 77.988000 msec
D/NATIVE_CODE: 29 duration: 78.555000 msec
D/NATIVE_CODE: 30 duration: 78.568000 msec
D/NATIVE_CODE: 31 duration: 78.566000 msec
D/NATIVE_CODE: 32 duration: 80.458000 msec
D/NATIVE_CODE: 33 duration: 79.139000 msec
D/NATIVE_CODE: 34 duration: 78.517000 msec
D/NATIVE_CODE: 35 duration: 78.168000 msec
D/NATIVE_CODE: 36 duration: 78.810000 msec
D/NATIVE_CODE: 37 duration: 78.541000 msec
D/NATIVE_CODE: 38 duration: 87.629000 msec
D/NATIVE_CODE: 39 duration: 89.959000 msec
D/NATIVE_CODE: 40 duration: 93.983000 msec
D/NATIVE_CODE: 41 duration: 97.401000 msec
D/NATIVE_CODE: 42 duration: 96.597000 msec
D/NATIVE_CODE: 43 duration: 90.576000 msec
D/NATIVE_CODE: 44 duration: 92.451000 msec
D/NATIVE_CODE: 45 duration: 97.267000 msec
D/NATIVE_CODE: 46 duration: 97.906000 msec
D/NATIVE_CODE: 47 duration: 97.167000 msec
D/NATIVE_CODE: 48 duration: 94.146000 msec
D/NATIVE_CODE: 49 duration: 94.663000 msec
D/NATIVE_CODE: 50 duration: 93.778000 msec
D/NATIVE_CODE: 51 duration: 99.721000 msec
D/NATIVE_CODE: 52 duration: 97.753000 msec
D/NATIVE_CODE: 53 duration: 97.071000 msec
D/NATIVE_CODE: 54 duration: 97.035000 msec
D/NATIVE_CODE: 55 duration: 101.194000 msec
D/NATIVE_CODE: 56 duration: 102.563000 msec
D/NATIVE_CODE: 57 duration: 99.331000 msec
D/NATIVE_CODE: 58 duration: 98.304000 msec
D/NATIVE_CODE: 59 duration: 91.064000 msec
D/NATIVE_CODE: 60 duration: 96.322000 msec
D/NATIVE_CODE: 61 duration: 94.546000 msec
D/NATIVE_CODE: 62 duration: 93.846000 msec
D/NATIVE_CODE: 63 duration: 98.800000 msec
D/NATIVE_CODE: 64 duration: 96.586000 msec
D/NATIVE_CODE: 65 duration: 99.268000 msec
D/NATIVE_CODE: 66 duration: 96.090000 msec
D/NATIVE_CODE: 67 duration: 96.336000 msec
D/NATIVE_CODE: 68 duration: 99.795000 msec
D/NATIVE_CODE: 69 duration: 100.935000 msec
D/NATIVE_CODE: 70 duration: 99.763000 msec
D/NATIVE_CODE: 71 duration: 98.782000 msec
D/NATIVE_CODE: 72 duration: 101.088000 msec
D/NATIVE_CODE: 73 duration: 101.943000 msec
D/NATIVE_CODE: 74 duration: 99.099000 msec
D/NATIVE_CODE: 75 duration: 99.384000 msec
D/NATIVE_CODE: 76 duration: 91.306000 msec
D/NATIVE_CODE: 77 duration: 93.050000 msec
D/NATIVE_CODE: 78 duration: 92.522000 msec
D/NATIVE_CODE: 79 duration: 91.041000 msec
D/NATIVE_CODE: 80 duration: 99.242000 msec
D/NATIVE_CODE: 81 duration: 99.460000 msec
D/NATIVE_CODE: 82 duration: 99.413000 msec
D/NATIVE_CODE: 83 duration: 99.119000 msec
D/NATIVE_CODE: 84 duration: 102.010000 msec
D/NATIVE_CODE: 85 duration: 98.925000 msec
D/NATIVE_CODE: 86 duration: 99.047000 msec
D/NATIVE_CODE: 87 duration: 95.927000 msec
D/NATIVE_CODE: 88 duration: 95.569000 msec
D/NATIVE_CODE: 89 duration: 94.219000 msec
D/NATIVE_CODE: 90 duration: 96.446000 msec
D/NATIVE_CODE: 91 duration: 96.937000 msec
D/NATIVE_CODE: 92 duration: 95.171000 msec
D/NATIVE_CODE: 93 duration: 90.502000 msec
D/NATIVE_CODE: 94 duration: 97.297000 msec
D/NATIVE_CODE: 95 duration: 100.063000 msec
D/NATIVE_CODE: 96 duration: 97.878000 msec
D/NATIVE_CODE: 97 duration: 97.199000 msec
D/NATIVE_CODE: 98 duration: 99.090000 msec
D/NATIVE_CODE: 99 duration: 92.561000 msec
```

## Expected behavior

Inference should be performed at the same level of inference speed with no latency degradation.

## Environment

 - PyTorch Version (e.g., 1.0): 1.9.1
 - OS (e.g., Linux): macOS 10.15.7 (x86_64)
 - How you installed PyTorch (`conda`, `pip`, source): source
 - Build command you used (if compiling from source): `BUILD_LITE_INTERPRETER=1 ./scripts/build_pytorch_android.sh arm64-v8a`
 - Python version: 3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 13:42:17)  [GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] (64-bit runtime)
 - CUDA/cuDNN version: No CUDA
 - GPU models and configuration: No CUDA
 - Any other relevant information:
     - Android deivce: Xiaomi Mi 11
     - Android version: 11 RKQ11.200928.002


cc @VitalyFedyunin @ngimel