Repository: flutter
Issue: Flutter onPanStart invokes late when the widget is wrapped inside InteractiveViewer
Labels: framework,f: material design,f: gestures,has reproducible steps,found in release: 3.3,found in release: 3.7,team-design,triaged-design

## Steps to Reproduce

1. Run `flutter create bug`.
2. Update `main.dart` as follows: 
```
import 'package:flutter/material.dart';
import 'package:flutter/rendering.dart';

void main() {
  debugPaintSizeEnabled = true;
  runApp(MyApp());
}

class MyApp extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Flutter Demo',
      home: MyHomePage(),
    );
  }
}

class MyHomePage extends StatefulWidget {
  @override
  _MyHomePageState createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> {
  double _width = 300;
  double _height = 200;

  @override
  Widget build(BuildContext context) {
    final m = MediaQuery.of(context);
    return Scaffold(
      body: SafeArea(
        child: Container(
          height: m.size.height,
          width: m.size.width,
          child: InteractiveViewer(
            maxScale: 30,
            child: Stack(
              children: <Widget>[
                Container(height: _height, width: _width, color: Colors.red),
                Positioned(
                  top: _height / 2 - 10,
                  left: _width - 20,
                  child: GestureDetector(
                    onPanStart: (details) {
                      print("PAN STARTED");
                    },
                    onPanUpdate: (details) {
                      final delta = details.delta;
                      setState(() {
                        _width = _width + delta.dx;
                      });
                    },
                    child: Container(height: 20, width: 20, color: Colors.blue),
                  ),
                )
              ],
            ),
          ),
        ),
      ),
    );
  }
}
```

I just have a simple container which has a resize handle on center right which means the container can be resized using that handle. It is wrapped inside InteractiveViewer widget.

**Expected results:**
I expect the `onPanStart` callback to invoke right away.

**Actual results:**
`onPanStart` callback invokes after certain delay.

Note: This problem can only be replicated in simulators. In DartPad, it seems to work fine but still has a very little delay.

<details>
  <summary>Logs</summary>

```
[✓] Flutter (Channel stable, 1.22.5, on macOS 11.0.1 20B29 darwin-x64, locale en-US)
    • Flutter version 1.22.5 at /Users/jaden/development/tools/flutter
    • Framework revision 7891006299 (2 weeks ago), 2020-12-10 11:54:40 -0800
    • Engine revision ae90085a84
    • Dart version 2.10.4


[✓] Android toolchain - develop for Android devices (Android SDK version 29.0.2)
    • Android SDK at /Users/jaden/Library/Android/sdk
    • Platform android-29, build-tools 29.0.2
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6222593)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 12.2)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 12.2, Build version 12B45b
    • CocoaPods version 1.8.4

[!] Android Studio (version 4.0)
    • Android Studio at /Applications/Android Studio.app/Contents
    ✗ Flutter plugin not installed; this adds Flutter specific functionality.
    ✗ Dart plugin not installed; this adds Dart specific functionality.
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6222593)

[✓] VS Code (version 1.52.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.18.0

[✓] Connected device (2 available)
    • Android SDK built for x86 (mobile) • emulator-5554                        • android-x86 • Android 7.0 (API 24) (emulator)
    • iPhone 12 Pro Max (mobile)         • A2D7546A-0E0D-430F-92F0-F9EF1ADBEFF9 • ios         • com.apple.CoreSimulator.SimRuntime.iOS-14-2 (simulator)

! Doctor found issues in 1 category.
```

</details>