Repository: flutter
Issue: Zooming makes it difficult to scroll to the top and bottom ends.
Labels: framework,f: scrolling,f: gestures,has reproducible steps,P3,team-framework,triaged-framework,found in release: 3.19,found in release: 3.20

### Steps to reproduce

1. Run the attached sample,
2. Zoom using two fingers,
3. Then scroll using one finger to the top and bottom,
4. Can't able to scroll till bottom and top.

### Expected results

When the widget is zoomed in, it should be able to scroll to the top and bottom.

### Actual results

When the widget is zoomed in, you cannot able to scroll to the top and bottom.

### Code sample

<details open><summary>Code sample</summary>

```dart
import 'package:flutter/material.dart';

void main() {
  runApp(const MyApp());
}

class MyApp extends StatelessWidget {
  const MyApp({super.key});

  @override
  Widget build(BuildContext context) {
    return const MaterialApp(
      title: 'Zooming using InteractiveViewer',
      home: MyHomePage(title: 'InteractiveViewer sample'),
    );
  }
}

class MyHomePage extends StatefulWidget {
  const MyHomePage({super.key, required this.title});

  final String title;

  @override
  State<MyHomePage> createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> {
  ScrollController scrollingController = ScrollController();
  bool isScrolling = true;
  final touchEvent = [];

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: Text(widget.title),
      ),
      body: Center(
          child: Listener(
        onPointerDown: (event) {
          touchEvent.add(event.pointer);
        },
        onPointerUp: (event) {
          touchEvent.clear();
          if (!isScrolling) {
            setState(() {
              isScrolling = true;
            });
          }
        },
        onPointerMove: (event) {
          if (touchEvent.length == 2) {
            if (isScrolling) {
              setState(() {
                isScrolling = false;
              });
            }
          }
        },
        child: InteractiveViewer(
          child: SingleChildScrollView(
            controller: scrollingController,
            physics: isScrolling
                ? const AlwaysScrollableScrollPhysics()
                : const NeverScrollableScrollPhysics(),
            child: Column(
              children: List.generate(10, (position) {
                return Padding(
                    padding: const EdgeInsets.all(8.0),
                    child: Container(
                      color: Colors.teal,
                      width: MediaQuery.of(context).size.width,
                      height: 350,
                      child: Center(
                        child: Text('Page number $position'),
                      ),
                    ));
              }),
            ),
          ),
        ),
      )),
    );
  }

  @override
  void dispose() {
    scrollingController.dispose();
    super.dispose();
  }
}

```

</details>


### Screenshots or Video

<details open>
<summary>Screenshots / Video demonstration</summary>

https://github.com/flutter/flutter/assets/105287978/0d991404-06d6-42d2-9269-225b5eb76d01


</details>


### Flutter Doctor output

<details open><summary>Doctor output</summary>

```console
[√] Flutter (Channel stable, 3.16.5, on Microsoft Windows [Version 10.0.19045.4046], locale en-US)
    • Flutter version 3.16.5 on channel stable at D:\Flutter SDK\flutter     
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 78666c8dc5 (2 months ago), 2023-12-19 16:14:14 -0800
    • Engine revision 3f3e560236
    • Dart version 3.2.3
    • DevTools version 2.28.4

[√] Windows Version (Installed version of Windows is version 10 or higher)   

[√] Android toolchain - develop for Android devices (Android SDK version 34.0.0) 
    • Android SDK at C:\Users\GowthamanRavichandra\AppData\Local\Android\sdk     
    • Platform android-34, build-tools 34.0.0
    • Java binary at: C:\Program Files\Android\Android Studio1\jbr\bin\java      
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-b2043.56-10027231)
    • All Android licenses accepted.

[√] Chrome - develop for the web
    • Chrome at C:\Program Files\Google\Chrome\Application\chrome.exe

[X] Visual Studio - develop Windows apps
    X Visual Studio not installed; this is necessary to develop Windows apps.
      Download at https://visualstudio.microsoft.com/downloads/.
      Please install the "Desktop development with C++" workload, including all of its default components

[!] Android Studio (version 2021.3)
    • Android Studio at C:\Program Files\Android\Android Studio
    • Flutter plugin can be installed from:
       https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
       https://plugins.jetbrains.com/plugin/6351-dart
    X Unable to determine bundled Java version.
    • Try updating or re-installing Android Studio.

[√] Android Studio (version 2022.3)
    • Android Studio at C:\Program Files\Android\Android Studio1
    • Flutter plugin can be installed from:
       https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
       https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-b2043.56-10027231)

[√] VS Code, 64-bit edition (version 1.85.0)
    • VS Code at C:\Program Files\Microsoft VS Code
    • Flutter extension version 3.82.0

[√] Connected device (4 available)
    • moto g32 (mobile) • ZD222FBQYC • android-arm64  • Android 13 (API 33)
    • Windows (desktop) • windows    • windows-x64    • Microsoft Windows [Version 10.0.19045.4046]
    • Chrome (web)      • chrome     • web-javascript • Google Chrome 122.0.6261.71
    • Edge (web)        • edge       • web-javascript • Microsoft Edge 122.0.2365.59

[√] Network resources
    • All expected network resources are available.

! Doctor found issues in 2 categories.
```

</details>