Repository: flutter
Issue: NestedScrollView with RefreshIndicator: dragging effect of RefreshIndicator seems to disappear with a slight drag
Labels: framework,f: material design,f: scrolling,has reproducible steps,P2,team-design,triaged-design,found in release: 3.19,found in release: 3.22

### Steps to reproduce

1.After placing RefreshIndicator inside NestedScrollView,
2.When there's a lot of content in NestedScrollView's headerSliverBuilder,
3.The dragging effect for pulling down to refresh disappears, and a slight drag immediately triggers the refresh action.

### Expected results

The consistent dragging effect when content is less in headerSliverBuilder

### Actual results

After nesting RefreshIndicator within NestedScrollView, when the content of NestedScrollView's headerSliverBuilder is extensive, the dragging effect of pulling down for refresh seems to disappear. A slight drag immediately triggers the refresh action.

### Code sample

<details open><summary>Code sample</summary>

```dart
class MyHomePage extends StatefulWidget {
  const MyHomePage({super.key, required this.title});

  final String title;

  @override
  State<MyHomePage> createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage>
    with SingleTickerProviderStateMixin {
  late TabController _tabController;

  @override
  void initState() {
    super.initState();
    _tabController = TabController(length: 3, vsync: this);
  }

  @override
  void dispose() {
    _tabController.dispose();
    super.dispose();
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        backgroundColor: Theme.of(context).colorScheme.inversePrimary,
        title: Text(widget.title),
      ),
      body: RefreshIndicator(
        notificationPredicate: (notification) {
          return notification.depth == 2;
        },
        onRefresh: () async {
          await Future.delayed(const Duration(seconds: 2), () {});
        },
        child: NestedScrollView(
          headerSliverBuilder: (
            BuildContext context,
            bool innerBoxIsScrolled,
          ) {
            return [
              SliverToBoxAdapter(
                child: Container(
                  height: 1000, // Adjusting the simulated height of content in headerSliverBuilder
                  color: Colors.amber,
                  alignment: Alignment.center,
                  child: const Text("header"),
                ),
              ),
            ];
          },
          body: TabBarView(
            controller: _tabController,
            children: [
              ListView(),
              ListView(),
              ListView(),
            ],
          ),
        ),
      ), // This trailing comma makes auto-formatting nicer for build methods.
    );
  }
}
```

</details>


### Screenshots or Video

https://github.com/flutter/flutter/assets/17977044/44ff84d3-b465-4871-b94e-d7e96735fafb


### Logs

<details open><summary>Logs</summary>

```console
```

</details>


### Flutter Doctor output

<details open><summary>Doctor output</summary>

```console
[✓] Flutter (Channel stable, 3.19.5, on macOS 14.4.1 23E224 darwin-x64, locale zh-Hans-CN)
[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.3)
[!] Xcode - develop for iOS and macOS (Xcode 14.2)
    ! CocoaPods 1.10.2 out of date (1.13.0 is recommended).
        CocoaPods is used to retrieve the iOS and macOS platform side's plugin code that responds to your plugin usage on the Dart side.
        Without CocoaPods, plugins will not work on iOS or macOS.
        For more info, see https://flutter.dev/platform-plugins
      To upgrade see https://guides.cocoapods.org/using/getting-started.html#updating-cocoapods for instructions.
[✓] Chrome - develop for the web
[✓] Android Studio (version 2021.2)
[✓] VS Code (version 1.87.2)
[✓] Connected device (3 available)
```

</details>