Repository: flutter
Issue: Tapping outside of drawer twice to dismiss it breaks swipe to go back
Labels: framework,f: gestures,has reproducible steps,P2,team-design,triaged-design,found in release: 3.13,found in release: 3.14

### Is there an existing issue for this?

- [X] I have searched the [existing issues](https://github.com/flutter/flutter/issues)
- [X] I have read the [guide to filing a bug](https://flutter.dev/docs/resources/bug-reports)

### Steps to reproduce

1. In a child Scaffold, add a right drawer and open it.
2. Tap twice outside the drawer to dismiss it.
3. Attempt iOS swipe to go back (doesn't work).

### Expected results

Swipe to go back should still work after dismissing the drawer given any number of dismiss requests.

### Actual results

Swipe to go back doesn't work.

### Code sample

<details><summary>Code sample</summary>

```dart
class AppDrawer extends ConsumerWidget {
  final Service currService;
  final Hour currHour;
  final void Function(int hourIdx, int index) jumpToWidget;
  final void Function() closeDrawer;
  final bool drawerBackButton;
  const AppDrawer({
    required this.currService,
    required this.currHour,
    required this.jumpToWidget,
    required this.closeDrawer,
    this.drawerBackButton = false,
  });

  @override
  Widget build(BuildContext context, WidgetRef ref) {
    var state = ref.read(contentViewBuildStateProvider);
    final startIdx = state.startIdx;
    return ClipRRect(
      borderRadius: BorderRadius.horizontal(left: Radius.circular(15)),
      child: Drawer(
        width: useMobileLayout
            ? Theme.of(context).drawerTheme.width
            : MediaQuery.of(context).size.width / 3,
        child: SafeArea(
          child: Column(
```

</details>


### Screenshots or Video

https://github.com/flutter/flutter/assets/9276839/11c8f485-5412-4c98-9a90-dad623a5d125



### Logs

_No response_

### Flutter Doctor output

<details><summary>Doctor output</summary>

```console
Flutter 3.13.2 • channel stable • https://github.com/flutter/flutter.git
Framework • revision ff5b5b5fa6 (7 days ago) • 2023-08-24 08:12:28 -0500
Engine • revision b20183e040
Tools • Dart 3.1.0 • DevTools 2.25.0

Running flutter doctor...
Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel stable, 3.13.2, on macOS 13.5.1 22G90 darwin-arm64, locale en-US)
[✓] Android toolchain - develop for Android devices (Android SDK version 33.0.0)
[✓] Xcode - develop for iOS and macOS (Xcode 14.3.1)
[✗] Chrome - develop for the web (Cannot find Chrome executable at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome)
    ! Cannot find Chrome. Try setting CHROME_EXECUTABLE to a Chrome executable.
[✓] Android Studio (version 2022.3)
[!] Android Studio (version unknown)
    ✗ Unable to determine Android Studio version.
[✓] VS Code (version 1.81.1)
[✓] Connected device (1 available)
    ! Error: (null) needs to connect to determine its availability. Check the connection between the device and its companion iPhone, and the connection between the iPhone and Xcode. Both devices may also need to be restarted and unlocked. (code 1)
[✓] Network resources```

</details>