Repository: flutter
Issue: [go_router] Pop is not functional when using `refreshListenable`
Labels: c: crash,package,has reproducible steps,P2,p: go_router,found in release: 3.0,found in release: 3.1,team-go_router,triaged-go_router

## Steps to Reproduce

1. Clone the repo https://github.com/sarbagyastha/router
2. Checkout the `refresh` branch
3. Run the app and tap on "Go to Second page"
4. Tap on the back button in App Bar.

**Expected results:** I should be able to pop to the first page.

**Actual results:** I see the following error.
```
======== Exception caught by gesture ===============================================================
The following assertion was thrown while handling a gesture:
have popped the last page off of the stack; there are no pages left to show
'package:go_router/src/go_router_delegate.dart':
Failed assertion: line 71 pos 12: '_matches.isNotEmpty'

When the exception was thrown, this was the stack: 
#2      GoRouterDelegate.pop (package:go_router/src/go_router_delegate.dart:71:12)
#3      GoRouter.pop (package:go_router/src/go_router.dart:167:32)
#4      GoRouterHelper.pop (package:go_router/go_router.dart:75:35)
#5      GoRouterHelper.pop.<anonymous closure> (package:go_router/go_router.dart:75:8)
#6      BackButton.build.<anonymous closure> (package:flutter/src/material/back_button.dart:104:21)

```

<details>
  <summary>flutter doctor -v</summary>

```
[✓] Flutter (Channel stable, 3.0.2, on macOS 12.4 21F79 darwin-arm, locale en-NP)
    • Flutter version 3.0.2 at /Users/sarbagya/Development/flutter
    • Upstream repository ssh://git@github.com/flutter/flutter.git
    • FLUTTER_GIT_URL = ssh://git@github.com/flutter/flutter.git
    • Framework revision cd41fdd495 (9 days ago), 2022-06-08 09:52:13 -0700
    • Engine revision f15f824b57
    • Dart version 2.17.3
    • DevTools version 2.12.2

[✓] Android toolchain - develop for Android devices (Android SDK version 32.1.0-rc1)
    • Android SDK at /Users/sarbagya/Library/Android/sdk
    • Platform android-32, build-tools 32.1.0-rc1
    • ANDROID_HOME = /Users/sarbagya/Library/Android/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jre/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.12+0-b1504.28-7817840)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 13.4.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • CocoaPods version 1.11.3

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2021.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.12+0-b1504.28-7817840)

[✓] VS Code (version 1.68.0)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.42.0

[✓] Connected device (3 available)
    • iPhone 13 Pro Max (mobile) • 52A89D3C-1D4B-42EE-817A-C77345806588 • ios            • com.apple.CoreSimulator.SimRuntime.iOS-15-5 (simulator)
    • macOS (desktop)            • macos                                • darwin-arm64   • macOS 12.4 21F79 darwin-arm
    • Chrome (web)               • chrome                               • web-javascript • Google Chrome 102.0.5005.115

[✓] HTTP Host Availability
    • All required HTTP hosts are available

• No issues found!
```

</details>