Repository: flutter
Issue: ScrollController jumpTo fails assertion 'activity.isScrolling': is not true
Labels: c: crash,framework,f: scrolling,has reproducible steps,P2,workaround available,found in release: 3.7,found in release: 3.9,team-framework,triaged-framework

## Steps to Reproduce

Code in words:

I start of with an CustomScrollView with an initial offset. If the user scrolls to the top from this initial offset, I will direct him to another screen. But if the user scrolls to the top from another offset, i want the ScrollController to jump back to the initial offset. 

Code to reproduce:

```dart
import 'package:flutter/material.dart';

void main() {
  runApp(MyApp());
}

class MyApp extends StatelessWidget {
  // This widget is the root of your application.
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Flutter Demo',
      theme: ThemeData(
        primarySwatch: Colors.blue,
        visualDensity: VisualDensity.adaptivePlatformDensity,
      ),
      home: TestScreen(),
    );
  }
}

class TestScreen extends StatefulWidget {
  static double initialScrollOffset = 50;

  _TestScreenState createState() => _TestScreenState();
}

class _TestScreenState extends State<TestScreen> {
  ScrollController _controller = ScrollController(
    initialScrollOffset: TestScreen.initialScrollOffset,
  );
  double scrollStartOffset;

  @override
  void initState() {
    super.initState();
    _controller.addListener(_scrollListener);
  }

  @override
  void dispose() {
    super.dispose();
    _controller.dispose();
  }

  void _scrollListener() {
    if (_controller.offset == 0) {
      if (scrollStartOffset == TestScreen.initialScrollOffset) {
        // move to the next screen
      }
      _controller.jumpTo(TestScreen.initialScrollOffset);
    }
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        centerTitle: true,
        title: Text(
          'TestScreen',
        ),
      ),
      body: NotificationListener<ScrollNotification>(
        onNotification: (ScrollNotification scrollNotification) {
          if (scrollNotification is ScrollStartNotification) {
            scrollStartOffset = scrollNotification.metrics.pixels;
          }
          return;
        },
        child: CustomScrollView(
          controller: _controller,
          slivers: <Widget>[
            SliverToBoxAdapter(
              child: Container(
                height: TestScreen.initialScrollOffset,
                color: Colors.yellow,
              ),
            ),
            SliverToBoxAdapter(
              child: Container(
                height: 1000,
                color: Colors.red,
              ),
            )
          ],
        ),
      ),
    );
  }
}
```

**Expected results:** <!-- what did you want to see? -->

If the user scrolls to the top from a different position than the initialScreenOffset, the controller should jump back to this position.

**Actual results:** <!-- what did you see? -->

```
════════ Exception caught by gesture ═══════════════════════════════════════════
The following assertion was thrown while handling a gesture:
'package:flutter/src/widgets/scroll_position.dart': Failed assertion: line 737 pos 12: 'activity.isScrolling': is not true.


Either the assertion indicates an error in the framework itself, or we should provide substantially more information in this error message to help you determine and fix the underlying cause.
In either case, please report this assertion by filing a bug on GitHub:
  https://github.com/flutter/flutter/issues/new?template=BUG.md

When the exception was thrown, this was the stack
#2      ScrollPosition.didOverscrollBy 
package:flutter/…/widgets/scroll_position.dart:737
#3      ScrollPosition.setPixels 
package:flutter/…/widgets/scroll_position.dart:248
#4      ScrollPositionWithSingleContext.setPixels 
package:flutter/…/widgets/scroll_position_with_single_context.dart:83
#5      ScrollPositionWithSingleContext.applyUserOffset 
package:flutter/…/widgets/scroll_position_with_single_context.dart:126
#6      ScrollDragController.update 
package:flutter/…/widgets/scroll_activity.dart:372
...
Handler: "onUpdate"
Recognizer: VerticalDragGestureRecognizer#7b432
    start behavior: start
════════════════════════════════════════════════════════════════════════════════
```

<details>
  <summary>Logs</summary>


<!--
     Run `flutter analyze` and attach any output of that command below.
     If there are any analysis errors, try resolving them before filing this issue.
-->

```
[flo@localhost bugtest]$ flutter analyze
Analyzing bugtest...                                                    
No issues found! (ran in 13.0s)
```

<!-- Finally, paste the output of running `flutter doctor -v` here. -->

```
[flutter] flutter doctor -v
[✓] Flutter (Channel master, v1.17.1-pre.10, on Linux, locale en_US.UTF-8)
    • Flutter version 1.17.1-pre.10 at /home/flo/Documents/flutter
    • Framework revision 63f8b9a4d5 (29 minutes ago), 2020-04-03 10:34:14 -0700
    • Engine revision f5127cc07a
    • Dart version 2.8.0 (build 2.8.0-dev.19.0 fae35fca47)

[✓] Android toolchain - develop for Android devices (Android SDK version 28.0.3)
    • Android SDK at /home/flo/Documents/android/sdk
    • Platform android-28, build-tools 28.0.3
    • ANDROID_SDK_ROOT = /home/flo/Documents/android/
    • Java binary at: /home/flo/Documents/android-studio/jre/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_212-release-1586-b4-5784211)
    • All Android licenses accepted.

[✓] Android Studio (version 3.6)
    • Android Studio at /home/flo/Documents/android-studio
    • Flutter plugin version 44.0.2
    • Dart plugin version 192.7761
    • Java version OpenJDK Runtime Environment (build 1.8.0_212-release-1586-b4-5784211)

[✓] VS Code (version 1.43.2)
    • VS Code at /usr/share/code
    • Flutter extension version 3.8.1

[✓] Connected device (1 available)
    • ONEPLUS A5010 • b6f09ad2 • android-arm64 • Android 9 (API 28)

• No issues found!
```

</details>