Repository: flutter
Issue: Automatic scroll when showing widgets larger than device screen
Labels: framework,f: scrolling,has reproducible steps,P2,found in release: 3.10,found in release: 3.11,team-framework,triaged-framework

For detail,
1. Showing widgets larger than device screen
2. Programmatically remove widgets before the widget that is showing
3. Automatic screen scroll happens

This case will occur when I want to swap widget position.

## Steps to Reproduce

** Sample Program **
<details>


```dart
import 'package:flutter/material.dart';

class SamplePage extends StatefulWidget {

  SamplePage(): super();

  @override
  SamplePageState createState() => SamplePageState();
}

class SamplePageState extends State<SamplePage> with WidgetsBindingObserver {

  ScrollController scrollController = ScrollController();

  List<String> _textList = [
"short",
"""long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
long
""",
"short1",
"short2",
"short3",
"short4",
"short5",
"short6",
"short7",
"short8",
"short9",
"short10",
"short11",
"short12",
"short13",
"short14",
"short15",
"short16",
"short17",
"short18",
"short19",
  ];

  @override
  void initState(){
    super.initState();

    Future.delayed(Duration(seconds: 3)).then((_) async {
      await scrollController.position.animateTo(500, duration: Duration(milliseconds: 500), curve: Curves.linear);
      await Future.delayed(Duration(seconds: 1));

      _textList.removeAt(0);

      await Future.delayed(Duration(seconds: 1));
      setState(() { });

      await Future.delayed(Duration(seconds: 1));

    });
  }

  Widget build(BuildContext context){
    Widget bodyScaffold = Scaffold(
      body: Container(
        padding : EdgeInsets.all(30),
        child   : ListView.builder(
          controller  : scrollController,
          itemBuilder : (BuildContext context, int index) { 
            return Card(
              color: Colors.grey.withAlpha(200),
              child: Container(
                padding   : EdgeInsets.only(right: 18.0, left: 18.0, top: 15, bottom: 15),
                child     : Text(_textList[index], style: TextStyle(fontWeight: FontWeight.bold,))
              )
            );
          },
          itemCount   : _textList.length,
        )
      ),
    );

    return bodyScaffold;
  }
}
```

</details>

1. Run Sample Program by `flutter run` 
2. just watch few seconds

**Expected results:** 
The current widget moves up by the size of the deleted widget
Not scrolled.

**Actual results:** 
Automatic scroll

this is the video
<img src="https://user-images.githubusercontent.com/7422251/116794275-c0ef6500-ab06-11eb-9cc1-2d7c507cc2cc.gif" height="450" />



<details>
  <summary>Logs</summary>

`flutter analyze`

```
Analyzing mangaplotter...                                               

   info - Don't import implementation files from another package - lib\generated_plugin_registrant.dart:9:8 - implementation_imports
   info - The library 'package:firebase_analytics_web/firebase_analytics_web.dart' is legacy, and should not be imported into a null safe library - lib\generated_plugin_registrant.dart:10:8 -
          import_of_legacy_library_into_null_safe
   info - Don't import implementation files from another package - lib\generated_plugin_registrant.dart:15:8 - implementation_imports
   info - The library 'package:flutter_facebook_login/src/web/facebook_login_plugin_web.dart' is legacy, and should not be imported into a null safe library - lib\generated_plugin_registrant.dart:15:8 -          import_of_legacy_library_into_null_safe
```

`flutter doctor -v`

```
[√] Flutter (Channel stable, 2.0.6, on Microsoft Windows [Version 10.0.19042.928], locale ja-JP)
    • Flutter version 2.0.6 at C:\src\flutter
    • Framework revision 1d9032c7e1 (2 days ago), 2021-04-29 17:37:58 -0700
    • Engine revision 05e680e202
    • Dart version 2.12.3

[√] Android toolchain - develop for Android devices (Android SDK version 29.0.2)
    • Android SDK at C:\Users\hatadev\AppData\Local\Android\sdk
    • Platform android-30, build-tools 29.0.2
    • Java binary at: C:\Program Files\Android\Android Studio1\jre\bin\java
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b01)
    • All Android licenses accepted.

[√] Chrome - develop for the web
    • Chrome at C:\Program Files (x86)\Google\Chrome\Application\chrome.exe

[√] Android Studio (version 4.1.0)
    • Android Studio at C:\Program Files\Android\Android Studio1
    • Flutter plugin can be installed from:
       https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
       https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b01)

[√] VS Code (version 1.55.2)
    • VS Code at C:\Users\hatadev\AppData\Local\Programs\Microsoft VS Code
    • Flutter extension version 3.21.0

[√] Connected device (3 available)
    • AOSP on IA Emulator (mobile) • emulator-5554 • android-x86    • Android 9 (API 28) (emulator)
    • Chrome (web)                 • chrome        • web-javascript • Google Chrome 90.0.4430.93
    • Edge (web)                   • edge          • web-javascript • Microsoft Edge 90.0.818.51
```

</details>