Repository: flutter
Issue: Performance regression with canvas.drawPoints in stable releases after 2.10.5 
Labels: engine,c: performance,P2,team-engine,triaged-engine

**TLDR**: I believe there is a performance regression with `canvas.drawPoints`, in commit 02a182dd96 "Roll Engine from 605454e20143 to 0fd7713f2b5f" resulting in a 50%+ decrease in frame rendering speed in Flutter 3. It will take some additional time for me to collect the minimal repro, but I figured it was still worth while to bring to the attention of the community in case others encounter it while I work on it. Currently my solution/recommendation is to stay on 2.10.5, while also working on concurrently helping improve this in 3.0+.

## Details

<!--
Pending items:
4.  Record a video of the performance issue using another phone so we
     can have an intuitive understanding of what happened. Don’t use
     "adb screenrecord", as that affects the performance of the profile run.

5.  Open Observatory and save a timeline trace of the performance issue
     so we know which functions might be causing it. See "How to Collect
     and Read Timeline Traces" on this blog post:
       https://medium.com/flutter/profiling-flutter-applications-using-the-timeline-a1a434964af3#a499
     Make sure the performance overlay is turned OFF when recording the
     trace as that may affect the performance of the profile run.
     (Pressing ‘P’ on the command line toggles the overlay.)
-->


**Target Platform:**
- Android

**Target OS version/browser:**
- Android 10/11/12

**Devices:**
- Pixel5a
- Pixel3 Emulator
- Other emulators

**Steps to reproduce**
I am working on ironing out the specifics and a minimum viable repro. My specific use case is a bit custom, but it can be summarized like the following:
1. Have a "background" that is constantly animated, such as a CustomPainter, with a few objects constantly moving. Think clouds in the sky or similar.
2. Have a "foreground" layer with game pieces, where each game piece groups together lines and dots paints with `drawPoints` to simulate various angles of the game pieces.
    1. For example it might be something like dice where each dice has 10+ lines and 10+ dots.
    2. These are repaint only on change, and are using LeafRenderObjectWidget with repaint boundary set to true.
3. Run the code with many pieces in the foreground on 3.0+, even though not animated, and notice the background rendering become choppy. Then test again on 2.10.5 and see it buttery smooth.

Update 6/2 - Here is a link to the minimal repro: https://gist.github.com/dfrtz/216dda1d84591e4745da3db3a76ee45d

**Investigation**
At first I thought I may be imagining it, but after I rolled back to 2.10.5 I confirmed an FPS improvement while idling on some heavier animated routes. The primary app I am working on has quite a bit of constant animations (it's a casual game), and I have tried to push the boundaries of what Flutter can handle with things like 2D particle effects, mock 3D perspectives, etc. A secondary concern I have is that it may be impacting other scenarios, they are just harder to notice whereas this one stands out easier. On 2.10.5 I can achieve ~60 FPS, whereas after upgrading to 3.0 the same pages often only manage ~30 FPS. It appears to be related to Skia itself based on the commit I narrowed it down to, but I am not 100% certain and appreciate any feedback.

I was curious if I could narrow down the impacted versions to when it first start, and I was able to isolate down to 2 tags to start:
```
5.11.0-0.1.pre # Good, ~60 fps
4.12.0-4.0.pre # Bad, ~30 fps
```

To attempt to narrow it down further, I checked out the 2.12 tag, and listed all the commits in between. This was ~800 commits, so I did a binary tree type checkout and performance test until I could narrow it down specifically to 02a182dd96:
```
# git checkout 2.12.0-4.0.pre
# git log --oneline 2.10.0-0.0.pre^..2.12.0-4.0.pre
commit 64a0138c937b1f7bd67e596571f4f33cd19c2b88 (HEAD, tag: 2.12.0-4.0.pre) - Bad perf.
...
02a182dd96 Roll Engine from 605454e20143 to 0fd7713f2b5f (3 revisions) (#98727) - First bad perf.
c50b02f2c3 Roll Engine from c5e89cbb7377 to 605454e20143 (3 revisions) (#98718) - Last good perf.
...
commit 95e555344dc746ad46ada9de9cc475042c63ec55 (tag: 2.10.0-0.0.pre) - Good perf.
```

I also tested the latest 3.1.0 tag on stable and master channels as well, but they behave the same. The last fast/safe flutter version details:
```
$ flutter --version
Flutter 2.11.0-0.0.pre.612 • channel unknown • unknown source
Framework • revision c50b02f2c3 (3 months ago) • 2022-02-17 22:44:23 -0500
Engine • revision 605454e201
Tools • Dart 2.17.0 (build 2.17.0-118.0.dev) • DevTools 2.10.0
```

First slow flutter version details:
```
$ flutter --version
Flutter 2.11.0-0.0.pre.613 • channel unknown • unknown source
Framework • revision 02a182dd96 (3 months ago) • 2022-02-18 02:24:27 -0500
Engine • revision 0fd7713f2b
Tools • Dart 2.17.0 (build 2.17.0-119.0.dev) • DevTools 2.11.0
```

This commit appears to be minor changes, but based on the PR there were 3 engine changes in this commit spanning Dart, Skia, and Flutter:
https://github.com/flutter/flutter/pull/98727
```
2022-02-18 skia-flutter-autoroll@skia.org Roll Dart SDK from 6894cf5d241c to 9ac87a1a1ca8 (1 revision) (flutter/engine#31534)
2022-02-18 gw280@google.com Add a complexity scoring class for Metal and OpenGL (flutter/engine#31417)
2022-02-18 skia-flutter-autoroll@skia.org Roll Skia from b5282fea2312 to 418fdebf1cca (1 revision) (flutter/engine#31531)
```

If I had to guess, it is probably related to the Skia rollup or the scoring changes? But I looked at 418fdebf1cca in Skia, and 9ac87a1a1ca8 in Dart, and I didn't see anything that stood out. I also checked my third party dependencies for any changes, and they showed the exact same libs in the pubspec.lock when switching between the fast/slow commits.

I have used the CPU profiler to narrow down some performance issues in the past, so I did a 10 second capture on these pages on the good and bad commit for comparison. The graphs look almost identical, leading me to further believe the issue may be in Skia. I checked for comparisons to flutter 2.10.5 and 2.5.0, the previous versions I was running, and while 2.5 looks drastically different in the flame charts, 2.10 seems to work great and looks similar.

Here are screen caps of the "stable" perfs when I remove `drawPoint` references, and keep all other animations going. The perf appears consistent between the 2 versions at a basic level:
<details>
<summary>Perf Timeline Snapshots Without drawPoint Calls (Stable perf with other animations)</summary>
Good c50b02f2c3 commit:
<img width="969" alt="c50b02_without_points" src="https://user-images.githubusercontent.com/29335974/171052830-5f516fdb-001f-47a7-939d-932b25f9a48d.png">
Problematic 02a182dd96 commit:
<img width="963" alt="02a182_without_points" src="https://user-images.githubusercontent.com/29335974/171052874-dd2d2ef6-55b9-4a8c-a989-6b7aecc02b10.png">
</details>

Here are with `drawPoint` calls included. There is some overhead from the profiling/tracking turned on, since I can hit 60 FPS on the good version without it, but even so there is about a 2x increase (16 vs 32 ms) in the raster time:
<details>
<summary>Perf Timeline Snapshots With drawPoint Calls (2x regression)</summary>
Good c50b02f2c3 commit:
<img width="971" alt="c50b02_with_points" src="https://user-images.githubusercontent.com/29335974/171053406-53d6af54-97eb-4b80-b2f9-dcf238cfdf92.png">
Problematic 02a182dd96 commit:
<img width="960" alt="02a182_with_points" src="https://user-images.githubusercontent.com/29335974/171053416-efe30bff-e159-4ce9-8209-2ee7497ca040.png">
</details>

And to push the boundaries further, here is a comparison of what the "maximum strength" of these `drawPoints` calls could theoretically be in one of my app routes that are currently smooth. There is some overhead from the profiling/tracking turned on, since I can hit 60 FPS on the good version without it, but there is about 4-5x increase (32 vs 139 ms) in raster time:
<details>
<summary>Perf Timeline Snapshots With "max" drawPoint Calls (4-5x regression)</summary>
Good c50b02f2c3 commit:
<img width="967" alt="c50b02_with_points_max" src="https://user-images.githubusercontent.com/29335974/171053543-f987d7c7-dc0b-40d5-a2a5-5800734e80c4.png">
Problematic 02a182dd96 commit:
<img width="968" alt="02a182_with_points_max" src="https://user-images.githubusercontent.com/29335974/171053523-b210544c-1d23-484e-803a-9e01af92918c.png">
</details>

I have tested debug/profile/release modes, and they all behave the same way. This behavior is noticeable on both my physical Pixel 5a and Android emulators. For example, debug mode on 2.10.5 runs smoother on these animated pages than release mode on 3.0.

I was hoping someone may have encountered this before me, but I do not believe so after looking over open perf related issues and based on the timeframe around when it happened (likely in last 2 months based on the commit). I am working on a minimum viable repro to fill in the blanks, but I wanted to see if there were any initial thoughts about the commit I isolated it too, or things to try. I understand if there is not much feedback that can be provided until I can come up with a "minimal" app, so I am working on isolating it further and providing additional details. However, I wanted to try to bring some attention to it while I try to narrow it down even further, in case someone else hits similar issues and/or may be able to isolate a minimum repro before I can.

I appreciate all the hard work that has gone in to Flutter, it's great stuff! I like to push boundaries and make things even more awesome 😄 

## Logs

<details>
<summary>Logs</summary>

Minimal repro:
https://gist.github.com/dfrtz/216dda1d84591e4745da3db3a76ee45d
```
$ flutter analyze
Analyzing repro...                                                      
No issues found! (ran in 2.3s)
```
</details>

<details>
<summary>Doctor Output</summary>

Details about my dev setup when on the first slow commit (02a182dd96). I am not working on mac/ios versions currently, even though I plan to at some point, so I know my xcode is not set up and I havent tested if iOS is impacted.
```
$ flutter doctor -v
[✓] Flutter (Channel unknown, 2.11.0-0.0.pre.613, on macOS 11.6 20G165 darwin-x64, locale en-US)
    • Flutter version 2.11.0-0.0.pre.613 at /Users/dfritz/Library/flutter
    • Upstream repository unknown
    • Framework revision 02a182dd96 (3 months ago), 2022-02-18 02:24:27 -0500
    • Engine revision 0fd7713f2b
    • Dart version 2.17.0 (build 2.17.0-119.0.dev)
    • DevTools version 2.11.0

[✓] Android toolchain - develop for Android devices (Android SDK version 33.0.0-rc4)
    • Android SDK at /Users/dfritz/Library/Android/sdk
    • Platform android-32, build-tools 33.0.0-rc4
    • Java binary at: /Users/dfritz/Library/Application
      Support/JetBrains/Toolbox/apps/AndroidStudio/ch-0/212.5712.43.2112.8609683/Android
      Studio.app/Contents/jre/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.12+0-b1504.28-7817840)
    • All Android licenses accepted.

[✗] Xcode - develop for iOS and macOS
    ✗ Xcode installation is incomplete; a full installation is necessary for iOS development.
      Download at: https://developer.apple.com/xcode/download/
      Or install Xcode via the App Store.
      Once installed, run:
        sudo xcode-select --switch /Applications/Xcode.app/Contents/Developer
        sudo xcodebuild -runFirstLaunch
    ✗ CocoaPods not installed.
        CocoaPods is used to retrieve the iOS and macOS platform side's plugin code that responds to your plugin usage
        on the Dart side.
        Without CocoaPods, plugins will not work on iOS or macOS.
        For more info, see https://flutter.dev/platform-plugins
      To install see https://guides.cocoapods.org/using/getting-started.html#installation for instructions.

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2021.2)
    • Android Studio at /Users/dfritz/Library/Application
      Support/JetBrains/Toolbox/apps/AndroidStudio/ch-0/212.5712.43.2112.8609683/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.12+0-b1504.28-7817840)

[✓] Android Studio (version 2020.3)
    • Android Studio at /Users/dfritz/Library/Application
      Support/JetBrains/Toolbox/apps/AndroidStudio/ch-0/203.7678000/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.10+0-b96-7281165)

[✓] Connected device (2 available)
    • Pixel 5a (mobile) • 17311JECB09831 • android-arm64  • Android 12 (API 32)
    • Chrome (web)      • chrome         • web-javascript • Google Chrome 102.0.5005.61

[✓] HTTP Host Availability
    • All required HTTP hosts are available

! Doctor found issues in 1 category.
```
</details>