Repository: flutter
Issue: Seamless System-Level Compositor Integration
Labels: engine,perf: energy,P3,team-engine,triaged-engine

## Use case

On platforms like iOS and Windows native applications rely heavily on the system compositor for speed and power efficiency. These compositors have the ability to offload blending, blitting, etc. to dedicated scanout hardware, and can otherwise efficiently GPU composite visuals.

Since these are system-level, they may be further enhanced by relying on newer hardware features, or newer software-side optimizations.

Layer effects that work on bitmaps, either raster cached or offscreen, can benefit from being composited by the system. For example, the opacity of a route during a route transition, the offset of said route during the transition, or even arbitrary bitmap transforms. Some compositors, like [iOS CoreAnimation](https://developer.apple.com/documentation/quartzcore/calayer) and [Windows DirectComposition](https://learn.microsoft.com/en-us/windows/win32/api/dcomp/nn-dcomp-idcompositionvisual), can represent clip masks, blurs, image filters, and animations too. 

Ultimately, on supported platforms Flutter can move to swapchain-less rendering, where it renders a set of GPU-side images and hands them over to the system compositor. Instead of Flutter views utilizing a fullscreen surface, it can instead render to a smaller surface when possible. This would be great for add-to-app scenarios.

### Native Application Compositor Usage

On iOS, practically every application relies upon CALayer. Metal and OpenGL layers are most likely lifted into the system compositor, while regular application content is rasterized within the application itself. CoreAnimation may choose to lift application rastered content into the system compositor when appropriate. Safari relies on CoreAnimation to heavily accelerate page rendering. So does [Firefox, with incredible power efficiency wins](https://mozillagfx.wordpress.com/2019/10/22/dramatically-reduced-power-usage-in-firefox-70-on-macos-with-core-animation/).

On Windows, DirectComposition is mostly intended to offload GPU work directly to DWM. Practically, it works like CoreAnimation minus the ability for applications to draw into layers with callbacks. Instead applications are expected to utilize Direct2D, Direct3D, or MediaFoundation to produce layer contents. Windows.UI.Composition works a bit closer to CoreAnimation, in that it supports some drawing primitives like [CompositionSpriteShape](https://learn.microsoft.com/en-us/uwp/api/windows.ui.composition.compositionspriteshape?view=winrt-22621).

But all of this is best described by @raphlinus's [blog post](https://raphlinus.github.io/rust/gui/2022/07/15/next-dozen-guis.html#tradeoff-use-of-system-compositor), which also has lots of great notes about non-compositor things as well.

## Proposal

Flow currently has the following phases: Preroll & Raster. In this proposal, two new phases will be introduced: Segmentation and Composition.

### Segmentation

This phase occurs after Preroll. It decides what top-level surfaces are needed to render the content. Currently, this would result in a single fullscreen surface without platform views, and multiple fullscreen surfaces with (nit: flow doesn't handle the latter today, but it should be in charge of this :).

In the future, smarter decisions can be made about segmentation:

* PlatformViews don't need to split rendering into multiple surfaces if nothing is drawn on top of it
* Surfaces can avoid rendering things behind completely opaque platform views, and even utilize hole punching for rendering
* Surfaces can fit the bounds of their content, instead of being sized to the whole screen
* Raster cached layers using transforms & effects the system compositor supports can be passed straight to the compositor as-is, no copies needed
* Avoid updating surfaces that haven't changed
* Partial rendering optimizations without relying on broken EGL extensions

### Composition

After rastering the display lists to the surfaces provided by the Segmentation phase, the Composition phase is in charge of communicating with the system compositor to place the surfaces on screen. Since this is done all at once, there are no problems with unsynchronized rendering.

### SystemCompositor

In Flow, a new virtual class is created to communicate the capabilities of the system-level compositor, so the Segmentation phase can understand what is and is not possible. This class is then also used by the Composition phase to update compositor visuals associated with surfaces.

#### Compositor Capabilities

Compositors like CoreAnimation and DirectComposition support many CSS-compliant effects.

Compositors like the Android View system support a subset of effects, and of course alpha transparency and transforms.

And on Wayland, it only supports simple pixel offsets, simple rotations & flips, .

But, to avoid creating a rigid hierarchy of "support levels", it should be queriable from the SystemCompositor class itself. Then platform implementations and embedders can choose to support what they can trivially support. And everything else falls back to the Engine to render.

### What about the existing PlatformViews Segmentation and Composition code?

Don't throw it out! Just partially move it into Flow :)

It already belongs there anyways.

cc @flar @chinmaygarde