Repository: flutter
Issue: (Doing PR) "Session replay" / pseudo screen recording for Flutter, which is helpful for debugging, and already exists for Web
Labels: c: new feature,engine,would be a good package,a: debugging,c: proposal,P3,team-engine,triaged-engine

### Is there an existing issue for this?

- [X] I have searched the [existing issues](https://github.com/flutter/flutter/issues)
- [X] I have read the [guide to filing a bug](https://flutter.dev/docs/resources/bug-reports)

### Use case

Recently, I saw Sentry released a new feature https://sentry.io/for/session-replay/. After a quick search, that feature seems to be there already with many companies and products providing it. Shortly speaking, the so-called "session replay" tries to record the user screen as a (possibly pseudo) video. Then, when a bug occurs, the user can upload that already-recorded video, and thus we can happily debug it. There are many products supporting the Web already. Many companies providing this feature says they follows GDPR, so I guess this feature does not harm to user privacy. Needless to say, this seems quite helpful for debugging ;)

### Proposal

Therefore, I am proposing that, maybe we can add a session replay feature to Flutter. Or, if Flutter is unhappy to add so many first-party code, Flutter only needs to expose some low-level API, and such high-level replay features can be third-party open-source libraries. Again, I am willing to PR.

#### Proposal 1: At RenderObject level

Simply dump (the diff of) the RenderObject tree. However, the tree may be large. Moreover, it may not be easy to recover the UI from the tree, since we may have to store RenderObject's various fields, some of them can be very heavy or even non-serializable (such as callbacks).

#### Proposal 2: At Layer level

For example, a naive solution is that, make a delegate of `Canvas` inside `PaintingContext`, and record every operation like `drawRect` and so on.

However, since `Path` does not expose any Dart method to read the underlying data, `drawPath` may need more modifications in order to get the data. Similar things happen for `Paragraph`, which is needed when painting text. For example, one way is to add a `Path.dump()` function which dumps its underlying state.

If we can record the whole (diff) layer tree and the pictures, we can store it and replay it in the future, thus implemented the proposed feature.

Cons:

* If we have quite complex scenes, such as very heavy Paths, it may be large in disk size.
* It may be hard to shrink resolution and "image" quality if we want to reduce recorded file size, because everything is vector instead of bitmap.

#### Proposal 3: At Impeller / Archivist level

As mentioned in https://github.com/flutter/flutter/issues/53602#issuecomment-1577111167, Impeller is going to have https://github.com/flutter/engine/tree/main/impeller/archivist. Therefore, one possibility is to utilize this (or some other Impeller features) to extract rendering data. I have not checked into details, whether it will be too low-level and verbose.

Discord link: https://discord.com/channels/608014603317936148/608021234516754444/1120191067195723826

#### Proposal 4: Transfer resized rasterized images

Hack the rasterizer. Just before it is going to present the drawn UI onto user's screen, somehow get the rasterized screen data from the GPU buffer. If possible, resize it down, such that the transfer from device to host (if any) and later postprocessing can be faster. After getting one bitmap per frame, use standard encoding techniques (e.g. ffmpeg?) to compress it.

This is better than repeatedly calling `Scene.toImage`, since the latter will duplicate the whole rasterization process, which can be very slow.

#### Proposal 5: Get bitmaps at native Android/iOS layer

It seems most existing non-open source projects utilize this approach (not checked into details though), since their Flutter integration is just a simple wrapper of Android/iOS integration.

For example, we can convert android View into a Bitmap, using standard "take a screenshot in Android" technique - `PixelCopy`. I guess this will not be slow, because Flutter does not have a native View hierarchy, but only one single FlutterView with rasterized bitmaps (not checked details yet).

### Alternative solutions

#### Alternative 1: `Scene.toImage`

This seems to be CPU, memory and disk heavy. In order to encode the sequence of images into a compressed format (say, video format? or zip of pngs?), it will need a lot of CPU. In order to transfer the ui.Image from GPU to CPU, it needs memory transfer. In addition, the resultant video can be big and requires much disk storage.

#### Alternative 2: Existing products

There exists some products that supports Flutter. However, I briefly looked at them, and they seem to use native Android/iOS and record screens. I am worried this can make the resulting video quite huge, needs some CPU (especially on low-end devices), and lacks fine-grained optimizations.

As a simple example of fine-grained control, we may want to replace real images with blank placeholders when doing (pseudo) screen recording, because we know vivid images are huge to encode and transfer, while those images may add little value to our debugging process. This may not be very easy with those native Android/iOS solutions (though I have not done thorough experiments).