Repository: flutter
Issue: Better implementation of FlutterTexture
Labels: c: new feature,c: proposal,P3,team-engine,triaged-engine

### Use case

To [simplify](https://github.com/misos1/packages/blob/24002603a885533831cd343624aab5df2639e933/packages/video_player/video_player_avfoundation/darwin/video_player_avfoundation/Sources/video_player_avfoundation/FVPVideoPlayerPlugin.m#L596-L600) its use in plugins like `video_player` https://github.com/flutter/packages/pull/7466 and make it better for plugins like `camera`.

### Proposal

Currently it is done in a way that `textureFrameAvailable` is used to signal that we have new texture frame and engine then eventually calls `copyPixelBuffer` to obtain it. `textureFrameAvailable` does two different thing in two classes in engine. In `FlutterDarwinExternalTextureMetal.mm` is used flag `_textureFrameAvailable`  and `textureFrameAvailable` sets it to `true`, but in `embedder_external_texture_metal.mm` it sets last stored image to NULL so if `copyPixelBuffer` returns NULL it will not show anything and will periodically call it until it returns non-NULL. This means returning NULL from `copyPixelBuffer` after first non-NULL can cause different results on different platforms and it is better to [avoid that](https://github.com/misos1/packages/blob/24002603a885533831cd343624aab5df2639e933/packages/video_player/video_player_avfoundation/darwin/video_player_avfoundation/Sources/video_player_avfoundation/FVPVideoPlayerPlugin.m#L607).

Multiple calls to `textureFrameAvailable` will cause only single call to `copyPixelBuffer` and `_textureFrameAvailable` is reset at some unknown after time that, thus it can be harder (or even impossible without depending on internals) to supply external texture reliably at display refresh rate. For example now `video_player` calls `textureFrameAvailable` from its own display link and on an iOS device it can be called at approximately the same time as `copyPixelBuffer` but the order in which they get called can change rapidly. Now let's suppose that at the current screen refresh is first called `copyPixelBuffer` as a reaction to some previous `textureFrameAvailable` and then is called `textureFrameAvailable` which triggers `copyPixelBuffer` at the next screen refresh. But at the next screen refresh is first called `textureFrameAvailable` and only after that `copyPixelBuffer` as reaction so now we had two calls to `textureFrameAvailable` but only a single call to `copyPixelBuffer`. Another `copyPixelBuffer` can be called only after some another `textureFrameAvailable` so this resulted in dropped video frame (first number is the counter of screen refreshes):

```
1 copyPixelBuffer
1 textureFrameAvailable
2 textureFrameAvailable
2 copyPixelBuffer
3 (engine sees that _textureFrameAvailable is false and will not call copyPixelBuffer at this frame)
3 textureFrameAvailable
4 copyPixelBuffer
4 textureFrameAvailable
```

There is race between `textureFrameAvailable` and `copyPixelBuffer` in a sense that call to `textureFrameAvailable` may or may not have any effect depending on whether engine already cleared that flag or not (or image) so it is unknown whether `copyPixelBuffer` will be called as reaction to `textureFrameAvailable` or not.

This can be resolved by engine calling `copyPixelBuffer` unconditionally for every screen refresh (as another mode of operation together with current one) as is https://github.com/flutter/packages/pull/7466 trying to achieve. Of course with such mode of operation it would be good if returning NULL from `copyPixelBuffer` was well defined (showing latest texture) and also engine could avoid calling `wrapExternalPixelBuffer` or similar (if expensive) every time and instead does it only if it got new texture. But this would not solve it (at least not in an easy way) for plugins like `camera` where there is also this same problem although maybe not so visible or not so often.

But maybe a much better possibility, which would work (better) also for plugins like `camera`, would be implementing it as a consumer-producer queue where instead of `textureFrameAvailable` we could send our texture into that queue and the engine would pull them from it. This queue needs to be able to contain at least 2 frames (if it is limited which it probably should be). Pushing a new frame into that queue can erase the frame at the other side of the queue (the oldest) if the limit is reached (how it is currently is like a queue with limit 1 which is not enough). Or as possible third alternative (like something between this and that previous approach) if every single call to `textureFrameAvailable` caused a call to `copyPixelBuffer` by implementing some counter instead of flag (of course never calling `copyPixelBuffer` more than once per screen refresh, and also here can be limit).

And maybe they can be combined because that first approach would make implementation in plugins like in `video_player` simpler because it would not require them to actively produce frames on their own display links as is currently done in `video_player` and that second approach would make it easier for plugins like `camera`. Difference is that `video_player` needs to actively pull frames from `AVPlayerItemVideoOutput` while in `camera` frames are pushed through `AVCaptureVideoDataOutputSampleBufferDelegate` by the system. So maybe the best would be to have both approaches available (possibility for `FlutterTexture` to pull frames from us, and also us to push frames into it).