Repository: flutter
Issue: Feature Request for a Synchronous way to Rasterize Images / Draw Pixels to Canvas
Labels: c: new feature,engine,c: performance,c: proposal,P3,a: gamedev,team-engine,triaged-engine

# Use case

This feature request is about the [`Canvas` class in `engine/painting.dart`](https://github.com/flutter/engine/blob/master/lib/ui/painting.dart#L3105).

As extensively discussed in #31598, using regular canvas operations to draw pixel data is *extremely slow*.  
This makes the only option we have decoding an image from our raw pixel data, e.g. using [`decodeImageFromPixels`](https://api.flutter.dev/flutter/dart-ui/decodeImageFromPixels.html). This has another problem, [pointed out by @andrewackerman in the mentioned issue](https://github.com/flutter/flutter/issues/31598#issuecomment-511510444), which is that creating an `Image` in Dart is an `async` operations.  
This means that we will have to postpone drawing our raw pixel data to the next frame if we wish to use real-time operations.

## The `async` `Image` problem

`Image`s in Flutter currently have the inherent problem that rasterizing / decoding them is **always** an asynchronous operation. This is why using them for drawing pixel data does not work synchronously, but it also creates a whole nother class of problems:

Any other canvas operation in Flutter that involves `Image`s is also **asynchronous**. So not only can you not draw pixel data synchronously, but you cannot rasterize _any_ `Picture` synchronously, which is necessary for achieving various effects (e.g. [this](https://youtu.be/qlVNSMVv7Qk)). Also [Canvas.drawAtlas](https://api.flutter.dev/flutter/dart-ui/Canvas/drawAtlas.html) suffers from this problem.

### `drawPixels` proposal

Here is an example proposal ([written by @andrewackerman in #31598]((https://github.com/flutter/flutter/issues/31598#issuecomment-516050204))) for directly drawing pixel data.

Note that this might not cover all use cases because often we _do_ need to have a decoded image (`dart:ui.Image`) available to us, e.g. when using [Canvas.drawAtlas](https://api.flutter.dev/flutter/dart-ui/Canvas/drawAtlas.html). It should, however, give an idea as to what we are looking for.

> [...] it should have a similar signature to `decodImageFromPixels` [sic], so something like:

```dart
class Canvas {
  ...

  void drawPixels(
    Offset c, 
    Uint8List bytes,
    int width,
    int height,
    PixelFormat pixelFormat, {
    int rowBytes,
    int targetWidth,
    int targetHeight,
  }) {
    assert(c != null && bytes != null && width != null && height != null && pixelFormat != null);
    assert(bytes.length == width * height * 4);
    if (rowBytes == null) rowBytes = width * 4;
    if (targetWidth == null && targetHeight == null) {
      targetWidth = width;
      targetHeight = height;
    } else if (targetWidth == null) {
      targetWidth = (targetHeight * (width / height)).toInt();
    } else if (targetHeight == null) {
      targetHeight = (targetWidth * (height / width)).toInt();
    }
    _drawPixels(c, pixelData, width, height, pixelFormat, rowBytes, targetWidth, targetHeight);
  }
}
```

<details><summary>Click to expand details</summary>

> `c` is the position that the image will be drawn at.
>
> `bytes` is the pixel data that will be drawn to the canvas, represented as byte groups of 4 bytes per pixel. The color channels will be extrapolated based on the value of `pixelFormat`.
>
> `width` and `height` are the dimensions of the image represented by the pixel data. If `width * height * 4` is not equal to `pixelData.length`, an error will be thrown.
>
> `rowBytes` is the number of bytes consumed by each row of pixels in the data buffer. If unspecified, it defaults to width multiplied by the number of bytes per pixel in the provided `format`.
>
> The `targetWidth and `targetHeight` arguments specify the size of the output image, in image pixels. If they are not equal to the intrinsic dimensions of the image, then the image will be draw. If exactly one of these two arguments is specified, then the aspect ratio will be maintained while forcing the image to match the other given dimension. If neither is specified, then the image maintains its real size. (Debatable as to whether these parameters should be supported as they don't really fall into the whole "just draw these pixels" mentality of this method.)**

@andrewackerman assumed that the byte data will always have 32 bits per pixel, which makes sense because the only [`PixelFormat` values](https://api.flutter.dev/flutter/dart-ui/PixelFormat-class.html) that currently exist are `bgra8888` and `rgba8888`, which use 32 bits per pixel.  
The [`decodeImageFromPixels` function](https://api.flutter.dev/flutter/dart-ui/decodeImageFromPixels.html) is open to any `PixelFormat` that might be supported in the future:

> `rowBytes` is the number of bytes consumed by each row of pixels in the data buffer. If unspecified, it defaults to width multiplied by the number of bytes per pixel in the provided `format`.

I think that this approach would make sense to also be used for the `Canvas` method (if it will use `PixelFormat`). Consequently, the assertion for `width * height * 4` should probably be `width * height * bytesPerPixel` and similarly, `bytes` should be pixel data, represented as byte groups of `bytesPerPixel` bytes per pixel.

</details>