Repository: deno
Issue: Fast streams
Labels: perf

_This issue acts as a design doc for a set of performance optimizations and clean-ups that can be performed in Deno to make piping data between consumers and producers of data significantly faster than we have previously been able to, while preserving the pristine APIs we expose to users for streaming data._

### Introducing streams

_Streams_ represent a combination of a single data _source_ and a data _sink_. Streams are abstract - they do not exist as objects in JS. Streams are a logical "pipe" that is able to transfer data between between a _source_ and a _sink_. On these streams the action of _streaming_ is performed. _Streaming_ is the operation of transferring data from the _source_ to the _sink_.

Sources are the producing end of a stream. They are most often represented as `ReadableStream` objects in JS. There are two types of sources: JS backed, and resource backed. JavaScript backed sources are streams that are constructed manually in JS, where the chunks sent on the stream stem from user code. Resource backed sources are producers backed by a Rust side resource. A `ReadableStream` representing file handle contents (`FsFile#readable`) is such a 

- `fsFile.readable` (a source representing data read from a file handle)
- `req.body` (a source representing the data received in an HTTP server request)
- `resp.body` (a source representing the data received as the response to an outbound `fetch` request)

Sinks are consuming end of a stream. In JS they are sometimes represented as `WritableStream`, but more often than not as an abstract resource that consumes data. There are once again two types of sinks: JS backed, and resource backed. JS backed sinks read chunks from the stream in JS. Resource backed sinks consume streams in Rust - they usually send this data to some IO sink, like a network socket or open file.

To make streams optimally fast, we need to optimize them specifically for the pair of source and sink they consist of. For example, a stream with a Rust backed sink & source, can perform the entire stream in Rust - no copies to JavaScript are necessary. If the reverse is true (both sides of the stream are JS backed), no copies to Rust are necessary.

Some sinks are also special: they want to read _all_ data from the stream into a single combined chunk for example. In this case if the source can provide all data in a single operation more efficiently than streaming into chunks, cpu cycles and thus time can be saved resulting faster streams.

### Streaming operation

To optimize streams what we really need to do is optimize the streaming operation. This means that every place we "perform a streaming operation" we need to decide if we can perform an optimized operation.

More concretely the baseline operation we are trying to optimize is the following:

1. get a reader from the `ReadableStream` (source)
2. while the reader has more chunks:
  1. read a chunk from the reader
  2. emit the chunk into the underlying sink
3. unlock the stream by closing the reader

This operation exists in various places. Some examples:

- `ReadableStream#pipeTo(writable)`
- `ReadableStream#pipeThrough(transformer)`
- `req.body` reading in `fetch`
- `resp.body` reading in HTTP server

Currently all of the above implementations of the operation (except for the HTTP server case) are completely un-optimized, and are implemented in pure JS. This means that if a stream consists of a rust backed sink and source (e.g file upload via fetch), all data needs to be copied once from Rust into JS, and then from JS back into Rust.

### Optimizations

This is exactly where the core of the optimization lies. We can significantly improve performance by skipping the step where we copy data from / to JS.

We have some practical experience with the performance improvements this unlocks in the HTTP server. We have implemented an optimization there that allows some Rust backed sources to be directly used as a HTTP response, without having to get copied through JS. This significantly improves throughput.

### The implementation

Deno uses WHATWG streams as the primitive for sources and sinks. Our implementation work will focus exclusively on these (`ReadableStream` and `WritableStream`). These stream primitives have the benefit that they are able to be locked. This allows us to take full ownership of the underlying source / sink, allowing us to move the ownership from JS to Rust. This enables all following optimizations.

To make the optimization work, we need to be able to effectively identify which sources and sinks are resource backed, and can have their streaming operation offloaded to Rust. To do this we need to brand all JS objects that represent a source or sink that is backed by a resource. Luckily this is easy, as the API for all source and sink resources is the same: it is the `deno_core::Resource` read & write API. To brand a source or sink object thus, we only need to attach a hidden `[rid]` property to it. This property can then hold the resource ID backing the source or sink.

For sources, this branding is already implemented for many streams through the `readableStreamForRid` helper function in `__bootstrap.streams.readableStreamForRid`. We need to implement a similar helper function for `WritableStream` sinks, probably called `writableStreamForRid`. A non branding version of this helper already exists in `ext/net`.

We then need to update all code paths that perform a streaming operation to go into a special case path if both source and sink are resource backed. The most generic case of this is `ReadableStream#pipeTo` and `ReadableStream#pipeThrough`. The latter is similar to the former, except that it has to make two connections rather than just one (src -> transformer sink & transfomer src -> sink). For these generic cases a new op would be introduced (`op_pipe`) that takes a source and sink resource and performs a simple streaming operation in Rust. More specialized logic and op integration is required when piping the `req.body` into `fetch`, or `resp.body` into a HTTP response.

Finally, more specialized streaming-like code paths can be optimized. A good example of this is the "read all" operation that is performed on `req.body` and `resp.body` when `req.arrayBuffer()` and `resp.arrayBuffer()` are called. Here we can add an op that performs the entire read operation in Rust (`op_read_all`), only passes a single aggregated chunk to JavaScript, rather than aggregating chunks manually in JS. @marcosc90 came up with this optimization in #16038. There are likely other specialized streaming-like operations that could use similar optimizations. 

- Branding sources
  - [x] `FsFile#readable`
  - [x] `Deno.stdin#readable`
  - [x] `Deno.Process#stdout#readable`
  - [x] `Deno.Process#stderr#readable`
  - [ ] `Deno.Child#stdout` (difficult, because unrefable)
  - [ ] `Deno.Child#stderr` (difficult, because unrefable)
  - [x] `Conn#readable`
  - [x] `resp.body` returned from `Cache#match`
  - [ ] `req.body` in `Deno.serveHttp`
  - [x] `req.body` in `Deno.serve`
  - [x] `resp.body` returned from `fetch`
  - [ ] `CompressionStream#readable`
  - [ ] `WebSocketStream#readable`
  - [ ] `TextEncoderStream#readable`
- Branding sinks
  - [ ] `FsFile#writable`
  - [ ] `Deno.stdout`
  - [ ] `Deno.stderr`
  - [ ] `Deno.Process#stdin#writable`
  - [ ] `Deno.Child#stdin`
  - [ ] `Conn#writable`
  - [ ] `CompressionStream#writable`
  - [ ] `WebSocketStream#writable`
  - [ ] `TextDecoderStream#writable`
- Optimize streaming operations
  - [x] `resp.body` consumption in `Deno.serveHttp`
  - [x] `resp.body` consumption in `Deno.serve`
  - [ ] `resp.body` consumption in `cache.put`
  - [ ] `req.body` consumption in `fetch`
  - [ ] `ReadableStream#pipeTo`
  - [ ] `ReadableStream#pipeThrough`
- Optimize streaming-like operations
  - [x] Collecting a `req.body` or `resp.body` (`req.arrayBuffer()` for example)