The weirdly obscure art of Streamed HTML
Tags: webdev, performance, marko, svelte
Published: 2022-03-15 15:30:58

---
title: The weirdly obscure art of Streamed HTML
published: true
description: Literally “one weird trick” to massively improve Time To First Byte
tags: webdev, performance, marko, svelte
series: Streets
cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jqs50vvpgkt08zsrgckw.jpg
//old_cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8hbbzn18es5s0pzqa52d.png
---

My goal from last time: reuse our existing APIs in a demo of the fastest possible version of our ecommerce website… and keep it under 20 kilobytes.

I decided this called for an MPA. (aka a traditional web app. Site. Thang. Not-SPA. Whatever.)

## And with that decision, I doomed the site to feel slow and clunky

In theory, there’s no reason MPA interactions need be as slow as commonly encountered. But in practice, there are _many_ reasons.

Here’s an example. At Kroger, product searches take two steps:

1. Send user’s query to an API to get matching product codes
2. Send those product codes to an API to get product names, prices, etc.

Using those APIs to generate a search results page would look something like this:

```js
const resultsData = fetch(`/api/search?${new URLSearchParams({
    query: usersSearchString
  })}`)
  .then(r => r.json())
  .then(({ upcs }) =>
    fetch(`/api/products/details?${new URLSearchParams({ upcs })}`
  )
  .then(r => r.json())

res.writeHead(resultsData.success ? 200 : 500, {
  'content-type': 'text/html;charset=utf-8'
})

const htmlResponse = searchPageTemplate.render({
    searchedQuery: usersSearchString,
    results: resultsData
  })

res.write(htmlResponse)
res.end()
```

Each fetch takes time, and `/api/products/details` only happens _after_ `/api/search` finishes. Plus, those requests traveled from my computer to the datacenter and back. A real server making those calls would [sit next to the others for _very_ fast requests](https://serverfault.com/questions/953169/what-is-the-latency-within-a-data-center-i-ask-this-assuming-there-are-orders-o).

But on my demo machine, the calls usually took ~200 milliseconds, sometimes spiking as high as 800ms. Combined with the target 3G network, server processing time, and other inconvenient realities, I was frequently flouting [the 100–1000ms limit for “natural and continuous progression of tasks”](https://web.dev/rail/#focus-on-the-user).

## So the problem is high Time to First Byte, huh?

No worries! High [Time to First Byte (TTFB)](https://developer.mozilla.org/en-US/docs/Glossary/time_to_first_byte) is a known performance culprit. Browser devtools, Lighthouse, and other speed utensils all warn about it, so there’s lots of advice for fixing it!

Except, none of the [easily-found advice for improving TTFB](https://web.dev/ttfb/#how-to-improve-ttfb) helps:

<dl>
<dt>Optimize the server application to prepare pages faster
<dd><p>Node.js spent 30ms or less handling the request and sending HTML. Very little to be gained there.

<dt>Optimize database queries or migrate to faster database systems
<dd><p>I was not allowed to touch our databases or API servers.

<dt>Upgrade server hardware to have more memory or CPU
<dd><p>This ran on a MacBook Pro with plenty of unused RAM and CPU.

<dt>Cache the expensive lookups
<dd><p>Caching can’t help the first requests, unless I precache all known endpoints or something. Even then, that wouldn’t work for search: users can and will search for strings never seen before.
</dl>

## The problem: web performance is other people

If only two API calls were a struggle, I was in for ruination. Here are some data sources our homepage uses:

- Authentication and user info
- Items in shopping cart
- Selected store, pickup vs. delivery, etc.
- Recommended products
- Products on sale
- Previous purchases
- Sponsored products
- Location-specific promotions
- Recommended coupons
- A/B tests
- Subscription to Kroger Boost
- …and so on. You get it, there’s a lot — and that’s only the stuff you can _see_.

Like many large companies, each data source may be owned by different teams. With their own schedules, SLAs, and bugs.

<figure>
<img alt="A unamused man in front of a whiteboard, upon which scads of shapes with silly labels like “Magic Baby” and “Hell Proxy”, festooned by arrows pointing every which way." src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jes7dwinkjn6qf7qzebi.jpg">

<figcaption><p>After you see real API charts, <a href="https://www.youtube.com/watch?v=y8OnoxKotPQ">Krazam’s satirical microservices diagram</a> gets either more or less funny. Still figuring out which.</figcaption>
</figure>

Let’s say the 10 data sources I listed are each one API call. What are the odds my server can respond quickly enough?

> Let’s say 1 client request creates 10 downstream requests to a longtail-latency affected subsystem. And assume it has a 1% probability of responding slowly to a single request. Then the probability that at least 1 of the 10 downstream requests are affected by the longtail latencies is equivalent to the complement of all downstream requests responding fast (99% probability of responding fast to any single request) which is:
>
> {% katex %}
> 1 - (0.99)^{10} = 0.095
> {% endkatex %}
>
> That’s 9.5 percent! This means that the 1 client request has an almost 10 percent chance of being affected by a slow response. That is equivalent to expecting 100,000 client requests being affected out of 1 million client requests. That’s a lot of members!
>
> —[Who moved my 99th percentile latency?](https://engineering.linkedin.com/performance/who-moved-my-99th-percentile-latency)

And since users visit multiple pages in MPAs, the chances of suffering a high TTFB approaches “guaranteed”:

> Gil walks through a simple, hypothetical example: a typical user session involves five page loads, averaging 40 resources per page. How many users will _not_ experience something worse than the 95th percentile? 0.003%.
>
> —[Everything You Know About Latency Is Wrong](https://bravenewgeek.com/everything-you-know-about-latency-is-wrong/)

I believe this is why Kroger.com used a SPA in the first place — if disparate teams’ APIs can’t be trusted, at least they won’t affect other teams’ code. (Similar insulation from other teams’ components is probably one reason for React’s industry adoption.)

## The solution: streamed HTML

It’s easier to show than to explain:

<video title="Head-to-head recording of two web pages with identical content, but loaded differently." src="https://assets.codepen.io/183091/HTML+streaming+vs.+non.mp4" muted controls crossorigin>
<track kind="descriptions" src="https://assets.codepen.io/183091/html-streaming.vtt" srclang="en-us" label="Text description">
<a href="https://assets.codepen.io/183091/HTML+streaming+vs.+non.mp4">Download video: HTML Streaming vs. Non-streaming</a>
</video>

<!-- data:text/vtt,WEBVTT%0A%0A1%0A00%3A00.000%20--%3E%2000%3A02.500%0ALoad%20start.%20Left%20page%20is%20blank%2C%20right%20page%20already%20shows%20header%2C%20search%20query%20%E2%80%9C%3Clang%20es%3Efajitas%3C%2Flang%3E%E2%80%9D%2C%20and%20skeleton%20placeholders%20for%20search%20results.%0A%0A2%0A00%3A02.500%20--%3E%2000%3A04.000%0ALoad%20end.%20Both%20pages%20are%20complete%20with%20header%2C%20search%20query%2C%20and%20search%20results. -->

Both pages show search results in 2.5 seconds. But they sure don’t _feel_ the same.

Not all sites have my API bottlenecking issue, but many have its cousins: database queries and reading files. **Showing pieces of a page as data sources finish is useful for almost any dynamic site.** For example…

- Showing the header before potentially-slow main content
- Showing main content before sidebars, related posts, comments, and other non-critical information
- Streaming paginated or batched queries as they progress instead of big expensive database queries

Beyond the obvious visual speedup, streamed HTML has other benefits:

<dl>
<dt>Interactive ASAP
<dd><p>If a user visits the homepage and immediately tries searching, they don’t have to wait for anything but the header to submit their query.

<dt>Optimized asset delivery
<dd><p>Even with no <code>&lt;body&gt;</code> to show, you can stream the <code>&lt;head&gt;</code>. That lets browsers download and parse styles, scripts, and other assets while waiting for the rest of the HTML.

<dt>Less server effort
<dd><p>Streamed HTML uses less memory. Instead of building the full response in RAM, it sends generated bytes immediately.

<dt>More robust and faster than incremental updates via JavaScript
<dd><p>Fewer roundtrips, happens before/while JS boots, immune to JS errors and <a href="https://adamsilver.io/blog/javascript-isnt-always-available-and-its-not-the-users-fault/">the other reasons 1% of visits have broken JavaScript</a>…
<dd>And because it’s more efficient, that leaves more CPU and RAM for the JavaScript we <em>do</em> run, not to mention painting, layout, and user interactions.
</dl>

But don’t take my word for it:

- [Progressive rendering via multiple flushes · Stoyan Stefanov](https://www.phpied.com/progressive-rendering-via-multiple-flushes/)
- [Fun hacks for faster content · Jake Archibald](https://jakearchibald.com/2016/fun-hacks-faster-content/)
- [Progressive Rendering — The Key to Faster Web · Dinesh Pandiyan](https://medium.com/the-thinkmill/progressive-rendering-the-key-to-faster-web-ebfbbece41a4)
- [Streams FTW · Jake Archibald](https://jakearchibald.com/2016/streams-ftw/)

Hopefully, you see why I considered HTML streaming a must.

## And that’s why not Svelte

Previously…

> Maybe if I sprinkled the HTML with just enough CSS to look good… and if I had any room left, some laser-focused JavaScript for the pieces that benefit most from complex interactivity.

That’s _exactly_ what Svelte excels at. So why didn’t I use it?

Because [Svelte does not stream HTML.](https://github.com/sveltejs/svelte/issues/958) (I hope it does someday.)

## If not Svelte, then what?

I found only 2 things on NPM that could stream HTML:

1. [Dust](https://www.dustjs.com/), a template language that seems to have died twice.
2. [Marko](https://markojs.com/), some library with an ungoogleable name and a rainbow logo… oh, and JSX-like syntax? And a client-side virtual DOM that fit in my budget? _And_ eBay has battle-tested it for _its_ ecommerce websites? **_And_ it only uses client-side JS for `state`ful components?** _You don’t say._

It’s nice when a decision makes itself.

[![And thus, Marko.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5kkg0zc4q52izorcb9hz.png)](https://markojs.com/)

### Marko’s `<await>` made streaming easy

Marko streams HTML with [its `<await>` tag](https://markojs.com/docs/core-tags/#await). I was pleasantly surprised at how easily it could optimize browser rendering, with all the control I wanted over HTTP, HTML, and JavaScript.

{% details Disclaimer %}
I now work for eBay, but I didn’t yet when I wrote this post. 
{% enddetails %}

<figure>
![Buffered pages don’t show content as it loads, but Marko’s streaming pages show content incrementally.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6ssntpycuf50nda90o5r.gif)
<figcaption><p><a href="https://markojs.com/#streaming">Source: markojs.com/#streaming</a></figcaption>
</figure>

As seen in [<cite>Skeleton screens, but fast</cite>](https://dev.to/tigt/skeleton-screens-but-fast-48f1):

```html
<SiteHead />

<h1>Search for “${searchQuery}”</h1>

<div.SearchSkeletons>
  <await(searchResultsFetch)> <!-- stalls the HTML stream until the API returns search results -->
    <@then|result|>
      <for|product| of=result.products>
        <ProductCard product=product />
      </for>
    </@then>
  </await>
</div>
```

### `<await>` for nice-to-haves

Imagine a component that displays recommended products. Fetching the recommendations is usually fast, but every once in a while, the API hiccups. `<await>`’s got your back:

```html
<await(productRecommendations)
    timeout=50> <!-- wait up to 50ms -->
  <@then|recs|>
    <RecommendedProductList of=recs />
  </@then>

  <@catch>
    <!-- don’t render anything; no big deal if this fails -->
  </@catch>
</await>
```

If you know how much money product recommendations make, you can fine-tune the `timeout` so the cost of the performance hit never exceeds that revenue.

And that’s not all!

```html
<await(productRecommendations) client-reorder>
  <@placeholder>
    <!-- immediately render placeholder to prevent content jumping around -->
    <RecommendedProductsPlaceholder /> 
  </@placeholder>

  <@then|recs|>
    <RecommendedProductList of=recs />
  </@then>
</await>
```

The `client-reorder` attribute turns the `<await>` into an HTML fragment that doesn’t delay the rest of the page behind it, but asynchronously renders when ready. `client-reorder` requires JavaScript, so you can weigh the tradeoffs of using it vs. a `timeout` with no fallback. (I think you can even combine them.)

That’s [how Facebook’s BigPipe renderer worked](https://engineering.fb.com/2010/06/04/web/bigpipe-pipelining-web-pages-for-high-performance/), which once lived on the same page as React. Wouldn’t it be nice to have the best of both?

Let me tell you: it _is_ nice.

### Marko’s `<await>` is awesome

Best of all, these `<await>` techniques are Marko’s golden path — heck, [its very reason for being](https://tech.ebayinc.com/engineering/async-fragments-rediscovering-progressive-html-rendering-with-marko/). Marko has stream control no other renderer makes easy, a way to automatically upgrade streamed HTML with JavaScript, and 8+ years of experience with the inevitable bugs and edge cases.

<small>(Yes, I was quite taken with Marko. Let me have my fun.)</small>

However, the fact that Marko was apparently my _one_ option does raise a certain question…

## Why is HTML streaming not common?

Or in the words of another developer after my demo: **“if Chunked `Transfer-Encoding` is so useful, how come I’ve never heard of it?”**

That is a very fair question. It’s not because it’s poorly-supported — [HTML rendered progressively in Netscape 1.0. _Beta_ Netscape 1.0](https://web.archive.org/web/20040617070232/http://www.boutell.com/newfaq/history/fbrowser.html). And it’s not because the technique is barely-used — Google search results stream after the top navbar, for instance.

### I think one reason is the inconsistent name

- [Steve Souders called it “early flushing”](https://www.stevesouders.com/blog/2009/05/18/flushing-the-document-early/), which is not… the _best_ name.
- “Chunked transfer-encoding” is the most unique, but it’s only in HTTP/1.1. HTTP/2, HTTP/3, and [even HTTP/0.9 stream](http://1997.webhistory.org/www.lists/www-talk.1994q3/1147.html) differently.
- It was known as “HTTP streaming” before HLS, DASH, and other forms of video-over-HTTP took up that mindspace. 
- The catch-all term is “progressive rendering”, but that applies to many other things: interlaced images, visualizing large datsets, video game engine optimizations, etc.

### Many languages/frameworks don’t care for streaming

Older languages/frameworks have long been able to stream HTML, but were never really _good_ at it. Some examples:

<dl>
<dt>PHP 🐘
<dd><p>Requires calling <a href="https://www.php.net/manual/en/book.outcontrol.php">inscrutable output-buffering functions</a> in a finicky order.

<dt>Ruby on Rails 🛤
<dd><p><a href="https://api.rubyonrails.org/classes/ActionController/Streaming.html"><code>ActionController::Streaming</code></a> has a lot of caveats. In particular:

<blockquote>This approach was introduced in Rails 3.1 and is still improving. Several Rack middlewares may not work and you need to be careful when streaming. Those points are going to be addressed soon.</blockquote>

<p>Rails hit 3.1 in 2011. There was clearly not much demand to address those points.

<p>(Rails’ modern way is <a href="https://turbo.hotwired.dev/handbook/streams">Turbo Streams</a>, but those need JS to render, so not the same thing.)

<dt>Django 🐍
<dd><p><a href="https://docs.djangoproject.com/en/4.0/ref/request-response/#django.http.StreamingHttpResponse">Django really doesn’t like streaming at all</a>:

<blockquote><code>StreamingHttpResponse</code> should only be used in situations where it is absolutely required that the whole content isn’t iterated before transferring the data to the client.</blockquote>

<dt>Perl 🐪
<dd><p><a href="https://www.perlmonks.org/?node_id=671400">Perl’s autostream behavior is controlled by a <code>$|</code> variable</a> (yes, that’s a pipe), but that sort of nonsense is normal for it. God I love Perl.

</dl>

Because streaming was never their default happy path, languages/frameworks considered it a last resort where you gained performance at the expense of the “real” render features. Here’s a telling quote:

> You can still write ASP.NET pages that properly stream data to the browser using `Response.Write` and `Response.Flush`. But you can’t do it within the normal ASP.NET page lifecycle. Maybe this is a natural consequence of the ASP.NET abstraction layer.
>
> Regardless, it still sucks for users.
>
> —[The Lost Art of Progressive HTML Rendering](https://blog.codinghorror.com/the-lost-art-of-progressive-html-rendering/) 

**Node.js is a happy exception.** As proudly described on [Node’s About page](https://nodejs.org/en/about/):

> HTTP is a first-class citizen in Node.js, designed with streaming and low latency in mind.

Despite that, the “new” hot JavaScript frameworks have been struggling to stream for a while:

- React’s had `renderToNodeStream` since 2016, but [using it was tricky](https://reactjs.org/docs/concurrent-mode-suspense.html). Ergonomic streaming is the focus of a _lot_ of [upcoming React SSR work](https://github.com/reactwg/react-18/discussions/37).
- Ember’s [Glimmer can stream HTML](https://blog.emberjs.com/glimmer-progress-report/#toc_server-side-rendering), but it’s been “experimental” since 2016, too.
- Vue _can_ stream, but [with caveats and incompatibilities since it’s not the default](https://github.com/nuxt/nuxt.js/issues/4753).

These frameworks have the staff, funding, and incentives to make streaming work, so the holdup must be something else. Maybe it’s hard to retrofit streaming onto their abstractions, especially without ruining established third-party integrations.

### Streaming rarely mentioned as a TTFB fix

As mentioned near the beginning, when high TTFB is detected, streaming is almost never suggested as a fix.

I think _that’s_ the biggest problem. [A Web API with a bad name](https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest) can become popular if it’s mentioned enough.

Personally, I’ve only seen streaming HTML recommended for TTFB _once_, and it’s [in chapter 10 of High-Performance Browser Networking](https://hpbn.co/primer-on-web-performance/#optimizing-time-to-first-byte-ttfb-for-google-search). In an aside. At the bottom.

<small>(Inside a <code>&lt;details&gt;</code> labeled <a href="https://hitchhikersguidetothegalaxy.wordpress.com/2006/04/09/beware-of-the-leopard/">“Beware of The Leopard”</a>.)</small>

## So that’s one silver bullet down

I had streaming HTML, but that was [no substitute for the other 999 lead bullets to back it up](https://a16z.com/2011/11/13/lead-bullets/). Now I had to… make the website.

You know, write the components, style the design, build the features. How hard could _that_ be? (Hint: people are paid to do those things.)