Re-Exploring Reactivity and Introducing the Observer API and Reflex Functions
Tags: webdev, javascript, tooling, webqit
Published: 2023-07-01 04:57:16

<!--
Previously on *The Web-Native Track*: [Rethinking the Modern Web &amp; Introducing Web-Native Development](https://dev.to/oxharris/rethinking-the-modern-web-5cn1)!

Now enter *reactivity*!
-->

In the last few years, I have spent an insane amount of time thinking through *reactivity* on the frontend! In all that we've achieved in the space, the State of Reactivity today still leaves much to be desired:

+ **reactive primitives** - in the various mechanisms for intercepting and reflecting change, there is still **no easy way** around the concept of change detection! From language-level primitives - proxies and accessors - to custom primitives, this has continued to be a tricky game!

+ **syntax and mental model** - in the overall language and programming paradigm around the idea, there's still **no *ergonomic* and *mentally efficient* way** to express reactive logic! (Much rigor here often as we try to model normal logic using functions; or try to wield a compiler!)

These challenges greatly impact our work, sometimes in less obvious ways, **given how much of the frontend *paradigm* space this useful magic now occupies**, and thus, how much of it accounts for the code we write and [our overall tooling budget](https://dev.to/oxharris/rethinking-the-modern-web-5cn1)! For a paradigm so intrinsic to frontend, it couldn't be more compelling to revisit the various painpoints!

To this end, I took a stab:

+ at re-exploring the fundamental idea of change detection in JavaScript (i.e. reactive primitives) and the overall language and programming paradigm around reactivity (i.e. syntax and mental model) for some "cleaner" magic!
+ at re-exploring full language support for all of it in each case, given how the idea seems to suffer from meager language support and thus, remains, for the most part, externalized to tooling!

This is a pretty long article - with 3 main sections:

1. [Re-Exploring Change Detection](#reexploring-change-detection) (The design discussion around the [Observer](https://github.com/webqit/observer) API)
2. [Re-Exploring the Language of Reactivity](#reexploring-the-language-of-reactivity) (The design discussion around [Reflex Functions](https://github.com/webqit/reflex-functions))
3. [The Duo and the Prospect for Reactivity](#the-duo-and-the-prospect-for-reactivity)

{% details **Show Full Outline** %}

+ [Re-Exploring Change Detection](#reexploring-change-detection)
    + [Accessors and Proxies; the Tricky Parts](#accessors-and-proxies-the-tricky-parts)
        + [Accessors Got a Flexibility Problem There](#accessors-got-a-flexibility-problem-there)
        + [Proxies Got an Identity Problem There](#proxies-got-an-identity-problem-there)
        + [The Whole Idea of Internal Traps](#the-whole-idea-of-magic-objects)
        + [The Whole Idea of "Magic Objects"](#the-whole-idea-of-magic-objects)
    + [A Way to Think About the Problem](#a-way-to-think-about-the-problem)
    + [Rediscovering Prior Art and Nailing the Problem](#rediscovering-prior-art-and-nailing-the-problem)
    + [Introducing the Observer API](#introducing-the-observer-api)
        + [An Overview](#an-overview)
        + [Featuring Path Observability](#featuring-path-observability)
        + [Documentation](#documentation)
        + [The Project](#the-project)
+ [Re-Exploring the Language of Reactivity](#reexploring-the-language-of-reactivity)
    + [The Toll On Syntax and Mental Model](#the-toll-on-syntax-and-mental-model)
        + [Solving for Syntax](#solving-for-syntax)
        + [Solving for Mental Model](#solving-for-mental-model)
        + [Solving for Both](#solving-for-both)
    + [Bringing Reactivity to Imperative JavaScript and Nailing the Problem](#bringing-reactivity-to-imperative-javascript-and-nailing-the-problem)
    + [Introducing Reflex Functions](#introducing-reflex-functions)
        + [An Overview](#an-overview-1)
        + [Change Propagation](#change-propagation)
        + [Documentation](#documentation-1)
        + [The Project](#the-project-1)
+ [The Duo and the Prospect for Reactivity](#the-duo-and-the-prospect-for-reactivity)
+ [Acknowledgements](#acknowledgements)
+ [References](#references)

{% enddetails %}

---------------------------

## Update Jan, '24

[Reflex Functions](https://dev.to/oxharris/re-exploring-reactivity-and-introducing-the-observer-api-and-reflex-functions-4h70#introducing-reflex-functions) has had a major update since this initial announcement! This includes:

+ A name change: being now [Quantum JS](https://github.com/webqit/quantum-js)!
+ A major API revamp: majorly in the return values of Quantum Functions - from being a tuple `[ value, reflect ] = sum()` to being a "State" object `state = sum()`!
+ An entire compiler overhaul towards a more runtime-driven form of reactivity!

Generally, while the ideas shared here under the term "Reflex Functions" are largely the same with its [Quantum JS](https://github.com/webqit/quantum-js) successor, there are important differences in the implementation!

---------------------------

> SECTION 1/3

## Re-Exploring Change Detection

Change detection comes as the first challenge to reactivity in JavaScript!

The general idea deals with detecting and responding to changes made to JavaScript objects to drive synchronization - sometimes data binding, but generally, any functionality that is dependent on object state. This in JavaScript has been approached in many different ways past and present.

This wouldn't be about Signals and Hooks, because - **paradigm mismatch**:

> Those "functional" primitives are designed with the [Functional Programming](https://en.wikipedia.org/wiki/Functional_programming) paradigm in mind, where immutable data and pure functions are emphasized. Object observability, on the other hand, is more closely associated with the [Object-Oriented](https://en.wikipedia.org/wiki/Object-oriented_computer_programming) paradigm, where mutable state and object interactions are the idea. And whereas the former basically represents an intermediary event system, the latter deals with object-level change detection via native mechanisms.
>
> Meanwhile, later in the second half of this article, we will be touching on Signals and Hooks.

{% details Compare also: **Mutability vs. Immutability** %}

> [Immutability](https://en.wikipedia.org/wiki/Immutable_object) is a programming principle that treats data as unchangeable. Here, the idea of change is modelled by functions that repesent a pipeline of transformations for *whole data* wherein each transformation creates a new instance, and observability happens at the event level, well, instead of at the fine-grained level. Mutability, on the other hand, embraces the concept of change, wherein objects are modified in place.

{% enddetails %}

Historically, certain techniques have had their day here! For example, [dirty-checking](https://docs.angularjs.org/guide/scope#scope-life-cycle) was the technique in Angular; and custom [pub/sub](https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern) mechanisms were the idea in [Backbone Models](https://backbonejs.org/#Model-View-separation) and [Ember Models](https://guides.emberjs.com/v2.11.0/models/defining-models/). These and probably other approaches at the time have now falling out of favour for newer, more straightforward solutions!

<!--
A form of the idea is a "pub/sub" object, wherein observability is based on dedicated instance methods for "get" and "set" operations over encapsulated data:

```js
const person = {
  name: 'John',
  age: 30,
};

const observable = new Observable(person);
observable.subscribe((prop, value) => {
  console.log(`Property "${prop}" changed to: ${value}`);
});

observable.set('name', 'Jane'); // Triggers the observe callback
```

Well, it turns out, their perfect usecase may be elsewhere, but not in object observability!

First, **we still aren't actually gaining observability at the real data level** - i.e. at the "`person` object" level, taking the *extra* outer medium out of the equation! In other words, the means to *observability* in this approach doesn't actually solve the problem itself!

Furthermore, in the context of object observability, **this wrapper-based approach doesn't always go without an object identity trade-off** - in which case the identity of the inner, actual object have to be re-expressed in the implementation of the outer wrapper, leading to a new type system:

```js
console.log(observable instanceof ObservablePerson);
console.log(observable instanceof ObservableProduct);
```

Similarly, other semantics of the inner, actual object often also end up being traded for wrapper-based `get()`/`set()` methods and similar alternatives - e.g. the beauty of interacting with the `person` object itself with *literal* property access - `person.name`, or something in the `person` API - e.g. `person.speak()` - that may have particular significance to the code.

There's now a ton of details to walk through and much clumsiness! But that's to say: **_object observability_ isn't just what you'd nail with pub/sub**!
-->

### Accessors and Proxies; the Tricky Parts

Today, [object accessors](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object/defineProperty) (since ES5) and [proxies](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Proxy) (since ES6) have greatly revolutionized the problem space! Being integral to the language, these primitives constitute the only way to detect change at the program level:

> *Accessors*

```js
const person = {
  // name accessors
  get name() { ... },
  set name(value) { ... },
  // age accessors
  get age() { ... },
  set age(value) { ... },
};
```

> *Proxies*

```js
// Original object
const $person = {
  name: 'John',
  age: 30,
};
// Proxy wrapper
const person = new Proxy($person, {
  get(target, key) { ... },
  set(target, key, value) { ... }
});
```

Such that we're able to hide *a ton* behind a dot syntax:

```js
person.name = 'James';
person.age = 40;
```

Giving a clean interface where you'd normally have needed intermediary `get()`/`set()` mechanisms is perhaps what makes these so nifty that you definitely want to use them! But then comes additional details to walk through, where you end up with an unexpected amount of boilerplate and a tricky implementation!

> Let's add here that there's something on the horizon: [Decorators](https://github.com/tc39/proposal-decorators)! (On which Dr. Axel Rauschmayer takes a [deep dive](https://2ality.com/2022/10/javascript-decorators.html) in their latest, stage 3, syntax.) But it turns out, while decorators might be solving something in the broader scope of meta programming in JavaScript, **they aren't adding anything to _change detection_ other than syntax sugars over existing possibilities**! So, if the question is object observabilty, I'm the least excited about decorators!

#### Accessors Got a Flexibility Problem There

It's probably the biggest deterrent to using them: **the inflexibility of working per-property and not supporting on-the-fly properties**! This requires you to make upfront design decisions around all the possible ways an object could change at runtime - an assumption that's too unrealistic for many dynamic applications; one that fails quickly if you tried!

> Accessors were the primary reactive mechanism in [Vue 2](https://v2.vuejs.org/v2/guide/reactivity.html), and that's probably the best place to see their reallife limitations - the biggest of which is perhaps their inability to detect new property additions! This became a big enough issue for the framework that [they didn't hesitate to move away](https://increment.com/frontend/making-vue-3/#:~:text=switching%20to%20proxy%20would%20allow%20us%20to%20eliminate%20vue%E2%80%99s%20existing%20limitations%2C%20such%20as%20the%20inability%20to%20detect%20new%20property%20additions%2C%20and%20provide%20better%20performance.) when they had the chance to!

#### Proxies Got an Identity Problem There

It turns out that for every "proxy" instance, you practically have to account for two different object identities, **and that comes very problematic for code that relies on type checks (`x instanceof y`) and equality checks (`x === y`)**. Heck, even two proxy instances with the same object wouldn't equate, due to their different identities!

> As [David Bruant](https://twitter.com/DavidBruant) would have me do here, let's add that with some hack, you could get "instanceof" working correctly: [via the `getPrototypeOf` trap](https://jsfiddle.net/Lr82xzpm/). (But I guess that doesn't make the whole idea any less tricky but more!)
>
> And maybe let's reference [Tom van Cutsem](https://twitter.com/tvcutsem) on [the concept of membranes](https://tvcutsem.github.io/js-membranes) and [Salesforce's implementation of the idea](https://github.com/salesforce/observable-membrane) - wherein you're able to have all references to the same target return the same proxy instance, and thus have references pass "equality checks"! (But I guess membranes aren't what you're really out here to do!)
>
> All of this is probably another way to see the problem *for reactivity* with proxies: *tricky*, *no easy way out*!

#### The Whole Idea of Internal Traps

It comes as a *gap* between the communication model that those internal methods enable and the typical model around reactivity: these primitives don't bring with them the regular subscription model wherein anyone can subscribe to changes on a subject! They just give you a way to *intercept* - in terms of *internal traps* - but not a way to *observe* from the outside - in something like a `.subscribe()` method! And in however you go from here, this becomes a tricky game all the way:

+ **Tricky if you were to take those internal methods for generic "observers"!** In the case of accessors, you couldn't have another "observer" without re-defining a property, and thus **inadvertently displacing an existing one**! And in the case of proxies, while you wouldn't be able to displace traps anyway as those are finalized at instance time (i.e. closed-in by the constructor), you also couldn't have another "observer" without, this time, creating *another proxy instance* over same object, and thus **inadvertently ending up with multiple redundant proxies that are each tracking very different interactions** - giving you a leaky, bypassable tracking mechanism each time!
    &ensp;

    > *Accessors*

    ```js
    Object.defineProperty(person, 'age', { set() { /* observer 1 */ } });
    // Singleton Error: A new observer inadvertently breaks the one before
    Object.defineProperty(person, 'age', { set() { /* observer 2 */ } });
    ```
    &ensp;

    > *Proxies*

    ```js
    let $person1 = new Proxy(person, { /* observer 1 */ });
    goEast($person1);
    // Singleton Error: A new observer inadvertently creates redundancy
    let $person2 = new Proxy(person, { /* observer 2 */ });
    goWest($person2);
    ```
    &ensp;

    Turns out, those internal methods *don't at all add up* to a reactive *system*! Until you actually build one, you don't have one!
    &ensp;

+ **Tricky in going from those internal methods to a "reactive system"!** You'd need to build an event dispatching mechanism that those internal methods get to talk to, and that couldn't happen without having to walk through new technical details! And there lies many pitfalls! For example, it's quick and easy to inadvertently introduce a breaking change to public object interfaces - e.g. `window.document` - by *polluting/patching* their namespace with setters and getters, or with something like a `.subscribe()` method!
    &ensp;

    > *Accessors*

    ```js
    Object.defineProperty(window.document, 'custom', { set() { /* observer 1 */ } });
    ```
    &ensp;

    ```js
    const person = {
      // Namespace pollution
      _callbacks: [],
      subscribe(callback) {
        this._callbacks.push(callback),
      },
      // Accessors
      get name() { ... },
      set name(value) { ... },
    };
    ```
    &ensp;

    > *Proxies*

    ```js
    const _callbacks: [],
    function subscribe(callback) {
      _callbacks.push(callback);
    }
    const $person = new Proxy(person, {
      get(target, key) {
        // Namespace pollution
        return key === 'subscribe' ? subscribe : target[key];
      },
      set(target, key, value) { ... }
    });
    ```
    &ensp;

    Turns out, there's no *direct/neat* path to go from "internal methods" to "reactive system"! 

#### The Whole Idea of "Magic Objects"

That comes as the narrative for the form of reactivity that these techniques enable, and the implication is in terms of the final model that these magic objects create: **a system where observability is an _object-level feature_, or in other words, an _object-level concern_**! This comes with **much complexity** and **multiple points of failure at the granular level**!

Being a concern at such a granular level of your architecture, "reactivity" now tends to govern every detail of your work: **objects have to be purpose-built for observability**, **and existing ones retrofitted**; **and in all you do, you must be _explicit_ about reactivity**! And because *it is hard to gain and easy to lose*, **you also have to be conscious about how each object flows through your application**; and there lies _the footguns and bottlenecks_:

+ **Tricky to account for across the various points of interaction!** Given a system where functionality relies on data and reactivity being bundled together and passed around together, it happens so easily that the latter is lost on transit without notice! That becomes a big problem in any fairly-sized codebase!
    &ensp;

    > *Accessors*

    ```js
    // Using the accessor-based ref() function in Vue.js
    import { ref, watchEffect } from 'vue';

    const count = ref(0); // Reactivity gained
    watchEffect(() => {
      console.log('Count value changed:', count.value);
    });

    function carelessHandling(count) {
        delete count.value;
        count.value = 2;
    }
    
    carelessHandling(count); // Reactivity lost
    ```
    &ensp;

    It turns out, any reactive system based on "magic objects" is susceptible to failure at the granular level! 
    &ensp;

+ **Tricky in dealing with object trees!** You couldn't gain reactivity beyond the surface without some form of recursive transformations, **and yet all of that would still be subject to the assumption that your object tree is a *static* structure**! Things get extra tricky this time because, in reality, **that assumption isn't always the case**!
    &ensp;

    > *Proxies*

    ```js
    // Using the depth-capable reactive() function in Vue.js
    import { reactive } from 'vue';

    // Creating a reactive object tree
    const reactiveTree = fetch(resourceUrl).then(res => res.json()).then(json => {
      return reactive(json);
    });

    // Problem
    reactiveTree.entries.push({ ... });
    ```
    &ensp;

    Turns out, any reactivity gained from stitching proxies or accessors together is most often than not *unpredictable* and *rickety*!

### A Way to Think About the Problem

Proxies and accessors constitute the only *native* way today to do *mutation-based* reactive programming - not particularly because they perfectly reflect the desired approach, but because they're the only native way! However, this isn't a case of poorly designed language features, **this is rather about usecase mismatch**! Whereas proxies and accessors have their perfect usecases across the broader subject of meta programming, **reactive programming represents a usecase not well captured**; or in other words: **something that isn't very much their usecase**!

> Proxies, for example, are mind blowing in what they can do across the broader subject of meta programming, having had a massive technical and academic work go into their design (as can been seen from [ES-lab's original design document](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/37741.pdf)), and it is on this backdrop they seem to be the silver bullet to reactivity! But it turns out that **it is on just a subset of that - typically something around three internal traps: `get`, `set`, `deleteProperty`** - that we're building our reactive world, **and well, the rest of which isn't fully captured in the design of proxies**!

While object observability as a technique may have an *overlap* with the usecase for proxies and accessors, **it isn't the main thing these were designed for, and these primitives conversely don't come ideal for the problem case**! If I were to represent that mathemaically, that would be 25% overlap, 75% mismatch:

<!--
This is not an esoteric or rare issue and will potentially affect a lot of people as private members begin to be adopted more broadly
+ https://exploringjs.com/deep-js/ch_proxies.html
+ http://soft.vub.ac.be/Publications/2012/vub-soft-tr-12-03.pdf
+ https://github.com/tvcutsem/es-lab/tree/master
+ https://esdiscuss.org/topic/implementing-membranes-using-proxies-and-getinheritance
-->

![Intersection between native primitives and reactivity](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tm4uuzktux4d1qmb7fpp.png)

This leaves us with a desire to find something made for the problem! And it turns out, this isn't the first time we're hitting this wall!

### Rediscovering Prior Art and Nailing the Problem

This is where history helps us!

Amidst the remnants lies a design precedent to object observability waiting to be re-discovered: the [`Object.observe()`](https://web.dev/es7-observe/) API, which was depreciated in November 2015! It probably was ahead of its time, but in its 8 years in the mud, there's something time has taught us: **_the platform needs native object observability_, and it's okay to have things coexist for different usecases**!

Consider the approach in this API:

```js
Object.observe(object, callback);
```

Notice that, now, **the tracking mechanism is _separate_ from the objects themselves**, and what's more, universal for every object! (It's much like the upgrade from [`.__defineSetter__()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object/__defineSetter__) and [`.__defineGetter__()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object/__defineGetter__) to [`Object.defineProperty()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object/defineProperty)!) **And this basically solves all of our problems today with primitives**!

+ Now, we can put all of that "magic object" idea behind us: observability shouldn't have to be an object-level feature (or object-level concern)!
    + We no more have to be explicit about reactivity; or be governed by that concern!
    + It shouldn't have to be gained or lost; or be passed around together with data!
+ Now everything just works on the fly; no more upfront work nor dealing with internal traps!
    + No more patching objects, nor transforming object trees!
+ Now we can interact with real objects across our codebase: things no more rely on intermediary functions or wrappers!
    + No more accounting for two different object identities or hacking around that!
    + No more dealing with leaky, bypassable tracking mechanism!

And doesn't that nail it? Why shouldn't we explore this further?

Having struggled around object observability myself over the years, I find it necessary to re-explore this prior art today as a new project: **the Observer API**!

### Introducing the Observer API

This is a new effort that re-explores object observability along the lines of `Object.observe()`, but this time, with a more wholistic approach! It takes a leap at what could be **a unifying API** over *related but disparate* APIs like `Object.observe()`, the "Proxy traps" API, and the [Reflect](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Reflect) API (which is a complementary API to proxies)!

This is an upcoming proposal!

#### An Overview

As an improvement over its original form - [`Object.observe()`](https://web.dev/es7-observe/), the object observability API is being re-imagined!

Whereas the previous API lived off the global "Object" object, the new idea is to have it on the global "Reflect" object as if being one of the missing pieces in a puzzle! (But this is subject to whether everything considered here still falls within the scope of the Reflect API.) But that's just one of two possibilitis considered! The other is to have it on a new Reflect-like object called `Observer`, which is, this time, a fully-featured reactivity API!

While that unfolds, a prototype of the Observer API exists today as `Observer`, featuring a superset of each Reflect method in addition to other specialized APIs. Much of the code here is based on this `Observer` namespace.

| Observer API    | Reflect API |
| -------- | ------- |
| `apply()`  | `apply()`    |
| `construct()`  | `construct()`    |
|  |           |
| `observe()`  | `-`    |
|  |          |
| `set()`  | `set()`    |
| `setPrototypeOf()`  | `setPrototypeOf()`    |

Being fully compatible with the Reflect API, the Observer API can be used today as a drop-in replacement for Reflect.

To begin, here's the `observe()` method:

└ Signatures

```js
// Observe all properties
Observer.observe(object, callback);
```

```js
// Observe a list of properties
Observer.observe(object, [ propertyName, ... ], callback);
```

```js
// Observe a value
Observer.observe(object, propertyName, inspect);
```

└ Handler

```js
function callback(mutations) {
  mutations.forEach(inspect);
}
```

```js
function inspect(m) {
  console.log(m.type, m.key, m.value, m.oldValue, m.isUpdate);
}
```

From here, additional features become necessary! We discuss some of these here and link to the rest.

#### Featuring Path Observability

A much needed feature for `Object.observe()` was path observability! The void that initially having it left out created required much boilerplate to fill. A few polyfills at the time (e.g. Polymer's [observe-js](https://github.com/googlearchive/observe-js)) took that to heart. And that certainly has a place in the new API!

This time, instead of following a string-based "path" approach - `level1.level2` - a path is represented by an array - a `Path` array instance:

```js
const path = Observer.path('level1', 'level2');
```

An array allows us to support property names that themselves have a dot in them. And by using Observer's `Path` array instance, we are able to distinguish between normal "property list" array - as seen earlier - and actual "path" array.

The idea here is to observe "a value" at a path in a given tree:

```js
// A tree structure that satisfies the path above
const object = {
  level1: {
    level2: 'level2-value',
  },
};
```

```js
Observer.observe(object, path, (m) => {
  console.log(m.type, m.path, m.value, m.isUpdate);
});
```

```js
object.level1.level2 = 'level2-new-value';
```

{% details Console %}

| type | path | value | isUpdate |
| ---- | ---- | ----- | -------- |
| `set` | [ `level1`, `level2`, ] | `level2-new-value` | `true` |

{% enddetails %}

And well, the initial tree structure can be whatever:

```js
// A tree structure that is yet to be built
const object = {};
```

```js
const path = Observer.path('level1', 'level2', 'level3', 'level4');
Observer.observe(object, path, (m) => {
  console.log(m.type, m.path, m.value, m.isUpdate);
});
```

Now, any operation that changes what "the value" at the path resolves to - either by tree extension or tree truncation - will fire our listener:

```js
object.level1 = { level2: {}, };
```

{% details Console %}

| type | path | value | isUpdate |
| ---- | ---- | ----- | -------- |
| `set` | [ `level1`, `level2`, `level3`, `level4`, ] | `undefined` | `false` |

{% enddetails %}

Meanwhile, this next one completes the tree, and the listener reports a value at its observed path:

```js
object.level1.level2 = { level3: { level4: 'level4-value', }, };
```

{% details Console %}

| type | path | value | isUpdate |
| ---- | ---- | ----- | -------- |
| `set` | [ `level1`, `level2`, `level3`, `level4`, ] | `level4-value` | `false` |

{% enddetails %}

If you were to find the exact point at which mutation happened in the path in an audit trail, use the event's `context` property to inspect the parent event:

```js
let context = m.context;
console.log(context);
```

And up again one level until the root event:

```js
let parentContext = context.context;
console.log(parentContext);
```

And you can observe trees that are built *asynchronously*! Where a promise is encountered along the path, further access is paused until promise resolves:

```js
object.level1.level2 = Promise.resolve({ level3: { level4: 'level4-new-value', }, });
```

#### Documentation

Visit the [docs](https://github.com/webqit/observer/wiki) for full details - including [Timing and Batching](https://github.com/webqit/observer/wiki#timing-and-batching), [full API Reference](https://github.com/webqit/observer/wiki#putting-it-all-together), and more.

#### The Project

The Observer API is being developed as something to be used today - via a polyfill. The polyfill features all of what's documented - with a few limitations!

{% embed https://github.com/webqit/observer %}

Show support with a star on github! Also, all forms of contributions are welcome at this time.

-----------------

> SECTION 2/3

## Re-Exploring the Language of Reactivity

While we may have fixed object observability, **we still need to find a way to actually write reactive logic - in how we normally would write JavaScript programs**! There's a big difference here, and it's like the difference between playing individual musical notes and composing a song!

<!--
> Where you combine notes, melodies, harmonies, and rhythms to create a cohesive musical piece.
-->

> But while this constitutes a different problem from what object observability solves, they've got be related, and that's definitely [where this journey culminates](#the-duo-and-the-prospect-for-reactivity)!

Come to writing application logic...

Here is how we learned to write applicatons (notice: in normal, imperative JavaScript):

```js
let count = 5;
let doubleCount = count * 2;
console.log(doubleCount); // 10
```

and the challenge for reactive programming is to find a way to express the same such that the logic...

```js
count -> doubleCount -> console.log(doubleCount)
```

...continues to hold even when a part of the dependency chain is updated!

This magic by itself has had no formal language, yet it has managed to become the most spoken *language* on the frontend today! But unfortunately, **reactivity is a _difficult_ language to speak:**

This has involved compiler-driven syntaxes and whole new [DSLs](https://en.wikipedia.org/wiki/Domain-specific_language), and on a more general note, making the shift to writing applications under an *entirely different* programming paradigm: [Functional Programming](https://en.wikipedia.org/wiki/Functional_programming)! And there lies the big problem for the average developer: **that *paradigm shift* from "normal" JavaScript**!

<!--
A flurry of [DSLs](https://en.wikipedia.org/wiki/Domain-specific_language) have been created for this:

some extending HTML with special directives to support various things like data binding, event handling, conditional rendering, looping, and more:
+ [Vue.js Syntax](https://vuejs.org/guide/essentials/template-syntax.html) - having directives like `v-if`, `v-for`, etc.
+ [Angular Syntax](https://angular.io/guide/architecture-components) - having directives like `ngIf`, `ngFor`, etc.

and some extending JavaScript for basically the same purpose, but with the full capabilities of JavaScript:

+ [JSX](https://en.wikipedia.org/wiki/JSX_(JavaScript)) - with angle brackets `<` `>` and curly braces `{` `}` for switching in and out of XML.
+ [Svelte](https://svelte.dev/docs/svelte-components) - with the dollar sign label `$:` for reactivity.

In the most common cases, while DSLs address *presentational* needs ("presentational" being the "D" in DSL), JavaScript itself remains the language for core, non-presentational logic - but this time, under a *different* programming paradigm 

Today, reactive logic **doesn't have an easy way out**; and 
-->

It just happens that there's no easy way around the idea! And when you look: **what *always* goes for reactivity is syntax and mental model**! Only, some approaches can be considered better off than others on either of those factors.

### The Toll On Syntax and Mental Model

Enter the typical language of reactivity: Functional Programming...

Whereas imperative programs are written in **literal terms** and are based on a **linear execution flow** (a sequential, line-by-line evaluation of statements and control flow structures), functional programming requires a series of change detection primitives to model the given application logic!

Given our sample program in context, here's what the "functional" equivalent could look like across frameworks, using a pretty identical set of primitives:

> React

```js
import { useState, useMemo, useEffect } from 'react';

// count
const [count, setCount] = useState(5);
// doubleCount
const doubleCount = useMemo(() => count * 2, [count]);
// console.log()
useEffect(() => {
  console.log(doubleCount);
}, [doubleCount]);

// Update
setCount(10);
```

> SolidJS

```js
import { createSignal, createMemo, createEffect } from "solid-js";

// count
let [count, setCount] = createSignal(5);
// doubleCount
let doubleCount = createMemo(() => count() * 2);
// console.log()
createEffect(() => {
  console.log(doubleCount());
});

// Update
setCount(10);
```

> Vue.js

```js
import { ref, computed, watchEffect } from 'vue';

// count
const count = ref(5);
// doubleCount
const doubleCount = computed(() => count.value * 2);
// console.log()
watchEffect(() => {
  console.log(doubleCount.value);
});

// Update
count.value = 10;
```

> Svelte

```js
import { writable, derived } from 'svelte/store';

// count
const count = writable(5);
// doubleCount
const doubleCount = derived(count, $count => $count * 2);
// console.log()
doubleCount.subscribe(value => {
  console.log(value);
});

// Update
count.set(10);
```

> MobX

```js
import { observable, computed, autorun } from 'mobx';

// count
const count = observable.box(5);
// doubleCount
const doubleCount = computed(() => count.get() * 2);
// console.log()
autorun(() => {
  console.log(doubleCount.get());
});

// Update
count.set(10);
```

What do we have here? A common reactive language, and, well, the problem case: **a *programming paradigm* shift**!

What is in traditional JavaScript a literal variable declaration is in the functional approach a very different type of declaration:

```js
// Variable
let items = ['one', 'two'];
```

```js
// Read/write segregation
const [items, setItems] = createSignal(['one', 'two']);
```

Also, what is to traditional JavaScript a "mutable" world is to the functional approach an "immutable" world (or you end up defying the reactivity system):

```js
// Mutation world
items.push('three');
```

```js
// Immutable world
setItems([...items(), 'three']);
```

And the story goes the same for conditional constructs, loops, etc.!

The focus entirely shifts now from writing idiomatic JavaScript to following paradigm-specific constraints, and in addition, other implementation-specific details:

+ In the Signals family: [that technique around fine-grained reactivity](https://dev.to/this-is-learning/making-the-case-for-signals-in-javascript-4c7i#ok-but-what-about-the-tradeoffs) - wherein you have to carefully craft, or "[group](https://dev.to/ryansolid/comment/256gh)", logic "around data flow" to actually stay fine-grained. Also, [that extra invokation syntax](https://twitter.com/acdlite/status/1628930249718202369#:~:text=we%20don't%20like%20the%20code%20you%20have%20to%20write%20to%20get%20it.) - `count()`, `count.get()`, `count.value` - that goes with what was supposed to be a literal variable.
+ In the Hooks family (React, Preact): [the underestimated *dependency* array](https://twitter.com/sebinsua/status/1628861236925407235), etc.

All of this on top of the required diligence needed to manually model the control flow and dependency graph of your application logic - by piecing together multiple primitives! Anyone who's been there can tell how the many *moving parts* and high amount of *abstraction* and *indirection* **significantly impact the authoring experience and cognitive load**!

It turns out that this is a problem with any "runtime" magic as there must always be a thing to put up with: **syntax noise and other ergonomic overheads**, **a difficult-to-grok execution model and other cognitive overheads**, etc.!

**This has been a strong enough case across frameworks that everyone has had to support their "functional" magic with some other type of magic that tries to solve either for syntax or for mental model!**

#### Solving for Syntax

Vue, for example, will help you [automatically unwrap](https://vuejs.org/guide/essentials/reactivity-fundamentals.html#ref-unwrapping-in-templates) certain refs used in `.vue` templates, and in a few other contexts. (The idea even made it as a more extensive [syntax transform](https://vuejs.org/guide/extras/reactivity-transform.html) project!) Similarly, in React, [work is underway](https://www.youtube.com/watch?v=lGEMwh32soc&t=2s) to [put memoization and more](https://twitter.com/acdlite/status/1628862794572374019) behind a compiler!

Svelte stands out here as it goes full-blown with the idea of a compiler to let you write whole logic in literal, imperative JavaScript in `.svelte` templates, on top of its "functional" core:

```js
<script>
  let count = 0;
  $: doubleCount = count * 2;
</script>

<main>
  <button on:click={() => count += 1}>Double Count: {doubleCount}</button>
</main>
```

Only, it suffers from being unintuitive by the "twist" - the magic - that went into it: compiling from *literal*, *linear* syntax to a *functional*, *non-linear* execution model, thus defying the general expectation of the "[imperative](https://en.wikipedia.org/wiki/Imperative_programming)" paradigm it appears to go by!

For example, as documented:

```html
<script>
export let person;

// this will update `name` when 'person' changes
$: ({ name } = person);

// don't do this. it will run before the previous line
let name2 = name;
</script>
```

And you can see that again here in how the program seems to go bottom-up:

```html
<script>
let count = 10;

$: console.log(count); // 20

// Update
$: count = 20;
</script>
```

There goes the non-linear execution model behind the imperative syntax! So, it turns out, what you get isn't *what you see* when you look in the face of your code, but *what's documented*! Indeed, Svelte scripts "are nothing like 'plain JavaScript' yet people seem to be accepting of those and some even advertising them as such." - [Ryan Carniato](https://dev.to/this-is-learning/the-quest-for-reactivescript-3ka3#:~:text=svelte's%20scripts%20are%20nothing%20like%20%22plain%20javascript%22%20yet%20people%20seem%20to%20be%20accepting%20of%20those%20and%20some%20even%20advertising%20them%20as%20such.)

Nonetheless, Svelte's vision underscores **the strong case for less ergonomic overheads in *plain*, *literal* syntax** - the bar that every "syntax" quest actually has in front of them! **Only, this cannot be rational with a *functional*, *non-linear* core - as seen from its current realities**!

#### Solving for Mental Model

React has long stood on what they consider a "simpler" rendering model:

Whereas in normal functional programming, changes propagate from function to function as explicitly modelled, React deviates from the idea in its "hooks" approach for a model where changes don't propagate from function to function as explicitly modelled, but propagate directly to the "component" and trigger a full "top-down" re-render of the component - giving us a "top-down" rendering model!

The pitch here is:

> *"You don't have to think about how updates flow through your UI anymore! Just rerender the whole thing and we'll make it fast enough."* - [Andrew Clark](https://twitter.com/acdlite/status/1628811936522543104)

And in relation to Signals:

> *"That's why I don't buy the argument that signals provide a better DX/mental model."* - [Andrew Clark](https://twitter.com/acdlite/status/1628811937797574657)

Which from this standpoint should remain highly-protected:

> *"I think you guys should be way more aggressive in pushing back against this stuff. Throwing out the react programming model and adopting something like signals (or worse, signals plus a dsl) is a huge step backwards and most people haven’t internalized that."* - [Pete Hunt](https://twitter.com/floydophone/status/1629009165422104576)

And when you look, you see a vision here for **a "linear" execution model**, much like the linear "top-down" flow of regular programs!

Only, it suffers from being counterintuitive by itself with the "twist" - the magic - that went into it: defying the general expectation of the "[functional](https://en.wikipedia.org/wiki/Functional_programming)" paradigm it appears to go by! So, it turns out, what you get isn't *what you see* when you look in the face of your code, but *what's documented*, which for many devs makes the Signals approach *more grokable*, wherein "functional" is "functional":

> *"its funny, the first time i ever used signals wasn't for perf, it was for understanding where change happens easier.  plus you get LSP references to change.*
>
> *just seems.... most rational"* - [ThePrimeagen](https://twitter.com/ThePrimeagen/status/1629221580344532992)

Being as it may:

> *"The React thought leadership had long cultivated the narrative of 'our model is more correct', almost becoming a dogma. In reality, many engineers are simply more productive with the signals mental model - arguing on a philosophical level is pointless when that happens."* - [Evan You](https://twitter.com/youyuxi/status/1631089471377801216)

And the whole idea seems to fall short again with the extra [useMemo/useCallback](https://twitter.com/acdlite/status/1628862794572374019) performance hack needed to control the "re-render" mdoel!

Nonetheless, React's vision underscores **the strong case for less mental overheads in terms of a _linear_ execution model** - the very thing we hope to regain someday! **Only, this doesn't seem to be what function primitives can provide rationally - as seen from its current realities**!

#### Solving for Both

Given the ongoing quest for reactivity without the ergonomic and mental overheads, the *new bar* for reactivity is to **not solve for one and fall short on the other**! We may not be hitting the mark in a single implmenentation now, but not until [more and more people](https://twitter.com/trueadm/status/1628866012786327552) discover the power of compilers and make the full shift!

> *"Smart compilers will be the next big shift. It needs to happen. Look at the optimization and wizardry of modern C compilers. The web needs similar sophistication so we can maintain simpler mental models but get optimized performance."* - [Matt Kruse](https://twitter.com/matt_kruse/status/1626654945775689729?s=20)

In their ability to take a piece of code and generate a different piece altogether, **compilers give us a blank check to write the code we want and get back the equivalent code for a particular problem**, becoming the hugest prospect for reactive programming! Soon on the Language of Reactivity, we may never again talk about "functional" primitives or "functional" core! And there's only one place this leads: **reactivity in just plain JavaScript; this time, in the *literal form* and *linear flow* of the language**!

> *"It’s building a compiler that understands the data flow of arbitrary JS code in components.*
>
> *Is that hard? Yeah!*
> *Is it possible? Definitely.*
>
> *And then we can use it for so many things…"* - [sophie alpert](https://twitter.com/sophiebits/status/1628952004725989377)

**There goes the *final bar* for reactivity!** It is what you see given half the vision in Svelte: "syntax" (assuming a path fully explored to the end) and half the vision in React: "top-down flow" (assuming a path fully explored to the end); the point where all quests on the Language of Reactivity culminate!

![Culmination of various quests in reactivity](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sd30oj6p8jv2i8e3k7na.png)

But what if *everything in this "reactivity of the future" thesis was possible today*, and what's more, *without the compile step*?

### Bringing Reactivity to Imperative JavaScript and Nailing the Problem

You'd realize that the idea of "building a compiler that understands arbitrary JS code" is synonymous to bringing reactivity to "arbitrary" JavaScript, **and if we could achieve that, we could literaly have reactivity as a native language feature**! Now this changes everything because **that wouldn't be a compiler thing anymore**!

Can we skip now to that fun part?

We explore this by revisiting where we've fallen short of *bringing reactivity to "arbitrary" JavaScript* and fixing that! Svelte comes as a good starting point:

In Svelte today, reactivity is based on the custom `.svelte` file extension! (Being what carries the idea of a *reactive programming context* in the application!)

> `calculate.svelte`

```html
<script>
  // Code here
</script>
```

{% details But what if we could bring that *reactive programming context* to a function... %}
...for a handy building block that can be easily composed into other things?
{% enddetails %}

```js
function calculate() {
  // Code here
}
```

And next is how you express reactivity!

That, in Svelte today, relies on the dollar sign `$` label - being another type of manual dependency plumbing, albeit subtle:

```html
<script>
let count = 10;
// Reactive expressions
$: doubleCount = count * 2;
$: console.log(doubleCount); // 20
</script>
```

{% details But what if we simply treated every expression as *potentially* reactive... %}
...having already moved into a reactive programming context?
{% enddetails %}

```js
function calculate() {
  let count = 10;
  // Reactive expressions
  let doubleCount = count * 2;
  console.log(doubleCount); // 20
}
```

And lastly, the execution model!

Updates propagate in Svelte today "functionally", and thus, in any direction, to any part of the program:

```html
<script>
let count = 10;
// Reactive expressions
$: doubleCount = count * 2;
$: console.log(doubleCount); // 40
// Update
$: count = 20;
</script>
```

{% details But what if we could get updates to propagate "top-down" the program... %}
...and actually regain the "top-down" linear flow of *actual* imperative programs?
{% enddetails %}

```js
function calculate() {
  let count = 10;
  // Reactive expressions
  let doubleCount = count * 2;
  console.log(doubleCount); // 20
  // Update
  count = 20;
}
```

Well, notice now that the update to `count` in the last line wouldn't be reactive! And that brings us to the question: **does anything ever react to anything in this model**? The answer lies in what "dependencies" mean in normal programs!

Conventionally, programs run in sequential, linear flow, along which references to "prior" identifiers in scope create "dependencies"! This means that **statements should *really* only be responding to changes happening *up the scope*, not down the scope** (as the case may be in Svelte today)! And that for a function scope would be: **changes happening "outside" the scope** - up to the global scope!

```js
function calculate(count) { // External dependency
  // Reactive expressions
  let doubleCount = count * 2;
  console.log(doubleCount); // 20
}
let value = 10;
calculate(value);
```

`count` is now a dependency from "up"/"outside" the function scope - by which reactivity inside the function scope is achieved! But what would that even look like?

Imagine where the function has the ability to just "statically" reflect updates to its external dependencies:

```js
// An update
value = 20;
// A hypothetical function
reflect('count'); // "count" being what the function sees
```

not particularly in their being a *parameter*, but particularly in their being a dependency:

```js
let count = 10; // External dependency
function calculate() {
  // Reactive expressions
  let doubleCount = count * 2;
  console.log(doubleCount); // 20
}
calculate();
```

```js
// An update
count = 20;
// A hypothetical function
reflect('count'); // "count" being what the function sees
```

And that brings us to our destination - this point where we've checked all the boxes; where:

+ The "reactive programming context" isn't some file, but a function - a handy new "reactivity primitive"!
+ Reactivity isn't some special language or special convention, but *just JavaScript* - the only real place you'd arrive if you tried!
+ Execution model isn't anything "functional" by which we defy the semantics of *literal* syntax, but "linear" - just in how imperative JavaScript works!

All of this in real life is **Reflex Functions**!

### Introducing Reflex Functions

Reflex Functions are a new type of JavaScript function that enables fine-grained Reactive Programming in the *imperative* form of the language - wherein reactivity is drawn entirely on the dependency graph of your own code!

This is an upcoming proposal! ([Introducing Imperative Reactive Programming](https://en.wikipedia.org/wiki/Reactive_programming#Imperative) (IRP) in JavaScript!)

#### An Overview

Reflex Functions have a distinguishing syntax: a double star notation.

```js
function** calculate() {
  // Function body
}
```

> See [Formal Syntax](https://github.com/webqit/reflex-functions/wiki#formal-syntax) for details.

Function body is any regular piece of code that should statically reflect changes to its external dependencies:

```js
let count = 10; // External dependency
function** calculate(factor) {
  // Reactive expressions
  let doubled = count * factor;
  console.log(doubled);
}
```

Return value is a two-part array that contains both the function's actual return value and a special `reflect` function for getting the function to reflect updates:

```js
let [ returnValue, reflect ] = calculate(2);
console.log(returnValue); // undefined
```

{% details Console %}

| doubled | returnValue |
| ------- | ----------- |
| `20`    | `undefined` |

{% enddetails %}

The `reflect()` function takes just the string representation of the external dependencies that have changed:

```js
count = 20;
reflect('count');
```

{% details Console %}

| doubled |
| ------- |
| `40`    |

{% enddetails %}

Path dependencies are expressed in array notation. And multiple dependencies can be reflected at once, if they changed at once:

```js
count++;
this.property = value;
reflect('count', [ 'this', 'property' ]);
```

#### Change Propagation

Reactivity exists with Reflex Functions where there are dependencies "up" the scope to respond to! And here's the mental model for that:

`┌─` a change *happens outside* of function scope

`└─` is *propagated into* function, then *self-propagates down* `─┐`

Changes within the function body itself *self-propagate* down the scope, but re-running only those expressions that depend on the specific change, and rippling down the dependency graph!

Below is a good way to see that: a Reflex Function having `score` as an external dependency, with "reflex lines" having been drawn to show the dependency graph for that variable, or, in other words, the deterministic update path for that dependency:

![Code with reflex lines](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gh6r6qb1pamvwy22fgqv.png)

<!--
```js
let score = 40;
```

```js
function** ui() {
  let divElement = document.createElement('div');
  // >>─────────┐
  let tense = score > 50 ? 'passed' : 'failed';
  //    └─>────────────────────────────────────┐
  let message = `Hi ${ p.firstName }, you ${ tense } this test!`;
  //    │
  let sp│anElement = document.createElement('span');
  //    └─>──────────────┐
  let fullMessage = [ message, ' ', 'Thank you!' ].join( '' );
  //    └─>─────────────────────────────┐
  let broadcast = { [ p.username ]: fullMessage };
  //    │
  let se│ctionElement = document.createElement('section');
  //    ├─>─────────────────────────────────────────┐
  let br│oadcastInstance = new BroadcastMessage( broadcast );
  //    └─>───────┐ └─>──────────┐
  console.log( broadcast, broadcastInstance );

  document.body.append(divElement, spanElement, sectionElement);
}
```

```js
let [ returnValue, reflect ] = ui();
```
-->

It turns out to be the very mental model you would have drawn if you set out to think about your own code! Everything works **in just how anyone would *predict* it**!

Plus, there's a hunble brag: that "pixel-perfect" level of fine-grained reactivity that the same algorithm translates to - which you could never model manually; that precision that means *no more*, *no less* performance - which you could never achieve with manual optimization; yet, all without working for it!

#### Documentation

Visit the [docs](https://github.com/webqit/reflex-functions/wiki) for details around [Formal Syntax](https://github.com/webqit/reflex-functions/wiki#formal-syntax), [Heuristics](https://github.com/webqit/reflex-functions/wiki#heuristics), [Flow Control](https://github.com/webqit/reflex-functions/wiki#flow-control) and [Functions](https://github.com/webqit/reflex-functions/wiki#functions), [API](https://github.com/webqit/reflex-functions/wiki#api), and more.

#### The Project

Reflex Functions is being developed as something to be used today - via a polyfill. The polyfill features a specialized compiler and a small *runtime* that work together to enable all of Reflex Functions as documented, with quite a few exceptions.

{% embed https://github.com/webqit/reflex-functions %}

Show support with a star on github! Also, all forms of contributions are welcome at this time.

-----------------------

> SECTION 3/3

## The Duo and the Prospect for Reactivity

The magic that drives Frontend has got all the more powerful and completely intuitive!

From where I am sitting, I can see the amount of heavy-lifting that the Observer API will do for large swaths of today's usecases for "internal traps" and the whole "magic objects" idea! (It's probably time for [that "revolution" that never took off](https://web.dev/es7-observe/#:~:text=a%20revolution%20is%20coming.)!)

And we've certainly needed language support for *reactive logic*! Think of the different proposals out there around JavaScript-XML-like DSLs - which of course don't translate well to a native ECMAScript feature! And consider how much we've had to rely on compilers today for the idea!

Now that we can get "arbitrary" JavaScript to be reactive with Reflex Functions, a huge amount of that can be met natively! I am more than excited at how much we can regain *using just JavaScript* in each of those areas where we've relied on tooling!

I presume that experimenting around these new ideas, using the polyfill in each case, will furnish us new learnings around the problem! *For now, I am happy to have started the conversation*!

Now, did you know that these two reactive primitives could individually, or together, solve very unlikely problems? Here's where you can find early examples:

+ [Reflex Functions / Usecases](https://github.com/webqit/reflex-functions#usecases)

If you'd find this an interesting idea, feel free to share!

-----------------------

## Acknowledgements

+ Much thanks to [David Bruant](https://twitter.com/DavidBruant), [Andrea Giammarchi](https://twitter.com/WebReflection), and others for the various feedbacks and insights on early drafts of this article.

## References

+ [What is reactivity?](https://vuejs.org/guide/extras/reactivity-in-depth.html#what-is-reactivity) - a Vue.js definition
+ [A Brief History of Reactivity](https://www.builder.io/blog/history-of-reactivity) - a Miško Hevery narrative
+ [`Object.observe()`](https://web.dev/es7-observe/) - web.dev
+ [On the design of the ECMAScript Reflection API](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/37741.pdf) - ES-Lab
+ [What is a membrane?](https://tvcutsem.github.io/js-membranes) - Tom van Cutsem
+ [Observable Membrane](https://github.com/salesforce/observable-membrane) - Salesforce
+ [A Hands-on Introduction to Fine-Grained Reactivity](https://dev.to/ryansolid/a-hands-on-introduction-to-fine-grained-reactivity-3ndf) - Ryan Carniato
+ [The Quest for ReactiveScript](https://dev.to/this-is-learning/the-quest-for-reactivescript-3ka3) - Ryan Carniato
+ [Rethinking Reactivity](https://www.youtube.com/watch?v=AdNJ3fydeao) - Rich Harris
+ [React Forget](https://www.youtube.com/watch?v=lGEMwh32soc&t=2s) - React
+ [Reactivity Transform](https://vuejs.org/guide/extras/reactivity-transform.html) - Vue.js